1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
|
## The impossible EFAULT
I have written a program; suppose it's called `worker`.
(While the program is written in Haskell, I don't think that's particularly relevant to this post.)
(EDIT: Reproducer can be found [here](https://git.tomsmeding.com/snap-efault/tree/).)
(EDIT 2: Diagnosis by `int-e` on irc [here](https://paste.tomsmeding.com/D22SvR2T).)
When run, `worker` starts a bunch of copies of a script.
Under normal circumstances this script sets up a container using Linux cgroups and Linux user namespaces, but none of that is relevant because the strange behaviour in question occurs just fine without all of that -- in fact, we'll let it start the following script, say `./sleep.sh`:
```bash
#!/bin/bash
sleep 10
```
Clearly, there is no weird behaviour here, assuming that the system has `bash` under `/bin`, and mine does.
The copies of `sleep.sh` are started by passing `./sleep.sh` to `posix_spawnp(3)`.
(The Haskell `process` library does this for me.)
The thing is, occasionally (once every 5 to 10 invocations of `./worker`, approximately), `posix_spawnp` returns `EFAULT` ("Bad Address").
The manpage for `posix_spawnp` says that:
> **ERRORS**
>
> The posix_spawn() and posix_spawnp() functions fail only in the case where the underlying fork(2), vfork(2) or clone(2) call fails; in these cases, these functions return an error number, which will be one of the errors described for fork(2), vfork(2) or clone(2).
>
> In addition, these functions fail if:
>
> **ENOSYS** Function not supported on this system.
Okay, so I should look for `EFAULT` in `fork(2)`, `vfork(2)` and `clone(2)` to figure out what goes wrong, right?
Wrong.
Or, in any case, none of those manpages mention `EFAULT`.
I've looked through the source code of `posix_spawnp` in glibc and it at least doesn't throw `EFAULT` directly; presumably, one of the subroutines it calls does.
glibc is large and I don't think looking through the entire call tree will be very productive, so I tried to diagnose the issue from the outside instead.
And this is where the weirdness starts.
Whenever my program encounters `EFAULT` from `posix_spawnp`, it prints `Oops EFAULT`; hence grepping for `EFAULT` gives output precisely if the error occurred in this run.
I get the following observations:
- `./worker 2>&1 | grep EFAULT`: errors occur.
- `./worker 2>&1 | grep EFAULT | cat`: errors DO NOT occur.
- `./worker 2>&1 | grep --line-buffered EFAULT | cat`: errors occur.
- `./worker 2>&1 | grep --line-buffered EFAULT`: errors occur.
("errors occur" means that once every few executions I get output indicating that `EFAULT` occurred; in the negative case I've run it for >20x the number of invocations that are necessary to produce `EFAULT` in the other cases, without any `EFAULT`.)
The only situation in which `posix_spawnp` seems to always succeed, is when `stdout` of the process that `worker`'s output is piped to, is block-buffered.
But this makes no sense: there shouldn't even be a reasonable way in which `worker` _can_ even determine whether this is the case!
Surely it can distinguish between `./worker | cat` and `./worker` (using `isatty(3)` -- this is precisely what `grep` does when not passed `--line-buffered`), but in all of the above cases the output is piped to another process anyway.
This is already spooky, but it gets even spookier: if I replace the invocation of `./sleep.sh` by an invocation of `sleep` (i.e. removing the indirection of the shell script), errors occur in none of the above setups.
Somehow, starting a script is different from starting a native process (and changing `bash` to `dash` in `sleep.sh` doesn't change anything).
`posix_spawnp` shouldn't care what it is starting!
That's the job of the loader, as far as I know.
So what gives?
### The cause
<s>I'll try to reduce my own program to a minimal reproducer, and if I find anything I'll post an update to this post.
In the meantime, spookiness.</s>
`snap-server` [modifies the environment](https://github.com/snapframework/snap-server/blob/8d89c10014d8d295bfbf5419bbb8551de32d7f85/src/Snap/Http/Server.hs#L161) to set the locale, and `setenv(3)` is not atomic.
In particular, it breaks `execve(2)` when they race, and this is what happens.
All possible solutions to this problem are hacks.
|
