Difficult to debug when `"terminal": true` is passed non-TTY stdio

[Michael Pratt]

Today I spent a long time debugging issues running a basic container with os/exec and runsc run. Something like this:

$ bazel build //runsc
$ mkdir /tmp/hello
$ cd /tmp/hello
$ mkdir rootfs
$ $RUNSC spec -- /hello
$ sudo docker export $(sudo docker create hello-world) | tar -xf - -C rootfs
$ $RUNSC -rootless -network=none run hello  # Works fine!

Hello from Docker!
...
$ cat > exec.go <<EOF
package main

import (
        "fmt"
        "os"
        "os/exec"
)

func main() {
        cmd := exec.Command("$RUNSC", "--rootless", "--network=none", "run", "hello")
        cmd.Stdout = os.Stdout
        cmd.Stderr = os.Stderr
        if err := cmd.Run(); err != nil {
                fmt.Fprintf(os.Stderr, "error: %v\n", err)
                os.Exit(1)
        }
}
EOF
$ go run exec.go
<no output>

We expect to see the same "Hello from Docker!" message when running as a subprocess from Go, but get no output. Enabling strace logging reveals:

I0920 17:02:30.852331  1383989 strace.go:591] [   1] hello E write(0x1 host:[1], 0x402000 "\nHello from Docker!\nThis <snip>", 0x327)
I0920 17:02:30.852382  1383989 strace.go:629] [   1] hello X write(0x1 host:[1], 0x402000 "\nHello from Docker!\nThis <snip>", 0x327) = 0x0 errno=9 (bad file number) (35.206µs)

The write is returning EBADF. Further digging reveals the EBADF is because FD 1 is not writable. More digging eventually leads us to here. Because the config.json sets "terminal": true, the sentry assumes app FDs 0, 1, and 2 are all identical and only uses the imported FD 0. In our case, it is /dev/null which is (apparently) not writable, thus our program fails.

The fix is simple, remove "terminal": true, but figuring this out is really difficult, and this is a really easy way to get bitten. I think this should be caught with an explicit error. Perhaps one of:

1. host.ImportFD(isTTY = true) returns an error if the imported FD is not actually a TTY. (I'm not sure this is possible, as it looks intentional that we treat arbitrary host FDs as TTYs?)
2. importFS returns an error when console=true and the first three app FDs aren't identical (perhaps by comparing device and inode numbers?).

cc @fvoznika

[Michael Pratt]

Note that this bug is not related to rootless. It fails the same way when run normally.

[Fabricio Voznika]

Option 1 should work. If terminal is set, stdio FDs must be a TTY. Here is where they are created in the normal case. We could also fail if terminal is set but --console-socket is not. I'm not sure if the OCI allows it, but runsc will not set up TTY if not. Lastly, runsc spec should not set "terminal": true.

[github-actions]

A friendly reminder that this issue had no activity for 120 days.

[Ayush Ranjan]

@nlacasse can you have a look?

[github-actions]

A friendly reminder that this issue had no activity for 120 days.

[github-actions]

This issue has been closed due to lack of activity.