Unable to get logs from running container

[Adam]

Description

I am running containers that complete jobs and then exit, and I'm moving those to run within a gVisor sandbox. That is working, but I have not been able to figure out how to get the logs that would contain stdout and stderr emitted by my running process.

Without gVisor (and no special logging driver) I would find my Docker log in a bash script as follows (replacing CONTAINERNAME with the container's name of course)

containerid="$(docker ps --no-trunc -aqf name=CONTAINERNAME)"
logfilename="/var/lib/docker/containers/$containerid/$containerid-json.log"

But using gVisor there's nothing there. Trying to figure it out from the docs I thought perhaps I needed to set preferences in /etc.docker/daemon.json so I set that to:

{
  "runtimes": {
    "runsc": {
      "path": "/usr/local/bin/runsc",
      "runtimeArgs": ["--debug-log=/tmp/runsc/","--debug","--strace"]
     }
   }
}  

...and restart Docker. However, after I run a container I still cannot find logs from the process. I do see some other logs in /tmp/runsc/ (such as runsc.log.20220307-010547.039195.create) but I can't see the outputs of my container.

Except for that the container runs fine, and I can use demsg option to confirm that the runtime is correct.

What am I missing? How does one obtain the Docker container outputs from the container that is running in the gVisor sandbox?

Steps to reproduce

This is replicable as follows: (1) Spin up a fresh EC2 server running the latest stable build of Amazon Linux 2. (2) Install the current version of Docker that's blessed for OS, using sudo amazon-linux-extras install docker. Then install gVisor (see below), and restart Docker. Here's the bash command I am using to fetch and install gVisor.

(
    set -e
    ARCH=$(uname -m)
    URL=https://storage.googleapis.com/gvisor/releases/release/latest/\${ARCH} 
    wget \${URL}/runsc \${URL}/runsc.sha512 \${URL}/containerd-shim-runsc-v1 \${URL}/containerd-shim-runsc-v1.sha512 
    sha512sum -c runsc.sha512 -c containerd-shim-runsc-v1.sha512
    rm -f *.sha512
    chmod a+rx runsc containerd-shim-runsc-v1
    sudo mv runsc containerd-shim-runsc-v1 /usr/local/bin
)

runsc version

runsc version release-20220228.0 / spec: 1.0.2-dev

docker version (if using docker)

Docker 20.10.7

uname

Linux ip-172-31-47-173.us-east-2.compute.internal 4.14.123-111.109.amzn2.x86_64 #1 SMP Mon Jun 10 1 9:37:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

kubectl (if using Kubernetes)

No response

repo state (if built from source)

No response

runsc debug logs (if available)

No response

[Adam]

This was due to my user error; no bugs.

[mayankchouhan1089]

Hi @Aadmaa what was the issue and how did you solve it? I am getting a similar issue but I do not understand how can I resolve it.

[Adam]

Hello @mayankchouhan1089,

I can't remember exactly but it was something dopey I was doing wrong. I can show you how I am doing it now.

1. I updated the bash setup command as follows, which runs when I sent up a VM.

(
    set -e
    ARCH=$(uname -m)
    URL=https://storage.googleapis.com/gvisor/releases/release/latest/\${ARCH} 
    wget \${URL}/runsc \${URL}/runsc.sha512 \${URL}/containerd-shim-runsc-v1 \${URL}/containerd-shim-runsc-v1.sha512 
    sha512sum -c runsc.sha512 -c containerd-shim-runsc-v1.sha512
    rm -f *.sha512
    chmod a+rx runsc containerd-shim-runsc-v1
    sudo mv runsc containerd-shim-runsc-v1 /usr/local/bin
    sudo mkdir /tmp/runsc/
    sudo mkdir /tmp/runsc-debug/
    sudo /usr/local/bin/runsc install --runtime runsc-debug -- --debug --debug-log=/tmp/runsc-debug/ --strace --log-packets
    sudo /usr/local/bin/runsc install --runtime runsc -- --debug --debug-log=/tmp/runsc/
    sudo systemctl restart docker
)

Note: I'm not using the runsc-debug listed in the third line from the bottom, but it's handy to have that set up.

2. Then I just add "--runtime=runsc" to the docker run command.

I am able to get the docker logs where you'd normally see them. Something like this will locate them, if you define the container name.

        containerid="$(docker ps --no-trunc -aqf name=COTNAINERNAME)"
        logfilename="/var/lib/docker/containers/$containerid/$containerid-json.log"