Debian bookworm upgrade renders container unable to start
boserup opened this issue · comments
The changes introduced in 7e4a3dd has broken our CI pipeline (Travis CI). We began seeing errors as the recent push to the 7.0 tag happened on DockerHub
https://hub.docker.com/layers/library/redis/7.0/images/sha256-178215249742b63308db1a5373a7c627714c582362f3dcd24b2eec849dc16e67?context=explore
It is worth mentioning that we resolved the issue temporarily by switching to the 7.0-bullseye image, but I still don't see a reason why the bookworm-based images should be crashing.
Starting a container using the redis:7.0 image produces the following output:
redis_1 | 1:C 15 Jun 2023 09:43:37.054 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1 | 1:C 15 Jun 2023 09:43:37.054 # Redis version=7.0.11, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1 | 1:C 15 Jun 2023 09:43:37.054 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1 | 1:M 15 Jun 2023 09:43:37.055 * monotonic clock: POSIX clock_gettime
redis_1 | 1:M 15 Jun 2023 09:43:37.055 * Running mode=standalone, port=6379.
redis_1 | 1:M 15 Jun 2023 09:43:37.055 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1 | 1:M 15 Jun 2023 09:43:37.055 # Server initialized
redis_1 | 1:M 15 Jun 2023 09:43:37.055 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1 | 1:M 15 Jun 2023 09:43:37.055 # Fatal: Can't initialize Background Jobs.
api_redis_1 exited with code 1In docker-compose.yml the container is defined as such:
version: '3.4'
services:
redis:
image: redis:7.0
hostname: redis
ports:
- "6379:6379"- Linux 4.15.0-1098-gcp 111~16.04.1-Ubuntu SMP Tue Apr 13 19:05:08 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
- Docker version 20.10.7, build f0df350
- docker-compose version 1.23.2, build 1110ad01
- MemTotal: 8164804 kB
Seen in our piplines running 6.x as well
We also notice this. The host is running on Ubuntu 18.04 with a 4.15.0-212-generic kernel. I assume it has something to do with the kernel version or the base OS?
Same issue in our Buildkite pipelines running version 7.0.11
Seen in our pipelines with 7.0.11
If you are having troubles, try using the redis:*-bullseye tags. If that fixes it, then you probably need to update Docker and libseccomp on the host (and maybe possibly kernel version, but try the others first). Newer base OS's use newer system calls and an older libseccomp can block them since they are unknown to it.
If you are having troubles, try using the
redis:*-bullseyetags. If that fixes it, then you probably need to update Docker andlibseccompon the host (and maybe possibly kernel version, but try the others first). Newer base OS's use newer system calls and an olderlibseccompcan block them since they are unknown to it.
I've tried updating to the latest docker version to no avail. I think it must be related to a specific kernel version.
The bullseye images are working fine.
I've tried updating to the latest docker version to no avail. I think it must be related to a specific kernel version.
Did you try updating libseccomp; not certain how exactly new it needs to be. You can verify that it is libseccomp by running the bookworm image with --security-opt seccomp=unconfined.
This is similar to the update to Ubuntu focal: docker-library/mongo#606 (comment)
Edit: This seems eerily similar to docker-library/python#835
Hi, it would be helpful to know more details here -- is the best short-term solution to pin to use bullseye? is bullseye going to break in the next release, so should we also pin a version?
Using recent AWS AMI's, docker-compose is no longer able to bring up redis:latest which (per docs) is the defacto image.. to which I would argue is a fairly jarring change
version: "3.2"
services:
redis:
image: redis
ports:
- 6379
The best short term solution for you is going to be pinning to bullseye images explicitly, yes. However, they are no longer actively maintained/supported in this case, meaning that when there are updates to Debian or Redis you will not be receiving them, so you do want to figure out the issues with your environment and get yourself upgraded to the bookworm-based images.
I have the same problem, it was working up to Wed 14th Jun 2023 at 20:43 UTC.
I am simply pulling latest,
[2023-06-14T20:58:50Z] Pulling redis (redis:)...
latest: Pulling from library/redis
759700526b78: Pull complete
eb5c5a6a60f0: Pull complete
12e7d88aa422: Pull complete
8f4ed3de3123: Pull complete
0705c505f99b: Pull complete
d943febfd435: Pull complete
Digest: sha256:6a5130174e14177c3777ea34f4989a91bb5a7d7562dd48578cbfa3ad8cd53743
Status: Downloaded newer image for redis:latest
is it possible to know which tag this corresponds to ?
Update: bullseye has the same sha256
Digest: sha256:6a5130174e14177c3777ea34f4989a91bb5a7d7562dd48578cbfa3ad8cd53743
Status: Downloaded newer image for redis:bullseye
The best short term solution for you is going to be pinning to bullseye images explicitly, yes. However, they are no longer actively maintained/supported in this case, meaning that when there are updates to Debian or Redis you will not be receiving them, so you do want to figure out the issues with your environment and get yourself upgraded to the bookworm-based images.
My issue is that there is no clarity on what are the requirements to make the new image work. I'm happy to make updates as necessary but nothing here tells me the root cause
We're in the same boat, all our pipelines started to fail yesterday when the latest version was released. We pinned to the bullseye version
image: redis:7.0.11-bullseye
Ref for anyone else coming to this issue fresh.
If this is something we need to change in our CI pipelines to support bookworm, can this be clarified?
We had the same issue on our side with the redis:6.2 image. However the strange thing is that it happened only on some of our test platform with slightly older versions of Docker. In that case updating Docker fixed the issue. I'm still puzzled by what is the root cause.
We had the same issue on our side with the
redis:6.2image. However the strange thing is that it happened only on some of our test platform with slightly older versions of Docker. In that case updating Docker fixed the issue. I'm still puzzled by what is the root cause.
it seems like there's some call in the build that requires the latest version of debian/docker, which wouldn't be a problem locally but the reality is the redis docker image is heavily used for CI pipelines and not production systems, where it makes sense to just stay on latest instead of pinning versions. This also makes it more jarring, since all of your builds start failing. Would be great to know a root cause so people can go back to using latest
If you want something "unversioned" that still works on systems which are not updated appropriately to run containers on newer distributions utilizing newer kernel system calls, you can use redis:bullseye, but with the caveat that it won't be updated (it's the same as using redis@sha256:6a5130174e14177c3777ea34f4989a91bb5a7d7562dd48578cbfa3ad8cd53743, which might even be a better idea so you're clearly choosing to rely on an unchanging/non-maintained image, but YMMV).
Root cause: it is Docker with libseccomp so a newer syscall used in Debian Bookworm packages/libs is being blocked.
libseccomp lets you configure allowed syscalls for a process. Docker sets a default seccomp profile for all containers such that only certain syscalls are allowed and everything else is blocked (so, newer syscalls that are not yet known to libseccomp or docker are blocked).
- verify that it is
libseccompby running the Bookworm-based image with--security-opt seccomp=unconfined- use only as a test; it is less secure to keep running them "unconfined"
- learn more: https://docs.docker.com/engine/security/seccomp/
- one fix:
- update
libseccompanddockeron the host running the containers
- update
- one workaround:
- switch to the
*bullseyeimages (in theredisimages, these are now unmaintained/unchanging)
- switch to the
It's not possible to update libseccomp on Mac it seems, so I just moved to bullseye images and that worked.
Mac OS cannot natively run Linux containers, so that would mean you need to update whichever system/VM is managing your Linux containers (Docker Desktop, etc). ❤️