QEMU issue with curl and SSL certificates on Debian
dubo-dubon-duponey opened this issue · comments
FROM debian:buster-slim
RUN apt-get update && apt-get install -y --no-install-recommends curl ca-certificates
RUN curl https://www.google.com
docker buildx build --platform="linux/arm64,linux/amd64,linux/386" -f Dockerfile . <- works as expected
docker buildx build --platform="linux/arm/v7" -f Dockerfile . <- fails with curl: (60) SSL certificate problem: unable to get local issuer certificate
Same problem with linux/arm/v6.
This problem is not visible on a native arm machine - so, I'm guessing it's a QEMU issue.
I believe this is this issue https://bugs.launchpad.net/qemu/+bug/1805913 which is kind of complicated to fix as it is a qemu/kernel/glibc combined issue that qemu on its own cannot fix...
"Same things happens with update-ca-certificates. It calls c_rehash through openssl, which ends up doing nothing. As a result, curl with https and probably anything else that uses SSL fails to work."
🤦🏽♂
@justincormack do you know if musl would present the same issue?
It seems to be mainly a problem on ext4 filesystems, so if it is an option, changing the filesystem that your system uses might be an option.
It should not apply not Musl, and it is fine with older glibc. (pre-Buster and Ubuntu 18.04 and earlier are fine for example).
Thanks.
It seems that if we compile qemu-static as a 32 bit binary this fixes the issue as it uses 32 bit syscalls. I opened linuxkit/linuxkit#3438
@justincormack: interesting and more logical workaround. For the record another workaround is to remove dir_index from ext4 filesystems but for me it does not work.
So I just have to cross-compile qemu-static as a 32 bit binary? That does not sound too hard, I'll try.
So I did apt install qemu-user-static:i386 but it still bugs 😢
The process still gets stuck in a loop involving getdents64.
root@earth:~# file /usr/bin/qemu-arm-static
/usr/bin/qemu-arm-static: ELF 32-bit LSB executable, Intel 80386, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, BuildID[sha1]=ff1224d87ca5dece8d0b0f5735cfee7fae97ee58, stripped
root@earth:~# ps afx | grep qemu
ports/org.qemu.guest_agent.0
10584 ? Ssl 0:00 | | \_ /usr/bin/qemu-arm-static /usr/bin/make install
28768 ? Sl 0:01 | | \_ /usr/bin/qemu-arm-static /usr/bin/make -C src VCSWITNESS=$(srcdir)/../.git/logs/HEAD all
16718 ? Sl 0:00 | | \_ /usr/bin/qemu-arm-static /usr/bin/make -C ../lisp compile-first EMACS=../src/bootstrap-emacs
16726 ? Rl 48:24 | | \_ /usr/bin/qemu-arm-static ../src/bootstrap-emacs -batch --no-site-file --no-site-lisp --eval (setq load-prefer-newer t) -f batch-byte-compile emacs-lisp/macroexp.el
10696 ? Ssl 0:00 | \_ /usr/bin/qemu-aarch64-static /usr/bin/make install
10972 ? Sl 0:02 | \_ /usr/bin/qemu-aarch64-static /usr/bin/make -C src VCSWITNESS=$(srcdir)/../.git/logs/HEAD all
20397 ? Sl 0:00 | \_ /usr/bin/qemu-aarch64-static /usr/bin/make -C ../lisp compile-first EMACS=../src/bootstrap-emacs
20405 ? Rl 24:09 | \_ /usr/bin/qemu-aarch64-static ../src/bootstrap-emacs -batch --no-site-file --no-site-lisp --eval (setq load-prefer-newer t) -f batch-byte-compile emacs-lisp/macroexp.el
root@earth:~# strace -p 16726
clock_gettime(CLOCK_REALTIME, {tv_sec=1584794027, tv_nsec=921230669}) = 0
getdents64(5, /* 0 entries */, 2048) = 0
_llseek(5, 0, [0], SEEK_SET) = 0
getdents64(5, /* 5 entries */, 2048) = 144
tgkill(29984, 29987, SIGRT_2) = -1 EAGAIN (Resource temporarily unavailable)
clock_gettime(CLOCK_REALTIME, {tv_sec=1584794027, tv_nsec=921642405}) = 0
getdents64(5, /* 0 entries */, 2048) = 0
_llseek(5, 0, [0], SEEK_SET) = 0
getdents64(5, /* 5 entries */, 2048) = 144
tgkill(29984, 29987, SIGRT_2) = -1 EAGAIN (Resource temporarily unavailable)
clock_gettime(CLOCK_REALTIME, {tv_sec=1584794027, tv_nsec=922333065}) = 0
getdents64(5, /* 0 entries */, 2048) = 0
_llseek(5, 0, [0], SEEK_SET) = 0
getdents64(5, /* 5 entries */, 2048) = 144
tgkill(29984, 29987, SIGRT_2) = -1 EAGAIN (Resource temporarily unavailable)
clock_gettime(CLOCK_REALTIME, ^C{tv_sec=1584794027, tv_nsec=923201432}) = 0
strace: Process 16726 detached
Looks like I need to dig deeper and actually debug QEMU 😟
Looks like this has been fixed in debian bullseye. Based on https://bugzilla.kernel.org/show_bug.cgi?id=205957 the kernel patches for it are not accepted and newer and properly configured glibc is needed to work around it.