for M1 Macs: the qemu-x86_64 emulates an ancient CPU model that doesn't support modern features, breaks Swift compiler / LLVM / ...

Question

for M1 Macs: the qemu-x86_64 emulates an ancient CPU model that doesn't support modern features, breaks Swift compiler / LLVM / ...

weissi opened this issue 3 years ago · comments

I have tried with the latest version of Docker Desktop
I have tried disabling enabled experimental features
I have uploaded Diagnostics

Expected behavior

I can run modern software that uses Intel instructions like pshufb in Docker for Mac (on M1 Macs) because the qemu-x86_64 that's installed using the binfmt_misc support can emulate it.

Actual behavior

I can't run modern software that makes use of newer Intel instructions because the qemu-x86_64 that gets run using binfmt_misc doesn't get passed a -cpu core2duo (or newer such as -cpu Skylake-Client/-cpu max) argument.

Information

The problem is that the qemu-x86_64 doesn't get passed a -cpu core2duo (or even better something like -cpu Skylake-Client or maybe -cpu max ("Enables all features supported by the accelerator in the current host")) argument. Really, I would recommend to just pass the newest CPU that qemu can emulate (probably -cpu max).

I got the Swift compiler to work just fine by compiling my own qemu with this patch applied:

sed -i 's/cpu_model = NULL/cpu_model = "core2duo"/g' linux-user/main.c

which just sets the default CPU model to core2duo. I then installed as the Docker for Mac default binfmt_misc handler for x86_64 using

/containers/services/binfmt/rootfs/usr/bin/binfmt -uninstall qemu-x86_64
echo ":qemu-x86_64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x3e\x00:\xff\xff\xff\xff\xff\xfe\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/var/lib/docker/overlay2/80s11o66od7msjkcmt4hal5ui/diff/amd64/usr/bin/qemu-x86_64-static-c2d:OCF" > /proc/sys/fs/binfmt_misc/register

in the actual VM which I entered using justincormack/nsenter.

Steps to reproduce the behavior

The quickest way to reproduce the problem is:

Get an M1 Mac and install Docker for Mac for M1 Macs
Run docker run -it --rm --platform linux/amd64 swift:5.3 bash -c 'echo "print(\"hello world\")" > /tmp/test.swift && cd /tmp && swiftc test.swift && ./test'

Expected: prints "hello world"
Actual: Output something similar to

Please submit a bug report (https://swift.org/contributing/#reporting-bugs) and include the project and the crash backtrace.
Stack dump:
0.	Program arguments: /swift-DEVELOPMENT-SNAPSHOT-2021-02-09-a-ubuntu18.04/usr/bin/swift-frontend -frontend -c -primary-file /tmp//test.swift -target x86_64-unknown-linux-gnu -disable-objc-interop -color-diagnostics -module-name test -o /tmp/test-d8e021.o
1.	Swift version 5.4-dev (LLVM 1e4181b99f530d2, Swift 842cc9c88f5463b)
 #0 0x0000000005cb5c13 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/swift-DEVELOPMENT-SNAPSHOT-2021-02-09-a-ubuntu18.04/usr/bin/swift-frontend+0x5cb5c13)
 #1 0x0000000005cb396e llvm::sys::RunSignalHandlers() (/swift-DEVELOPMENT-SNAPSHOT-2021-02-09-a-ubuntu18.04/usr/bin/swift-frontend+0x5cb396e)
 #2 0x0000000005cb5f9c SignalHandler(int) (/swift-DEVELOPMENT-SNAPSHOT-2021-02-09-a-ubuntu18.04/usr/bin/swift-frontend+0x5cb5f9c)
 #3 0x0000004000a48980 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x12980)
 #4 0x00000000018a7f63 std::vector<swift::DiagnosticState::Behavior, std::allocator<swift::DiagnosticState::Behavior> >::_M_fill_insert(__gnu_cxx::__normal_iterator<swift::DiagnosticState::Behavior*, std::vector<swift::DiagnosticState::Behavior, std::allocator<swift::DiagnosticState::Behavior> > >, unsigned long, swift::DiagnosticState::Behavior const&) (/swift-DEVELOPMENT-SNAPSHOT-2021-02-09-a-ubuntu18.04/usr/bin/swift-frontend+0x18a7f63)
 #5 0x00000000018a237f swift::DiagnosticState::DiagnosticState() (/swift-DEVELOPMENT-SNAPSHOT-2021-02-09-a-ubuntu18.04/usr/bin/swift-frontend+0x18a237f)
 #6 0x00000000006ae240 swift::CompilerInstance::CompilerInstance() (/swift-DEVELOPMENT-SNAPSHOT-2021-02-09-a-ubuntu18.04/usr/bin/swift-frontend+0x6ae240)
 #7 0x0000000000513165 swift::performFrontend(llvm::ArrayRef<char const*>, char const*, void*, swift::FrontendObserver*) (/swift-DEVELOPMENT-SNAPSHOT-2021-02-09-a-ubuntu18.04/usr/bin/swift-frontend+0x513165)
 #8 0x00000000004ad906 main (/swift-DEVELOPMENT-SNAPSHOT-2021-02-09-a-ubuntu18.04/usr/bin/swift-frontend+0x4ad906)
 #9 0x0000004002013bf7 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21bf7)
#10 0x00000000004ad57a _start (/swift-DEVELOPMENT-SNAPSHOT-2021-02-09-a-ubuntu18.04/usr/bin/swift-frontend+0x4ad57a)
qemu: uncaught target signal 4 (Illegal instruction) - core dumped
<unknown>:0: error: unable to execute command: Illegal instruction
<unknown>:0: error: compile command failed due to signal 4 (use -v to see invocation)

Please note the qemu: uncaught target signal 4 (Illegal instruction) - core dumped which qemu prints when it encounters the pshufb Intel instruction (which needs the Intel SSE3 feature which qemu (correctly) only emulates if you tell it to emulate a new-enough CPU using -cpu core2duo or better).

The problem here is that the Swift compiler runs a pshufb instruction which isn't available in the ancient CPU that qemu emulates if you don't pass -cpu core2duo or newer. See Swift compiler bug SR-14186 for all the details.

For a somewhat more complete test, it'd be probably good to run

docker run -it --rm --platform linux/amd64 swift:5.3 bash -c 'git clone https://github.com/apple/swift-nio.git && cd swift-nio && swift test'

which clones SwiftNIO, compiles it and runs its test suite.

Stephen Turner · Answer 1 · Mon Apr 12 2021 18:17:45 GMT+0800 (China Standard Time)

Thanks for the analysis, @weissi. Assigning to @djs55 for comments.

Tõnis Tiigi · Answer 2 · Thu Apr 15 2021 00:54:34 GMT+0800 (China Standard Time)

Passing -cpu max ~~does not seem to make a difference~~ for this case

root@0e92e16ee730:/tmp# /usr/bin/qemu-x86_64 /usr/bin/swiftc test.swift
Stack dump:
0.	Program arguments: /usr/bin/swiftc test.swift
1.	Swift version 5.3.3 (swift-5.3.3-RELEASE)
/usr/bin/swiftc[0x51fa1c4]
/usr/bin/swiftc[0x51f7dbe]
/usr/bin/swiftc[0x51fa49c]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x4000a45980]
/usr/bin/swiftc[0x192e333]
/usr/bin/swiftc[0x192735f]
/usr/bin/swiftc[0x4e9956]
/usr/bin/swiftc[0x4e8ce8]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x4002247bf7]
/usr/bin/swiftc[0x4e888a]
qemu: uncaught target signal 4 (Illegal instruction) - core dumped
Illegal instruction
root@0e92e16ee730:/tmp#



root@0e92e16ee730:/tmp# /usr/bin/qemu-x86_64 -cpu max /usr/bin/swiftc test.swift
Stack dump:
0.	Program arguments: /usr/bin/swift -frontend -c -primary-file test.swift -target x86_64-unknown-linux-gnu -disable-objc-interop -color-diagnostics -module-name test -o /tmp/test-7be3cf.o
1.	Swift version 5.3.3 (swift-5.3.3-RELEASE)
/usr/bin/swift[0x51fa1c4]
/usr/bin/swift[0x51f7dbe]
/usr/bin/swift[0x51fa49c]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x4000a45980]
/usr/bin/swift[0x192e333]
/usr/bin/swift[0x192735f]
/usr/bin/swift[0x6b0384]
/usr/bin/swift[0x55e3ad]
/usr/bin/swift[0x4e8ce8]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x4002247bf7]
/usr/bin/swift[0x4e888a]
qemu: uncaught target signal 4 (Illegal instruction) - core dumped
<unknown>:0: error: unable to execute command: Illegal instruction
<unknown>:0: error: compile command failed due to signal 4 (use -v to see invocation)

edit: looking into it more, looks like subprocess doesn't keep the args atm

Tõnis Tiigi · Answer 3 · Thu Apr 15 2021 01:40:17 GMT+0800 (China Standard Time)

Indeed -cpu max seems to allow this command to pass.

I've created an emulator image that enables cpu=max by default. To install it run:

docker run -it --rm --privileged  tonistiigi/binfmt --uninstall qemu-x86_64
docker run -it --rm --privileged  tonistiigi/binfmt:qemu-v5.0.1-cpu-max --install amd64

We need to figure out if there are any performance/functionality regressions before we can switch this image as default in desktop.

There is also a workaround you can do in your code without switching the system emulator. You just need to define QEMU_CPU=max. Eg. the following command works even with the current emulator:

docker run -it --rm --platform linux/amd64 swift:5.3 bash -c 'echo "print(\"hello world\")" > /tmp/test.swift && cd /tmp && export QEMU_CPU=max && swiftc test.swift && ./test'

Johannes Weiss · Answer 4 · Thu Apr 15 2021 02:13:24 GMT+0800 (China Standard Time)

Hey @tonistiigi, thanks for looking into this.

edit: looking into it more, looks like subprocess doesn't keep the args atm

Correct, Qemu does in fact do nothing for subprocesses. It just execves them normally. Because the executed binary is an x86 binary, this then invokes the binfmt_misc handler (which is installed by Docker for Mac) which defaults to an old micro-arch.

We need to figure out if there are any performance/functionality regressions before we can switch this image as default in desktop.

Awesome, this shouldn't make any difference as it should only affect software that currently doesn't work I think?

There is also a workaround you can do in your code without switching the system emulator. You just need to define QEMU_CPU=max. Eg. the following command works even with the current emulator:
docker run -it --rm --platform linux/amd64 swift:5.3 bash -c 'echo "print(\"hello world\")" > /tmp/test.swift && cd /tmp && export QEMU_CPU=max && swiftc test.swift && ./test'

Oh wow, that's awesome, thank you!

Stephen Turner · Answer 5 · Fri May 07 2021 16:19:09 GMT+0800 (China Standard Time)

We have implemented this in Docker Desktop 3.3.2. Thank you for the suggestion.

Johannes Weiss · Answer 6 · Fri May 07 2021 16:20:19 GMT+0800 (China Standard Time)

Awesome, thank you!

docker-robot · Answer 7 · Sun Jun 06 2021 11:01:09 GMT+0800 (China Standard Time)

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked