mmap_test_runsc_kvm_vfs2 fails in a VM if vdso and vvar are mapped to the guest phys address space
avagin opened this issue · comments
Description
//test/syscalls:mmap_test_runsc_kvm_vfs2 TIMEOUT in 1 out of 4 in 301.1s
Stats over 4 runs: max = 301.1s, min = 1.6s, avg = 76.5s, dev = 129.6s
/home/agent/.cache/bazel/_bazel_agent/2ad144a0e8d66d1c5f9bcc89004ae2c5/execroot/__main__/bazel-out/k8-fastbuild/testlogs/test/syscalls/mmap_test_runsc_kvm_vfs2/shard_1_of_4/test.log
exe-110348 [006] .... 410259.258828: kvm_userspace_exit: reason error (14)
exe-110348 [006] .... 410259.258840: kvm_fpu: load
exe-110348 [006] d... 410259.258844: kvm_entry: vcpu 1, rip 0x5600b39c8132
exe-110348 [006] d... 410259.258862: kvm_exit: vcpu 1 reason EPT_VIOLATION rip 0x5600b39c8132 info1 0x0000000000000181 info2 0x0000000000000000 intr_info 0x00000000 error_code 0x00000000
exe-110348 [006] .... 410259.258863: kvm_page_fault: address 3fef42812000 error_code 181
exe-110348 [006] d... 410259.258864: fast_page_fault: vcpu 1 gva 3fef42812000 error_code U sptep 00000000ce97aa9a old 0x0 new 0 spurious 0 fixed 0
exe-110348 [006] .... 410259.258867: kvm_fpu: unload
exe-110348 [006] .... 410259.258871: kvm_userspace_exit: reason error (14)
exe-110348 [006] .... 410259.258883: kvm_fpu: load
exe-110348 [006] d... 410259.258887: kvm_entry: vcpu 1, rip 0x5600b39c8132
exe-110348 [006] d... 410259.258906: kvm_exit: vcpu 1 reason EPT_VIOLATION rip 0x5600b39c8132 info1 0x0000000000000181 info2 0x0000000000000000 intr_info 0x00000000 error_code 0x00000000
exe-110348 [006] .... 410259.258907: kvm_page_fault: address 3fef42812000 error_code 181
exe-110348 [006] d... 410259.258907: fast_page_fault: vcpu 1 gva 3fef42812000 error_code U sptep 00000000ce97aa9a old 0x0 new 0 spurious 0 fixed 0
exe-110348 [006] .... 410259.258911: kvm_fpu: unload
exe-110348 [006] .... 410259.258914: kvm_userspace_exit: reason error (14)
exe-110348 [006] .... 410259.258927: kvm_fpu: load
exe-110348 [006] d... 410259.258931: kvm_entry: vcpu 1, rip 0x5600b39c8132
exe-110348 [006] d... 410259.258949: kvm_exit: vcpu 1 reason EPT_VIOLATION rip 0x5600b39c8132 info1 0x0000000000000181 info2 0x0000000000000000 intr_info 0x00000000 error_code 0x00000000
exe-110348 [006] .... 410259.258950: kvm_page_fault: address 3fef42812000 error_code 181
exe-110348 [006] d... 410259.258951: fast_page_fault: vcpu 1 gva 3fef42812000 error_code U sptep 00000000ce97aa9a old 0x0 new 0 spurious 0 fixed 0
exe-110348 [006] .... 410259.258954: kvm_fpu: unload
exe-110348 [006] .... 410259.258958: kvm_userspace_exit: reason error (14)
exe-110348 [006] .... 410259.258970: kvm_fpu: load
exe-110348 [006] d... 410259.258974: kvm_entry: vcpu 1, rip 0x5600b39c8132
exe-110348 [006] d... 410259.258993: kvm_exit: vcpu 1 reason EPT_VIOLATION rip 0x5600b39c8132 info1 0x0000000000000181 info2 0x0000000000000000 intr_info 0x00000000 error_code 0x00000000
exe-110348 [006] .... 410259.258993: kvm_page_fault: address 3fef42812000 error_code 181
exe-110348 [006] d... 410259.258994: fast_page_fault: vcpu 1 gva 3fef42812000 error_code U sptep 00000000ce97aa9a old 0x0 new 0 spurious 0 fixed 0
exe-110348 [006] .... 410259.258997: kvm_fpu: unload
exe-110348 [006] .... 410259.259001: kvm_userspace_exit: reason error (14)
00400000-01018000 r-xp 00000000 08:01 2065646 /home/avagin/.cache/bazel/_bazel_avagin/b81b35ebb4504fa5145835608a9b1744/execroot/__main__/bazel-out/k8-fastbuild-ST-4c64f0b3d5c7/bin/runsc/runsc_/runsc
01018000-01c63000 r--p 00c18000 08:01 2065646 /home/avagin/.cache/bazel/_bazel_avagin/b81b35ebb4504fa5145835608a9b1744/execroot/__main__/bazel-out/k8-fastbuild-ST-4c64f0b3d5c7/bin/runsc/runsc_/runsc
01c63000-01c98000 rw-p 01863000 08:01 2065646 /home/avagin/.cache/bazel/_bazel_avagin/b81b35ebb4504fa5145835608a9b1744/execroot/__main__/bazel-out/k8-fastbuild-ST-4c64f0b3d5c7/bin/runsc/runsc_/runsc
01c98000-01cd2000 rw-p 00000000 00:00 0
c000000000-c000400000 rw-p 00000000 00:00 0
c000400000-c000800000 rw-p 00000000 00:00 0
c000800000-c004000000 rw-p 00000000 00:00 0
3fee42811000-3fee42a11000 r--s 00000000 08:01 1548479 /home/avagin/.cache/bazel/_bazel_avagin/b81b35ebb4504fa5145835608a9b1744/sandbox/linux-sandbox/164/execroot/__main__/_tmp/17c44a2debad9383a1bd713efd562aa9/096880235/gvisor_test_temp_1_1634583688083586376
3fee42a11000-3fee42c11000 r--s 00000000 08:01 3440 /usr/lib/x86_64-linux-gnu/libc-2.31.so
3fee42c11000-3fee42ca1000 rw-p 00000000 00:00 0
3fee42ca1000-3fee42ea1000 r--s 00000000 08:01 3453 /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
3fee42ea1000-3fee430a1000 r--s 00000000 08:01 3433 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
3fee430a1000-3fee432a1000 r--s 00000000 08:01 3442 /usr/lib/x86_64-linux-gnu/libm-2.31.so
3fee432a1000-3fee434a1000 r--s 00000000 08:01 3434 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
3fee4365e000-3fee4369e000 rw-p 00000000 00:00 0
3fee4369e000-3fee436a1000 rw-s 00000000 00:0d 13440 anon_inode:kvm-vcpu:1
3fee436a1000-3fee438a1000 r--s 00000000 08:01 3436 /usr/lib/x86_64-linux-gnu/ld-2.31.so
3fee438a1000-3fee43aa1000 r--s 00200000 08:01 2581766 /home/avagin/.cache/bazel/_bazel_avagin/b81b35ebb4504fa5145835608a9b1744/execroot/__main__/bazel-out/k8-fastbuild/bin/test/syscalls/linux/mmap_test
3fee43aa1000-3fee43ca1000 r--s 00000000 08:01 2581766 /home/avagin/.cache/bazel/_bazel_avagin/b81b35ebb4504fa5145835608a9b1744/execroot/__main__/bazel-out/k8-fastbuild/bin/test/syscalls/linux/mmap_test
3fee43ca1000-3fee43da2000 rw-s 00303000 00:01 2366 /memfd:flipcall_packet_windows (deleted)
3fee43da2000-3fee43ea3000 rw-s 00202000 00:01 2366 /memfd:flipcall_packet_windows (deleted)
3fee43ea3000-3fee43fa4000 rw-s 00101000 00:01 2366 /memfd:flipcall_packet_windows (deleted)
3fee43fa4000-3fee440a5000 rw-s 00000000 00:01 2366 /memfd:flipcall_packet_windows (deleted)
3fee440a5000-3fee44235000 rw-p 00000000 00:00 0
3fee44235000-3fee84235000 rw-s 00000000 00:01 3430 /memfd:runsc-memory (deleted)
3fee84235000-3fee84238000 rw-s 00000000 00:0d 13440 anon_inode:kvm-vcpu:0
3fee84238000-7ff004238000 ---p 00000000 00:00 0
7ff004238000-7ff004239000 rw-s 00000000 00:01 3429 /memfd:memory-usage (deleted)
7ff004239000-7ff00670a000 rw-p 00000000 00:00 0
7ff00670a000-7ff01688a000 ---p 00000000 00:00 0
7ff01688a000-7ff01688b000 rw-p 00000000 00:00 0
7ff01688b000-7ff02873a000 ---p 00000000 00:00 0
7ff02873a000-7ff02873b000 rw-p 00000000 00:00 0
7ff02873b000-7ff02ab10000 ---p 00000000 00:00 0
7ff02ab10000-7ff02ab11000 rw-p 00000000 00:00 0
7ff02ab11000-7ff02af8a000 ---p 00000000 00:00 0
7ff02af8a000-7ff02af8b000 rw-p 00000000 00:00 0
7ff02af8b000-7ff02b00a000 ---p 00000000 00:00 0
7ff02b00a000-7ff02b06a000 rw-p 00000000 00:00 0
7ffd9c4e6000-7ffd9c507000 rw-p 00000000 00:00 0 [stack]
7ffd9c5c2000-7ffd9c5c6000 r--p 00000000 00:00 0 [vvar]
7ffd9c5c6000-7ffd9c5c8000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
I1018 19:01:27.882849 1 loader.go:531] Platform: kvm
I1018 19:01:27.883103 1 physical_map.go:124] region: virtual [3fee84238000,7ff004238000)
I1018 19:01:27.883136 1 physical_map.go:176] physicalRegion: virtual [1000,3fee84238000) => physical [100001000,3fef84238000)
I1018 19:01:27.883143 1 physical_map.go:176] physicalRegion: virtual [7ff004238000,7ffffffff000) => physical [3fef84238000,3fff7ffff000)
Steps to reproduce
$ git diff
diff --git a/pkg/sentry/platform/kvm/kvm_amd64.go b/pkg/sentry/platform/kvm/kvm_amd64.go
index 6f9d0af66..be4334bce 100644
--- a/pkg/sentry/platform/kvm/kvm_amd64.go
+++ b/pkg/sentry/platform/kvm/kvm_amd64.go
@@ -184,10 +184,6 @@ type cpuidEntries struct {
// updateGlobalOnce does global initialization. It has to be called only once.
func updateGlobalOnce(fd int) error {
- // VVAR pages are mapped as I/O memory and in case of netsted
- // virtualization, get_user_pages will fail on them.
- dontMapVVAR = cpuid.HostFeatureSet().HasFeature(cpuid.X86FeatureHypervisor)
-
physicalInit()
err := updateSystemValues(int(fd))
ring0.Init(cpuid.HostFeatureSet())
$ bazel test test/syscalls:mmap_test_runsc_kvm_vfs2
runsc version
No response
docker version (if using docker)
No response
uname
Linux gvisor-kvm-test-e2 5.11.0-1020-gcp #22~20.04.1-Ubuntu SMP Tue Sep 21 10:54:26 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
kubectl (if using Kubernetes)
No response
repo state (if built from source)
No response
runsc debug logs (if available)
No response
The fault address is 3fef42812000:
3fee42811000-3fee42a11000 r--s 00000000 08:01 1548479 /home/avagin/.cache/bazel/_bazel_avagin/b81b35ebb4504fa5145835608a9b1744/sandbox/linux-sandbox/164/execroot/main/_tmp/17c44a2debad9383a1bd713efd562aa9/096880235/gvisor_test_temp_1_1634583688083586376
The target test case is ReadWriteSharedPrivate/MMapFileParamTest.SigBusDeath/0.
This test case creates a file with the size of 2098, maps it to the memory and try to access the second page. It checks that a test process will be killed by SIGBUS.
The fault address is 3fef42812000:
This is a second page of the test file.
If I run the same test case via runsc do, it passes:
$ bazel-out/k8-fastbuild-ST-4c64f0b3d5c7/bin/runsc/runsc_/runsc --rootless --network none --platform kvm --vfs2 do bazel-bin/test/syscalls/linux/mmap_test --gtest_filter=ReadWriteSharedPrivate/MMapFileParamTest.SigBusDeath/0
Note: Google Test filter = ReadWriteSharedPrivate/MMapFileParamTest.SigBusDeath/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from ReadWriteSharedPrivate/MMapFileParamTest
[ RUN ] ReadWriteSharedPrivate/MMapFileParamTest.SigBusDeath/0
[ OK ] ReadWriteSharedPrivate/MMapFileParamTest.SigBusDeath/0 (114 ms)
[----------] 1 test from ReadWriteSharedPrivate/MMapFileParamTest (115 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (116 ms total)
[ PASSED ] 1 test.
Failed to match any benchmarks against regex: .
[pid 116262] <... ioctl resumed>, 0) = -1 EFAULT (Bad address)
[pid 116208] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 116262] ioctl(10, KVM_NMI, 0) = 0
[pid 116262] ioctl(10, KVM_RUN <unfinished ...>
[pid 116208] <... nanosleep resumed>NULL) = 0
[pid 116262] <... ioctl resumed>, 0) = -1 EFAULT (Bad address)
[pid 116208] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 116262] ioctl(10, KVM_NMI, 0) = 0
[pid 116262] ioctl(10, KVM_RUN <unfinished ...>
[pid 116208] <... nanosleep resumed>NULL) = 0
[pid 116262] <... ioctl resumed>, 0) = -1 EFAULT (Bad address)
[pid 116208] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 116262] ioctl(10, KVM_NMI, 0) = 0
[pid 116262] ioctl(10, KVM_RUN <unfinished ...>
[pid 116208] <... nanosleep resumed>NULL) = 0
[pid 116208] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 116262] <... ioctl resumed>, 0) = -1 EFAULT (Bad address)
All kvm tests pass on buildkite: https://buildkite.com/gvisor/pipeline/builds/14008