Stuck tasks detected in watchdog running Chromium
jseba opened this issue · comments
Description
We're upgrading Gvisor to release-20220328.0 and are seeing a mildly consistent hang in the sentry kernel. We're running a Chromium container that has a pair of stuck tasks being caught by the watchdog. Our previous Gvisor version was release-20211026.0. We're stuck trying to upgrade until we can resolve this hang but right now we're just trying to understand the cause.
I'm having trouble pinning down what task is actually hanging on since it's frequent-ish (a handful hits a day out of a hundred or so containers) but not frequent enough that I can reproduce it at will. I don't have a repro to share yet but am trying to come up with something.
These are stacks from the stuck tasks (the whole stack trace is attached at the bottom):
goroutine 29630 [syscall, 4 minutes, locked to thread]:
syscall.Syscall6(0xca, 0xc00003a39c, 0x80, 0x5, 0x0, 0x0, 0x0)
src/syscall/asm_linux_amd64.s:43 +0x5 fp=0xc00408b6e0 sp=0xc00408b6d8 pc=0x47e625
gvisor.dev/gvisor/pkg/sentry/platform/kvm.(*vCPU).waitUntilNot(0xe548cb, 0x2c41600)
pkg/sentry/platform/kvm/machine_unsafe.go:139 +0x56 fp=0xc00408b748 sp=0xc00408b6e0 pc=0xe5f7b6
gvisor.dev/gvisor/pkg/sentry/platform/kvm.(*vCPU).bounce(0xc00003a240, 0x0)
pkg/sentry/platform/kvm/machine.go:608 +0xf6 fp=0xc00408b7a0 sp=0xc00408b748 pc=0xe5d236
gvisor.dev/gvisor/pkg/sentry/platform/kvm.(*vCPU).BounceToKernel(...)
pkg/sentry/platform/kvm/machine.go:662
gvisor.dev/gvisor/pkg/sentry/platform/kvm.(*addressSpace).invalidate.func1(0xc00234d078)
pkg/sentry/platform/kvm/address_space_amd64.go:21 +0x2d fp=0xc00408b7c0 sp=0xc00408b7a0 pc=0xe58bcd
gvisor.dev/gvisor/pkg/sentry/platform/kvm.(*dirtySet).forEach(0xc003623158, 0xc0003e61e0, 0xc00408b818)
pkg/sentry/platform/kvm/address_space.go:45 +0xab fp=0xc00408b800 sp=0xc00408b7c0 pc=0xe57dab
gvisor.dev/gvisor/pkg/sentry/platform/kvm.(*addressSpace).invalidate(0xc00003bb00)
pkg/sentry/platform/kvm/address_space_amd64.go:19 +0x45 fp=0xc00408b838 sp=0xc00408b800 pc=0xe58b65
gvisor.dev/gvisor/pkg/sentry/platform/kvm.(*addressSpace).Unmap(0xc0004c0fe0, 0x11163c0, 0xc0003b6000)
pkg/sentry/platform/kvm/address_space.go:225 +0x153 fp=0xc00408b8b0 sp=0xc00408b838 pc=0xe588b3
gvisor.dev/gvisor/pkg/sentry/mm.(*MemoryManager).unmapASLocked(0xc0067a9200, {0x7000, 0xbd400000})
pkg/sentry/mm/address_space.go:239 +0x50 fp=0xc00408b8d8 sp=0xc00408b8b0 pc=0x835ad0
gvisor.dev/gvisor/pkg/sentry/mm.(*MemoryManager).getPMAsInternalLocked(0xc00277c000, {0x1447710, 0xc002d42a80}, {0xc0051b3000, 0x85ea0d}, {0xc0051b3000, 0x470757}, {0x0, 0x1, 0x0})
pkg/sentry/mm/pma.go:391 +0x97a fp=0xc00408bd00 sp=0xc00408b8d8 pc=0x846d3a
gvisor.dev/gvisor/pkg/sentry/mm.(*MemoryManager).getPMAsLocked(0x100060100, {0x1447710, 0xc002d42a80}, {0xc0051b3000, 0x45bb38}, {0xc0004c0fe0, 0xc00018a570}, {0x0, 0x1, 0x0})
pkg/sentry/mm/pma.go:125 +0xbf fp=0xc00408bd90 sp=0xc00408bd00 pc=0x845ddf
gvisor.dev/gvisor/pkg/sentry/mm.(*MemoryManager).HandleUserFault(0xc00277c000, {0x1447710, 0xc002d42a80}, 0x143d8d8, {0x0, 0xc0, 0x77}, 0x145bb38)
pkg/sentry/mm/syscalls.go:57 +0x173 fp=0xc00408be38 sp=0xc00408bd90 pc=0x856b13
gvisor.dev/gvisor/pkg/sentry/kernel.(*runApp).execute(0xc002dda580, 0xc002d42a80)
pkg/sentry/kernel/task_run.go:273 +0x9b1 fp=0xc00408bf60 sp=0xc00408be38 pc=0x9710d1
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).run(0xc002d42a80, 0x47d)
pkg/sentry/kernel/task_run.go:95 +0x1ac fp=0xc00408bfc0 sp=0xc00408bf60 pc=0x9700ac
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).Start·dwrap·236()
pkg/sentry/kernel/task_start.go:339 +0x2a fp=0xc00408bfe0 sp=0xc00408bfc0 pc=0x97ac4a
runtime.goexit()
src/runtime/asm_amd64.s:1581 +0x1 fp=0xc00408bfe8 sp=0xc00408bfe0 pc=0x469321
created by gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).Start
pkg/sentry/kernel/task_start.go:339 +0xfe
goroutine 29632 [semacquire, 4 minutes]:
runtime.gopark(0x1e74760, 0xe56ab2, 0x40, 0x97, 0xc000d74ce8)
GOROOT/src/runtime/proc.go:366 +0xd6 fp=0xc000d74c88 sp=0xc000d74c68 pc=0x4380d6
runtime.goparkunlock(...)
GOROOT/src/runtime/proc.go:372
runtime.semacquire1(0xc00277c864, 0x10, 0x3, 0x1)
GOROOT/src/runtime/sema.go:144 +0x20c fp=0xc000d74cf0 sp=0xc000d74c88 pc=0x44920c
sync.runtime_SemacquireMutex(0xc000d74d80, 0xd, 0xc006903800)
GOROOT/src/runtime/sema.go:71 +0x25 fp=0xc000d74d20 sp=0xc000d74cf0 pc=0x464de5
sync.(*Mutex).lockSlow(0xc00277c860)
GOROOT/src/sync/mutex.go:138 +0x165 fp=0xc000d74d70 sp=0xc000d74d20 pc=0x472305
sync.(*Mutex).Lock(...)
GOROOT/src/sync/mutex.go:81
gvisor.dev/gvisor/pkg/sync.(*CrossGoroutineMutex).Lock(...)
pkg/sync/mutex_unsafe.go:38
gvisor.dev/gvisor/pkg/sync.(*CrossGoroutineRWMutex).Lock(0xc002c41640)
pkg/sync/rwmutex_unsafe.go:146 +0x36 fp=0xc000d74d90 sp=0xc000d74d70 pc=0x52c976
gvisor.dev/gvisor/pkg/sync.(*RWMutex).Lock(...)
pkg/sync/rwmutex_unsafe.go:286
gvisor.dev/gvisor/pkg/sentry/mm.(*MemoryManager).HandleUserFault(0xc00277c000, {0x1447710, 0xc003fd0000}, 0x143d8d8, {0x0, 0xc0, 0x77}, 0x145bb38)
pkg/sentry/mm/syscalls.go:56 +0x125 fp=0xc000d74e38 sp=0xc000d74d90 pc=0x856ac5
gvisor.dev/gvisor/pkg/sentry/kernel.(*runApp).execute(0xc002ddb280, 0xc003fd0000)
pkg/sentry/kernel/task_run.go:273 +0x9b1 fp=0xc000d74f60 sp=0xc000d74e38 pc=0x9710d1
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).run(0xc003fd0000, 0x47f)
pkg/sentry/kernel/task_run.go:95 +0x1ac fp=0xc000d74fc0 sp=0xc000d74f60 pc=0x9700ac
gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).Start·dwrap·236()
pkg/sentry/kernel/task_start.go:339 +0x2a fp=0xc000d74fe0 sp=0xc000d74fc0 pc=0x97ac4a
runtime.goexit()
src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000d74fe8 sp=0xc000d74fe0 pc=0x469321
created by gvisor.dev/gvisor/pkg/sentry/kernel.(*Task).Start
pkg/sentry/kernel/task_start.go:339 +0xfe
One of the hits has Decommit (madvise(MADV_DONTNEED)) in the call stack instead of HandleUserFault, but that may not mean anything.
I'm pretty unfamiliar with the internals of the KVM platform but on the surface this seems like a deadlock, although I can't see why the KVM bounce would get stuck? Is there's any context someone could provide as to what's happening in this stack trace and any tools that could be of help tracking this down? We can also add some logging to Gvisor or modify the watchdog to print more information.
runsc version: release-20220328.0
host kernel version: Linux 5.15.32-cloudflare-kasan-2022.4.1
full stacktrace: https://gist.github.com/jseba/22c274b149270bf579b4b1f57660f5ef
Steps to reproduce
No response
runsc version
No response
docker version (if using docker)
No response
uname
No response
kubectl (if using Kubernetes)
No response
repo state (if built from source)
No response
runsc debug logs (if available)
No response
The watchdog report has 2 goroutines that are stuck for more than 3 mins:
Task tid: 67 (goroutine 468), entered RunSys state 3m40.19s ago.
Task tid: 63 (goroutine 441), entered RunSys state 3m40.19s ago
The first one is waiting for the MemoryManager.activeMu lock. The second one holds the lock and is waiting for the vCPU to bounce:
goroutine 441 [syscall, 3 minutes, locked to thread]:
syscall.Syscall6(0xca, 0xc0003aa5dc, 0x80, 0x5, 0x0, 0x0, 0x0)
src/syscall/asm_linux_amd64.s:43 +0x5 fp=0xc001f376e0 sp=0xc001f376d8 pc=0x47e625
gvisor.dev/gvisor/pkg/sentry/platform/kvm.(*vCPU).waitUntilNot(0xe548cb, 0x1e1a700)
pkg/sentry/platform/kvm/machine_unsafe.go:139 +0x56 fp=0xc001f37748 sp=0xc001f376e0 pc=0xe5f7b6
gvisor.dev/gvisor/pkg/sentry/platform/kvm.(*vCPU).bounce(0xc0003aa480, 0x0)
pkg/sentry/platform/kvm/machine.go:608 +0xf6 fp=0xc001f377a0 sp=0xc001f37748 pc=0xe5d236
gvisor.dev/gvisor/pkg/sentry/platform/kvm.(*vCPU).BounceToKernel(...)
pkg/sentry/platform/kvm/machine.go:662
gvisor.dev/gvisor/pkg/sentry/platform/kvm.(*addressSpace).invalidate.func1(0xc00231d088)
pkg/sentry/platform/kvm/address_space_amd64.go:21 +0x2d fp=0xc001f377c0 sp=0xc001f377a0 pc=0xe58bcd
gvisor.dev/gvisor/pkg/sentry/platform/kvm.(*dirtySet).forEach(0xc0021c5de8, 0xc0003b01e0, 0xc001f37818)
pkg/sentry/platform/kvm/address_space.go:45 +0xab fp=0xc001f37800 sp=0xc001f377c0 pc=0xe57dab
gvisor.dev/gvisor/pkg/sentry/platform/kvm.(*addressSpace).invalidate(0xc0003aab40)
pkg/sentry/platform/kvm/address_space_amd64.go:19 +0x45 fp=0xc001f37838 sp=0xc001f37800 pc=0xe58b65
gvisor.dev/gvisor/pkg/sentry/platform/kvm.(*addressSpace).Unmap(0xc0011a4ac0, 0x11163c0, 0xc000380000)
pkg/sentry/platform/kvm/address_space.go:225 +0x153 fp=0xc001f378b0 sp=0xc001f37838 pc=0xe588b3
gvisor.dev/gvisor/pkg/sentry/mm.(*MemoryManager).unmapASLocked(0xc00233a000, {0x7000, 0x3a600000})
pkg/sentry/mm/address_space.go:239 +0x50 fp=0xc001f378d8 sp=0xc001f378b0 pc=0x835ad0
gvisor.dev/gvisor/pkg/sentry/mm.(*MemoryManager).getPMAsInternalLocked(0xc00231e000, {0x1447710, 0xc0022b3500}, {0xc002320000, 0x85ea0d}, {0xc002320000, 0x470757}, {0x0, 0x1, 0x0})
pkg/sentry/mm/pma.go:391 +0x97a fp=0xc001f37d00 sp=0xc001f378d8 pc=0x846d3a
gvisor.dev/gvisor/pkg/sentry/mm.(*MemoryManager).getPMAsLocked(0x100070100, {0x1447710, 0xc0022b3500}, {0xc002320000, 0x45bb38}, {0xc0011a4ac0, 0xc0000d6580}, {0x0, 0x1, 0x0})
pkg/sentry/mm/pma.go:125 +0xbf fp=0xc001f37d90 sp=0xc001f37d00 pc=0x845ddf
gvisor.dev/gvisor/pkg/sentry/mm.(*MemoryManager).HandleUserFault(0xc00231e000, {0x1447710, 0xc0022b3500}, 0x143d8d8, {0x0, 0xe0, 0x31}, 0x145bb38)
...
@avagin, can you please take a look...
We're upgrading Gvisor to release-20220328.0
What was the previous version?
Previous release was release-20211026.0
@jseba I think I've fixed the problem. Could you verify that your problem is not reproduced with the fix?
I’ve updated the release in our testing environment, will monitor for a few days and follow up if it looks like it’s fixed. Thanks for the quick turnaround!
Looks like its fixed, we've rolled it out and haven't seen it pop back up, so I think this is closed