Panic in Linux thread_parker
agourlay opened this issue · comments
parking_lot version: 0.12.1
While doing performance tests with a lot of concurrent threads and RwLocks, I got the following panic due to a debug assertion.
/parking_lot_core-0.9.5/src/thread_parker/linux.rs at line 120: assertion failed: r == 0 || r == -1
Here is the whole trace.
[2023-06-26T13:46:11.974Z ERROR qdrant::startup] Panic occurred in file /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot_core-0.9.5/src/thread_parker/linux.rs at line 120: assertion failed: r == 0 || r == -1
[2023-06-26T13:46:11.958Z ERROR qdrant::startup] Panic backtrace:
0: qdrant::startup::setup_panic_hook::{{closure}}
at ./src/startup.rs:38:25
1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1987:9
2: std::panicking::rust_panic_with_hook
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:695:13
3: std::panicking::begin_panic_handler::{{closure}}
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:580:13
4: std::sys_common::backtrace::__rust_end_short_backtrace
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:150:18
5: rust_begin_unwind
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:578:5
6: core::panicking::panic_fmt
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:67:14
7: core::panicking::panic
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:117:5
8: parking_lot_core::thread_parker::imp::ThreadParker::futex_wait
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot_core-0.9.5/src/thread_parker/linux.rs:120:9
9: <parking_lot_core::thread_parker::imp::ThreadParker as parking_lot_core::thread_parker::ThreadParkerT>::park
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot_core-0.9.5/src/thread_parker/linux.rs:66:13
10: parking_lot_core::parking_lot::park::{{closure}}
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot_core-0.9.5/src/parking_lot.rs:635:17
11: parking_lot_core::parking_lot::with_thread_data
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot_core-0.9.5/src/parking_lot.rs:207:5
12: parking_lot_core::parking_lot::park
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot_core-0.9.5/src/parking_lot.rs:600:5
13: parking_lot::raw_rwlock::RawRwLock::lock_common
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:1115:17
14: parking_lot::raw_rwlock::RawRwLock::lock_exclusive_slow
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:633:26
15: <parking_lot::raw_rwlock::RawRwLock as lock_api::rwlock::RawRwLock>::lock_exclusive
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/parking_lot-0.12.1/src/raw_rwlock.rs:73:26
16: lock_api::rwlock::RwLock<R,T>::write
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/lock_api-0.4.9/src/rwlock.rs:480:9
17: segment::index::hnsw_index::graph_layers_builder::GraphLayersBuilder::link_new_point
at ./lib/segment/src/index/hnsw_index/graph_layers_builder.rs:368:33
18: <segment::index::hnsw_index::hnsw::HNSWIndex<TGraphLinks> as segment::index::vector_index_base::VectorIndex>::build_index::{{closure}}::{{closure}}
at ./lib/segment/src/index/hnsw_index/hnsw.rs:556:21
19: core::ops::function::impls::<impl core::ops::function::FnMut<A> for &F>::call_mut
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:274:13
20: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:310:13
21: core::option::Option<T>::map
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/option.rs:1099:29
22: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::next
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/iter/adapters/map.rs:103:9
23: rayon::iter::plumbing::Folder::consume_iter
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:178:21
24: <rayon::iter::map::MapFolder<C,F> as rayon::iter::plumbing::Folder<T>>::consume_iter
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/map.rs:248:21
25: rayon::iter::plumbing::Producer::fold_with
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:110:9
26: rayon::iter::plumbing::bridge_producer_consumer::helper
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:438:13
27: rayon::iter::plumbing::bridge_producer_consumer::helper::{{closure}}
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:427:21
28: rayon_core::join::join_context::call_b::{{closure}}
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:129:25
29: rayon_core::job::StackJob<L,F,R>::run_inline
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:102:9
30: rayon_core::join::join_context::{{closure}}
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:159:36
31: rayon_core::registry::in_worker
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:984:13
32: rayon_core::join::join_context
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:132:5
33: rayon::iter::plumbing::bridge_producer_consumer::helper
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:416:47
34: rayon::iter::plumbing::bridge_producer_consumer::helper::{{closure}}
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:427:21
35: rayon_core::join::join_context::call_b::{{closure}}
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:129:25
36: rayon_core::job::StackJob<L,F,R>::run_inline
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:102:9
37: rayon_core::join::join_context::{{closure}}
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:159:36
38: rayon_core::registry::in_worker
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:984:13
39: rayon_core::join::join_context
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:132:5
40: rayon::iter::plumbing::bridge_producer_consumer::helper
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:416:47
41: rayon::iter::plumbing::bridge_producer_consumer::helper::{{closure}}
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:418:21
42: rayon_core::join::join_context::call_a::{{closure}}
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:124:17
43: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panic/unwind_safe.rs:271:9
44: std::panicking::try::do_call
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
45: __rust_try
46: std::panicking::try
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
47: std::panic::catch_unwind
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
48: rayon_core::unwind::halt_unwinding
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/unwind.rs:17:5
49: rayon_core::join::join_context::{{closure}}
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:142:24
50: rayon_core::registry::in_worker
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:984:13
51: rayon_core::join::join_context
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:132:5
52: rayon::iter::plumbing::bridge_producer_consumer::helper
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:416:47
53: rayon::iter::plumbing::bridge_producer_consumer::helper::{{closure}}
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:427:21
54: rayon_core::join::join_context::call_b::{{closure}}
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:129:25
55: rayon_core::job::JobResult<T>::call::{{closure}}
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:218:41
56: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panic/unwind_safe.rs:271:9
57: std::panicking::try::do_call
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
58: __rust_try
59: std::panicking::try
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
60: std::panic::catch_unwind
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
61: rayon_core::unwind::halt_unwinding
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/unwind.rs:17:5
62: rayon_core::job::JobResult<T>::call
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:218:15
63: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:120:32
64: rayon_core::job::JobRef::execute
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:64:9
65: rayon_core::registry::WorkerThread::execute
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:874:9
66: rayon_core::registry::WorkerThread::wait_until_cold
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:820:17
67: rayon_core::registry::WorkerThread::wait_until
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:803:13
68: rayon_core::join::join_context::{{closure}}
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:167:17
69: rayon_core::registry::in_worker
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:984:13
70: rayon_core::join::join_context
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:132:5
71: rayon::iter::plumbing::bridge_producer_consumer::helper
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:416:47
72: rayon::iter::plumbing::bridge_producer_consumer::helper::{{closure}}
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.7.0/src/iter/plumbing/mod.rs:427:21
73: rayon_core::join::join_context::call_b::{{closure}}
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/join/mod.rs:129:25
74: rayon_core::job::JobResult<T>::call::{{closure}}
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:218:41
75: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panic/unwind_safe.rs:271:9
76: std::panicking::try::do_call
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
77: __rust_try
78: std::panicking::try
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
79: std::panic::catch_unwind
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
80: rayon_core::unwind::halt_unwinding
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/unwind.rs:17:5
81: rayon_core::job::JobResult<T>::call
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:218:15
82: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:120:32
83: rayon_core::job::JobRef::execute
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:64:9
84: rayon_core::registry::WorkerThread::execute
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:874:9
85: rayon_core::registry::WorkerThread::wait_until_cold
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:820:17
86: rayon_core::registry::WorkerThread::wait_until
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:803:13
87: rayon_core::registry::main_loop
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:948:5
88: rayon_core::registry::ThreadBuilder::run
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:54:18
89: <rayon_core::registry::DefaultSpawn as rayon_core::registry::ThreadSpawn>::spawn::{{closure}}
at /home/agourlay/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/registry.rs:99:20
90: std::sys_common::backtrace::__rust_begin_short_backtrace
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
91: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/mod.rs:526:17
92: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panic/unwind_safe.rs:271:9
93: std::panicking::try::do_call
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
94: __rust_try
95: std::panicking::try
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
96: std::panic::catch_unwind
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
97: std::thread::Builder::spawn_unchecked_::{{closure}}
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/thread/mod.rs:525:30
98: core::ops::function::FnOnce::call_once{{vtable.shim}}
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
99: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1973:9
100: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1973:9
101: std::sys::unix::thread::Thread::new::thread_start
at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys/unix/thread.rs:108:17
102: start_thread
at ./nptl/./nptl/pthread_create.c:444:8
103: __GI___clone3
at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
At the time of the error the system was not running out of memory.
I would love to know what kind of invariant was broken to know if I am maybe misusing this library.
Thanks!
This should not be possible. futex(FUTEX_WAIT) is documented (by its man page) to only ever return 0 or -1. What kernel version are you running? Do you have anything that might mess with system calls (e.g. seccomp filters, ptrace)?
Thank you for the fast answer!
Do you have anything that might mess with system calls (e.g. seccomp filters, ptrace)?
Bingo, I was profiling with Intel VTune at the same time, I guess we can assume this is responsible for that broken invariant.
I have never seen this behavior in the past, sorry for the noise.
I'm going to close this for now. Do reopen if you manage to reproduce this without vtune.