PostgresML is spawning too many threads.

Question

PostgresML is spawning too many threads.

higuoxing opened this issue 7 months ago · comments

Recently, we found that PostgresML was spawning too many threads and threads is waiting for each other. No deadlock but low performance.

Root cause

The root cause is that PyTorch spawns too many threads, which cause severe CPU contention.

This can be solved by limiting the degree of parallelism with e.g.,

import torch
torch.set_num_threads(4)

The root cause and workaround is found and provided by @xuebinsu

Probably we can add some GUC parameter to allow user tunning the performance?

Reproducing steps

Run test without limiting the number of threads. (Finished in 273.66s)

$ unset OMP_NUM_THREADS
$ cargo pgrx test pg16
...
test result: ok. 57 passed; 0 failed; 25 ignored; 0 measured; 0 filtered out; finished in 273.66s

Run test by limiting the number of threads that openmp can spawn with OMP_NUM_THREADS=1. (Finished in 7.81s)

$ export OMP_NUM_THREADS=1
$ cargo pgrx test pg16
...
test result: ok. 57 passed; 0 failed; 25 ignored; 0 measured; 0 filtered out; finished in 7.81s

Stack trace

process: /proc/180629/mem
thread: 0x7ff82594f740, lwp: 180629, type: 1
#0  0x00007ff82518ca1e in gomp_team_barrier_wait_end()+318 in /usr/lib/libgomp.so.1 at futex.h:111
#1  0x00007ff82518b35d in gomp_team_end()+44 in /usr/lib/libgomp.so.1 at team.c:956
#2  0x00007ff719a7935f in _ZN7xgboost6common13ParallelFor2dIZNS_4tree16HistogramBuilder13SyncHistogramEPKNS_7RegTreeERKSt6vectorIiSaIiEESB_EUlmNS0_7Range1dEE2_EEvRKNS0_14BlockedSpace2dEiOT_!()+302 in /home/v/.local/lib/python3.11/site-packages/xgboost/lib/libxgboost.so
#3  0x00007ff719a857bd in _ZN7xgboost4tree16HistogramBuilder13SyncHistogramEPKNS_7RegTreeERKSt6vectorIiSaIiEES9_!()+1404 in /home/v/.local/lib/python3.11/site-packages/xgboost/lib/libxgboost.so
#4  0x00007ff719a9c93c in _ZN7xgboost4tree21MultiHistogramBuilder18BuildHistLeftRightINS0_20CommonRowPartitionerENS0_14CPUExpandEntryEEEvPNS_7DMatrixEPKNS_7RegTreeERKSt6vectorIT_SaISB_EERKSA_IT0_SaISG_EENS_6linalg10TensorViewIKNS_6detail20GradientPairInternalIfEELi2EEERKNS_10BatchParamEb!()+1979 in /home/v/.local/lib/python3.11/site-packages/xgboost/lib/libxgboost.so
#5  0x00007ff719b49783 in _ZN7xgboost4tree10UpdateTreeINS0_14CPUExpandEntryENS0_11HistUpdaterEEEvPNS_6common7MonitorENS_6linalg10TensorViewIKNS_6detail20GradientPairInternalIfEELi2EEEPT0_PNS_7DMatrixEPKNS0_10TrainParamEPNS_16HostDeviceVectorIiEEPNS_7RegTreeE!()+4866 in /home/v/.local/lib/python3.11/site-packages/xgboost/lib/libxgboost.so
#6  0x00007ff719b4b610 in _ZN7xgboost4tree17QuantileHistMaker6UpdateEPKNS0_10TrainParamEPNS_16HostDeviceVectorINS_6detail20GradientPairInternalIfEEEEPNS_7DMatrixENS_6common4SpanINS5_IiEELm18446744073709551615EEERKSt6vectorIPNS_7RegTreeESaISJ_EE!()+3663 in /home/v/.local/lib/python3.11/site-packages/xgboost/lib/libxgboost.so
#7  0x00007ff719860c79 in _ZN7xgboost3gbm6GBTree13BoostNewTreesEPNS_16HostDeviceVectorINS_6detail20GradientPairInternalIfEEEEPNS_7DMatrixEiPSt6vectorINS2_IiEESaISB_EEPSA_ISt10unique_ptrINS_7RegTreeESt14default_deleteISG_EESaISJ_EE!()+3720 in /home/v/.local/lib/python3.11/site-packages/xgboost/lib/libxgboost.so
#8  0x00007ff71986176c in _ZN7xgboost3gbm6GBTree7DoBoostEPNS_7DMatrixEPNS_16HostDeviceVectorINS_6detail20GradientPairInternalIfEEEEPNS_20PredictionCacheEntryEPKNS_11ObjFunctionE!()+1019 in /home/v/.local/lib/python3.11/site-packages/xgboost/lib/libxgboost.so
#9  0x00007ff7198c54f7 in _ZN7xgboost11LearnerImpl13UpdateOneIterEiSt10shared_ptrINS_7DMatrixEE!()+758 in /home/v/.local/lib/python3.11/site-packages/xgboost/lib/libxgboost.so
#10 0x00007ff719561ef0 in XGBoosterUpdateOneIter!()+111 in /home/v/.local/lib/python3.11/site-packages/xgboost/lib/libxgboost.so
#11 0x00007ff822d034f6 in <unknown>() in /usr/lib/libffi.so.8
#12 0x00007ff822cfff5e in <unknown>() in /usr/lib/libffi.so.8
#13 0x00007ff822d02b73 in ffi_call!()+290 in /usr/lib/libffi.so.8
#14 0x00007ff8228e900f in <unknown>() in /usr/lib/python3.11/lib-dynload/_ctypes.cpython-311-x86_64-linux-gnu.so
#15 0x00007ff8228e8831 in <unknown>() in /usr/lib/python3.11/lib-dynload/_ctypes.cpython-311-x86_64-linux-gnu.so
#16 0x00007ff820dd953c in _PyObject_MakeTpCall!()+651 in /usr/lib/libpython3.11.so.1.0
#17 0x00007ff820de3839 in _PyEval_EvalFrameDefault!()+1880 in /usr/lib/libpython3.11.so.1.0
#18 0x00007ff820e09560 in _PyFunction_Vectorcall!()+399 in /usr/lib/libpython3.11.so.1.0
#19 0x00007ff820e13bca in PyObject_Call!()+169 in /usr/lib/libpython3.11.so.1.0
#20 0x00007ff820de722b in _PyEval_EvalFrameDefault!()+16714 in /usr/lib/libpython3.11.so.1.0
#21 0x00007ff820e09560 in _PyFunction_Vectorcall!()+399 in /usr/lib/libpython3.11.so.1.0
#22 0x00007ff820e13bca in PyObject_Call!()+169 in /usr/lib/libpython3.11.so.1.0
#23 0x00007ff820de722b in _PyEval_EvalFrameDefault!()+16714 in /usr/lib/libpython3.11.so.1.0
#24 0x00007ff820e09560 in _PyFunction_Vectorcall!()+399 in /usr/lib/libpython3.11.so.1.0
#25 0x00007ff821d364f7 in pyo3::instance::Py<pyo3::types::any::PyAny>::call<pyo3::types::any::PyAny, &pyo3::types::tuple::PyTuple>()+214 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so at instance.rs:631
#26 0x00007ff821d36d5a in pyo3::instance::Py<pyo3::types::any::PyAny>::call1<pyo3::types::any::PyAny, &pyo3::types::tuple::PyTuple>()+41 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so at instance.rs:642
#27 0x00007ff821c91a22 in pgml::bindings::sklearn::fit::{closure#0}()+1233 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so at mod.rs:154
#28 0x00007ff821a18aa0 in pyo3::marker::Python::with_gil<pgml::bindings::sklearn::fit::{closure_env#0}, core::result::Result<(pyo3::instance::Py<pyo3::types::any::PyAny>, pyo3::instance::Py<pyo3::types::any::PyAny>, pyo3::instance::Py<pyo3::types::any::PyAny>), anyhow::Error>>()+159 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so at marker.rs:320
#29 0x00007ff82196a54e in pgml::bindings::sklearn::fit()+205 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so at mod.rs:135
#30 0x00007ff82196b3b7 in pgml::bindings::sklearn::xgboost_classification()+38 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so at mod.rs:24
#31 0x00007ff821a8b5b2 in pgml::orm::model::Model::get_bindings_and_metrics()+1057 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so at model.rs:713
#32 0x00007ff821a8c93e in pgml::orm::model::Model::fit()+2941 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so at model.rs:812
#33 0x00007ff821a84ba9 in pgml::orm::model::Model::create()+2408 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so at model.rs:148
#34 0x00007ff821b0e401 in pgml::api::train_joint()+5376 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so at api.rs:229
#35 0x00007ff821b07e11 in pgml::api::train()+1008 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so at api.rs:117
#36 0x00007ff821cf92b7 in pgml::api::tests::test_train_binary_classification()+2646 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so at api.rs:1564
#37 0x00007ff821cf988a in pgml::api::tests::test_train_binary_classification_wrapper::test_train_binary_classification_wrapper_inner()+9 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so at api.rs:1552
#38 0x00007ff821d7f181 in pgml::api::tests::test_train_binary_classification_wrapper::{closure#0}()+16 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so at api.rs:1552
#39 0x00007ff821d198c9 in std::panicking::try::do_call<pgml::api::tests::test_train_binary_classification_wrapper::{closure_env#0}, ()>()+40 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so at panicking.rs:500
#40 0x00007ff821d2240b in __rust_try!()+26 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so
#41 0x00007ff821d15baa in std::panicking::try<(), pgml::api::tests::test_train_binary_classification_wrapper::{closure_env#0}>()+57 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so at panicking.rs:464
#42 0x00007ff821a7241e in std::panic::catch_unwind<pgml::api::tests::test_train_binary_classification_wrapper::{closure_env#0}, ()>()+13 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so at panic.rs:142
#43 0x00007ff821c3beff in pgrx_pg_sys::submodules::panic::run_guarded<pgml::api::tests::test_train_binary_classification_wrapper::{closure_env#0}, ()>()+46 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so at panic.rs:408
#44 0x00007ff821c50540 in pgrx_pg_sys::submodules::panic::pgrx_extern_c_guard<pgml::api::tests::test_train_binary_classification_wrapper::{closure_env#0}, ()>()+31 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so at panic.rs:385
#45 0x00007ff821cf986e in pgml::api::tests::test_train_binary_classification_wrapper()+29 in /home/v/.pgrx/16.0/pgrx-install/lib/postgresql/pgml.so at api.rs:1552
#46 0x000056371e54fc2f in ExecInterpExpr()+1774 in /home/v/.pgrx/16.0/pgrx-install/bin/postgres at execExprInterp.c:734
#47 inlined            in ExecEvalExprSwitchContext at executor.h:355
#47 inlined            in ExecProject at executor.h:389
#47 0x000056371e5884a5 in ExecResult()+308 in /home/v/.pgrx/16.0/pgrx-install/bin/postgres at nodeResult.c:136
#48 inlined            in ExecProcNode at executor.h:273
#48 inlined            in ExecutePlan at execMain.c:1670
#48 0x000056371e5549fa in standard_ExecutorRun()+313 in /home/v/.pgrx/16.0/pgrx-install/bin/postgres at execMain.c:365
#49 0x000056371e712fbf in PortalRunSelect()+510 in /home/v/.pgrx/16.0/pgrx-install/bin/postgres at pquery.c:924
#50 0x000056371e71472e in PortalRun()+621 in /home/v/.pgrx/16.0/pgrx-install/bin/postgres at pquery.c:768
#51 0x000056371e7105f6 in exec_simple_query()+821 in /home/v/.pgrx/16.0/pgrx-install/bin/postgres at postgres.c:1274
#52 0x000056371e711fd4 in PostgresMain()+5523 in /home/v/.pgrx/16.0/pgrx-install/bin/postgres at postgres.c:4637
#53 inlined            in BackendRun at postmaster.c:4464
#53 inlined            in BackendStartup at postmaster.c:4192
#53 0x000056371e67ab86 in ServerLoop()+7813 in /home/v/.pgrx/16.0/pgrx-install/bin/postgres at postmaster.c:1782
#54 0x000056371e67bbaf in PostmasterMain()+3582 in /home/v/.pgrx/16.0/pgrx-install/bin/postgres at postmaster.c:1466
#55 0x000056371e38b57a in main()+521 in /home/v/.pgrx/16.0/pgrx-install/bin/postgres at main.c:198
#56 0x00007ff825445cd0 in <unknown>() in /usr/lib/libc.so.6
#57 0x00007ff825445d8a in __libc_start_main!()+137 in /usr/lib/libc.so.6
#58 0x000056371e38b8b5 in _start!()+36 in /home/v/.pgrx/16.0/pgrx-install/bin/postgres

thread: 0x7ff6ffef16c0, lwp: 180964, type: 1
#0  0x00007ff82518c7ae in gomp_barrier_wait_end()+110 in /usr/lib/libgomp.so.1 at futex.h:97
#1  inlined            in gomp_simple_barrier_wait at simple-bar.h:60
#1  0x00007ff825189c70 in gomp_thread_start()+303 in /usr/lib/libgomp.so.1 at team.c:133
#2  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#3  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7006f26c0, lwp: 180963, type: 1
#0  0x00007ff82518c7ae in gomp_barrier_wait_end()+110 in /usr/lib/libgomp.so.1 at futex.h:97
#1  inlined            in gomp_simple_barrier_wait at simple-bar.h:60
#1  0x00007ff825189c70 in gomp_thread_start()+303 in /usr/lib/libgomp.so.1 at team.c:133
#2  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#3  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff700ef36c0, lwp: 180962, type: 1
#0  0x00007ff82518c782 in gomp_barrier_wait_end()+66 in /usr/lib/libgomp.so.1 at wait.h:56
#1  inlined            in gomp_simple_barrier_wait at simple-bar.h:60
#1  0x00007ff825189c70 in gomp_thread_start()+303 in /usr/lib/libgomp.so.1 at team.c:133
#2  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#3  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7016f46c0, lwp: 180961, type: 1
#0  0x00007ff82518c782 in gomp_barrier_wait_end()+66 in /usr/lib/libgomp.so.1 at wait.h:56
#1  inlined            in gomp_simple_barrier_wait at simple-bar.h:60
#1  0x00007ff825189c70 in gomp_thread_start()+303 in /usr/lib/libgomp.so.1 at team.c:133
#2  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#3  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff701ef56c0, lwp: 180960, type: 1
#0  0x00007ff82518c7ae in gomp_barrier_wait_end()+110 in /usr/lib/libgomp.so.1 at futex.h:97
#1  inlined            in gomp_simple_barrier_wait at simple-bar.h:60
#1  0x00007ff825189c70 in gomp_thread_start()+303 in /usr/lib/libgomp.so.1 at team.c:133
#2  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#3  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7026f66c0, lwp: 180959, type: 1
#0  0x00007ff82518c7ae in gomp_barrier_wait_end()+110 in /usr/lib/libgomp.so.1 at futex.h:97
#1  inlined            in gomp_simple_barrier_wait at simple-bar.h:60
#1  0x00007ff825189c70 in gomp_thread_start()+303 in /usr/lib/libgomp.so.1 at team.c:133
#2  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#3  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff702ef76c0, lwp: 180958, type: 1
#0  0x00007ff82518c7ae in gomp_barrier_wait_end()+110 in /usr/lib/libgomp.so.1 at futex.h:97
#1  inlined            in gomp_simple_barrier_wait at simple-bar.h:60
#1  0x00007ff825189c70 in gomp_thread_start()+303 in /usr/lib/libgomp.so.1 at team.c:133
#2  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#3  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7036f86c0, lwp: 180957, type: 1
#0  0x00007ff82518c7ae in gomp_barrier_wait_end()+110 in /usr/lib/libgomp.so.1 at futex.h:97
#1  inlined            in gomp_simple_barrier_wait at simple-bar.h:60
#1  0x00007ff825189c70 in gomp_thread_start()+303 in /usr/lib/libgomp.so.1 at team.c:133
#2  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#3  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff703ef96c0, lwp: 180956, type: 1
#0  0x00007ff82518c7ae in gomp_barrier_wait_end()+110 in /usr/lib/libgomp.so.1 at futex.h:97
#1  inlined            in gomp_simple_barrier_wait at simple-bar.h:60
#1  0x00007ff825189c70 in gomp_thread_start()+303 in /usr/lib/libgomp.so.1 at team.c:133
#2  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#3  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7046fa6c0, lwp: 180955, type: 1
#0  0x00007ff82518c782 in gomp_barrier_wait_end()+66 in /usr/lib/libgomp.so.1 at wait.h:56
#1  inlined            in gomp_simple_barrier_wait at simple-bar.h:60
#1  0x00007ff825189c70 in gomp_thread_start()+303 in /usr/lib/libgomp.so.1 at team.c:133
#2  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#3  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff704efb6c0, lwp: 180954, type: 1
#0  0x00007ff82518c782 in gomp_barrier_wait_end()+66 in /usr/lib/libgomp.so.1 at wait.h:56
#1  inlined            in gomp_simple_barrier_wait at simple-bar.h:60
#1  0x00007ff825189c70 in gomp_thread_start()+303 in /usr/lib/libgomp.so.1 at team.c:133
#2  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#3  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7056fc6c0, lwp: 180953, type: 1
#0  0x00007ff82518c7ae in gomp_barrier_wait_end()+110 in /usr/lib/libgomp.so.1 at futex.h:97
#1  inlined            in gomp_simple_barrier_wait at simple-bar.h:60
#1  0x00007ff825189c70 in gomp_thread_start()+303 in /usr/lib/libgomp.so.1 at team.c:133
#2  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#3  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff705efd6c0, lwp: 180952, type: 1
#0  0x00007ff82518c782 in gomp_barrier_wait_end()+66 in /usr/lib/libgomp.so.1 at wait.h:56
#1  inlined            in gomp_simple_barrier_wait at simple-bar.h:60
#1  0x00007ff825189c70 in gomp_thread_start()+303 in /usr/lib/libgomp.so.1 at team.c:133
#2  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#3  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7066fe6c0, lwp: 180951, type: 1
#0  0x00007ff82518c786 in gomp_barrier_wait_end()+70 in /usr/lib/libgomp.so.1 at wait.h:56
#1  inlined            in gomp_simple_barrier_wait at simple-bar.h:60
#1  0x00007ff825189c70 in gomp_thread_start()+303 in /usr/lib/libgomp.so.1 at team.c:133
#2  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#3  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff706eff6c0, lwp: 180950, type: 1
#0  0x00007ff82518c782 in gomp_barrier_wait_end()+66 in /usr/lib/libgomp.so.1 at wait.h:56
#1  inlined            in gomp_simple_barrier_wait at simple-bar.h:60
#1  0x00007ff825189c70 in gomp_thread_start()+303 in /usr/lib/libgomp.so.1 at team.c:133
#2  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#3  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff733c066c0, lwp: 180941, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff737532e3f in th_worker()+942 in /home/v/.local/lib/python3.11/site-packages/numexpr/interpreter.cpython-311-x86_64-linux-gnu.so at module.cpp:62
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7344076c0, lwp: 180940, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff737532e3f in th_worker()+942 in /home/v/.local/lib/python3.11/site-packages/numexpr/interpreter.cpython-311-x86_64-linux-gnu.so at module.cpp:62
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff734c086c0, lwp: 180939, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff737532e3f in th_worker()+942 in /home/v/.local/lib/python3.11/site-packages/numexpr/interpreter.cpython-311-x86_64-linux-gnu.so at module.cpp:62
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7354096c0, lwp: 180938, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff737532e3f in th_worker()+942 in /home/v/.local/lib/python3.11/site-packages/numexpr/interpreter.cpython-311-x86_64-linux-gnu.so at module.cpp:62
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff735c0a6c0, lwp: 180937, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff737532e3f in th_worker()+942 in /home/v/.local/lib/python3.11/site-packages/numexpr/interpreter.cpython-311-x86_64-linux-gnu.so at module.cpp:62
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff73640b6c0, lwp: 180936, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff737532e3f in th_worker()+942 in /home/v/.local/lib/python3.11/site-packages/numexpr/interpreter.cpython-311-x86_64-linux-gnu.so at module.cpp:62
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff736c0c6c0, lwp: 180935, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff737532e3f in th_worker()+942 in /home/v/.local/lib/python3.11/site-packages/numexpr/interpreter.cpython-311-x86_64-linux-gnu.so at module.cpp:62
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff73740d6c0, lwp: 180934, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff737532e3f in th_worker()+942 in /home/v/.local/lib/python3.11/site-packages/numexpr/interpreter.cpython-311-x86_64-linux-gnu.so at module.cpp:62
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff73cdff6c0, lwp: 180924, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff73f5069a5 in <unknown>() in /usr/lib/libarrow.so.1300
#3  0x00007ff73f506383 in <unknown>() in /usr/lib/libarrow.so.1300
#4  0x00007ff73f50625e in <unknown>() in /usr/lib/libarrow.so.1300
#5  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#6  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff762df16c0, lwp: 180879, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff76a1407db in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/scipy/sparse/linalg/_isolve/../../../../scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7635f26c0, lwp: 180878, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff76a1407db in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/scipy/sparse/linalg/_isolve/../../../../scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff763df36c0, lwp: 180877, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff76a1407db in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/scipy/sparse/linalg/_isolve/../../../../scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7645f46c0, lwp: 180876, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff76a1407db in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/scipy/sparse/linalg/_isolve/../../../../scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff764df56c0, lwp: 180875, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff76a1407db in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/scipy/sparse/linalg/_isolve/../../../../scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7655f66c0, lwp: 180874, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff76a1407db in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/scipy/sparse/linalg/_isolve/../../../../scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff765df76c0, lwp: 180873, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff76a1407db in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/scipy/sparse/linalg/_isolve/../../../../scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7665f86c0, lwp: 180872, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff76a1407db in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/scipy/sparse/linalg/_isolve/../../../../scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff766df96c0, lwp: 180871, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff76a1407db in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/scipy/sparse/linalg/_isolve/../../../../scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7675fa6c0, lwp: 180870, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff76a1407db in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/scipy/sparse/linalg/_isolve/../../../../scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff767dfb6c0, lwp: 180869, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff76a1407db in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/scipy/sparse/linalg/_isolve/../../../../scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7685fc6c0, lwp: 180868, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff76a1407db in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/scipy/sparse/linalg/_isolve/../../../../scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff768dfd6c0, lwp: 180867, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff76a1407db in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/scipy/sparse/linalg/_isolve/../../../../scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7695fe6c0, lwp: 180866, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff76a1407db in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/scipy/sparse/linalg/_isolve/../../../../scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff769dff6c0, lwp: 180865, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff76a1407db in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/scipy/sparse/linalg/_isolve/../../../../scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff78b1f16c0, lwp: 180849, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff792f3d77b in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-7a851222.3.23.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff78b9f26c0, lwp: 180848, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff792f3d77b in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-7a851222.3.23.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff78c1f36c0, lwp: 180847, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff792f3d77b in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-7a851222.3.23.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff78c9f46c0, lwp: 180846, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff792f3d77b in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-7a851222.3.23.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff78d1f56c0, lwp: 180845, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff792f3d77b in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-7a851222.3.23.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff78d9f66c0, lwp: 180844, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff792f3d77b in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-7a851222.3.23.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff78e1f76c0, lwp: 180843, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff792f3d77b in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-7a851222.3.23.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff78e9f86c0, lwp: 180842, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff792f3d77b in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-7a851222.3.23.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff78f1f96c0, lwp: 180841, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff792f3d77b in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-7a851222.3.23.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff78f9fa6c0, lwp: 180840, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff792f3d77b in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-7a851222.3.23.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7901fb6c0, lwp: 180839, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff792f3d77b in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-7a851222.3.23.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7909fc6c0, lwp: 180838, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff792f3d77b in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-7a851222.3.23.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7911fd6c0, lwp: 180837, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff792f3d77b in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-7a851222.3.23.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7919fe6c0, lwp: 180836, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff792f3d77b in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-7a851222.3.23.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

thread: 0x7ff7921ff6c0, lwp: 180835, type: 1
#0  0x00007ff8254a74ae in <unknown>() in /usr/lib/libc.so.6
#1  0x00007ff8254a9d40 in pthread_cond_wait!()+511 in /usr/lib/libc.so.6
#2  0x00007ff792f3d77b in blas_thread_server!()+154 in /home/v/.local/lib/python3.11/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-7a851222.3.23.so
#3  0x00007ff8254aa9eb in <unknown>() in /usr/lib/libc.so.6
#4  0x00007ff82552e7cc in <unknown>() in /usr/lib/libc.so.6

kczimm · Answer 1 · Thu Nov 16 2023 22:31:52 GMT+0800 (China Standard Time)

Hi @higuoxing, thanks for filing this issue. We are aware that OMP_NUM_THREADS environment variable can control the threading. Have you confirmed that the torch.set_num_threads provides the same results?

Xing Guo · Answer 2 · Fri Nov 17 2023 14:46:38 GMT+0800 (China Standard Time)

Hi @higuoxing, thanks for filing this issue. We are aware that OMP_NUM_THREADS environment variable can control the threading. Have you confirmed that the torch.set_num_threads provides the same results?

Sorry. Not only PyTorch but also sklearn is spawning many threads (See the stack trace from my issue description). The thread number of sklearn can also be controlled by setting the OMP_NUM_THREADS environment variable. Probably we can adding some GUC parameter for setting it?

Xing Guo · Answer 3 · Fri Nov 17 2023 14:48:13 GMT+0800 (China Standard Time)

But I do see there're paramters for limiting the degree of parallelism from pgml. E.g., pgml.predict_concurrency. Can we use a simple GUC to controll them all?

Lev Kokotov · Answer 4 · Sat Nov 18 2023 03:39:22 GMT+0800 (China Standard Time)

pgml.predict_concurrency is being passed to XGBoost (and LightGBM I believe) directly via an argument. OMP_NUM_THREADS will only work if it's set before the Python runtime is loaded I believe, so setting it in _PG_init from a GUC value should work.

Xing Guo · Answer 5 · Mon Nov 20 2023 15:56:51 GMT+0800 (China Standard Time)

pgml.predict_concurrency is being passed to XGBoost (and LightGBM I believe) directly via an argument. OMP_NUM_THREADS will only work if it's set before the Python runtime is loaded I believe, so setting it in _PG_init from a GUC value should work.

Thanks! Many packages are using openmp as their multi-threaded backend for CPU runtime. What do you think of introducing a new GUC pgml.omp_num_threads (or some better name) to limit the degree of parallelism for them? Namely, XGBoost, LightGBM, scikit-learn, Pytorch, openblas (it uses the system implementation and sometimes its backend is not openmp and probably we can statically link it to force it to use the openmp backend?). If the proposal sounds good I can raise a PR for it after #1146 gets merged (since it refactors the GUC initialization codes).

Montana Low · Answer 6 · Tue Nov 21 2023 01:01:49 GMT+0800 (China Standard Time)

Since this has bitten a few people, I think it's worth having the pgml.omp_num_threads config setting as a GUC, just for discoverability and documentation on the best practices for parallelism in a Postgres multi-process server. PR would be welcome, and we should add a section in the docs mentioning all of these new settings.

kczimm · Answer 7 · Tue Mar 05 2024 04:52:40 GMT+0800 (China Standard Time)

Environment variables like OMP_NUM_THREADS and OPENBLAS_NUM_THREADS need to be set in the environment before starting Postgres. This can easily be done as mentioned earlier: OMP_NUM_THREADS=1 cargo pgrx run but is not as easily achieved when running Postgres using systemd. In this situation, we've not found any success when setting these variables from the extension, either in the postmaster process via _PG_init or the client process.

Xing Guo · Answer 8 · Wed Mar 06 2024 20:05:54 GMT+0800 (China Standard Time)

Environment variables like OMP_NUM_THREADS and OPENBLAS_NUM_THREADS need to be set in the environment before starting Postgres. This can easily be done as mentioned earlier: OMP_NUM_THREADS=1 cargo pgrx run but is not as easily achieved when running Postgres using systemd. In this situation, we've not found any success when setting these variables from the extension, either in the postmaster process via _PG_init or the client process.

Ah, you're right. When the program gets started, openmp simply ignores modified environment variables¹.

This chapter describes the OpenMP environment variables that specify the settings of the ICVs that affect the execution of OpenMP programs (see Chapter 2). The names of the environment variables must be upper case. Unless otherwise specified, the values assigned to the environment variables are case insensitive and may have leading and trailing white space. Modifications to the environment variables after the program has started, even if modified by the program itself, are ignored by the OpenMP implementation. However, the settings of some of the ICVs can be modified during the execution of the OpenMP program by the use of the appropriate directive clauses or OpenMP API routines.

One possible solution is to call omp_set_num_threads() manually, but requires introducing an unsafe {} block.

diff --git a/pgml-extension/src/lib.rs b/pgml-extension/src/lib.rs
index 6c2884ce..c024ddee 100644
--- a/pgml-extension/src/lib.rs
+++ b/pgml-extension/src/lib.rs
@@ -21,9 +21,16 @@ pg_module_magic!();

 extension_sql_file!("../sql/schema.sql", name = "schema");

+extern "C" {
+    fn omp_set_num_threads(num_threads: i32);
+}
+
 #[cfg(not(feature = "use_as_lib"))]
 #[pg_guard]
 pub extern "C" fn _PG_init() {
+    unsafe {
+        omp_set_num_threads(1);
+    }
     bindings::python::activate().expect("Error setting python venv");
     orm::project::init();
 }

https://www.openmp.org/wp-content/uploads/OpenMP-API-Specification-5-2.pdf (Chapter 21) ↩

Lev Kokotov · Answer 9 · Thu Mar 07 2024 02:04:53 GMT+0800 (China Standard Time)

I've considered this, is there a way to ensure that the function exists before we call it? Nothing wrong with unsafe if we've made sure that the code works beforehand (on multiple platforms). Otherwise, what I've actually done on our side is forked Postgres and set the OMP_NUM_THREADS manually in pg_ctl. Seemed like the only way to register an environment variable before the pgml .so is loaded into the postmaster.

Xing Guo · Answer 10 · Thu Mar 07 2024 07:57:57 GMT+0800 (China Standard Time)

I've considered this, is there a way to ensure that the function exists before we call it? Nothing wrong with unsafe if we've made sure that the code works beforehand (on multiple platforms).

Since the OpenMP spec mentions that there's a function called omp_set_num_threads, I think most platforms shall provide it?

BTW, PostgreSQL is loading libraries with the RTLD_NOW flag and all of needed symbols are resolved at that time. Even if we're referencing an undefined symbol, we will know it as soon as possible.
https://github.com/postgres/postgres/blob/099ca50bd41c1a72fe5a086bb095858ea74c88de/src/backend/utils/fmgr/dfmgr.c#L239-L249

pgml=# create extension pgml;
ERROR:  could not load library "/home/v/.pgrx/16.2/pgrx-install/lib/postgresql/pgml.so": /home/v/.pgrx/16.2/pgrx-install/lib/postgresql/pgml.so: undefined symbol: omp_unknown_func

Otherwise, what I've actually done on our side is forked Postgres and set the OMP_NUM_THREADS manually in pg_ctl. Seemed like the only way to register an environment variable before the pgml .so is loaded into the postmaster.

IMHO, modifying pg_ctl sounds a little bit invasive.

Lev Kokotov · Answer 11 · Fri Mar 08 2024 01:19:01 GMT+0800 (China Standard Time)

I agree. We'll accept a patch using omp_set_num_threads.

Xing Guo · Answer 12 · Thu Apr 04 2024 11:19:19 GMT+0800 (China Standard Time)

Hi, I submit a PR to resolve this issue, could you please take a look? #1362

PostgresML is spawning too many threads.

Root cause

Reproducing steps

Stack trace

Footnotes