Loky test suite is much slower on macOS than on Linux or Windows
ogrisel opened this issue · comments
For instance on:
The macOS runs (install + test + results upload) take around 30 min while the Windows and Linux builds run in 7 min.
But it could be the case that the hardware is not comparable (or that the macOS VMs are more oversubscribed and run slower generally than the Windows and Linux VMs on Azure).
On my local macOS (M1) machine, the test suite run in 7 min which is not too bad but the same test suite runs in 3 min on the same machine within a linux docker container.
So there is something macOS specific that makes loky run 2x slower on that OS.
I think we need to isolated an minimal case where loky macOS is significantly slower than linux and then check whether this is also the case with concurrent.futures.ProcessPoolExecutor and the spawn start method.
Something weird, I just observed on a running macos build that pytest has finished running 100% of the tests in 17 min (which is already quite slow) but then waits several minutes joining a subprocess in a the atexit finalizers in MainProcess:MainThread.
=========== 292 passed, 22 skipped, 1 warning in 1028.79s (0:17:08) ============
[INFO:MainProcess:MainThread] process shutting down
[DEBUG:MainProcess:MainThread] running all "atexit" finalizers with priority >= 0
[DEBUG:MainProcess:MainThread] telling queue thread to quit
[DEBUG:MainProcess:QueueFeederThread] feeder thread got sentinel -- exiting
[INFO:MainProcess:MainThread] calling join() for process LokyProcess-1334
Here is the link of the build:
This also happened on this run (the merge of #394 in the master branch):
The tests ran in 20 min according to pytest but then an extra 10 minutes were wasted in joining LokyProcess instances in the atexit finalizers.