The aim of this project is to provide a robust, cross-platform and
cross-version implementation of the ProcessPoolExecutor
class of
concurrent.futures
. It features:
-
Deadlock free implementation: one of the major concern in standard
multiprocessing
andconcurrent.futures
libraries is the ability of thePool/Executor
to handle crashes of worker processes. This library intends to fix those possible deadlocks and send back meaningful errors. -
Consistent spawn behavior: All processes are started using fork/exec on POSIX systems. This ensures safer interactions with third party libraries.
-
Reusable executor: strategy to avoid respawning a complete executor every time. A singleton pool can be reused (and dynamically resized if necessary) across consecutive calls to limit spawning and shutdown overhead. The worker processes can be shutdown automatically after a configurable idling timeout to free system resources.
import os
from time import sleep
from loky.reusable_executor import get_reusable_executor
def say_hello(k):
pid = os.getpid()
print("hello from {} with arg {}".format(pid, k))
sleep(.01)
return pid
if __name__ == "__main__":
# Create an executor with max_workers workers, that will shutdown after 2s
max_workers = 4
executor = get_reusable_executor(max_workers=max_workers, timeout=2)
res = executor.submit(say_hello, 1)
print("got result", res.result())
results = executor.map(say_hello, range(50))
n_workers = len(set(results))
print("# used processes:", n_workers)
assert n_workers == max_workers
executor.shutdown(wait=True)
This work is supported by the Center for Data Science, funded by the IDEX Paris-Saclay, ANR-11-IDEX-0003-02