ProcessPoolExecutor behaviour on MacOS in interactive environment changed between Python versions
basnijholt opened this issue · comments
Bas Nijholt commented
The following code works in an interactive environment (ipython
) on MacOS Python 3.6 and Python 3.7 but not Python 3.8 and 3.9:
from concurrent.futures import ProcessPoolExecutor
def peak(x, offset=0):
a = 0.01
return x + a**2 / (a**2 + (x - offset)**2)
if __name__ == '__main__':
ex = ProcessPoolExecutor(max_workers=1)
fut = ex.submit(peak, 1)
fut.result()
Which with Python 3.7 and 3.8 raises:
Process SpawnProcess-1:
Traceback (most recent call last):
File "/Users/basnijholt/miniconda3/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/Users/basnijholt/miniconda3/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/Users/basnijholt/miniconda3/lib/python3.8/concurrent/futures/process.py", line 233, in _process_worker
call_item = call_queue.get(block=True)
File "/Users/basnijholt/miniconda3/lib/python3.8/multiprocessing/queues.py", line 116, in get
return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'peak' on <module '__main__' (built-in)>
---------------------------------------------------------------------------
BrokenProcessPool Traceback (most recent call last)
<ipython-input-2-1f1b9bfb79b2> in <module>
7 ex = ProcessPoolExecutor(max_workers=1)
8 fut = ex.submit(peak, 1)
----> 9 fut.result()
10
~/miniconda3/lib/python3.8/concurrent/futures/_base.py in result(self, timeout)
437 raise CancelledError()
438 elif self._state == FINISHED:
--> 439 return self.__get_result()
440 else:
441 raise TimeoutError()
~/miniconda3/lib/python3.8/concurrent/futures/_base.py in __get_result(self)
386 def __get_result(self):
387 if self._exception:
--> 388 raise self._exception
389 else:
390 return self._result
BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
This means we need to change the default executor to Loky on MacOS.