python-adaptive / adaptive

:chart_with_upwards_trend: Adaptive: parallel active learning of mathematical functions

Home Page:http://adaptive.readthedocs.io/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

ProcessPoolExecutor behaviour on MacOS in interactive environment changed between Python versions

basnijholt opened this issue · comments

The following code works in an interactive environment (ipython) on MacOS Python 3.6 and Python 3.7 but not Python 3.8 and 3.9:

from concurrent.futures import ProcessPoolExecutor
def peak(x, offset=0):
    a = 0.01
    return x + a**2 / (a**2 + (x - offset)**2)

if __name__ == '__main__':
    ex = ProcessPoolExecutor(max_workers=1)
    fut = ex.submit(peak, 1)
    fut.result()

Which with Python 3.7 and 3.8 raises:

Process SpawnProcess-1:
Traceback (most recent call last):
  File "/Users/basnijholt/miniconda3/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/basnijholt/miniconda3/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/basnijholt/miniconda3/lib/python3.8/concurrent/futures/process.py", line 233, in _process_worker
    call_item = call_queue.get(block=True)
  File "/Users/basnijholt/miniconda3/lib/python3.8/multiprocessing/queues.py", line 116, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'peak' on <module '__main__' (built-in)>
---------------------------------------------------------------------------
BrokenProcessPool                         Traceback (most recent call last)
<ipython-input-2-1f1b9bfb79b2> in <module>
      7     ex = ProcessPoolExecutor(max_workers=1)
      8     fut = ex.submit(peak, 1)
----> 9     fut.result()
     10

~/miniconda3/lib/python3.8/concurrent/futures/_base.py in result(self, timeout)
    437                 raise CancelledError()
    438             elif self._state == FINISHED:
--> 439                 return self.__get_result()
    440             else:
    441                 raise TimeoutError()

~/miniconda3/lib/python3.8/concurrent/futures/_base.py in __get_result(self)
    386     def __get_result(self):
    387         if self._exception:
--> 388             raise self._exception
    389         else:
    390             return self._result

BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

This means we need to change the default executor to Loky on MacOS.