Pickling error on OS X / Python 3.8 (but not in 3.7)
avirshup opened this issue · comments
I am getting very similar errors to those described in #54, but in a different context:
I'm using Python 3.8 on MacOS.
Note that everything works fine with python 3.7!
$ py.test -x
========================================== test session starts ==========================================
platform darwin -- Python 3.8.1, pytest-5.3.2, py-1.8.1, pluggy-0.13.1
rootdir: [...], inifile: pytest.ini
plugins: typeguard-2.7.1, flask-0.15.0
test_cli.py E
================================================ ERRORS =================================================
_______________________________ ERROR at setup of test_idempotent_upload ________________________________
[...]
/Users/avirshup/.pyenv/versions/anaconda3-2019.10/envs/wcdb-py38/lib/python3.8/site-packages/pytest_flask/fixtures.py:72: in start
self._process.start()
/Users/avirshup/.pyenv/versions/anaconda3-2019.10/envs/wcdb-py38/lib/python3.8/multiprocessing/process.py:121: in start
self._popen = self._Popen(self)
/Users/avirshup/.pyenv/versions/anaconda3-2019.10/envs/wcdb-py38/lib/python3.8/multiprocessing/context.py:224: in _Popen
return _default_context.get_context().Process._Popen(process_obj)
/Users/avirshup/.pyenv/versions/anaconda3-2019.10/envs/wcdb-py38/lib/python3.8/multiprocessing/context.py:283: in _Popen
return Popen(process_obj)
/Users/avirshup/.pyenv/versions/anaconda3-2019.10/envs/wcdb-py38/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
super().__init__(process_obj)
/Users/avirshup/.pyenv/versions/anaconda3-2019.10/envs/wcdb-py38/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
self._launch(process_obj)
/Users/avirshup/.pyenv/versions/anaconda3-2019.10/envs/wcdb-py38/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
obj = <Process name='Process-1' parent=61378 initial>, file = <_io.BytesIO object at 0x7ff570abf9f0>
protocol = None
def dump(obj, file, protocol=None):
'''Replacement for pickle.dump() using ForkingPickler.'''
> ForkingPickler(file, protocol).dump(obj)
E AttributeError: Can't pickle local object 'LiveServer.start.<locals>.worker'
/Users/avirshup/.pyenv/versions/anaconda3-2019.10/envs/wcdb-py38/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
======================================== short test summary info ========================================
ERROR test_cli.py::test_idempotent_upload - AttributeError: Can't pickle local object 'LiveServer.star...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=================================== 393 deselected, 1 error in 4.41s ====================================
This is probably because the defaults for multiprocessing changed in python 3.8. Now, by default, OS X multiprocessing starts processes with spawn instead of fork (like Windows). And with spawning, data is sent by pickling.
The workaround is simply to run multiprocessing.set_start_method("fork")
exactly once per test session before calling the live_server
fixture.
@avirshup thanks for posting the solution so others can find it.
Closing it for now.