Pickling functions defined in doctests fail
Erotemic opened this issue · comments
New ubelt futures test failed when trying to run in xdoctest:
>>> # xdoctest: +SKIP
>>> # Note: while this works in IPython, this does not work when running
>>> # in xdoctest.
>>> # xdoctest: +REQUIRES(module:timerit)
>>> # Does my function benefit from parallelism?
>>> def my_function(arg1, arg2):
... return (arg1 + arg2) * 3
>>> #
>>> def run_process(inputs, mode='serial', max_workers=0):
... from concurrent.futures import as_completed
... import ubelt as ub
... # The executor interface is the same regardless of modes
... executor = ub.Executor(mode=mode, max_workers=max_workers)
... # submit returns a Future object
... jobs = [executor.submit(my_function, *args) for args in inputs]
... # future objects will contain results when they are done
... results = [job.result() for job in as_completed(jobs)]
... return results
>>> # The same code tests our method in serial, thread, or process mode
>>> import timerit
>>> ti = timerit.Timerit(100, bestof=10, verbose=2)
>>> # Setup test data
>>> import random
>>> rng = random.Random(0)
>>> max_workers = 4
>>> inputs = [(rng.random(), rng.random()) for _ in range(100)]
>>> for mode in ['serial', 'process', 'thread']:
>>> for timer in ti.reset('mode={} max_workers={}'.format(mode, max_workers)):
>>> with timer:
>>> run_process(inputs, mode=mode, max_workers=max_workers)
>>> print(ub.repr2(ti))
Error is:
DOCTEST TRACEBACK
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/queues.py", line 239, in _feed
obj = _ForkingPickler.dumps(obj)
File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function my_function at 0x7f7f27f5e430>: attribute lookup my_function on ubelt.util_futures failed
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<doctest:/home/joncrall/code/ubelt/ubelt/util_futures.py::__doc__:0>", line rel: 27, abs: 37, in <module>
>>> run_process(inputs, mode=mode, max_workers=max_workers)
File "<doctest:/home/joncrall/code/ubelt/ubelt/util_futures.py::__doc__:0>", line rel: 14, abs: 24, in run_process
... results = [job.result() for job in as_completed(jobs)]
File "<doctest:/home/joncrall/code/ubelt/ubelt/util_futures.py::__doc__:0>", line rel: 14, abs: 24, in <listcomp>
... results = [job.result() for job in as_completed(jobs)]
File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
raise self._exception
File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/queues.py", line 239, in _feed
obj = _ForkingPickler.dumps(obj)
File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function my_function at 0x7f7f27f5e430>: attribute lookup my_function on ubelt.util_futures failed
DOCTEST REPRODUCTION
CommandLine:
python -m xdoctest /home/joncrall/code/ubelt/ubelt/util_futures.py __doc__:0
=== Failed tests ===
python -m xdoctest /home/joncrall/code/ubelt/ubelt/util_futures.py __doc__:0
=== 1 failed, 6 passed in 0.14 seconds ===
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/concurrent/futures/process.py", line 104, in _python_exit
Process ForkProcess-4:
Process ForkProcess-2:
Process ForkProcess-1:
Process ForkProcess-3:
This is related to the issue I just saw in scriptconfig.
The fix is that we need to allow xdoctest to create a dummy module scope for it to execute its code in. See: https://stackoverflow.com/questions/5122465/can-i-fake-a-package-or-at-least-a-module-in-python-for-testing-purposes
Not exactly sure how to do this yet.