Pickling functions defined in doctests fail

[Jon Crall]

New ubelt futures test failed when trying to run in xdoctest:

    >>> # xdoctest: +SKIP
    >>> # Note: while this works in IPython, this does not work when running
    >>> # in xdoctest. 
    >>> # xdoctest: +REQUIRES(module:timerit)
    >>> # Does my function benefit from parallelism?
    >>> def my_function(arg1, arg2):
    ...     return (arg1 + arg2) * 3
    >>> #
    >>> def run_process(inputs, mode='serial', max_workers=0):
    ...     from concurrent.futures import as_completed
    ...     import ubelt as ub
    ...     # The executor interface is the same regardless of modes
    ...     executor = ub.Executor(mode=mode, max_workers=max_workers)
    ...     # submit returns a Future object
    ...     jobs = [executor.submit(my_function, *args) for args in inputs]
    ...     # future objects will contain results when they are done
    ...     results = [job.result() for job in as_completed(jobs)]
    ...     return results
    >>> # The same code tests our method in serial, thread, or process mode
    >>> import timerit
    >>> ti = timerit.Timerit(100, bestof=10, verbose=2)
    >>> # Setup test data
    >>> import random
    >>> rng = random.Random(0)
    >>> max_workers = 4
    >>> inputs = [(rng.random(), rng.random()) for _ in range(100)]
    >>> for mode in ['serial', 'process', 'thread']:
    >>>     for timer in ti.reset('mode={} max_workers={}'.format(mode, max_workers)):
    >>>         with timer:
    >>>             run_process(inputs, mode=mode, max_workers=max_workers)
    >>> print(ub.repr2(ti))

Error is:

    DOCTEST TRACEBACK
    concurrent.futures.process._RemoteTraceback: 
    """
    Traceback (most recent call last):
      File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/queues.py", line 239, in _feed
        obj = _ForkingPickler.dumps(obj)
      File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
        cls(buf, protocol).dump(obj)
    _pickle.PicklingError: Can't pickle <function my_function at 0x7f7f27f5e430>: attribute lookup my_function on ubelt.util_futures failed
    """
    
    
    The above exception was the direct cause of the following exception:
    
    
    Traceback (most recent call last):
    
      File "<doctest:/home/joncrall/code/ubelt/ubelt/util_futures.py::__doc__:0>", line rel: 27, abs: 37, in <module>
        >>>             run_process(inputs, mode=mode, max_workers=max_workers)
    
      File "<doctest:/home/joncrall/code/ubelt/ubelt/util_futures.py::__doc__:0>", line rel: 14, abs: 24, in run_process
        ...     results = [job.result() for job in as_completed(jobs)]
    
      File "<doctest:/home/joncrall/code/ubelt/ubelt/util_futures.py::__doc__:0>", line rel: 14, abs: 24, in <listcomp>
        ...     results = [job.result() for job in as_completed(jobs)]
    
      File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/concurrent/futures/_base.py", line 432, in result
        return self.__get_result()
    
      File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
        raise self._exception
    
      File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/queues.py", line 239, in _feed
        obj = _ForkingPickler.dumps(obj)
    
      File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
        cls(buf, protocol).dump(obj)
    
    _pickle.PicklingError: Can't pickle <function my_function at 0x7f7f27f5e430>: attribute lookup my_function on ubelt.util_futures failed
    DOCTEST REPRODUCTION
    CommandLine:
        python -m xdoctest /home/joncrall/code/ubelt/ubelt/util_futures.py __doc__:0

=== Failed tests ===
python -m xdoctest /home/joncrall/code/ubelt/ubelt/util_futures.py __doc__:0
=== 1 failed, 6 passed in 0.14 seconds ===
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/joncrall/.pyenv/versions/3.8.6/lib/python3.8/concurrent/futures/process.py", line 104, in _python_exit
Process ForkProcess-4:
Process ForkProcess-2:
Process ForkProcess-1:
Process ForkProcess-3:

[Jon Crall]

This is related to the issue I just saw in scriptconfig.

The fix is that we need to allow xdoctest to create a dummy module scope for it to execute its code in. See: https://stackoverflow.com/questions/5122465/can-i-fake-a-package-or-at-least-a-module-in-python-for-testing-purposes

Not exactly sure how to do this yet.