KeyError with peterbe.com easy example on WinPython 3.6.3.0-64

[graphedge]

I'm getting this in Jupyter notebok 5.7.8 with WinPython 3.6.8-64 on Windows 10. This is the first deco'd example from here

---

RemoteTraceback Traceback (most recent call last)
RemoteTraceback:
"""
Traceback (most recent call last):
File "c:\wpy64-368zro\python-3.6.8.amd64\lib\multiprocessing\pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "c:\wpy64-368zro\python-3.6.8.amd64\lib\site-packages\deco\conc.py", line 10, in concWrapper
result = concurrent.functions[f](*args, **kwargs)
KeyError: 'slow'
"""

The above exception was the direct cause of the following exception:

KeyError Traceback (most recent call last)
in
4 slow(index)
5
----> 6 run()

c:\wpy64-368zro\python-3.6.8.amd64\lib\site-packages\deco\conc.py in call(self, *args, **kwargs)
59 exec(out, scope)
60 self.f = scope[self.orig_f.name]
---> 61 return self.f(*args, **kwargs)
62
63

in run()

c:\wpy64-368zro\python-3.6.8.amd64\lib\site-packages\deco\conc.py in wait(self)
134 results = []
135 while self.results:
--> 136 result, operations = self.results.pop().get()
137 self.apply_operations(operations)
138 results.append(result)

c:\wpy64-368zro\python-3.6.8.amd64\lib\site-packages\deco\conc.py in get(self)
154
155 def get(self):
--> 156 return self.async_result.get(3e+6)
157
158 def result(self):

c:\wpy64-368zro\python-3.6.8.amd64\lib\multiprocessing\pool.py in get(self, timeout)
642 return self._value
643 else:
--> 644 raise self._value
645
646 def _set(self, i, obj):

KeyError: 'slow'

# after.py

from deco import concurrent, synchronized

@concurrent
def slow(index):
    time.sleep(5)

@synchronized
def run():
    for index in list('123'):
        slow(index)

run()

[Alex Sherman]

I believe this is an issue with the way multiprocessing creates new processes with this specific configuration. Historically I've avoided pickling function objects by assuming that the script where they're defined gets executed and I can simply look them up by name. This appears to not be happening in this configuration, so investigating pickling the concurrent methods might be the way to go.

[graphedge]

Thanks. I got Peter's example running by removing the synchronized function. Now I'm trying to get something useful back, but I'm only getting:

[<deco.conc.ConcurrentResult at 0x10d13310>,
<deco.conc.ConcurrentResult at 0x10d13430>,
<deco.conc.ConcurrentResult at 0x10d134d0>]

from deco import concurrent
resultlist=[]

@concurrent
def slow(index):
    result[index]= float(index**25 )
    return result[index]

for index in [1,2,3]:
    #print(index)
    resultlist.append(slow(index))

[Alex Sherman]

Please take a look at the wiki on what exactly @concurrent and @synchronized do, removing these decorators essentially means not using the library.

The root issue still remains that deco does not have good support for this runtime configuration.

[Alex Sherman]

I have verified that indeed jupyter will not execute the main module in the subprocesses that get spawned. This appears to be a "feature" of multiprocessing, and people suggest defining the target methods in separate files. See ipython/ipython#10894

An option for deco is to add a dependency on a module like dill which is capable of serializing functions. My preference will be to not add the dependency at this point, and live with the mitigation.