Windows Error: `cannot pickle '_thread.RLock' ` (concurrency)

[maria carrizo mascarell]

When using
simulation.compute(observed=True)

obtained following message

TypeError: cannot pickle '_thread.RLock' object

[Dieter Werthmüller]

Thanks @mariacarrizo

As you mention, it looks like it is related to Windows (and maybe Mac); possibly something along the lines of Koed00/django-q#389 (comment) ; https://docs.python.org/3/whatsnew/3.8.html#multiprocessing . Definitely a concurrent.futures thing, but not exactly sure what the problem is.

Help from anyone with a Windows machine would be greatly appreciated ;-)

[Dieter Werthmüller]

Here is a work-around, put this somewhere before the compute() command (probably best at the top of your file, just after the imports):

import emg3d  # You have that probably already, then there is no need to repeat it.

def process_map(fn, *iterables, max_workers, **kwargs):
    return list(map(fn, *iterables))

emg3d.simulations.process_map = process_map

It is not very satisfactory, as you basically loose all parallel functionality. But it should work. If you have tqdm installed and would like to have a progress bar, do the following instead of the above:

import emg3d
import tqdm

def process_map(fn, *iterables, max_workers, **kwargs):
    out = []
    for it in tqdm.tqdm(iterables[0]):
        out.append(fn(it))
    return out

emg3d.simulations.process_map = process_map

[maria carrizo mascarell]

Thank you @prisae this addition solved the issue!

[Dieter Werthmüller]

@mariacarrizo , could you test if this is the issue?: tqdm/tqdm#920

Hence:

• Does it work with Python < 3.7, but not with Python >= 3.7?
• Does it work if you de-install tqdm?

You actually don't have to de-install tqdm to test the latter point, just run

emg3d.simulations.process_map = emg3d.utils._process_map

That would be very useful to know.

[maria carrizo mascarell]

Hello @prisae

Using just

emg3d.simulations.process_map = emg3d.utils._process_map

the simulation works fine. I am using Jupyter notebook with Python version 3.8.3

[Dieter Werthmüller]

Interesting, thanks for reporting. So it looks like it is a problem with tqdm. So you have all parallel functionality, just not the nice progress bar. So for now you can use two different approaches then:

• emg3d.simulations.process_map = emg3d.utils._process_map
• uninstall tqdm (if tqdm is not installed the above line happens internally in emg3d)

[Dieter Werthmüller]

Another work-around is given in tqdm/tqdm#920 (comment) :

from multiprocessing import RLock
from tqdm import tqdm
tqdm.set_lock(RLock())

tqdm/tqdm#1046 will close this issue!

[Dieter Werthmüller]

tqdm v4.50.2 got released, so I am closing this.