can't pickle C data structures

[Shao Ming]

Hi, I use pathos.pools instead of concurrent.futures.ProcessPoolExecutor. And my codes are as follows, python_od is the Python interface of some C++ class obtained by Pybind11.

import pathos.pools as pp
from libs.python_od import OdImlp

batch_size=16
od = OdImlp(batch_size)
od.Load("models", 1, batch_size)

# here are some codes to make `batch_process_input_mp`
batch_process_input_mp=[]
for item in input_list:
    input1 = func1(item)
    input2 = func2(item)
    batch_process_input_mp.append((inut1, input2))

# I use 8 processes
with pp.ProcessPool(8) as pool:
    pool.map(od.process, batch_process_input_mp)

Then, I got a error message: /opt/conda/lib/python3.8/site-packages/dill/_dill.py:1890: PicklingWarning: Pickling a PyCapsule (None) does not pickle any C data strutures and could cause segmentation faults or other memory errors when unpickling.
I use pathos version: 0.3.0, dill version:0.3.6

[Mike McKerns]

This is just a warning. Are you receiving an error, or is there a question about the warning?

[Shao Ming]

This is just a warning. Are you receiving an error, or is there a question about the warning?

Yes, I get a error. Here is the error message:

TypeError: cannot pickle 'libs.python_od.OdImlp' object

[Mike McKerns]

Ah, ok. This is because python cannot serialize a C data structure unless the C object has a method that explicitly states how the state is to be serialized. So, unfortunately, this is not a problem that dill can solve -- the developer of OdImlp could add one of the state-saving methods (like reduce).

You could try one of the other pools that don't need to serialize the object (ParallelPool or ThreadPool).

[Shao Ming]

Ah, ok. This is because python cannot serialize a C data structure unless the C object has a method that explicitly states how the state is to be serialized. So, unfortunately, this is not a problem that dill can solve -- the developer of OdImlp could add one of the state-saving methods (like reduce).

You could try one of the other pools that don't need to serialize the object (ParallelPool or ThreadPool).

Ok, I will try the methods you mentioned above. Thanks a lot!

[Mike McKerns]

I'll close this, but feel free to return to comment or reopen as needed.