PicklingError when input pyspider

[Matrix_Jia]

• pyspider version:
  pyspider-0.3.10
  Python 3.8.1

• Operating system:
  MAC Catalina V10.15.4

• Start up command:
  pyspider

Actual behavior

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/bin/pyspider", line 11, in
load_entry_point('pyspider==0.3.10', 'console_scripts', 'pyspider')()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pyspider/run.py", line 754, in main
cli()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/click/core.py", line 1236, in invoke
return Command.invoke(self, ctx)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pyspider/run.py", line 165, in cli
ctx.invoke(all)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pyspider/run.py", line 467, in all
threads.append(run_in(ctx.invoke, phantomjs, **phantomjs_config))
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pyspider/libs/utils.py", line 68, in run_in_subprocess
thread.start()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
return Popen(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function cli at 0x10820e0d0>: it's not the same object as pyspider.run.cli

Please help to check how to fix the issue?
Thank you.

[haima95]

I have the same problem

[siyhgame]

I have the same problem how fix it

[gorpher]

me too 😭😭😭😭

[LittlefingerJessic]

maybe the problem is the version of Python, you can try lower version

[LittlefingerJessic]

I am sure about the problem is the version of python.First uninstall your python, Then choose your python version,I have chosen the version of python3.7.5. hah , you will find things be easier

[hanswu]

me too , used python 3.9

[iphantom]

so sad ,used 3.9 have the same problem

[buaashn]

you can use "multiprocessing_on_dill" to replace "multiprocessing".
open file "/Library/Frameworks/Python.framework/Versions/3.x/lib/python3.x/site-packages/pyspider/libs/utils.py". function "run_in_subprocess" change to:

def run_in_subprocess(func, *args, **kwargs):
    """Run function in subprocess, return a Process object"""
    from multiprocessing_on_dill import Process
    thread = Process(target=func, args=args, kwargs=kwargs)
    thread.daemon = True
    thread.start()
    return thread

[Jintop1]

me too , used python 3.9