Setting up Auto-reload for celery work

[Leon Kim]

I have replaced main.py with run_process and callback to run_work function as specified here: https://testdriven.io/courses/fastapi-celery/auto-reload/

I also updated the start script to use python main.py instead of calling celery directly celery -A main.celery workier --loglevel=info

When I build and re-run, I get the following error message:

celery_worker_1  | Traceback (most recent call last):
celery_worker_1  |   File "/app/main.py", line 19, in <module>
celery_worker_1  |     celery_worker()
celery_worker_1  |   File "/app/main.py", line 15, in celery_worker
celery_worker_1  |     run_process("./project", run_worker)
celery_worker_1  |   File "/usr/local/lib/python3.9/site-packages/watchgod/main.py", line 184, in run_process
celery_worker_1  |     process = _start_process(target=target, args=args, kwargs=kwargs)
celery_worker_1  |   File "/usr/local/lib/python3.9/site-packages/watchgod/main.py", line 148, in _start_process
celery_worker_1  |     process.start()
celery_worker_1  |   File "/usr/local/lib/python3.9/multiprocessing/process.py", line 121, in start
celery_worker_1  |     self._popen = self._Popen(self)
celery_worker_1  |   File "/usr/local/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
celery_worker_1  |     return Popen(process_obj)
celery_worker_1  |   File "/usr/local/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
celery_worker_1  |     super().__init__(process_obj)
celery_worker_1  |   File "/usr/local/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
celery_worker_1  |     self._launch(process_obj)
celery_worker_1  |   File "/usr/local/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
celery_worker_1  |     reduction.dump(process_obj, fp)
celery_worker_1  |   File "/usr/local/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
celery_worker_1  |     ForkingPickler(file, protocol).dump(obj)
celery_worker_1  | AttributeError: Can't pickle local object 'celery_worker.<locals>.run_worker'

I am running Ubuntu on WSL (although the error message doesn't seem related to this issue at all)

[Michael Yin]

@BetweenTwoTests

From the log it seems you launch Celery worker using Docker compose.

Could you please test again with the package version in this file (celery, watchgod) https://github.com/testdrivenio/fastapi-celery-project/blob/master/requirements.txt

If the issue is resolved, then the error is related with some package number. If not, then the error might be related with your custom code.

Thx.