Setting up Auto-reload for celery work
BetweenTwoTests opened this issue · comments
I have replaced main.py with run_process and callback to run_work function as specified here: https://testdriven.io/courses/fastapi-celery/auto-reload/
I also updated the start script to use python main.py
instead of calling celery directly celery -A main.celery workier --loglevel=info
When I build and re-run, I get the following error message:
celery_worker_1 | Traceback (most recent call last):
celery_worker_1 | File "/app/main.py", line 19, in <module>
celery_worker_1 | celery_worker()
celery_worker_1 | File "/app/main.py", line 15, in celery_worker
celery_worker_1 | run_process("./project", run_worker)
celery_worker_1 | File "/usr/local/lib/python3.9/site-packages/watchgod/main.py", line 184, in run_process
celery_worker_1 | process = _start_process(target=target, args=args, kwargs=kwargs)
celery_worker_1 | File "/usr/local/lib/python3.9/site-packages/watchgod/main.py", line 148, in _start_process
celery_worker_1 | process.start()
celery_worker_1 | File "/usr/local/lib/python3.9/multiprocessing/process.py", line 121, in start
celery_worker_1 | self._popen = self._Popen(self)
celery_worker_1 | File "/usr/local/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
celery_worker_1 | return Popen(process_obj)
celery_worker_1 | File "/usr/local/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
celery_worker_1 | super().__init__(process_obj)
celery_worker_1 | File "/usr/local/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
celery_worker_1 | self._launch(process_obj)
celery_worker_1 | File "/usr/local/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
celery_worker_1 | reduction.dump(process_obj, fp)
celery_worker_1 | File "/usr/local/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
celery_worker_1 | ForkingPickler(file, protocol).dump(obj)
celery_worker_1 | AttributeError: Can't pickle local object 'celery_worker.<locals>.run_worker'
I am running Ubuntu on WSL (although the error message doesn't seem related to this issue at all)
From the log it seems you launch Celery worker using Docker compose.
Could you please test again with the package version in this file (celery
, watchgod
) https://github.com/testdrivenio/fastapi-celery-project/blob/master/requirements.txt
If the issue is resolved, then the error is related with some package number. If not, then the error might be related with your custom code.
Thx.