Decouple Dash & Celery
user1584 opened this issue · comments
I use Dash in combination with Celery to deal with long-running background callbacks. Within these callbacks, data is being retrieved using Spark. Since I have to run the frontend (Dash) and the backend (Spark) in separate containers anyway, I'd like to decouple the dependencies. The frontend does not need a full Spark installation (quite big!) and the backend does not need any web-related dependencies.
In other projects, I followed the concept outlined in decoupled-celery-example. However, due to the very integrated nature of Dash's callback-decorators, I struggle to decouple the workers.
Is this somehow possible?
@user1584 great question! I can see this being either a new class of background_callback_manager, or a new constructor arg to CeleryManager like named_tasks=True.
The main challenge will be defining an API for it because as you point out, the decorator pattern assumes the callback code is defined within the app and to make this work it needs to be defined in the worker and referenced in the app just by name. This means that unlike the existing background callback managers, you won't be able to just add background=True to move a callback from foreground to background, but that's unavoidable and I guess if you're at this point you wouldn't be writing those callbacks in the app code in the first place.
My gut reaction is with this callback manager we'd want to make callback function NOT as a decorator when used with background=True, but instead it gets an additional argument like task_name="my_worker_task" - making the definition look more like clientside_callback.
If you're interested in contributing we'd happily help getting a PR to completion 😎