Background callbacks with celery + redis getting 204s

[Shiguang(Allen) Zhang]

My dash app uses background callbacks with Celery and Redis for long-running requests. It's being tested in Dev container.

Issue

I'm getting 204s intermittently when triggering the background callback. Sometimes the callback and caching work. Sometimes the dash/_dash-update-component endpoint just returns 204, and nothing on the frontend gets updated.
Below is the gunicorn log

Basic code structure

main.py

redis_conn_str = "..."
app_celery = Celery(__name__, broker=redis_conn_str, backend=redis_conn_str)
background_callback_manager = CeleryManager(app_celery, expire=1200)
app_server = Flask(__name__)
app = Dash(
    __name__,
    use_pages=True,
    suppress_callback_exceptions=True,
    server=app_server,
    background_callback_manager=background_callback_manager,
)

run_server.sh

nohup celery -A app.app_celery worker --loglevel=INFO -c 2 | tee /dev/null &

gunicorn --bind=0.0.0.0:8000 --workers=2 --worker-class=gthread --threads=20 app:app_server

callbacks.py

...
register_page(__name__, path="/dashboard")
layout = html.Div(...)

@callback(
    Output("output1", "data"),
    Output("output2", "children"),
    [
        Input("button", "n_clicks"),
        State("date-picker-range", "start_date"),
        State("date-picker-range", "end_date"),
        State("start1", "value"),
        State("start1", "value"),
        State("start1", "value"), 
        ... # 12 states in total
    ],
    prevent_initial_call=True,
    background=True,
    cache_args_to_ignore=[0],
    interval=9000,
)
def run_sql_query_callback(...):
    ...
    return dcc.send_data_frame(res.to_csv, "result.csv"), None

I've tried everything I can to debug, but it doesn't work. So, I need your help here. Much appreciate.

[Shiguang(Allen) Zhang]

Issue resolved.
The root cause is that the same Celery queue is used by multiple versions of running dash instances, which causes a task submitted by a producer to be consumed by a worker of a different version.
A better practice is to use different queues for different versions of workers.