celery / django-celery-results

I detect this issue occurring after i set up multiple database with no default DB in

ImproperlyConfigured 
celery.backend_cleanup
settings.DATABASES is improperly configured. Please supply the ENGINE value. Check..

My Django setting as below

CELERY_QUEUE = "queue_one"
CELERY_RESULT_BACKEND = "django-db"
DATABASES = {
    "default": {},
    "main": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": "..",
        "USER": "..",
        "PASSWORD": "..",
        "HOST": "..",
        "PORT": "..",  
    },
    "datalake": {
        "ENGINE": "timescale.db.backends.postgresql",
        "NAME": "..",
        "USER": "..",
        "PASSWORD": "..",
        "HOST": "..",
        "PORT": "..",  
    },
}
DATABASE_ROUTERS = [
    "app.db_routers.TimeScaleRouter",
    "app.db_routers.MainRouter",
]

The issue occurs because we are trying to perform an atomic transaction at

django-celery-results/django_celery_results/managers.py

Line 90 in 16a18d9

with transaction.atomic():

The line transaction.atomic() will always point to the default db if you look at the django source code at:
https://github.com/django/django/blob/2887b9f67cadc5295ef6a0574de2c2c8fdd66905/django/db/transaction.py#L315

I tried to read the documentation and see if we can specify which DATABASES we can manually specify which db but i failed to find it, in case of settings above main or datalake.

Im assuming the database is determined using the DATABASE_ROUTERS, in this case can we somehow inject the db name into the atomic transaction like this:

transaction.atomic(using=self.db)

I think https://github.com/celery/django-celery-beat/pull/451/files is somehow related to what you are thinking

released

Make Celery Result Backend work when there are multiple database but no default