Giters

revsys / django-health-check

a pluggable app that runs a full check on the deployment, using a number of plugins to check e.g. database, queue server, celery processes, etc.

Home Page:https://readthedocs.org/projects/django-health-check/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Celery health check timeout exceptions and TaskRevokedError

miaoz2001 opened this issue · comments

commented

When I use the celery check, I will get some errors every 2-3 minutes. there are two kinds of errors like below. However my celery runs well without a problem.

2023-10-07T00:02:53.476107594Z unavailable: TaskRevokedError: The task was revoked, likely because it spent too long in the queue
2023-10-07T00:02:53.476143094Z Traceback (most recent call last):
2023-10-07T00:02:53.476149594Z   File "/usr/local/lib/python3.9/site-packages/health_check/contrib/celery/backends.py", line 20, in check_status
2023-10-07T00:02:53.476164894Z     result.get(timeout=result_timeout)
2023-10-07T00:02:53.476168894Z   File "/usr/local/lib/python3.9/site-packages/celery/result.py", line 250, in get
2023-10-07T00:02:53.476172794Z     return self.backend.wait_for_pending(
2023-10-07T00:02:53.476176494Z   File "/usr/local/lib/python3.9/site-packages/celery/backends/base.py", line 763, in wait_for_pending
2023-10-07T00:02:53.476180394Z     return result.maybe_throw(propagate=propagate, callback=callback)
2023-10-07T00:02:53.476184194Z   File "/usr/local/lib/python3.9/site-packages/celery/result.py", line 364, in maybe_throw
2023-10-07T00:02:53.476188094Z     self.throw(value, self._to_remote_traceback(tb))
2023-10-07T00:02:53.476191695Z   File "/usr/local/lib/python3.9/site-packages/celery/result.py", line 357, in throw
2023-10-07T00:02:53.476195595Z     self.on_ready.throw(*args, **kwargs)
2023-10-07T00:02:53.476199495Z   File "/usr/local/lib/python3.9/site-packages/vine/promises.py", line 234, in throw
2023-10-07T00:02:53.476203395Z     reraise(type(exc), exc, tb)
2023-10-07T00:02:53.476206995Z   File "/usr/local/lib/python3.9/site-packages/vine/utils.py", line 30, in reraise
2023-10-07T00:02:53.476210795Z     raise value
2023-10-07T00:02:53.476214395Z celery.exceptions.TaskRevokedError: expired

and

2023-10-07T00:43:51.514487089Z unavailable: TimeoutError: The task took too long to return a result
2023-10-07T00:43:51.514537089Z Traceback (most recent call last):
2023-10-07T00:43:51.514543189Z   File "/usr/local/lib/python3.9/site-packages/health_check/contrib/celery/backends.py", line 20, in check_status
2023-10-07T00:43:51.514547189Z     result.get(timeout=result_timeout)
2023-10-07T00:43:51.514558590Z   File "/usr/local/lib/python3.9/site-packages/celery/result.py", line 250, in get
2023-10-07T00:43:51.514563590Z     return self.backend.wait_for_pending(
2023-10-07T00:43:51.514567390Z   File "/usr/local/lib/python3.9/site-packages/celery/backends/base.py", line 755, in wait_for_pending
2023-10-07T00:43:51.514571290Z     meta = self.wait_for(
2023-10-07T00:43:51.514574890Z   File "/usr/local/lib/python3.9/site-packages/celery/backends/base.py", line 791, in wait_for
2023-10-07T00:43:51.514588490Z     raise TimeoutError('The operation timed out.')
2023-10-07T00:43:51.514592190Z celery.exceptions.TimeoutError: The operation timed out.

Is it expected or I am missing something?
Thanks!

commented

We have the same problem:

unavailable: TaskRevokedError: The task was revoked, likely because it spent too long in the queue

Although the issue is happening randomly, the service and the connection still work properly.

Do we know why that is happening suddenly? Is there any workaround we can use?