Celery worker silently stops working in some conditions under high load
fruttasecca opened this issue · comments
Describe the bug
After a period of very high resource contention the celery worker stopped processing tasks. The probe and liveness checks did not lead to a restart of the pod, which is what fixed the issue. We definitively want the probes to notice the problem and restart the pod in case this happens, moreover, some tuning related to settings like the CELERY_BROKER_HEARTBEAT could be done to reduce load.
Environment
Happened in the cloud where our smaller instances are on 2 vcpus, but could happen anywhere of course.