Giters

revsys / django-health-check

a pluggable app that runs a full check on the deployment, using a number of plugins to check e.g. database, queue server, celery processes, etc.

Home Page:https://readthedocs.org/projects/django-health-check/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

When RabbitMQ active queues is none, it throws an exception instead of failing gracefully

lifenautjoe opened this issue · comments

commented

Hi Kristian,

First of all, thank you for the amazing package! If you have a place where we can invite you some coffee's, we're thrilled to do so.

We're seeing some exceptions and thought someone else might also benefit from a resolution.

Bug

We were experiencing issues with our RabbitMQ instance but instead of notifying us that there were issues with it, our API was throwing 500 errors.

Logs

[somus-api-production-sfo2] [2022-03-12 17:37:49] AttributeError: 'NoneType' object has no attribute 'values'
[somus-api-production-sfo2] [2022-03-12 17:37:49] 2022-03-12 18:37:49,145 django.request ERROR    Internal Server Error: /health/
[somus-api-production-sfo2] [2022-03-12 17:37:49] Traceback (most recent call last):
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/site-packages/asgiref/sync.py", line 451, in thread_handler
[somus-api-production-sfo2] [2022-03-12 17:37:49]     raise exc_info[1]
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 38, in inner
[somus-api-production-sfo2] [2022-03-12 17:37:49]     response = await get_response(request)
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 233, in _get_response_async
[somus-api-production-sfo2] [2022-03-12 17:37:49]     response = await wrapped_callback(request, *callback_args, **callback_kwargs)
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/site-packages/asgiref/sync.py", line 414, in __call__
[somus-api-production-sfo2] [2022-03-12 17:37:49]     ret = await asyncio.wait_for(future, timeout=None)
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/asyncio/tasks.py", line 455, in wait_for
[somus-api-production-sfo2] [2022-03-12 17:37:49]     return await fut
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
[somus-api-production-sfo2] [2022-03-12 17:37:49]     result = self.fn(*self.args, **self.kwargs)
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/site-packages/asgiref/sync.py", line 455, in thread_handler
[somus-api-production-sfo2] [2022-03-12 17:37:49]     return func(*args, **kwargs)
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/django/views.py", line 67, in sentry_wrapped_callback
[somus-api-production-sfo2] [2022-03-12 17:37:49]     return callback(request, *args, **kwargs)
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
[somus-api-production-sfo2] [2022-03-12 17:37:49]     return self.dispatch(request, *args, **kwargs)
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/site-packages/django/views/generic/base.py", line 98, in dispatch
[somus-api-production-sfo2] [2022-03-12 17:37:49]     return handler(request, *args, **kwargs)
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/site-packages/django/utils/decorators.py", line 43, in _wrapper
[somus-api-production-sfo2] [2022-03-12 17:37:49]     return bound_method(*args, **kwargs)
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
[somus-api-production-sfo2] [2022-03-12 17:37:49]     response = view_func(request, *args, **kwargs)
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/site-packages/health_check/views.py", line 86, in get
[somus-api-production-sfo2] [2022-03-12 17:37:49]     status_code = 500 if self.errors else 200
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/site-packages/health_check/mixins.py", line 16, in errors
[somus-api-production-sfo2] [2022-03-12 17:37:49]     self._errors = self.run_check()
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/site-packages/health_check/mixins.py", line 40, in run_check
[somus-api-production-sfo2] [2022-03-12 17:37:49]     for plugin in executor.map(_run, self.plugins):
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
[somus-api-production-sfo2] [2022-03-12 17:37:49]     yield fs.pop().result()
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 444, in result
[somus-api-production-sfo2] [2022-03-12 17:37:49]     return self.__get_result()
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
[somus-api-production-sfo2] [2022-03-12 17:37:49]     raise self._exception
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
[somus-api-production-sfo2] [2022-03-12 17:37:49]     result = self.fn(*self.args, **self.kwargs)
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/site-packages/health_check/mixins.py", line 32, in _run
[somus-api-production-sfo2] [2022-03-12 17:37:49]     plugin.run_check()
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/site-packages/health_check/backends.py", line 30, in run_check
[somus-api-production-sfo2] [2022-03-12 17:37:49]     self.check_status()
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/site-packages/health_check/contrib/celery_ping/backends.py", line 33, in check_status
[somus-api-production-sfo2] [2022-03-12 17:37:49]     self._check_ping_result(ping_result)
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/site-packages/health_check/contrib/celery_ping/backends.py", line 50, in _check_ping_result
[somus-api-production-sfo2] [2022-03-12 17:37:49]     self._check_active_queues(active_workers)
[somus-api-production-sfo2] [2022-03-12 17:37:49]   File "/usr/local/lib/python3.8/site-packages/health_check/contrib/celery_ping/backends.py", line 61, in _check_active_queues
[somus-api-production-sfo2] [2022-03-12 17:37:49]     for queues in app.control.inspect(active_workers).active_queues().values():
[somus-api-production-sfo2] [2022-03-12 17:37:49] AttributeError: 'NoneType' object has no attribute 'values'