Uncaught exception in write_error

[Charles]

Describe the bug
I've deployed flower in GCP Kubenetes and it periodically throws a Uncaught exception in write_error which seems to stem from tornado. The admin dashboard still loads fine.

System information

>>> print(bugreport())
flower   -> flower:1.2.0 tornado:6.2 humanize:4.6.0
software -> celery:5.2.7 (dawn-chorus) kombu:5.2.4 py:3.10.10
            billiard:3.6.4.0 py-amqp:5.1.1
platform -> system:Linux arch:64bit
            kernel version:5.10.161+ imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:disabled

Logs

[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] [E 230301 17:00:48 web:1221] Uncaught exception in write_error
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     Traceback (most recent call last):
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1711, in _execute
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         result = method(*self.path_args, **self.path_kwargs)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/flower/views/error.py", line 11, in post
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         raise tornado.web.HTTPError(404)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     tornado.web.HTTPError: HTTP 404: Not Found
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     During handling of the above exception, another exception occurred:
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     Traceback (most recent call last):
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/flower/views/__init__.py", line 73, in get_current_user
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         basic, credentials = auth_header.split()
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     ValueError: not enough values to unpack (expected 2, got 0)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     During handling of the above exception, another exception occurred:
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     Traceback (most recent call last):
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1219, in send_error
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         self.write_error(status_code, **kwargs)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/flower/views/__init__.py", line 43, in write_error
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         self.render('404.html', message=message)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/flower/views/__init__.py", line 36, in render
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         super(BaseHandler, self).render(*args, **kwargs)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 865, in render
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         html = self.render_string(template_name, **kwargs)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1012, in render_string
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         namespace = self.get_template_namespace()
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1028, in get_template_namespace
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         current_user=self.current_user,
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1345, in current_user
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         self._current_user = self.get_current_user()
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/flower/views/__init__.py", line 78, in get_current_user
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         raise tornado.web.HTTPError(401)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     tornado.web.HTTPError: HTTP 401: Unauthorized
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] 01-03-23 17:00:48.784 - tornado.application - ERROR: Uncaught exception in write_error (web.py:1221)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] Traceback (most recent call last):
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1711, in _execute
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     result = method(*self.path_args, **self.path_kwargs)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/flower/views/error.py", line 11, in post
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     raise tornado.web.HTTPError(404)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] tornado.web.HTTPError: HTTP 404: Not Found
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] During handling of the above exception, another exception occurred:
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] Traceback (most recent call last):
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/flower/views/__init__.py", line 73, in get_current_user
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     basic, credentials = auth_header.split()
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] ValueError: not enough values to unpack (expected 2, got 0)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] During handling of the above exception, another exception occurred:
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] Traceback (most recent call last):
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1219, in send_error
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     self.write_error(status_code, **kwargs)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/flower/views/__init__.py", line 43, in write_error
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     self.render('404.html', message=message)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/flower/views/__init__.py", line 36, in render
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     super(BaseHandler, self).render(*args, **kwargs)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 865, in render
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     html = self.render_string(template_name, **kwargs)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1012, in render_string
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     namespace = self.get_template_namespace()
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1028, in get_template_namespace
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     current_user=self.current_user,
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1345, in current_user
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     self._current_user = self.get_current_user()
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/flower/views/__init__.py", line 78, in get_current_user
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     raise tornado.web.HTTPError(401)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] tornado.web.HTTPError: HTTP 401: Unauthorized
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] [E 230301 17:06:05 web:1221] Uncaught exception in write_error
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     Traceback (most recent call last):
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1711, in _execute
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         result = method(*self.path_args, **self.path_kwargs)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/flower/views/error.py", line 8, in get
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         raise tornado.web.HTTPError(404)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     tornado.web.HTTPError: HTTP 404: Not Found
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     During handling of the above exception, another exception occurred:
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     Traceback (most recent call last):
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/flower/views/__init__.py", line 73, in get_current_user
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         basic, credentials = auth_header.split()
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     ValueError: not enough values to unpack (expected 2, got 0)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     During handling of the above exception, another exception occurred:
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     Traceback (most recent call last):
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1219, in send_error
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         self.write_error(status_code, **kwargs)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/flower/views/__init__.py", line 43, in write_error
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         self.render('404.html', message=message)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/flower/views/__init__.py", line 36, in render
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         super(BaseHandler, self).render(*args, **kwargs)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 865, in render
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         html = self.render_string(template_name, **kwargs)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1012, in render_string
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         namespace = self.get_template_namespace()
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1028, in get_template_namespace
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         current_user=self.current_user,
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1345, in current_user
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         self._current_user = self.get_current_user()
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]       File "/usr/local/lib/python3.10/site-packages/flower/views/__init__.py", line 78, in get_current_user
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]         raise tornado.web.HTTPError(401)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     tornado.web.HTTPError: HTTP 401: Unauthorized
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] 01-03-23 17:06:05.418 - tornado.application - ERROR: Uncaught exception in write_error (web.py:1221)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] Traceback (most recent call last):
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1711, in _execute
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     result = method(*self.path_args, **self.path_kwargs)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/flower/views/error.py", line 8, in get
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     raise tornado.web.HTTPError(404)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] tornado.web.HTTPError: HTTP 404: Not Found
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] During handling of the above exception, another exception occurred:
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] Traceback (most recent call last):
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/flower/views/__init__.py", line 73, in get_current_user
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     basic, credentials = auth_header.split()
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] ValueError: not enough values to unpack (expected 2, got 0)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] During handling of the above exception, another exception occurred:
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] Traceback (most recent call last):
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1219, in send_error
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     self.write_error(status_code, **kwargs)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/flower/views/__init__.py", line 43, in write_error
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     self.render('404.html', message=message)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/flower/views/__init__.py", line 36, in render
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     super(BaseHandler, self).render(*args, **kwargs)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 865, in render
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     html = self.render_string(template_name, **kwargs)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1012, in render_string
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     namespace = self.get_template_namespace()
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1028, in get_template_namespace
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     current_user=self.current_user,
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1345, in current_user
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     self._current_user = self.get_current_user()
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/flower/views/__init__.py", line 78, in get_current_user
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     raise tornado.web.HTTPError(401)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] tornado.web.HTTPError: HTTP 401: Unauthorized
^C
charles@cloudshell:~$ kubectl logs --all-containers --ignore-errors --follow --prefix --tail=20 --selector "application in (web,flower-app,celery-beat,celery-worker,redis)"
[pod/celery-worker-6869bc5d78-stk25/celery-worker] [2023-03-01 17:00:02,973: WARNING/ForkPoolWorker-1]
[pod/celery-worker-6869bc5d78-stk25/celery-worker] [2023-03-01 17:00:02,973: WARNING/ForkPoolWorker-1] raise RuntimeError('This event loop is already running')
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] During handling of the above exception, another exception occurred:
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] Traceback (most recent call last):
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1219, in send_error
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     self.write_error(status_code, **kwargs)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/flower/views/__init__.py", line 43, in write_error
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     self.render('404.html', message=message)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/flower/views/__init__.py", line 36, in render
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     super(BaseHandler, self).render(*args, **kwargs)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 865, in render
[pod/celery-beat-75b8c768-d56gp/celery-beat] Waiting for redis...
[pod/celery-beat-75b8c768-d56gp/celery-beat] redis started
[pod/celery-beat-75b8c768-d56gp/celery-beat] reading envs from GOOGLE_CLOUD_PROJECT
[pod/celery-beat-75b8c768-d56gp/celery-beat] celery beat v5.2.7 (dawn-chorus) is starting.
[pod/celery-beat-75b8c768-d56gp/celery-beat] __    -    ... __   -        _
[pod/celery-beat-75b8c768-d56gp/celery-beat] LocalTime -> 2023-03-01 12:42:35
[pod/celery-beat-75b8c768-d56gp/celery-beat] Configuration ->
[pod/celery-beat-75b8c768-d56gp/celery-beat]     . broker -> redis://redis:6379//
[pod/celery-beat-75b8c768-d56gp/celery-beat]     . loader -> celery.loaders.app.AppLoader
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     html = self.render_string(template_name, **kwargs)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1012, in render_string
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     namespace = self.get_template_namespace()
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1028, in get_template_namespace
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     current_user=self.current_user,
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/tornado/web.py", line 1345, in current_user
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     self._current_user = self.get_current_user()
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]   File "/usr/local/lib/python3.10/site-packages/flower/views/__init__.py", line 78, in get_current_user
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app]     raise tornado.web.HTTPError(401)
[pod/flower-app-5c77c7b4bc-b5f8s/flower-app] tornado.web.HTTPError: HTTP 401: Unauthorized

[Andrii]

Have the same issue, tried yo update flower from 1.2.0 to 2.0.1 version but it did't help. Any solutions?