Problem when set worker state <Response [404]>
fightpf opened this issue · comments
Dear Developer,
I am using HTTPS Docker and have enabled Authentication user management. I often encounter a 404 error at nbworker/rest.py line 179. I printed out the contents of the header after response.post, and here's what I found:
Error response:
File "/app/mercury/apps/../apps/nbworker/rest.py", line 182, in set_worker_state
raise Exception(f"Problem when set worker state {response}")
Exception: Problem when set worker state <Response [404]>
NB 2024-03-18 02:44:47,706 1 - goodbye
Error response header:
{'Server': 'nginx', 'Date': 'Mon, 18 Mar 2024 02:44:47 GMT', 'Content-Length': '0', 'Connection': 'keep-alive', 'Vary': 'Accept, Origin', 'Allow': 'POST, OPTIONS', 'X-Content-Type-Options': 'nosniff', 'Referrer-Policy': 'same-origin', 'Cross-Origin-Opener-Policy': 'same-origin'}
Normal response header:
{'Server': 'nginx', 'Date': 'Mon, 18 Mar 2024 03:03:28 GMT', 'Content-Type': 'application/json', 'Content-Length': '112', 'Connection': 'keep-alive', 'Vary': 'Accept, Origin', 'Allow': 'POST, OPTIONS', 'X-Content-Type-Options': 'nosniff', 'Referrer-Policy': 'same-origin', 'Cross-Origin-Opener-Policy': 'same-origin', 'X-Frame-Options': 'SAMEORIGIN'}
How can I resolve this issue? THX
Hi @fightpf,
Mercury is working but sometimes you get above error? or it is not working at all?
yes, mercury is working but sometimes i get errors and currently don't know how to trigger this error again.
When error occurs, Nginx logs was not showing any error or warning messages
There might be a situation when worker is closed in client but it is not yet closed in the backend, in this situation server will return 404 error, but it is not a bug. It is expected. I will close the issue for now. Thank you for reporting, if you will experience any problems about this issue please reopen. Thanks!
[2024-03-18 07:04:09,124: INFO/MainProcess] Task apps.ws.tasks.task_start_websocket_worker[1a549221-cfde-4fad-b624-a3205d7cb599] received
[2024-03-18 07:04:09,165: INFO/MainProcess] Task apps.ws.tasks.task_start_websocket_worker[1a549221-cfde-4fad-b624-a3205d7cb599] succeeded in 0.03348259627819061s: None
NB 2024-03-18 07:05:26,242 1 - goodbye
namespace(username='testuser', email='testuser@test.com')
NB 2024-03-18 07:06:47,569 Exception when set worker state
Traceback (most recent call last):
File "/app/mercury/apps/../apps/nbworker/rest.py", line 182, in set_worker_state
raise Exception(f"Problem when set worker state {response}")
Exception: Problem when set worker state 404
NB 2024-03-18 07:06:47,604 1 - goodbye
namespace(username='testuser', email='testuser@test.com')
[2024-03-18 07:07:47,876: INFO/MainProcess] Task apps.ws.tasks.task_start_websocket_worker[0aa2a851-124f-4d91-ac4f-78a37de55cbe] received
[2024-03-18 07:07:47,917: INFO/MainProcess] Task apps.ws.tasks.task_start_websocket_worker[0aa2a851-124f-4d91-ac4f-78a37de55cbe] succeeded in 0.03227986395359039s: None
NB 2024-03-18 07:13:09,118 1 - goodbye
namespace(username='testuser', email='testuser@test.com')
[2024-03-18 08:01:46,951: INFO/MainProcess] Task apps.ws.tasks.task_start_websocket_worker[f15486e5-344e-46c1-b2ec-1ea7f725afb6] received
[2024-03-18 08:01:47,008: INFO/MainProcess] Task apps.ws.tasks.task_start_websocket_worker[f15486e5-344e-46c1-b2ec-1ea7f725afb6] succeeded in 0.04924239683896303s: None
NB 2024-03-18 08:03:47,575 Exception when set worker state 404
Traceback (most recent call last):
File "/app/mercury/apps/../apps/nbworker/rest.py", line 182, in set_worker_state
raise Exception(f"Problem when set worker state {response}")
Exception: Problem when set worker state 404
NB 2024-03-18 08:03:47,606 1 - goodbye
namespace(username='testuser', email='testuser@test.com')
[2024-03-18 08:04:48,241: INFO/MainProcess] Task apps.ws.tasks.task_start_websocket_worker[9c765109-d34f-4081-917e-95574cc0b33a] received
[2024-03-18 08:04:48,282: INFO/MainProcess] Task apps.ws.tasks.task_start_websocket_worker[9c765109-d34f-4081-917e-95574cc0b33a] succeeded in 0.03667271416634321s: None
NB 2024-03-18 08:10:09,432 1 - goodbye
I figure out time interval always 2 minutes, it's possible relative about the mercury.py or ws setting?
beat_command = [
py_executable,
"-m",
"celery",
"-A",
"mercury.server" if sys.argv[0].endswith("mercury") else "server",
"beat",
"--loglevel=error",
"--max-interval",
"60", # sleep 60 seconds
]