ENGINE A fatal exception happened
ZoltrixGFC opened this issue · comments
SABnzbd version
4.3.1
Operating system
Windows 11
Using Docker image
None
Description
Recently have been seeing repeated hanging of sabnzbd. Seems to happen only when downloading. Error message in logs just before the hang shows:
Please, report this on the Cheroot tracker at <https://github.com/cherrypy/cheroot/issues/new/choose>, providing a full reproducer with as much context and details as possible.
Traceback (most recent call last):
File "cheroot\workers\threadpool.py", line 119, in run
File "cheroot\workers\threadpool.py", line 216, in _process_connections_until_interrupted
File "cheroot\server.py", line 1364, in close
File "cheroot\server.py", line 1484, in _close_kernel_socket
File "ssl.py", line 1291, in shutdown
OSError: [WinError 10022] An invalid argument was supplied
2024-05-12 16:40:03,087::ERROR::[_cplogging:213] [12/May/2024:16:40:03] ENGINE Error in HTTP server: shutting down
Traceback (most recent call last):
File "cherrypy\process\servers.py", line 225, in _start_http_thread
File "cheroot\server.py", line 1845, in start
File "cheroot\server.py", line 1833, in serve
File "threading.py", line 1073, in _bootstrap_inner
File "cheroot\workers\threadpool.py", line 119, in run
File "cheroot\workers\threadpool.py", line 216, in _process_connections_until_interrupted
File "cheroot\server.py", line 1364, in close
File "cheroot\server.py", line 1484, in _close_kernel_socket
File "ssl.py", line 1291, in shutdown
OSError: [WinError 10022] An invalid argument was supplied
Only real change I've done recently is added a new "news server" to the servers section. I had my two top ones at priority 0 with 50 connections on one and 100 on the other. I thought maybe this was too much of a load, so have removed having two 0 priority servers, and have lowered the connections on each, but it keeps continuing to hang.
This is a bug in the webserver depency.
The first like describes what to do:
Please, report this on the Cheroot tracker at https://github.com/cherrypy/cheroot/issues/new/choose, providing a full reproducer with as much context and details as possible.
Thanks have posted cherrypy/cheroot#710
Though I was unsure what to list as the Cheroot and CherryPy versions?
cheroot==10.0.1
cherrypy==18.9.0
Python version is 3.12.3, not 3.8.
Thank you 👍
Just adding a more info (will add to thread on cherrypy as well).
It had been sitting idle for a while, then threw out the errors below and crashed.
2024-05-12 20:36:11,033::INFO::[postproc:138] Saving postproc queue
2024-05-12 20:36:11,033::INFO::[downloader:409] Post-processing finished, resuming download
2024-05-12 20:36:15,047::INFO::[nzbqueue:230] Saving queue
2024-05-12 20:36:15,049::INFO::[postproc:138] Saving postproc queue
2024-05-12 20:36:15,051::INFO::[notifier:157] Sending notification: SABnzbd - Queue finished (type=queue_done, job_cat=None)
2024-05-12 20:36:21,625::INFO::[_cplogging:213] [12/May/2024:20:36:21] ENGINE Client ('192.168.0.26', 58176) lost peer dropped the TLS connection suddenly, during handshake: (8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')
2024-05-12 20:36:21,628::INFO::[_cplogging:213] [12/May/2024:20:36:21] ENGINE Client ('192.168.0.26', 58164) lost peer dropped the TLS connection suddenly, during handshake: (8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')
2024-05-12 20:37:29,339::INFO::[database:208] [N/A] Moved job SABnzbd_nzo_2a1nfhb6 to archive
2024-05-12 21:10:33,102::INFO::[notifier:157] Sending notification: Error - [12/May/2024:21:10:33] ENGINE A fatal exception happened. Setting the server interrupt flag to OSError(22, 'An invalid argument was supplied', None, 10022, None) and giving up.
Please, report this on the Cheroot tracker at <https://github.com/cherrypy/cheroot/issues/new/choose>, providing a full reproducer with as much context and details as possible.
Traceback (most recent call last):
File "cheroot\workers\threadpool.py", line 119, in run
File "cheroot\workers\threadpool.py", line 216, in _process_connections_until_interrupted
File "cheroot\server.py", line 1364, in close
File "cheroot\server.py", line 1484, in _close_kernel_socket
File "ssl.py", line 1291, in shutdown
OSError: [WinError 10022] An invalid argument was supplied
(type=error, job_cat=None)
2024-05-12 21:10:33,102::CRITICAL::[_cplogging:213] [12/May/2024:21:10:33] ENGINE A fatal exception happened. Setting the server interrupt flag to OSError(22, 'An invalid argument was supplied', None, 10022, None) and giving up.
Please, report this on the Cheroot tracker at <https://github.com/cherrypy/cheroot/issues/new/choose>, providing a full reproducer with as much context and details as possible.
Traceback (most recent call last):
File "cheroot\workers\threadpool.py", line 119, in run
File "cheroot\workers\threadpool.py", line 216, in _process_connections_until_interrupted
File "cheroot\server.py", line 1364, in close
File "cheroot\server.py", line 1484, in _close_kernel_socket
File "ssl.py", line 1291, in shutdown
OSError: [WinError 10022] An invalid argument was supplied
2024-05-12 21:10:33,250::ERROR::[_cplogging:213] [12/May/2024:21:10:33] ENGINE Error in HTTP server: shutting down
Traceback (most recent call last):
File "cherrypy\process\servers.py", line 225, in _start_http_thread
File "cheroot\server.py", line 1845, in start
File "cheroot\server.py", line 1833, in serve
File "threading.py", line 1073, in _bootstrap_inner
File "cheroot\workers\threadpool.py", line 119, in run
File "cheroot\workers\threadpool.py", line 216, in _process_connections_until_interrupted
File "cheroot\server.py", line 1364, in close
File "cheroot\server.py", line 1484, in _close_kernel_socket
File "ssl.py", line 1291, in shutdown
OSError: [WinError 10022] An invalid argument was supplied
2024-05-12 21:10:33,250::INFO::[_cplogging:213] [12/May/2024:21:10:33] ENGINE Bus STOPPING
2024-05-12 21:10:33,354::INFO::[_cplogging:213] [12/May/2024:21:10:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('192.168.0.31', 10443)) shut down
2024-05-12 21:10:33,354::INFO::[_cplogging:213] [12/May/2024:21:10:33] ENGINE Bus STOPPED
2024-05-12 21:10:33,354::INFO::[_cplogging:213] [12/May/2024:21:10:33] ENGINE Bus EXITING
2024-05-12 21:10:33,354::INFO::[_cplogging:213] [12/May/2024:21:10:33] ENGINE Bus EXITED
basically fallout of ddos to cause it to crash.. make sure you dont have your sab exposed to the global internet.. and if you must.. do the needful to try and prevent people from being able to hammer it. so use non standard port / use firewall/reverse proxy... use something like fail2ban to block repeat attacks. and if its fallout from misconfiguration, fix that
@thezoggy thanks for the tips. I definitely don't have it exposed to the internet. Internal network access only. And it is really only internal services that interact with it. Radarr and Sonarr (via Overseerr). I do normally have approx 6 news servers configured with various priorities. Is that too much of a load for the CherryPy webservice?
@thezoggy thanks for the tips. I definitely don't have it exposed to the internet. Internal network access only. And it is really only internal services that interact with it. Radarr and Sonarr (via Overseerr). I do normally have approx 6 news servers configured with various priorities. Is that too much of a load for the CherryPy webservice?
no, cherrpy is used for sab gui/api. nothing to do with usenet downloading itself with your servers
Interesting point regarding only used for gui/api. In the errors I posted above, 192.168.0.26 is a docker host running nginx proxy. I had SSL enabled on SABnzbd running via the reverse proxy, and I noticed the SSL error listed above:
2024-05-12 20:36:21,625::INFO::[_cplogging:213] [12/May/2024:20:36:21] ENGINE Client ('192.168.0.26', 58176) lost peer dropped the TLS connection suddenly, during handshake: (8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')
2024-05-12 20:36:21,628::INFO::[_cplogging:213] [12/May/2024:20:36:21] ENGINE Client ('192.168.0.26', 58164) lost peer dropped the TLS connection suddenly, during handshake: (8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')
Last night I did a fresh install of SABnzbd, left SSL off, and set my other services to correct directly to SABnzbd (not via nginx), so far it has stayed up.
No idea if the reverse proxy could have been an issue at all?
It could be!
Interesting point regarding only used for gui/api. In the errors I posted above, 192.168.0.26 is a docker host running nginx proxy. I had SSL enabled on SABnzbd running via the reverse proxy, and I noticed the SSL error listed above:
2024-05-12 20:36:21,625::INFO::[_cplogging:213] [12/May/2024:20:36:21] ENGINE Client ('192.168.0.26', 58176) lost peer dropped the TLS connection suddenly, during handshake: (8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)') 2024-05-12 20:36:21,628::INFO::[_cplogging:213] [12/May/2024:20:36:21] ENGINE Client ('192.168.0.26', 58164) lost peer dropped the TLS connection suddenly, during handshake: (8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')Last night I did a fresh install of SABnzbd, left SSL off, and set my other services to correct directly to SABnzbd (not via nginx), so far it has stayed up.
No idea if the reverse proxy could have been an issue at all?
Click on the wrench and set the sab logging level to debug. That if the issue happens again you will have a bit more info.
If you need to triage further with cherrypy you can enable the webserver logging with -w on startup.
Closing this as an upstream issue :)