Race condition in dashboard signal deque

Question

Race condition in dashboard signal deque

lampep opened this issue 3 years ago · comments

Mar 23 13:34:14 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:14] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:14 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:14] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:14 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:14] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:14 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:14] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:14 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:14] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:14 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:14] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:14 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:14] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:15 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:15] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:15 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:15] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:15 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:15] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:15 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:15] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:15 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:15] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:15 mof-rts-00011 bash[1708]: INFO:werkzeug:127.0.0.1 - - [23/Mar/2021 13:34:15] "POST /radiotracking/_dash-update-component HTTP/1.1" 200 -
Mar 23 13:34:16 mof-rts-00011 bash[1708]: Exception on /radiotracking/_dash-update-component [POST]
Mar 23 13:34:16 mof-rts-00011 bash[1708]: Traceback (most recent call last):
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 2447, in wsgi_app
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     response = self.full_dispatch_request()
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1952, in full_dispatch_request
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     rv = self.handle_user_exception(e)
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1821, in handle_user_exception
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     reraise(exc_type, exc_value, tb)
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/_compat.py", line 39, in reraise
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     raise value
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1950, in full_dispatch_request
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     rv = self.dispatch_request()
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1936, in dispatch_request
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     return self.view_functions[rule.endpoint](**req.view_args)
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/dash/dash.py", line 1078, in dispatch
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     response.set_data(func(*args, outputs_list=outputs_list))
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/dash/dash.py", line 1009, in add_context
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     output_value = func(*args, **kwargs)  # %% callback invoked %%
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/home/pi/pyradiotracking/radiotracking/present.py", line 448, in update_signal_variance
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     sigs = self.select_sigs(power, snr, freq, duration)
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/home/pi/pyradiotracking/radiotracking/present.py", line 382, in select_sigs
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     return [sig for sig in self.signal_queue
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/home/pi/pyradiotracking/radiotracking/present.py", line 382, in <listcomp>
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     return [sig for sig in self.signal_queue
Mar 23 13:34:16 mof-rts-00011 bash[1708]: RuntimeError: deque mutated during iteration
Mar 23 13:34:16 mof-rts-00011 bash[1708]: ERROR:radiotracking.present:Exception on /radiotracking/_dash-update-component [POST]
Mar 23 13:34:16 mof-rts-00011 bash[1708]: Traceback (most recent call last):
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 2447, in wsgi_app
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     response = self.full_dispatch_request()
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1952, in full_dispatch_request
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     rv = self.handle_user_exception(e)
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1821, in handle_user_exception
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     reraise(exc_type, exc_value, tb)
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/_compat.py", line 39, in reraise
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     raise value
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1950, in full_dispatch_request
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     rv = self.dispatch_request()
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1936, in dispatch_request
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     return self.view_functions[rule.endpoint](**req.view_args)
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/dash/dash.py", line 1078, in dispatch
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     response.set_data(func(*args, outputs_list=outputs_list))
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/usr/local/lib/python3.7/dist-packages/dash/dash.py", line 1009, in add_context
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     output_value = func(*args, **kwargs)  # %% callback invoked %%
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/home/pi/pyradiotracking/radiotracking/present.py", line 448, in update_signal_variance
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     sigs = self.select_sigs(power, snr, freq, duration)
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/home/pi/pyradiotracking/radiotracking/present.py", line 382, in select_sigs
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     return [sig for sig in self.signal_queue
Mar 23 13:34:16 mof-rts-00011 bash[1708]:   File "/home/pi/pyradiotracking/radiotracking/present.py", line 382, in <listcomp>
Mar 23 13:34:16 mof-rts-00011 bash[1708]:     return [sig for sig in self.signal_queue
Mar 23 13:34:16 mof-rts-00011 bash[1708]: RuntimeError: deque mutated during iteration

Jonas Höchst · Answer 1 · Tue Apr 13 2021 20:01:11 GMT+0800 (China Standard Time)

I share this observation, which is indeed a race condition inserting new signals while accessing the signal deque. However it is not interrupting operations, other than skipping a single dashboard refresh.