hpguppi_proc timeout leaves us in "processing" state
lacker opened this issue · comments
I think this will leave the automator wedged so that it can't do anything any more?
[2022-10-18 07:23:37,478 - INFO - proc_hpguppi.py:69] Processing file: guppi_59867_42817_000358_J1830-3602_0001.0000.raw
[2022-10-18 07:27:52,555 - INFO - automator.py:198] New state: not-tracking
[2022-10-18 07:29:38,426 - INFO - automator.py:198] New state: tracking
[2022-10-18 07:29:38,426 - INFO - automator.py:326] Processing still in progress, therefore not recording current track.
[2022-10-18 07:31:42,562 - INFO - automator.py:198] New state: not-tracking
[2022-10-18 07:31:42,562 - INFO - automator.py:380] No active recording for endof track for array_1
[2022-10-18 07:32:02,162 - INFO - automator.py:198] New state: tracking
[2022-10-18 07:32:02,162 - INFO - automator.py:326] Processing still in progress, therefore not recording current track.
[2022-10-18 07:33:37,548 - ERROR - proc_hpguppi.py:124] Timeout of 600 seconds exceeded
[2022-10-18 07:33:37,548 - ERROR - proc_hpguppi.py:74] Timed out, processing has not started.
[2022-10-18 07:33:37,548 - ERROR - automator.py:432] hpguppi_proc timed out. Stopping automator for debugging.
[2022-10-18 08:02:06,161 - INFO - automator.py:198] New state: not-tracking
[2022-10-18 08:02:06,161 - INFO - automator.py:380] No active recording for endof track for array_1
[2022-10-18 08:02:24,961 - INFO - automator.py:198] New state: tracking
[2022-10-18 08:02:24,961 - INFO - automator.py:326] Processing still in progress, therefore not recording current track.
I changed it so that it goes into "paused" mode, where it keeps running but does nothing on new messages. this still wedges the automator, but it's intentionally wedging the automator, so that we can debug if need be. to get out of pause, you just have to restart. 524a9cd