Mismatch between input and output directories
lacker opened this issue · comments
From automator logs:
[2022-10-18 06:22:55,830 - INFO - proc_hpguppi.py:54] Creating output directory...
[2022-10-18 06:23:14,437 - INFO - proc_hpguppi.py:65] Processing:
inputdir: /buf0ro/20221014/0014/Unknown/GUPPI
outputdir: /scratch/data/20221017/0022/hpguppi_beamformer
[2022-10-18 06:23:14,438 - INFO - proc_hpguppi.py:69] Processing file: guppi_59867_42817_000358_J1830-3602_0001.0000.raw
Seems like inputdir and outputdir are supposed to be the same, though.
buf0 is completely empty.
An odd series of events earlier:
[2022-10-18 05:20:57,528 - INFO - proc_hpguppi.py:153] Incomplete agreement of PROCSTAT across hosts. Retrying in 1s.
[2022-10-18 05:20:58,535 - WARNING - proc_hpguppi.py:160] 3 straggler(s).
[2022-10-18 05:20:58,535 - INFO - proc_hpguppi.py:161] Gathered proc status: END
[2022-10-18 05:20:58,535 - INFO - proc_hpguppi.py:120] Upchanneliser/beamformer at END
[2022-10-18 05:20:58,536 - INFO - automator.py:437] New recording processed by hpguppi_proc. Output data are available in /scratch/data/20221017/0022
[2022-10-18 05:20:58,536 - INFO - automator.py:439] Proceeding to processing-complete
[2022-10-18 05:20:58,536 - INFO - automator.py:198] New state: processing-complete
[2022-10-18 05:20:58,537 - INFO - automator.py:471] ['srun', '-w', 'blpn0 blpn1 blpn2 blpn3 blpn4 blpn5 blpn6 blpn7 blpn8 blpn9 blpn10 blpn11 blpn12 blpn13 blpn14 blpn15 blpn16 blpn17 blpn18 blpn19 blpn20 blpn21 blpn22 blpn23 blpn24 blpn25 blpn26 blpn27 blpn28 blpn29 blpn30 blpn31 blpn32 blpn33 blpn34 blpn35 blpn36 blpn37 blpn38 blpn39 blpn40 blpn41 blpn42 blpn43 blpn44 blpn45 blpn46 blpn47 blpn48 blpn49 blpn50 blpn51 blpn52 blpn53 blpn54 blpn55 blpn56 blpn57 blpn58 blpn59 blpn60 blpn61 blpn62 blpn63', 'bash', '-c', '/home/obs/bin/cleanmybuf0.sh --force']
[2022-10-18 05:21:44,657 - INFO - automator.py:503] Resetting nshot after processing: coordinator:trigger_mode coordinator:trigger_mode:array_1:nshot:1
[2022-10-18 06:20:04,255 - INFO - automator.py:198] New state: capture-stop
[2022-10-18 06:20:04,256 - INFO - automator.py:264] Ignoring irrelevant state for array_1
[2022-10-18 06:20:14,356 - INFO - automator.py:198] New state: not-tracking
[2022-10-18 06:20:20,320 - INFO - automator.py:198] New state: capture-done
[2022-10-18 06:20:20,320 - INFO - automator.py:264] Ignoring irrelevant state for array_1
[2022-10-18 06:21:43,671 - INFO - automator.py:198] New state: deconfigure
[2022-10-18 06:21:43,671 - INFO - automator.py:348] array_1 deconfigured. Proceeding to processing.
[2022-10-18 06:21:43,671 - INFO - automator.py:198] New state: processing
I'm not sure if it is supposed to be able to go from state processing-complete -> capture-stop -> not-tracking -> capture-done -> deconfigure -> processing - i would have expected it to gather data sometime between the two processing phases.
more similar trouble:
[2022-10-18 07:23:19,867 - INFO - proc_hpguppi.py:54] Creating output directory...
[2022-10-18 07:23:37,478 - INFO - proc_hpguppi.py:65] Processing:
inputdir: /buf0ro/20221014/0014/Unknown/GUPPI
outputdir: /scratch/data/20221017/0022/hpguppi_beamformer
We tracked this down to a case where keys were still in bluse_raw_watch:* although the raw files were already removed. @david-macmahon is going to fix up the raw file watcher to fix these keys in the situation where the files were removed while the watcher wasn't running, which is our best guess. For now these zombie keys are deleted so let's cross our fingers that it doesn't happen again.