controller needs a heartbeat function
caseyjlaw opened this issue · comments
When submitting lots of data reading jobs, the controller can queue up lots of data to read. If it tries fails to clean out old results and release data in memory, then it can be overwhelmed and workers start to fail.
The realfast controller has a "cleanup" function that looks at futures status and runs final analysis, indexing, etc. on candidates. Currently, it runs only when new data arrives. If lots of data arrives and then no more, then data can queue up and overload memory.
A "heartbeat" feature could run this function at regular intervals, regardless of whether data is coming in.
Not clear if the current design can do this. May require going to Python 3 with asyncio or trio.
Can the evla_mcast library be made to trigger a heartbeat call to handle_scan? I actually want the controller to do something periodically, but maybe it has to be driven by evla_mcast?