Add configuration for janitor cleanup batch size
JaredNeil opened this issue · comments
The janitor cleans a max of 10 invocations per cycle, so to keep up, we had to set the cleanup_interval down to 10s. It seems a bit wasteful to query the database so frequently for such small batches.
Increasing the number of cleanup_workers doesn't make it go any faster because the Ticker still only sends one signal per interval. More workers just means they take turns. If cleaning one batch takes longer than the cleanup_interval, then more workers would allow them to run in parallel, but I think they'll just try to delete the same invocations because the database query is going to return the same set of invocations to both workers.
Should I create a PR to add cleanup_batch_size configuration value instead of the hard-coded 10?
Hey @JaredNeil, yeah if you want to send a CL to add a flag to configure the batch size, that'd be great!