Automatic cleanup / decrease of no longer used Pools / Test-DBs
majodev opened this issue · comments
Automatically cleanup x
previous template databases and it's test databases after a new template database was requested, without manually invoking the DELETE /templates
endpoint.
Enable this handling via the env var INTEGRESQL_KEEP_PREVIOUS_POOLS_COUNT
(int), default 0
(unlimited).
(Initial proposal via @MorpheusXAUT , cc aaa_internal)
I consider this to be the most critical feature for the next release (v1.2.x). As mentioned in the CHANGELOG.md:
- We still have no mechanism to limit the global (cross-pool) number of test-databases.
- This is especially problematic if you have many pools running at the same time.
- This could lead to situations where the pool manager is unable to create a new test-databases because the limit (e.g. disk size) is reached even tough some pools/test-databases will probably never be used again.
- This is a known issue and will be addressed in a future release.
Specifically, here are some first thoughts about it:
- I want that downscaling to happen in a dynamic manner, especially for test databases. Meaning, after a certain time of no active (/ or even very limited) pool usage, a kind of descaler worker should start to do its work:
- Reduce the always ready test database count in the unused pool for the incoming next minutes
- Reduce the number of dirty databases in the unused pool for the incoming next minutes
- Both all the way down to 0
- Full pool removal / template database removal is more destructive, but after a certain time (e.g. full 24h) - why not, it's just testing anyways.
- It might be hard to come up with good defaults that work for small and big (template / test) databases. Therefore a quite important metric to include in the descaling behavior might be the target maximal size (MB vs. GB) to use for all databases.
- Anyways, all values of this de-scaler have to be configurable.