Metrics sucking up disk space and no way to configure a retention policy or limit
duhmojo opened this issue · comments
I discovered a couple of VMs we run ongdb on using 15GB of disk space due to the size of the metrics csv files. I had to manually delete them to free up space. I couldn't find much information on how metrics are managed or if a retention policy could be set.
If deleting the files occasionally is what's required, I can deal with that but it would be nice if the system would limit usage like it does with log rotation. More over, is deleting the csv files save?
ONgDB Version: 1.0.4 (currently we intend to move to 1.0.5 soon)
Operating System: Alma Linux 8.something
Expected behavior
Disk usage is limited to a reasonable amount
Actual behavior
Disk usage is outsized for the size of the DB. e.g. 15GB
Have you tried the following config?
# The file size in bytes at which the csv files will auto-rotate. If set to zero then no
# rotation will occur. Accepts a binary suffix `k`, `m` or `g`.
metrics.csv.rotation.size=100m
# Maximum number of history files for the csv files.
metrics.csv.rotation.keep_number=7
No I haven't but I'll give it a shot right now. Where did you dig those settings up from? The confusion that comes with 1.0.4 at least, didn't include them.
After adding them to my ongdb.conf and restarting, it didn't execute any rotation. My 2nd instance has 7gb of metrics files. Should be rotation prune these old files outside the normal rotation window? I'll set it on my 1st instance where I deleted the old csv files and see if it stays under control over the week.