m3db / m3

M3 monorepo - Distributed TSDB, Aggregator and Query Engine, Prometheus Sidecar, Graphite Compatible, Metrics Platform

Home Page:https://m3db.io/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[BUG] disk space does not shrink after change aggregated namespace retention period to smaller

naughtyGitCat opened this issue · comments

commented

My Env:

binary deploy

m3dbnode version v1.4.2 = 3*(1+1) = 6
m3coordinator version v1.5.0
m3query version v1.5.0

My Case

the m3dbnode disk usage warnings, so I try to change the retention and aggregation options on May 10

after changing the aggregated namespace retention period from 26888h to 17520h and changing the aggregation downsample config from null => 1m => 90s (do restart the coordinator, and one of the m3dbnode) does not see disk usage shrinks

even the fileset-{timestamp}-0-data.db minion does not shrink any more

image

-rw-rw-r-- 1 dba dba 187M Mar  7 08:35 fileset-1678017600000000000-0-data.db
-rw-rw-r-- 1 dba dba   20 Mar  7 08:35 fileset-1678017600000000000-0-digest.db
-rw-rw-r-- 1 dba dba  49M Mar  7 08:35 fileset-1678017600000000000-0-index.db
-rw-rw-r-- 1 dba dba   44 Mar  7 08:35 fileset-1678017600000000000-0-info.db
-rw-rw-r-- 1 dba dba 748K Mar  7 08:35 fileset-1678017600000000000-0-summaries.db
-rw-rw-r-- 1 dba dba 120K Mar  8 20:23 fileset-1678147200000000000-0-bloomfilter.db
-rw-rw-r-- 1 dba dba    4 Mar  8 20:23 fileset-1678147200000000000-0-checkpoint.db
-rw-rw-r-- 1 dba dba 188M Mar  8 20:23 fileset-1678147200000000000-0-data.db
-rw-rw-r-- 1 dba dba   20 Mar  8 20:23 fileset-1678147200000000000-0-digest.db
-rw-rw-r-- 1 dba dba  49M Mar  8 20:23 fileset-1678147200000000000-0-index.db
-rw-rw-r-- 1 dba dba   44 Mar  8 20:23 fileset-1678147200000000000-0-info.db
-rw-rw-r-- 1 dba dba 755K Mar  8 20:23 fileset-1678147200000000000-0-summaries.db
-rw-rw-r-- 1 dba dba 120K Mar 10 08:35 fileset-1678276800000000000-0-bloomfilter.db
-rw-rw-r-- 1 dba dba    4 Mar 10 08:35 fileset-1678276800000000000-0-checkpoint.db
-rw-rw-r-- 1 dba dba 187M Mar 10 08:35 fileset-1678276800000000000-0-data.db
-rw-rw-r-- 1 dba dba   20 Mar 10 08:35 fileset-1678276800000000000-0-digest.db
-rw-rw-r-- 1 dba dba  49M Mar 10 08:35 fileset-1678276800000000000-0-index.db
-rw-rw-r-- 1 dba dba   44 Mar 10 08:35 fileset-1678276800000000000-0-info.db
-rw-rw-r-- 1 dba dba 753K Mar 10 08:35 fileset-1678276800000000000-0-summaries.db
-rw-rw-r-- 1 dba dba 120K Mar 11 20:14 fileset-1678406400000000000-0-bloomfilter.db
-rw-rw-r-- 1 dba dba    4 Mar 11 20:14 fileset-1678406400000000000-0-checkpoint.db
-rw-rw-r-- 1 dba dba 187M Mar 11 20:14 fileset-1678406400000000000-0-data.db
-rw-rw-r-- 1 dba dba   20 Mar 11 20:14 fileset-1678406400000000000-0-digest.db
-rw-rw-r-- 1 dba dba  49M Mar 11 20:14 fileset-1678406400000000000-0-index.db
-rw-rw-r-- 1 dba dba   44 Mar 11 20:14 fileset-1678406400000000000-0-info.db
-rw-rw-r-- 1 dba dba 751K Mar 11 20:14 fileset-1678406400000000000-0-summaries.db
-rw-rw-r-- 1 dba dba 121K Mar 13 08:37 fileset-1678536000000000000-0-bloomfilter.db
-rw-rw-r-- 1 dba dba    4 Mar 13 08:37 fileset-1678536000000000000-0-checkpoint.db
-rw-rw-r-- 1 dba dba 186M Mar 13 08:37 fileset-1678536000000000000-0-data.db
-rw-rw-r-- 1 dba dba   20 Mar 13 08:37 fileset-1678536000000000000-0-digest.db
-rw-rw-r-- 1 dba dba  50M Mar 13 08:37 fileset-1678536000000000000-0-index.db
-rw-rw-r-- 1 dba dba   44 Mar 13 08:37 fileset-1678536000000000000-0-info.db
-rw-rw-r-- 1 dba dba 756K Mar 13 08:37 fileset-1678536000000000000-0-summaries.db

this is my final namespace option get from the coordinator instance API
namespace.json.zip

commented

it shrinks on last day 20:00, may be it works after I reboot the coordinator(last day 18:00)

commented

image
Maybe we can add the tip to the document?

ps, the minion data file size still doesn't shrink anymore