m3db / m3

M3 monorepo - Distributed TSDB, Aggregator and Query Engine, Prometheus Sidecar, Graphite Compatible, Metrics Platform

Home Page:https://m3db.io/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

m3db node memory leak(v1.3)

wisre opened this issue · comments

commented

memory of cluster node increate slowly, like memo leak

General Issues

m3db node memory increase slowly, i want to know which config can slove this problem. this is the trend, the node version is v1.3

image

namespace config:

namespaces": {
"default": {
"bootstrapEnabled": true,
"flushEnabled": true,
"writesToCommitLog": true,
"cleanupEnabled": true,
"repairEnabled": false,
"retentionOptions": {
"retentionPeriodNanos": "8640000000000000",
"blockSizeNanos": "7200000000000",
"bufferFutureNanos": "600000000000",
"bufferPastNanos": "600000000000",
"blockDataExpiry": true,
"blockDataExpiryAfterNotAccessPeriodNanos": "300000000000",
"futureRetentionPeriodNanos": "0"
},
"snapshotEnabled": true,
"indexOptions": {
    "enabled": true,
"blockSizeNanos": "7200000000000"
},
"schemaOptions": null,
"coldWritesEnabled": false,
"runtimeOptions": null,
"cacheBlocksOnRetrieve": false,
"aggregationOptions": {
"aggregations": [
{
    "aggregated": false,
    "attributes": null
}
]
},
"stagingState": {
    "status": "READY"
},
"extendedOptions": null
}

cache config

image

if you need any other data, please tell me ,thanks

commented

memory of cluster node increate slowly, like memo leak

General Issues

m3db node memory increase slowly, i want to know which config can slove this problem. this is the trend, the node version is v1.3

image

namespace config:

namespaces": {
"default": {
"bootstrapEnabled": true,
"flushEnabled": true,
"writesToCommitLog": true,
"cleanupEnabled": true,
"repairEnabled": false,
"retentionOptions": {
"retentionPeriodNanos": "8640000000000000",
"blockSizeNanos": "7200000000000",
"bufferFutureNanos": "600000000000",
"bufferPastNanos": "600000000000",
"blockDataExpiry": true,
"blockDataExpiryAfterNotAccessPeriodNanos": "300000000000",
"futureRetentionPeriodNanos": "0"
},
"snapshotEnabled": true,
"indexOptions": {
    "enabled": true,
"blockSizeNanos": "7200000000000"
},
"schemaOptions": null,
"coldWritesEnabled": false,
"runtimeOptions": null,
"cacheBlocksOnRetrieve": false,
"aggregationOptions": {
"aggregations": [
{
    "aggregated": false,
    "attributes": null
}
]
},
"stagingState": {
    "status": "READY"
},
"extendedOptions": null
}

cache config

image

if you need any other data, please tell me ,thanks

heap pprof

image

Thanks for reporting! We'll be cutting a release with a fix included (from #3806).

commented

Thanks for reporting! We'll be cutting a release with a fix included (from #3806).

thank you @wesleyk , i will wait next release

@wisre v1.4.0 is cut now! Closing this out, please re-open if you still experience issues.

@wisre if you still observe the issue, could you ping me on the M3 slack? Would be good to get debug profiles from you from startup to when you observe memory growth.

commented

@wesleyk the issue is exists too, and i found that #3806 not merge in 1.4.0?
image