M3DB nodes stuck in bootstraping and restarted with OOM signal (many large commitlogs)

[Ghasem Pahlavan]

Hi all,

General Issues

1. What service is experiencing the issue? M3DB
2. What is the configuration of the service? https://pastebin.pl/view/7e4b79e2
3. How are you using the service? via Prometheus
4. Is there a reliable way to reproduce the behavior? high cardinal data with large block size and small size RAM.

We have 3 nodes M3DB cluster, all 3 of which are restarting due to OOM. During this time a large number of commitlogs are being generated in large volumes which has made the bootstrapping operation very slow and looped.

Is there a way to bootstrap the cluster and then apply commitlogs manually?
Or move commitlogs to another folder and return it to the commitlogs folder after bootstrap to be applied by m3db?

Any help appreciated.

[yuan]

same question, Have you solved your problem yet？how？