Limit RocksDB memory requirements

[Martin]

Especially during initial index import the RocksDB memory usage of Blockbook is large and unpredictable. Find a way (options) that will make the memory usage limited.

[Roman Zeyde]

I can suggest disabling auto-compactions and using smaller values for block_size and max_open_files (I had similar issue at romanz/electrs#30).

[Martin]

Thank you very much for your suggestions. I am going to try some of them.

I am afraid, with disabled auto-compactions the DB size will grow very much as we are overwriting data in the database.

As I understand the rockdb documentation, with larger values of block_size one should get better memory footprint. You are certainly using very large block size (512kB) in comparison to our 16kB or 32kB.

After I experiment with the options, I will post my findings here.

[Yura Pakhuchiy]

I was able to sync blockbook from scratch for GRS mainnet on 2GB VPS by restarting daemon every minute:

while true; do date; systemctl restart blockbook-groestlcoin; sleep 60; done

Without restarts it was killed due to OOM after several minutes. So my guess is that GC happens not often enough, thus resulting in much higher memory requirements during initial sync. I suggest to force GC after every 1000 or 10k blocks to reduce memory requirements.

[Yura Pakhuchiy]

Not that easy. I've tried doing runtime.GC() after every 1k blocks. Does not really helps. :-(

[Martin]

A good test of graceful shutdown procedure :)

Actually, there are probably better ways how to reduce memory footprint of the initial sync. Unfortunately, we did not have time to document them yet.

1. disable rocksdb cache by parameter -dbcache=0, the default size is 500MB
2. run blockbook with parameter -workers=1. This disables bulk import mode, which caches a lot of data in memory (not in rocksdb cache). It will run about twice as slowly but especially for smaller blockchains it is no problem at all.

[Yura Pakhuchiy]

Martin, thank you, these options helped to reduce memory usage. Still blockbook caches enough to force me restart it once in the middle of the sync to prevent OOM killer.

[Martin]

It is probably not Blockbook itself but Rocksdb, who is taking memory. This is exactly why this issue exists, we are not able to control memory usage of Rocksdb as much as we would like.

[WakiyamaP]

If it is only the initial synchronization, is it a good solution to make swapfile?
I solved my initial sync using 4GB swapfile.

[Pavol Travnik]

This issue is really fatal, I have 16 GB RAM, 4 cores and it always fails anyway.

Jan 08 01:48:45 instance-5 blockbook[22720]: E0108 01:48:29.950974   22720 sync.go:293] getBlockWorker 0 connect block error hash 4bf49e2661325894a8287a2dd2a49c22e17a401f5d77cc
Jan 08 01:48:45 instance-5 blockbook[22720]: E0108 01:48:36.717507   22720 sync.go:330] GetBlockHash error height 1521750: Post http://127.0.0.1:8034: net/http: request cancele
Jan 08 01:52:32 instance-5 blockbook[22720]: E0108 01:52:32.117822   22720 sync.go:330] GetBlockHash error height 1527204: Post http://127.0.0.1:8034: net/http: request cancele
Jan 08 02:04:34 instance-5 systemd[1]: blockbook-litecoin.service: Main process exited, code=killed, status=9/KILL

I

[Martin]

16 GB RAM should be more than enough for Litecoin. Is it really a memory problem? Have you tried to run the inital import with settings mentioned in this comment? Especially the flag -workers=1 will dramatically reduce the memory footprint of the initial import.

[Pavol Travnik]

I can confirm, that parameters -workers=1 -dbcache=0 in /lib/systemd/system/blockbook-litecoin.service file helped. Thank you a lot @martinboehm.

[kolya182]

I can confirm, that parameters -workers=1 -dbcache=0 in /lib/systemd/system/blockbook-litecoin.service file helped. Thank you a lot @martinboehm.

I did:
sudo systemctl stop blockbook-dogecoin
vim /lib/systemd/system/blockbook-dogecoin.service
changed -workers=1 => -workers=6
sudo systemctl start blockbook-dogecoin

but still showing 1 worker
ps aux | grep blockbook

/opt/coins/blockbook/dogecoin/bin/blockbook -blockchaincfg=/opt/coins/blockbook/dogecoin/config/blockchaincfg.json -datadir=/opt/coins/data/dogecoin/blockbook/db -sync -internal=:9038 -public=:9138 -certfile=/opt/coins/blockbook/dogecoin/cert/blockbook -explorer= -log_dir=/opt/coins/blockbook/dogecoin/logs -resyncindexperiod=30011 -resyncmempoolperiod=2011 -workers=1 -dbcache=0

[Martin]

Hi, the workers are in the same process as goroutines, you cannot see them using ps.

[Slush]

We should give a try to this:

https://blog.cloudflare.com/the-effect-of-switching-to-tcmalloc-on-rocksdb-memory-use/

It basically explain all our issues, including why having only one worker makes the memory footprint much better.

[Martin]

There is now an option to build Blockbook with TCMalloc, see 212b767 and in the documentation.
Unfortunately, the tests we have done do not show any advantage of using TCMalloc. Neither the initial sync nor long term usage of Blockbook showed any significant differences in the memory footprint.

[quietnan]

Does anybody have experience with hardware requirements during initial sync of bcashsv?

It's "supported", as in there is a config file, but since trezor does not seem to run an instance, I'm not sure if there is actually experience with stability and resource requirements in the presence of 1GB blocks.

[h4x3rotab]

How much memory is required to sync Bitcoin and how long does it take?

[Martin]

Hi, with the bulk import, the memory required is at least 32GB and will take about 24-36hours.
Without the bulk import (with the setting -workers=1), the required memory is about 16GB and will take about double time.

The advantage of running without bulk import mode is that even if you run out of memory and the process is killed, you can just restart it and it will continue (with bulk import you have to start from the beginning). If your computer is memory restricted, I would opt for the non bulk mode.