Experiment on applying block that already in local disk

[zoro]

Level db

How to start

./bnbchaind start --home ./testnoded

The log

I[2020-03-03|07:49:38.128] start to collect publish information         module=pub height=1

I[2020-03-03|07:49:52.625] Executed block                               module=consensus height=10000 validTxs=0 invalidTxs=0

I[2020-03-03|07:53:20.049] Executed block                               module=consensus height=13364 validTxs=0 invalidTxs=0

Analysis

The first 10000 block is very quick, only use 14 second. But after that the speed slow down to 16 blocks/second quickly. This is possibly related to the prune strategy, we have discussed in the #551, the speed of no prune is 3 times faster than prune in the long run.

Mem db

How to start

./bnbchaind start --home ./testnoded --pruning everything &, Have to use prune everything strategy to avoid memory usage grow huge.

Also need change the code in main.go a bit:

func newApp(logger log.Logger, db dbm.DB, storeTracer io.Writer) abci.Application {
	mendb := dbm.NewMemDB()
	return app.NewBinanceChain(logger, mendb, storeTracer)
}

Proof that use memdb indeed:

$du -l --max-depth=1
20      ./evidence.db
16220   ./state.db
61544   ./blockstore.db
20      ./tx_index.db
16      ./application.db  #it is empty when use mem db, even start with no prune strategy
11660   ./application.db_back  #level db

The log

I[2020-03-03|08:02:50.461] Executed block                               module=consensus height=1 validTxs=0 invalidTxs=0
I[2020-03-03|08:02:55.230] Executed block                               module=consensus height=1000 validTxs=0 invalidTxs=0
I[2020-03-03|08:04:41.864] start to collect publish information         module=pub height=5000
I[2020-03-03|08:08:36.001] start to collect publish information         module=pub height=8725

Analysis

The first 1000 block, it is quick. But after 5000 block, only 15.8 blocks/second.

Reason:
We can see that the bnbchaind process use 100 CPU resouces (means one core), we doubt that
there is one routine that is serial and cpu intensive.
We profile the cpu usage for 120seconds, and get that:

Each time commit wil save SaveOrphans:

func (ndb *nodeDB) SaveOrphans(version int64, orphans map[string]int64) {
	....
	toVersion := ndb.getPreviousVersion(version)
        ....

func (ndb *nodeDB) getPreviousVersion(version int64) int64 {
	itr := ndb.db.ReverseIterator(
		rootKeyFormat.Key(1),
		rootKeyFormat.Key(version),
	)
	...

	return 0
}

func (db *MemDB) ReverseIterator(start, end []byte) Iterator {
         ....
	keys := db.getSortedKeys(start, end, true)
	return newMemDBIterator(db, keys, start, end)
}

func (db *MemDB) getSortedKeys(start, end []byte, reverse bool) []string {
	keys := []string{}
	for key := range db.db {
                 inDomain := IsKeyInDomain([]byte(key), start, end)
		....
	}
	sort.Strings(keys)
         ...
        reurn keys
}

And let's see what is memdb's structure:

type MemDB struct {
	mtx sync.Mutex
	db  map[string][]byte
}

It means will iterate all the keys in db when save SaveOrphans, obviously IsKeyInDomain and sort.Strings(keys) functions is cpu intensive