MemTable is not get freed while creating index

[rockeet]

We run myrocks with sysbench:

sysbench --db-driver=mysql --threads=20 --mysql-host=192.168.100.10  \
         --mysql-port=3306 --mysql-user=test  --mysql-password=1234 \
         --mysql-db=test --tables=100 --table_size=100000000 oltp_write_only \
         --mysql_storage_engine='rocksdb default charset latin1' prepare

When creating index, data insertion on other tables is still running, we noticed that MemTables were not get freed during this period, the memory usage is very high, once index creation is done, the memory usage is decreased. On small memory machine, the mysqld process were kill for OOM.

[Yoshinori Matsunobu]

Yes, by default DDL runs as a single transaction, so it is constrained by memory.

We recommend using bulk loading --
https://github.com/facebook/mysql-5.6/wiki/Data-Loading -- while this is not transactional, it uses RocksDB's bulk loading feature that is not constrained by memory.
For online schema change, we recommend using our online schema change tool https://github.com/facebookincubator/OnlineSchemaChange -- which supports the bulk loading feature.

[rockeet]

During creating index, an RocksDB iterator is used for fetching table rows, and there is a Version object held by the Iterator, the Version object pinned a set of MemTable and SST.

So I have a solution: refresh(delete and recreate with same snapshot for simple) an RocksDB iterator periodically, thus the obsoleted Version object can be released and the MemTable and deleted SST pinned by the Version object can be freed.

[Mark Callaghan]

Yes, by default DDL runs as a single transaction, so it is constrained by memory.

We recommend using bulk loading -- https://github.com/facebook/mysql-5.6/wiki/Data-Loading -- while this is not transactional, it uses RocksDB's bulk loading feature that is not constrained by memory. For online schema change, we recommend using our online schema change tool https://github.com/facebookincubator/OnlineSchemaChange -- which supports the bulk loading feature.

From current sysbench I assume this happens during secondary index creation after the table has been loaded.

I appreciate your response but it implies a user can't create secondary indexes on existing large tables -- because you encounter this problem if you do that.

[rockeet]

Yes, by default DDL runs as a single transaction, so it is constrained by memory.
We recommend using bulk loading -- https://github.com/facebook/mysql-5.6/wiki/Data-Loading -- while this is not transactional, it uses RocksDB's bulk loading feature that is not constrained by memory. For online schema change, we recommend using our online schema change tool https://github.com/facebookincubator/OnlineSchemaChange -- which supports the bulk loading feature.

From current sysbench I assume this happens during secondary index creation after the table has been loaded.

I appreciate your response but it implies a user can't create secondary indexes on existing large tables -- because you encounter this problem if you do that.

Users can create index on large tables, but they can not writing too much (new) data during creating index, if there is no data written to MyRocks during creating index, this issue will not be triggered.