Trying to access a table file deleted by compaction process

[Alan Verbner]

Hi! First I would like to thank you for your lib. It's great and we are heavily using it in our Scala based ethereum client called Mantis.

Sometimes we are experiencing an error while querying for a key: RuntimeException: Could not open table. I'm attaching an issue description (including logs) and a proposed solution. We would love to push our fix and create a PR if you want to.

Description

When reading from DB, it seems that it's trying to open a file that was already deleted as it was marked as obsolete.

How to reproduce

Unfortunately we are not able to reproduce this error on a deterministic test as it seems to be due to a race condition error. That being said, we found that when stressing the database (reads and writes) it happens more often.

Sample Log

We have added some leveldb logs. Some examples:

• Full stacktrace without logs:

22:25:00 Could not open table 20954
java.lang.RuntimeException: Could not open table 20954
 at org.iq80.leveldb.impl.TableCache.getTable(TableCache.java:97)
 at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:78)
 at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:73)
 at org.iq80.leveldb.impl.Level0.get(Level0.java:100)
 at org.iq80.leveldb.impl.Version.get(Version.java:168)
 at org.iq80.leveldb.impl.VersionSet.get(VersionSet.java:222)
 at org.iq80.leveldb.impl.DbImpl.get(DbImpl.java:614)
 at org.iq80.leveldb.impl.DbImpl.get(DbImpl.java:575)

• Table 351 was deleted and tried to access it after that

18:10:06.196 [leveldb-compaction-0] INFO  i.i.e.d.d.LevelDBDataSource$  - [deleteObsoleteFiles]: /tmp/testdb8717364116035090466/000361.sst FileInfo{fileType=TABLE, fileNumber=361}
18:10:06.196 [leveldb-compaction-0] INFO  i.i.e.d.d.LevelDBDataSource$  - [deleteObsoleteFiles]: /tmp/testdb8717364116035090466/000351.sst FileInfo{fileType=TABLE, fileNumber=351}
18:10:06.196 [leveldb-compaction-0] INFO  i.i.e.d.d.LevelDBDataSource$  - [deleteObsoleteFiles]: /tmp/testdb8717364116035090466/000353.sst FileInfo{fileType=TABLE, fileNumber=353}
18:10:06.197 [leveldb-compaction-0] INFO  i.i.e.d.d.LevelDBDataSource$  - [deleteObsoleteFiles]: /tmp/testdb8717364116035090466/000354.sst FileInfo{fileType=TABLE, fileNumber=354}
18:10:06.197 [leveldb-compaction-0] INFO  i.i.e.d.d.LevelDBDataSource$  - [deleteObsoleteFiles]: /tmp/testdb8717364116035090466/000352.sst FileInfo{fileType=TABLE, fileNumber=352}
[info] MerklePatriciaTreeIntegrationSuite:
[info] - MPT benchmark *** FAILED *** (25 seconds, 813 milliseconds)
[info]   java.lang.RuntimeException: Could not open table 351
[info]   at org.iq80.leveldb.impl.TableCache.getTable(TableCache.java:97)
[info]   at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:78)
[info]   at org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:73)

Proposed fix

We think the issue is because in a concurrent modification, current version is being changed and previous files are not being retained https://github.com/dain/leveldb/blob/master/leveldb/src/main/java/org/iq80/leveldb/impl/DbImpl.java#L614 and when deleting obsolete files, a file being about to be read is deleted https://github.com/dain/leveldb/blob/master/leveldb/src/main/java/org/iq80/leveldb/impl/DbImpl.java#L285