cockroachdb / pebble

RocksDB/LevelDB inspired key-value database in Go

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Support values larger than 4 GB

mitar opened this issue · comments

#717 added clear limits on max value (and batch) size: 4 GB. I wonder if there could be a way to raise those limits so that Pebble could be used for larger values as well and be more useful as a general data store (e.g., for uploading data from users which might range in different sizes).

There are offsets which are currently 32bit which seems to currently prevent this, but could they be increased to 64bit on 64bit systems?

I have read somewhere in issues that it might be reasonable to implement handling of large values by skipping wal altogether for them and directly writing into sstable.

Current suggested workaround seems to be DB.Ingest, but the ergonomic of this seems to be tricky. If this is the way (and it seems it is similar to the idea of "directly writing into sstable", it might be useful to have some high-level function, something like DB.SetLarge which would take a Reader and copy it into a temporary sstable and then call DB.Ingest on that file? (At least to me it is not clear how to create a sstable to be used by DB.Ingest myself.)

I do not expect that we'll ever support values larger than 4 GB. It's ill-advised to store such large blobs within the LSM itself, because the write amplification of the LSM will be extremely costly. Clients that require large blobs should store them outside the LSM and commit a pointer to the blobs to the LSM.