Large SST files (ignoring VersionSet.TARGET_FILE_SIZE?)
lwhite1 opened this issue · comments
In my databases, SSTable file sizes are more or less the same within an individual db. Varying either the content or db options, can cause sst files to be ~2MB in one database, ~23MB in another, ~39MB in a third, etc. That's the approximate size of every sst file in the system, in databases with dozens to hundreds of SST files.
Is there something I can do to control this? The large db files perform poorly on read-heavy loads.
thanks,
tables in levels 1+ conform to the TARGET_FILE_SIZE; tables written to level 0 will be around the writeBufferSize, which is configurable in Options
All of the files are approximately the same size. They can't all be level
0, could they?
On Monday, December 14, 2015, rnarubin notifications@github.com wrote:
tables in levels 1+ conform to the TARGET_FILE_SIZE; tables written to
level 0 will be around the writeBufferSize, which is configurable in Options—
Reply to this email directly or view it on GitHub
#65 (comment).
how large are your keys/values? they don't get split up across tables; for example, if a table being built is 1MB and it's given a 4MB key-value pair, it'll grow to 5MB before checking the size breakpoint
Values are around 1k
On Monday, December 14, 2015, rnarubin notifications@github.com wrote:
how large are your keys/values? they don't get split up across tables; for
example, if a table being built is 1MB and it's given a 4MB key-value pair,
it'll grow to 5MB before checking the size breakpoint—
Reply to this email directly or view it on GitHub
#65 (comment).
incidentally, you can inspect the MANIFEST file using google's leveldbutil
from their package. if you clone their repo, make leveldbutil creates a
tool with which you can dump the contents of certain leveldb files. In this
case, leveldbutil dump /path/to/MANIFEST-xxxxxx will list all live files,
which for table files includes their level number. The output is a bit
cryptic, but you'll have something like
"AddFile: 2 1510 1923066 '\xfc\xdf\xfb;Qs+\xbeK6\xb5\x1e\xf0..."
the 2 here is the level, the 1510 is the file number, and the 1923066 is
the file size in bytes. the hex string that i've cut off here is the
beginning of the smallest key in the table, it'll also have the largest key
On Mon, Dec 14, 2015 at 1:21 PM, Larry White notifications@github.com
wrote:
Values are around 1k
On Monday, December 14, 2015, rnarubin notifications@github.com wrote:
how large are your keys/values? they don't get split up across tables;
for
example, if a table being built is 1MB and it's given a 4MB key-value
pair,
it'll grow to 5MB before checking the size breakpoint—
Reply to this email directly or view it on GitHub
#65 (comment).—
Reply to this email directly or view it on GitHub
#65 (comment).