cockroachdb / pebble

RocksDB/LevelDB inspired key-value database in Go

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

db: Metrics.DiskSpaceUsage incorrect with virtual, shared or external files

jbowens opened this issue · comments

CockroachDB uses (*pebble.Metrics).DiskSpaceUsage to calculate the disk space used by Cockroach within the data directory. Today, the sstable portion of the calculation assumes that the Size of all the fileMetadatas in the current version accurately reflect the local physical file size. With the introduction of virtual SSTables (eg, created from ingest-time splitting or excises), the underlying physical size of the file may be larger than the logical size of the file metadata. With shared and external files, it may be larger (if the entire backing file is copied locally) or smaller (if the file only exists remotely).

We should adapt Pebble to keep an accurate, incrementally-updated record of the physical size of local sstables and use it in DiskSpaceUsage. We must take care to avoid any O(n) work during GetMetrics(), instead incrementally updating statistics as version edits are applied.