cockroachdb / pebble

RocksDB/LevelDB inspired key-value database in Go

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

replay: add support for capturing and replaying read operations

itsbilal opened this issue · comments

Currently, the replay package in Pebble works to capture and replay write batches and ingestions - all write operations to pebble. The purpose of this is to measure compaction performance, specifically how effectively Pebble can keep up with paced/unpaced write traffic - traffic that is actually CockroachDB-representative. This addressed a common issue with Pebble-specific ycsb benchmarks that didn't stress the LSM in the same way CRDB does.

However, the replay package is mostly only good for measuring changes to WAL/compaction write throughput. The read side of this equation is currently missing; we don't capture or replay read operations (eg. iterator creation and iteration on top of the db, batches, or snapshots). Capturing these operation(s) at the cockroach/pebble interface level, and measuring their latencies and/or ops/sec in addition to everything else already in replay would basically get us a Pebble-level, CockroachDB-representative benchmarking suite.

This is an umbrella issue that consists of many smaller pieces of work; as design progresses on this project, we will turn this into a meta issue and link subtasks here.