sharded-log
A batch-oriented multi-threaded sharded log for workloads that occasionally flush logs into some other system. All batches have a 32-bit CRC written with them. Even though writes may happen to different log shards concurrently, recovery is single-threaded and will recover batches in the order that a monotonic ID was written to them during the write process. See the concurrent tests for examples of how this may be reasoned about.
Built specifically to complement the marble
on-disk garbage-collecting object store by buffering updates before
flushing them to storage. Intended to be used in a "rolling" manner
where you create a new ShardedLog instance and swap it out
while serving concurrent write requests, so that flushing to
secondary storage can happen in the background, followed by a removal
of the sharded log directory after it's stable.
use sharded_log::Config;
let config = Config {
path: "path/to/logs".into(),
..Default::default()
};
// purge
config.purge().unwrap();
// logs must be created in fresh directories (after a purge)
let sharded_log = config.create().unwrap();
sharded_log.write_batch(&[b"a", b"b", b"c", b"d"]).unwrap();
for write_batch in config.recover().unwrap() {
// recover contents to secondary storage
assert_eq!(write_batch, vec![b"a", b"b", b"c", b"d"]);
}
// purge contents when important data is safe elsewhere
sharded_log.purge().unwrap()