justanotherdot / sharded-log

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool


A batch-oriented multi-threaded sharded log for workloads that occasionally flush logs into some other system. All batches have a 32-bit CRC written with them. Even though writes may happen to different log shards concurrently, recovery is single-threaded and will recover batches in the order that a monotonic ID was written to them during the write process. See the concurrent tests for examples of how this may be reasoned about.

Built specifically to complement the marble on-disk garbage-collecting object store by buffering updates before flushing them to storage. Intended to be used in a "rolling" manner where you create a new ShardedLog instance and swap it out while serving concurrent write requests, so that flushing to secondary storage can happen in the background, followed by a removal of the sharded log directory after it's stable.

use sharded_log::Config;

let config = Config {
    path: "path/to/logs".into(),

// purge

// logs must be created in fresh directories (after a purge)
let sharded_log = config.create().unwrap();

sharded_log.write_batch(&[b"a", b"b", b"c", b"d"]).unwrap();

for write_batch in config.recover().unwrap() {
  // recover contents to secondary storage
  assert_eq!(write_batch, vec![b"a", b"b", b"c", b"d"]);

// purge contents when important data is safe elsewhere
ezoic increase your site revenue


License:GNU General Public License v3.0


Language:Rust 100.0%