Margaret is go-ssb's append-only
log* provider, and greatly inspired by flumedb. Compatible with Go 1.13+.
the project name is inspired by Twin Peaks's character Margaret aka the log lady
Margaret has the following facilities:
- an append-only log interface
.Append(interface{}),.Get(int64) - queries
.Query(...QuerySpec)for retrieving ranges based on sequence numbers e.g..Gt(int64), or limiting the amount of data returned.Limit(int64) - a variety of index mechanisms, both for categorizing log entries into buckets and for creating virtual logs (aka sublogs)
Margaret is one of a few key components that make the go implementation of ssb tick, for example:
ssb/sbotuses margaret for storing each peer's data
Margaret outputs data according to the offset2 format, which is inspired by (but significantly differs from) flumelog-offset.
In brief: margaret stores the data of all logs in the three following files:
datastores the actual data (with a length-prefix before each entry)ofstindexes the starting locations for each data entry indatajrnlan integrity checking mechanism for all three files; a checksum of sorts, more details
There are a few concepts that might be tough to digest for newcomers on first approach:
- multilogs, a kind of tree-based index, where each leaf is a margaret.Log
- in other words: it creates virtual sublogs that map to entries in an offset log (see log storage above)
margaret/indexessimilar to leveldb indexes (arbitrary key-value stores)- sublogs (and rxLog/receiveLog/offsetLog and its equivalence to offset.log)
- queries
- zeroing out, or replacing, written data
For more on these concepts, visit the dev.scutttlebutt.nz portal for in-depth explanations.
* margaret is technically an append-based log, as there is support for both zeroing out and replacing items in the log after they have been written. Given the relative ubiquity of append-only logs & their uses, it's easier to just say append-only log.