atomix / copycat

A novel implementation of the Raft consensus algorithm

Home Page:http://atomix.io/copycat

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Remove indexing in logs

kuujo opened this issue · comments

The current implementation of Copycat's log uses in-memory indexes to support random access of entries in the log. However, this feature is a product of the evolution of the log that is no longer necessary and thus represents unnecessary overhead.

Because servers tend to read logs sequentially, Copycat's log should instead expose an iterator-like interface that allows for such sequential reading. When indexes are reset - e.g. when a follower returns a lower index to a leader - iterators should be reset by scanning segments. This is a rare occurrence that typically only takes place shortly after a leader election, so the overhead of scanning a segment is acceptable.

However, removing indexes will also require refactoring the Entry abstraction. The problem is, offsets change when segments are compacted, and the index currently allows indexes to be mapped to offsets when an entry is released for compaction. In order to ensure entries can be mapped back to offsets for compaction, the Entry should instead store a reference to a segment/offset that can be updated by compaction processes.