`shared-log`: Efficient sync
marcus-pousette opened this issue · comments
Problem
Currently the syncing mechanisms is not very efficient.
- Memory consumption: It requires participants to keep all heads hashes and other metadata in RAM
- Syncing performance: When syncing entries with an remote without any prior information, it needs to check all relevant hashes with the remote. Once the first sync has been performed, the cache will make subsequent syncs faster at it will only look at changes from the last sync. The issue is still that many syncs need to be efficient also when syncing without any previous information about the peer
Solution
Implement a syncing mechanism that have following features:
- Uses disc storage
- Uses some kind of set reconciliation
bump
@eL1x00r Working on a PR here #278 but not done yet. Seems like a little bigger task then initially thought since I also am considering implementing live query features #200 at the same time, so I don't close any doors when doing the abstraction for index engines. Another estimation for completion is around 2-3 weeks
For this specific issue (efficient sync), it is more about improving the protocol for syncing activities which might become a bottleneck later, but not a one that is troubling you now. The work for a solution on this issue will commence once #274 and #200 are resolved