Enable dot compression in state (memory leak)
derekkraan opened this issue · comments
The tricky thing is making this work with network partitions. If a neighbouring node goes away and later comes back, and there is a gap in the deltas that we have, then we need to be able to recover from this situation.
Under normal operation (static membership, no churn or network splits), each node has in the delta-buffer only the delta-groups that haven't been acknowledged by all peers.
But this means that the delta-buffer will never shrink if a single peer stops responding.
In this case, we can safely drop it from the map of acks (meaning that now we can shrink the buffer), as long as, if we reconnect to this neighbor, we can detect it. And when this happens, we should send the full CRDT state in order to maintain causal-consistency [1].
This situation can be generalized: if we're sending deltas to a peer that is not in the map of acks (maybe it was before or not, it doesn't matter), we send the full state.
The problem with this is that the peer receiving the full state (that can be big) should add it to the delta-buffer for further propagation. But with the join-decomposition trick [2], we will prune redundant state from the full state, and only add to the buffer what's strictly needed for convergence.
[1] https://arxiv.org/pdf/1603.01529.pdf
[2] https://arxiv.org/pdf/1803.02750.pdf
Current state of this is: we currently do remove deltas, but we do not optimize storage of dots. So dots will remain forever in the CRDT. Luckily, they are small, but it does represent a memory leak that will grow with mutations to the CRDT.