Implement join decomposition
derekkraan opened this issue · comments
Join decomposition, as described in https://arxiv.org/pdf/1803.02750.pdf will reduce the problem of redundant deltas being sent around in the network.
I haven't quite grokked it 100% yet, so if anyone has a good understanding of it and wants to help, then discussion is VERY welcome (or a PR).
Hi. I can try to answer any question regarding that paper.
That would be great! I will dive into this again soon and try to figure it out again and let you know which questions I come up with. I think my main question was "how does one implement this?" But that's not a very specific question 😅
@vitorenesduarte I was wondering if I could abuse your offer of help to ask an unrelated question?
The computational complexity of join for a DotMap is really poor (O(n)), because you have to consider the union of keys in both deltas. It's impossible to tell from only {m, c}, {m', c'} which keys are commonly affected (because of possible deltas like {{}, {a1]}).
In delta_crdt, I've resorted to tracking the affected keys, so that we only have to consider the intersection of the affected keys of the deltas instead of the union of all keys of the two deltas. This reduces the computational complexity to O(1).
So my question: is there a better way I could be doing this?
Hi @vitorenesduarte, I have found another of your papers that goes into a bit more depth on how join decompositions work, and I think I've worked it out. At least, my property tests pass!
I have a related question, however. I see that checking for an inflation is very expensive for the case of a removal that has already been applied (we have to then check every single sub-CRDT of the ORMap to find out if the dot still appears anywhere). This is further complicated by the fact that I can't limit the check in a meaningful way, since I can't derive the key that the removal is removing in the join decomposition function.
If you had any thoughts on this, I'd be happy to hear them. I will let this problem simmer in the back of my mind, maybe I'll come up with something tomorrow.
I'm also still curious what your opinion is of my "hack" to include the affected keys in delta intervals.
It turns out that I can limit the check for removals in a meaningful way. In the decomposition, I consider that the removal can be for any of the keys present in the entire delta interval. If it is a small delta, this will still greatly limit the amount of checking I have to do.
I'm closing this issue, but I am still curious to hear your thoughts on the issue of tracking affected keys in the delta interval.