CRDTExtraMeta refactoring

[mweidner037]

Current approach

Currently, CRDTExtraMeta incorporates:

• causal-order broadcast
• tagged causal-order broadcast, i.e., exposing a vector clock to receivers
• wall-clock time

This metadata must then be attached to every message, even if the recipients have no intention of using it. E.g., wall-clock time is ~8 bytes per message, which is only a little, but adds up to a lot of waste across 100,000 single-character typing messages. Tagged causal-order broadcast is tricky to do efficiently given that replicaIds change frequently. In the future, we might want to add other metadata like Lamport timestamps, and users might want to add their own.

Straw man alternative

At the other extreme, we could provide no metadata and let CPrimitive's attach it themselves if needed. However, this can lead to duplication in batched messages. E.g., if you update many LwwCRegister's in one batch, each of them would need to attach its own wall-clock time. It would be much more efficient to instead attach the wall-clock time just once as "global metadata", like in the current approach.

Proposed solution

I propose refactoring CRDTExtraMeta to include such metadata only an "as-needed" basis. By default, we would just provide (untagged) causal-order broadcast, which is cheap, especially in the common case of low concurrency. CPrimitives sending messages could then request extra global metadata that should be added to the message, either using our built-ins or a custom extension. E.g.:

• wall-clock time
• specific vector clock entries
• Lamport timestamps

Multiple requests for the same metadata would only result in its addition once, hopefully recovering the efficiency of our current approach. It also allows garbage collection of Crdts, like LwwCRegister, that depend on the "current" vector clock, since they don't have to track that themselves. Finally, it voids the need for tagged causal broadcast. Hopefully attaching specific vector clock entries as needed won't be too expensive.

runLocally

One final issue is runLocally: if you runLocally an operation, it might request global metadata for the just-received message that you don't have. Possible solutions:

• On the original sender, observe what metadata runLocally requests and attach that. (Might not always work, in case recipients perform slightly different operations, but maybe we can rule that out?)
• Attach all metadata for any message whose recipients might runLocally. That is at most as bad as the current CRDTExtraMeta approach. It does mean that we would have to figure out how to do tagged causal broadcast somewhat efficiently, which I had been hoping to avoid (it's a hard problem).

[mweidner037]

Once this is implemented, we can optimize ContainerRuntimeSource: it can use a CausalBroadcastNetwork that does nothing, since this proposal only needs it to provide causal ordering, and that would already be guaranteed by the outer CausalBroadcastNetwork.

[mweidner037]

Closed by #213