eventually-sled: new backend based on sled db

[Paul Woolcock]

I brought up a possible backend implementation using the sled database, and danilo suggested I open a tracking issue to discuss it. (I'll post more specific thoughts later, just wanted to get this open for now so I didn't forget)

[Danilo Cianfrone]

Hey @pwoolcoc,
thanks for opening the issue!

For a new backend implementation, you need to implement the following traits:

1. EventStore, for appending and streaming events on request
2. EventSubscriber, for opening event streams receiving all new events committed to the store from when the stream has been opened
3. (Optional) A persistent Subscription implementation (e.g. https://ar3s3ru.github.io/eventually-rs/eventually/postgres/struct.Persistent.html)

For now, let's only consider the first two traits.

For point 2. it seems pretty easy to implement using the reactive semantics offered by sled.

For point 1. we need to figure out how to store the events and how to stream them back. When storing and streaming, we need double indexing: one on a single aggregate level (aggregate id), and one on aggregate type level (aggregate type id).

Also for point 1., we need to figure out a way to use optimistic concurrency in the write operations, in order to prevent concurrent writes scenarios.

What do you think? @pwoolcoc

[Danilo Cianfrone]

Hey @pwoolcoc, if you're still interested in adding this feature, you can check out #128 for reference: it adds support for Redis backend.

[Paul Woolcock]

@ar3s3ru thanks! sorry I didn't respond to this thread earlier, but I'm definitely still interested in doing this and I appreciate the pointers and the reference!

[Paul Woolcock]

Ok, initial thoughts, in no particular order:

• Sled keys/values are all just bytes, so I'm planning on using serde + bincode instead of serde_json, in order to save some db space since the bincode format is a little more efficient than taking the bytes of a string of JSON
• The EventStore implementation will take a sled::Db. From there, it will open sled::Trees as needed for each index.
• sled has transactions, and can do transactions across trees, so we can be assured that we can write to both an aggregate tree and an aggregate type tree and stay ACID compliant
• sled::Tree has a method to pull out keys that are greater than a specific value, so if we incorporate the sequence value into the key correctly when writing, we can use that mechanism to stream from a specific point.

[Paul Woolcock]

@ar3s3ru quick question: I'm hoping to have a POC of this soon, should I open a PR and try to get it in-tree or should I just keep in my own repo?

[Danilo Cianfrone]

Hey @pwoolcoc 👋

Feel free to open a PR to have the CI tests run 😄

[Danilo Cianfrone]

Also, now that I've created an org for eventually, I was considering moving eventually-postgres and eventually-rust out in their own respective repositories in the org.

Maybe we can do the same with eventually-sled.

Issues #131 #132