RAM tables

[Luke Bakken]

Currently any changes in mnevis will have to go through the ra log, which takes some time to persist (write to disk). This makes disc_copies tables in mnesia useless, because data is already stored on disc. But performance of raft log is not that good because of that.
If you want to move your ram_copies table from mnesia replication to mnevis - you will see significant slow down, because all data is stored on disc.

I have an idea on how it should be possible to support ram tables as well as disc tables.

Essentially we will require to manage two raft quorums of nodes (or one raft and one other consensus), while only one of them would be actually writing data to disc and another will use a ram-based log.

When a client commits a transaction, which only affects ram tables - it will be submitted to the ram quorum, otherwise - the disc quorum.

Both quorums should share the same locker process, which should only allow lock operations when registered in both of them. While it registers there will be no locks aquied, but there may be transactions refused from one or another quorums, which is fine.

The challenge here is to re-merge the ram data in case of failures. When there are no nodes left to recover from or majority partition does not have the most recent data.
Since ram tables should be used only for transient data - application should be able to handle this data disappearance in case of partitions.

Originally proposed by @hairyhum

@michaelklishin commented:

This could be useful for a small subset of features (e.g. connection or MQTT client ID tracking). I don't know if it would be worth the effort, though.

@hairyhum responded:

There was an idea to implement some sort of a cache, which would return a result before changes are committed in the raft cluster. The main challenge would be consistency, because mnevis does not guarantee that commit will be applied. In case of locker process issues the commit may be refused and transaction needs to re-run. This makes the cache incorrect. If the cache is used by other systems, this may cause data inconsistencies. The only reason why we need a cache is to be used by other systems.