Coordinate (and merge?) these efforts with Jepsen
dumblob opened this issue · comments
Jepsen does very good job in finding safety issues in databases (most well known databases failed their safety tests pretty hard).
It might make sense to somehow cooperate with Jepsen or even merge your efforts?
Thanks for your suggestion. I'm a big fan of Jepsen, and I gather Kyle sometimes refers to Hermitage as well. However, while the two projects support each other, I think they are quite different in many ways, and thus I think it makes sense to keep them separate.
@ept as someone who is unfamiliar with Jepsen 😅 do you mind documenting what is the difference with it? Thanks!
@miparnisari Sure. Jepsen is a test framework that automatically generates lots of inputs to a system, and concurrently controls the network and the clocks on the nodes to try to cause trouble. It then checks whether the results are correct. Hermitage is much simpler – it doesn't run tests automatically, it's just some queries that you can manually copy and paste into a terminal window. As such, Hermitage isn't really software, it's more like documentation. In principle, one could write an automated test runner based on the examples in this repo, but that would be a separate project. I like the fact that Hermitage tests are run manually, because the process of running them manually helps us build an intuition for how concurrency control mechanisms work.
(The FoundationDB tests are an exception – they are run by a shell script because the transaction would time out if the queries were copied and pasted by hand.)