Comparative Benchmarks Results
echarles opened this issue · comments
Hi, Have you run any kind of comparative benchmarks to assess the performance?
See e.g. https://github.com/dmonad/crdt-benchmarks that compares yjs vs automerge vs delta-crdts
There are benchmarks in the benchmarks folder but there are not comparative.
I see #188 was attempting to compare vs automerge and yjs but it has been closed and not merged. Any reason for that? Any result from that branch to give a taste of the collabs performance?
oops, there are scripts to run the benchmark for automerge and yjs. Let me try these.
I have rename this issue to "Comparative Benchmarks Results". It would be useful to have on the README the results so folks should not clone and run the benchmarks.
Yeah, there's comparative benchmarks as you found, we just need to make them more obvious and put the results somewhere visible. Sorry about that!
For now:
- You can view the benchmark descriptions in https://github.com/composablesys/collabs/blob/master/benchmarks/README.md
- Latest benchmark results (from our own server): benchmarks.zip.
My tasks
- Document benchmarks in main (?) readme
- Host benchmark results publicly
- Link to https://github.com/composablesys/compoventuals-chart (source of the attached charts) and add docs to it
- PR for https://github.com/dmonad/crdt-benchmarks
No issue, and thx for the zip. I find the results encouraging. Maybe worth mentioning in the legend which version is used (e.g. is it automerge 1+ which was the performance branch?)
It's Automerge v0. I have it on my todo-list to try v1 again. (When last I tried it a couple months ago, the text editing was actually slower, and the todo-list demo broke. But they may have fixed those issues by now.)
Subsumed by #164