Could use updated benchmarks; identify OS
JoeUX opened this issue · comments
A few notes on the benchmarks:
They're out of date
-
The benchmarks here on the main GitHub page have been stuck on Zstd 0.8.2 and brotli 0.4 for a while.
-
The benchmarks on the separate Zstd website use Zstd 1.0.0, and still brotli 0.4. (The website I'm referring to is listed as zstd.net, but redirects to http://facebook.github.io/zstd/ – github.io is not the GitHub you are on right now, but is their custom web hosting.)
Given all the releases since those benchmarks, it would be helpful to have updated results for Zstd ≥ 1.1.4 and brotli ≥ 0.5.2.
Different compilers
Note also that the benchmarks here on GitHub use two different compilers: gcc 5.4 for the table, and gcc 5.2.1 for the graphs. Unless there's a compelling reason, there's no point in introducing a confound like two different compilers.
No operating system
There's no mention of an OS in any of the benchmarks, not even broad families like Linux vs. Windows.
Summary
Ideally, what we want is: updated builds, same and updated compiler, identify the OS, report the amount of memory (and type) the system has and whether it's SSD or spinner (unless the benchmarks are in-memory only).
I've got some useful machines for benchmarking, but lzbench only works with gcc (on Windows). I think @inikep and others would be better for gcc benchmarks. I could do Visual Studio 2017 or 2015 benchmarks.
If your comment apply to zstd project,
please note that this issue is filled on the fse issue board.
Oh damn, yeah that's a mistake. Sorry. I'll move it over.