Request for feature: Cross-Versions benchmarking
diesalbla opened this issue · comments
For most Scala projects, when I submit a Pull Request and I have to evaluate the performance impact of a Pull Request, I often have to manually do the following process:
- Checkout each distinct version, with and without the changes.
- Run the
sbt:jmhcommand, a same command, in each of them. - Retrieve and store the results from each run.
- Arrange the results of both runs onto a spreadsheet, which I then use to show the relative error of each measure, and the observed improvement.
It may be desirable if sbt:jmh plugin could offer a command to carry ouy these operations directly, and were able to generate a single summary table, which included the score and error for each commit, and the relative difference between them.
Benchmark Mode Cnt Score(master) Error Score(branch) Error Change Units
Fili thrpt 10 45000 42 42000 41 93.3 % B/s
Why closing @diesalbla? Is it supported now? Did you find a solution?
@guizmaii This is an issue I opened a few years ago during the contributions with fs2, due to the small inconvenience of switching branches and keeping results. It is not something I am aiming at doing soon, and it did not seem right to burden sbt maintainers with it. Maybe all that all is needed is a shell script, to do all the switch branch, setup, compilation, run benchmarks and save results.
I have not found any solution to do this locally. I understand, though, that there are some CI workflows to run benchmarks on each commit or PR. That may alleviate the need for this.
Leaving open for now.