Move benchmarks out of tests into their own binary target
stephentyrone opened this issue · comments
So that we do not have to run tests with -c release
, and to separate the two more cleanly.
This also lets us avoid the Xcode measure
infrastructure, which offers some nice conveniences, but also isn't totally ideal for our intended use.
In SwiftNIO we use a make-shift benchmarking "framework" in NIOPerformanceTester
.
If this project will create a SwiftPM compatible benchmarking framework, we would like to steal that too 😁.
NIOPerformanceTester is pretty trivial, it runs every test 11 times, throws away the first and prints the run times of the remaining 10 runs. We then use various scripts (such as compare_perf_of_swift_versions.sh
and analyze_performance_results.rb
) to give us performance reports.
The performance reports can looks like
- this in CI
- that as images:
- or like this as output from
compare_perf_of_swift_versions.sh
:
benchmark: write_http_headers
===========
- with /usr/local/swift/swift-5.0.3-RELEASE-ubuntu18.04/usr/bin/swift
0.004265284 0.004260747 0.004255175 0.004286377 0.004254612 0.004276966 0.004255307 0.004276315 0.00426916 0.004264282
Min. 1st Qu. Median Mean 3rd Qu. Max.
0.004255 0.004257 0.004265 0.004266 0.004275 0.004286
- with /usr/local/swift/swift-5.1.2-RELEASE-ubuntu18.04/usr/bin/swift
0.004201598 0.004138115 0.004132022 0.004122942 0.004106163 0.004108991 0.004109322 0.00409021 0.004111035 0.004089371
Min. 1st Qu. Median Mean 3rd Qu. Max.
0.004089 0.004107 0.004110 0.004121 0.004130 0.004202
winner: /usr/local/swift/swift-5.1.2-RELEASE-ubuntu18.04/usr/bin/swift
comparison: /usr/local/swift/swift-5.0.3-RELEASE-ubuntu18.04/usr/bin/swift to /usr/local/swift/swift-5.1.2-RELEASE-ubuntu18.04/usr/bin/swift : -3.88 %
benchmark: bytebuffer_write_12MB_short_string_literals
===========
- with /usr/local/swift/swift-5.0.3-RELEASE-ubuntu18.04/usr/bin/swift
0.6138896 0.6210803 0.6273048 0.6317553 0.6265068 0.622431 0.6114756 0.6239886 0.6304719 0.6265637
Min. 1st Qu. Median Mean 3rd Qu. Max.
0.6115 0.6214 0.6252 0.6235 0.6271 0.6318
- with /usr/local/swift/swift-5.1.2-RELEASE-ubuntu18.04/usr/bin/swift
0.6234693 0.6091164 0.6028338 0.6157596 0.623367 0.6083817 0.6123535 0.6225484 0.6033183 0.6210232
Min. 1st Qu. Median Mean 3rd Qu. Max.
0.6028 0.6086 0.6141 0.6142 0.6222 0.6235
winner: /usr/local/swift/swift-5.1.2-RELEASE-ubuntu18.04/usr/bin/swift
comparison: /usr/local/swift/swift-5.0.3-RELEASE-ubuntu18.04/usr/bin/swift to /usr/local/swift/swift-5.1.2-RELEASE-ubuntu18.04/usr/bin/swift : -1.41 %
[...]
This is "done" in that the time-consuming tests have been removed. I think the long-term solution I want here is to adopt Karoy's benchmarking setup from collections, but that will require a separate Numerics-Benchmark project; it can't live in Numerics because that will end up giving us layering violations (alternatively, SPM might change how dependency graphs are tracked, which would also provide a solution).