Giters

apple / swift-numerics

Advanced mathematical types and functions for Swift

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Move benchmarks out of tests into their own binary target

stephentyrone opened this issue · comments

commented

So that we do not have to run tests with -c release, and to separate the two more cleanly.

commented

This also lets us avoid the Xcode measure infrastructure, which offers some nice conveniences, but also isn't totally ideal for our intended use.

commented

In SwiftNIO we use a make-shift benchmarking "framework" in NIOPerformanceTester.

If this project will create a SwiftPM compatible benchmarking framework, we would like to steal that too 😁.

NIOPerformanceTester is pretty trivial, it runs every test 11 times, throws away the first and prints the run times of the remaining 10 runs. We then use various scripts (such as compare_perf_of_swift_versions.sh and analyze_performance_results.rb ) to give us performance reports.

The performance reports can looks like

  • this in CI
  • that as images: getPlot
  • or like this as output from compare_perf_of_swift_versions.sh:
benchmark:  write_http_headers 
===========
- with /usr/local/swift/swift-5.0.3-RELEASE-ubuntu18.04/usr/bin/swift 
0.004265284 0.004260747 0.004255175 0.004286377 0.004254612 0.004276966 0.004255307 0.004276315 0.00426916 0.004264282 
   Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
0.004255 0.004257 0.004265 0.004266 0.004275 0.004286 
- with /usr/local/swift/swift-5.1.2-RELEASE-ubuntu18.04/usr/bin/swift 
0.004201598 0.004138115 0.004132022 0.004122942 0.004106163 0.004108991 0.004109322 0.00409021 0.004111035 0.004089371 
   Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
0.004089 0.004107 0.004110 0.004121 0.004130 0.004202 
winner:  /usr/local/swift/swift-5.1.2-RELEASE-ubuntu18.04/usr/bin/swift 
comparison: /usr/local/swift/swift-5.0.3-RELEASE-ubuntu18.04/usr/bin/swift to /usr/local/swift/swift-5.1.2-RELEASE-ubuntu18.04/usr/bin/swift : -3.88 %

benchmark:  bytebuffer_write_12MB_short_string_literals 
===========
- with /usr/local/swift/swift-5.0.3-RELEASE-ubuntu18.04/usr/bin/swift 
0.6138896 0.6210803 0.6273048 0.6317553 0.6265068 0.622431 0.6114756 0.6239886 0.6304719 0.6265637 
  Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
0.6115  0.6214  0.6252  0.6235  0.6271  0.6318 
- with /usr/local/swift/swift-5.1.2-RELEASE-ubuntu18.04/usr/bin/swift 
0.6234693 0.6091164 0.6028338 0.6157596 0.623367 0.6083817 0.6123535 0.6225484 0.6033183 0.6210232 
  Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
0.6028  0.6086  0.6141  0.6142  0.6222  0.6235 
winner:  /usr/local/swift/swift-5.1.2-RELEASE-ubuntu18.04/usr/bin/swift 
comparison: /usr/local/swift/swift-5.0.3-RELEASE-ubuntu18.04/usr/bin/swift to /usr/local/swift/swift-5.1.2-RELEASE-ubuntu18.04/usr/bin/swift : -1.41 %
[...]
commented

This is "done" in that the time-consuming tests have been removed. I think the long-term solution I want here is to adopt Karoy's benchmarking setup from collections, but that will require a separate Numerics-Benchmark project; it can't live in Numerics because that will end up giving us layering violations (alternatively, SPM might change how dependency graphs are tracked, which would also provide a solution).