brainstorm / bencher.dev

🐰 Bencher - Continuous Benchmarking

Bencher

Bencher is a suite of continuous benchmarking tools designed to catch performance regressions in CI. That is, Bencher allows you to detect and prevent performance regressions before they make it to production.

For the same reasons that unit tests are run in CI to prevent feature regressions, benchmarks should also be run in CI to prevent performance regressions. Performance bugs are bugs!

Bencher consists of:

• bencher CLI
• Bencher API Server
• Bencher Web UI

The best place to start is the Bencher Quick Start tutorial.

Though Bencher is open source, there is also a hosted version available Bencher Cloud.

🐰 Use the GitHub Action with your project

Documentation

• Tutorial
  • Quick Start
• How To
  • Install CLI
  • Track Benchmarks
  • GitHub Actions
  • GitLab CI/CD
• Explanation
  • Benchmarking Overview
  • Benchmark Adapters
  • Branch Selection
  • Thresholds & Alerts
  • Continuous Benchmarking
  • Talks
• Reference
  • REST API
  • Server Config
  • Prior Art
  • Roadmap
  • Changelog

Supported Benchmark Harnesses

• {...} JSON
  • Custom benchmark harness support
• #️⃣ C#
  • BenchmarkDotNet
• ➕ C++
  • Catch2
  • Google Benchmark
• 🕳 Go
  • go test -bench
• ☕️ Java
  • Java Microbenchmark Harness (JMH)
• 🕸 JavaScript
  • Benchmark.js
  • console.time/console.timeEnd
• 🐍 Python
  • airspeed velocity
  • pytest-benchmark
• ♦️ Ruby
  • Benchmark
• 🦀 Rust
  • libtest bench
  • Criterion
  • Iai

For more details see the explanation of benchmark harness adapters.

Share Your Benchmarks

All public projects have their own perf page. These results can easily be shared with an auto-updating perf image. Perfect for your README!

GitHub Actions

Install the Bencher CLI using the GitHub Action. Use it to track your benchmarks. See how to use GitHub Actions for more details.

name: Track benchmarks with Bencher
on: [push]
jobs:
  benchmark_with_bencher:
    name: Benchmark with Bencher
    runs-on: ubuntu-latest
    env:
      - BENCHER_API_TOKEN: ${{ secrets.BENCHER_API_TOKEN }}
    steps:
      - uses: actions/checkout@v3
      - uses: bencherdev/bencher@v0.2.42
      - run: bencher run --project my-project-slug "bencher mock"

Repository Secrets

Add BENCHER_API_TOKEN to you Repository secrets (ex: https://github.com/my-user-slug/my-repo/settings/secrets/actions). You can find your API tokens by running bencher token ls --user my-user-slug or by going to the Bencher Console (ex: https://bencher.dev/console/users/my-user-slug/tokens).

Specify CLI Version

- uses: bencherdev/bencher@v0.2.42

There is also an optional version argument to specify an exact version of the Bencher CLI to use. Otherwise, it will default to using the latest CLI version.

- uses: bencherdev/bencher@main
  with:
    version: 0.2.42

These two commands are functionally equivalent. However, the first protects you from any breaking changes in the Bencher CLI GitHub action by using a tagged release. Therefore, the former is the suggested method (ex: - uses: bencherdev/bencher@v0.2.42)

Contributing

The easiest way to contribute is to open this repo as a Dev Container in VSCode by simply clicking one of the buttons below. Everything you need will already be there! Once set up, both the UI and API should be built, running, and seeded at localhost:3000 and localhost:61016 respectively. To make any changes to the UI or API though, you will have to exit the startup process and restart the UI and API yourself.

License

All content that resides under any directory or feature named "plus" is licensed under the Bencher Plus License.

All other content is license under either of Apache License, Version 2.0 or MIT license at your discretion.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Bencher by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.