bduffany / ab

Randomized a/b testing tool written in Rust

ab

Simple tool written in Rust to perform A/B experiments on commands.

Similar to hyperfine, but with the following advantages:

• Randomly alternates between commands, to rule out execution ordering as a possible reason for performance differences.
• Shows live output of results, including a heatmap.

Usage

• The first argument must be a comma-separated list of parameter values that you would like to test.
• The remaining arguments specify the command to be run.
• One of the arguments should be {} - the parameter values will be substituted here.

ab --flag-to-test=value1,--flag-to-test=value2 my-command {} other-flags...

Demo

(Click to see animation)

Installation

(TODO: publish release)

If you have cargo installed, just copy and paste this into your shell:

bash -ec '
  cd $(mktemp -d)
  git clone https://github.com/bduffany/ab "$PWD"
  cargo build -r
  sudo cp target/release/ab /usr/local/bin
'