Run benchmarks in Github Actions on a standalone EC2 instance
alexklibisz opened this issue · comments
Background
I recently switched to an M1 laptop and running benchmarks on this is making me question everything I know about computers. I've heard similar from others, so I don't think I'm crazy. I'd like to setup the benchmarks to run in a Github Actions workflow, on a standalone EC2 instance.
It should use a standalone EC2 instance so that the results are consistent. In the past I've tried benchmarking on Github Actions but found the results extremely noisy, likely due to noisy neighbors or mixed underlying hardware on Github's runners. I think I should be able to use this action to spin up and tear down the runner. The workflow should have 3 steps: start the EC2 runner, run the benchmark tasks on the EC2 runner, terminate the runner.
Finally, the workflow should have an optional checkbox input. If checked, the workflow will make a draft PR with the results.
Deliverables
- Benchmark workflow running on a standalone EC2 runner
- Make a PR to main with new results if the checkbox input is checked
Related Issues
No response
Decided to just stand up my own personal EC2 instance and use that for benchmarks. Much simpler. Re-generated benchmark results in #587