Regression testing
dblalock opened this issue · comments
We should automatically do full training runs for all of the benchmarks sometimes to detect accuracy regressions. This will be more challenging than regular testing since we'll have to:
- Launch multi-GPU jobs
- Monitor the results
- Track previous results as a reference
This should actually be pretty good MCLI dogfooding.
Closing as we're not tracking this here.