Fail on record creation ?
njoyard opened this issue · comments
When using perf-rec, committing new tests without committing performance records will result in passing tests without having actually tested performance.
Shouldn't perf-rec fail (or throw) on record creation?
I thought of this. [[ https://vcrpy.readthedocs.io/en/latest/usage.html#record-modes | vcrpy has 4 record modes ]] and perf-rec currently does the same as 'once' mode, which is its default. The mode you suggest is 'none' mode. We could support it with a Django setting which users activate during their CI process.
N.B. at YPlan we've never had this problem because everything goes through code review on Phabricator, which also doesn't allow code to be submitted with uncommitted files hanging around.
I like the setting idea. I'm also working on displaying a diff between records on failure (as TestCase.assertEqual does), maybe this can be a django setting, too.
That would be cool. I don't think we need a setting for that, just do it automatically for all the poor non-pytest users 😈
So I'm wondering what's the best option here:
- just add a boolean setting that makes tests fail when record does not exist (and create it)
- add 3 modes:
- 'once' : only fail when record exists and is different ; create record quietly
- 'none' : fail when record does not exist, but do not create it
- 'all' : fail when record does not exist, and create it
Thoughts ?
I'd probably go with a single setting dict PERF_REC, as that's the trend these days to keep settings files tidy. Then we can have a mode key within that dict that takes the 3 modes you posted there. The new_episodes mode that VCR does seem to be less useful and was clearly problematic as a default.
Fixed in cde3be1