If parametric is in PR title, or of a file is changed in tests/parametric, add run-parametric-scenario-label
cbeauchesne opened this issue · comments
Can we just do it based on the files changed? Relying on a specific PR title seems error prone.
Unless you want to enforce that some "scope" strings are in the PR title with another check?
If you are not able to automate based on the files changes, you could add a check to ensure that specific labels exist on the PR, and block the PR if someone doesn't add parametric to the PR or something. Could be more user friendly approach.
Just same random musings/ideas.
It's not meant to be error prone, It's only an helper to the also-error-prone actual process (asking user to add the label).
The issue is that based on the files changed logic is also error prone, as it's impossible to guarantee that it'll handle all use-cases. We can at least include any change in tests/parametric, but it won't be enough.
To be more precise, the fact that it'll work most of the time without any conscious action from the user is the dangerous point : as it'll work quit often, it'll let people think it's error prone without doing nothing, and they'll rely too much on it, leading to one day or another to an issue.
I prefer a small helper, requiring an action that everybody knows and understand the limit, that something silent that let people think it's rock solid where it's not.
And the proposed got the huge advantage to be very very simple to implement 😄
For more details about why based on the files changed logic is hard, the question is : how can you ensure that a given file is not used by parametric scenario?
You can easily do the "opposite" : if this file is modified, then I need to run parametric scenario. But it's not the same.
Actually, it's impossible theoretically to do the first point. The better you'll have is a list of file that answer the second point
For more details about why
based on the files changedlogic is hard, the question is : how can you ensure that a given file is not used by parametric scenario?You can easily do the "opposite" : if this file is modified, then I need to run parametric scenario. But it's not the same.
Actually, it's impossible theoretically to do the first point. The better you'll have is a list of file that answer the second point
Makes sense, we have this problem too. This is what Intelligent Tester Runner is meant to solve... I wonder if it works here... probably not.
We also introduced our own mapping to try and cover this in dd-trace-py, definitely error prone even defining it statically in the repo.
I've change the issue the effect will be add the label (which will trigger the good scenario run), so we keep the label logic in front of user's eyes.
Done two weeks ago