Only trigger experiments when a comment starts with `/gcbrun`
DonggeLiu opened this issue · comments
Currently, a GCB
experiment will be triggered if our comment mentions /gcbrun
.
This is problematic because experiments will be launched unintendedly (e.g., when we quote previous /gcb
commands, or even when we mention /gcb
in any form).
Root cause
GCB
trigger simply checks for/gcb
without any constraints.- Our
gcbrun_experiment.py
iterates all previous comments to find the latest valid/gcb
command, which might have already been run.
Proofs
- This example quotes an old command and modifies its experiment name. However, it triggers an experiment with the old name, indicating we launch the latest valid experiment command.
- This example includes a random
/gcb
command in the middle of the sentence. It also triggers the same old experiment, confirming the same conclusion.
Propose fix
Change
fuzzbench/service/gcbrun_experiment.py
Lines 56 to 57 in 56caa83
to
if not body.startswith(RUN_EXPERIMENT_COMMAND_STR):
return None
Justification
- Based on our setting on Google Cloud, all
/gcb
commands will immediately trigger an experiment, there is no need to search back in history for an old command. - If we need to run an old command, we can always add a new comment for it.
@jonathanmetzman: Did I miss anything?
If it looks good to you, I will implement the fix proposed above.
you mean [/]gcbrun
run right?
I think this is a good idea.
you mean
[/]gcbrun
run right?
Yep, fixed.