google / fuzzbench

FuzzBench - Fuzzer benchmarking as a service.

Home Page:https://google.github.io/fuzzbench/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Only trigger experiments when a comment starts with `/gcbrun`

DonggeLiu opened this issue · comments

Currently, a GCB experiment will be triggered if our comment mentions /gcbrun.

This is problematic because experiments will be launched unintendedly (e.g., when we quote previous /gcb commands, or even when we mention /gcb in any form).

Root cause

  1. GCB trigger simply checks for /gcb without any constraints.
  2. Our gcbrun_experiment.py iterates all previous comments to find the latest valid /gcb command, which might have already been run.

Proofs

  1. This example quotes an old command and modifies its experiment name. However, it triggers an experiment with the old name, indicating we launch the latest valid experiment command.
  2. This example includes a random /gcb command in the middle of the sentence. It also triggers the same old experiment, confirming the same conclusion.

Propose fix

Change

if not body.startswith(RUN_EXPERIMENT_COMMAND_STR):
continue

to

 if not body.startswith(RUN_EXPERIMENT_COMMAND_STR): 
     return None 

Justification

  1. Based on our setting on Google Cloud, all /gcb commands will immediately trigger an experiment, there is no need to search back in history for an old command.
  2. If we need to run an old command, we can always add a new comment for it.

@jonathanmetzman: Did I miss anything?
If it looks good to you, I will implement the fix proposed above.

you mean [/]gcbrun run right?

I think this is a good idea.

you mean [/]gcbrun run right?

Yep, fixed.