Roadmap for future features
DonggeLiu opened this issue · comments
To keep track of the desirable features that we'd like to add in the future, here is a list of them in the order of priority:
- Make CI tests OSS-Fuzz-like trial builds.
- Revive more bug-based benchmarks and test them thoroughly with core fuzzers.
- Save experiment logs to an externally accessible bucket for fuzzer owners to debug.
- Versionize FuzzBench: We use a stable version of FuzzBench for production (experiments, etc.) and periodically replace it with new versions.
- Update benchmarks yearly to ensure the timeliness of FuzzBench.
- Allow users to add fuzzer flags from the command line or configuration file.
- Parallelize the coverage evaluation process to save time, fixed by #1672.
- More sophisticated ranking for bug-based benchmarks.
- When a fuzzer instance crashes during an experiment, fill the rest of the experiment with its latest data before the crash. This avoids sudden increases/drops in performance in the cov/bug vs. time plot (and avoids cheating).
- A script to run private experiment. Given a patch file that contains the new fuzzer, the script applies the patch, requests a private cloud experiment, and print the link to the experiment data and result.
- Notify us when an experiment finishes or crashes. This can be done via email.
Maintenance
- Omit empty strings in /gcbrun commands.
- For example, this command failed due to a mysterious extra space between
--fuzzersand the first fuzzer. After being converted to a list, the command becomes[..., '--fuzzers', '', 'aflplusplus_um_parallel'], making our script think that we passed an empty string as a fuzzer's name. - The solution is to remove all empty strings in the list.
- For example, this command failed due to a mysterious extra space between
Happy to discuss the necessity/priority of each, feel free to add more if I missed any : )
Allow users to add fuzzer flags from the command line or configuration file.
No I don't think this is worth it. I'd rather people make new fuzzers. We tried this and it's more trouble to implement than is worth it. Keep it Simple S*****
Another feature: something to write all stackdriver logs from an experiment to a bucket that can be viewed by users so we can shift some debugging burden
Another feature: something to write all stackdriver logs from an experiment to a bucket that can be viewed by users so we can shift some debugging burden
Yep I agree this could be very helpful, let me add it to the list.
Allow users to add fuzzer flags from the command line or configuration file.
No I don't think this is worth it. I'd rather people make new fuzzers. We tried this and it's more trouble to implement than is worth it. Keep it Simple S*****
Could this be as simple as passing a JSON file (e.g., in format {libfuzzer: '--flag_a --flag b'}) as an argument of run_experiment.py, or having extra items in the experiment-config.yaml (e.g., in format extra_flags: -libfuzzer: '--flag_a --flag b')?
We will also make sure to document these flags in the experiment report, though.
Allow users to add fuzzer flags from the command line or configuration file.
No I don't think this is worth it. I'd rather people make new fuzzers. We tried this and it's more trouble to implement than is worth it. Keep it Simple S*****
Could this be as simple as passing a JSON file (e.g., in format
{libfuzzer: '--flag_a --flag b'}) as an argument ofrun_experiment.py, or having extra items in theexperiment-config.yaml(e.g., in formatextra_flags: -libfuzzer: '--flag_a --flag b')?We will also make sure to document these flags in the experiment report, though.
This seems complex to me. And lets say you are comparing libfuzzer --flaga and libfuzzer --flagb, what do you name them?
I think fuzzbench doesn't have a lot of people working on it so it makes sense to choose simple easy-to-maintain solutions.
Related: #1636
- Could we save logs to the experiment data bucket? In that case, we don't have to worry about exposing private logs.
- Could we also save dispatcher logs? Currently, we need to ssh to the VM to access it.
Could we save logs to the experiment data bucket? In that case, we don't have to worry about exposing private logs.
I don't think we can save it to the experiment bucket but it is being saved right now to a different bucket gs://fuzzbench-logs/
However, I haven't made this bucket public because we need a way to filter private experiments. I'm not sure I like another change I had to make to make this work https://github.com/google/fuzzbench/pull/1638/files
Could we also save dispatcher logs? Currently, we need to #1806 (comment).
Maybe, can you think of a way to do this?
However, I haven't made this bucket public because we need a way to filter private experiments.
I see, could we have a private-logs bucket just like private experiment data and reports?
Could we also save dispatcher logs? Currently, we need to #1806 (comment).
Maybe, can you think of a way to do this?
I noticed build logs are in the experiment data bucket. Would it be OK for me to create another directory to save dispatcher log file?
However, I haven't made this bucket public because we need a way to filter private experiments.
I see, could we have a
private-logsbucket just like private experiment data and reports?
I think there's a misunderstanding here. We're using a GCS router to route logs to the bucket and we must not route logs from a private experiment into a public bucket.
Could we also save dispatcher logs? Currently, we need to #1806 (comment).
Maybe, can you think of a way to do this?
I noticed build logs are in the experiment data bucket. Would it be OK for me to create another directory to save dispatcher log file?
absolutely. I think it's OK even to put it in the top level folder.