Giters

google / fuzzbench

FuzzBench - Fuzzer benchmarking as a service.

Home Page:https://google.github.io/fuzzbench/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Mruby is broken

andreafioraldi opened this issue · comments

commented

I'm using mruby to test things in my local checkout of fuzzbench and it I noticed that it is currently broken as it changed from the version that I uploaded months ago.

  • The build script is ignoring the compiler wrappers overriding CC and CXX

    fuzzbench/benchmarks/mruby-2018-05-23/build.sh

    Line 24 in 5d7362a

    LD=/usr/local/bin/clang CC=/usr/local/bin/clang CXX=/usr/local/bin/clang++ rake -m || true
    this make any fuzzer with a compiler wrapper like AFL++ and libafl incompatible. These changes don't come from the OSS-Fuzz build script as AFL++ works perfectly there.

  • The commit was updated and it is not anymore the version 2018-05-23 (so the name has no sense now) and thus there are no known bugs that can be triggered in a bug-based experiment.

commented

Thanks, @andreafioraldi,

The build script is ignoring the compiler wrappers overriding CC and CXX

Yes, good catch. I am more than happy to switch to that build script later.
Feel free to submit a PR if you are interested in fixing it, I would really appreciate that as well : )

The commit was updated and it is not anymore the version 2018-05-23

Yep, long story short, we recently upgraded the version of Ubuntu used by FuzzBench, and that requires us to update many benchmarks to build them. Eventually, we will update all benchmarks to their newer versions.
As you said, bug-based benchmarks are a bit special, as their latest version might not have any known bugs, so we will need a bit of time to identify the best version that is compatible with the new Ubuntu but still has as many known bugs as possible. At this moment, we only revived 5 bug benchmarks, but I promise I will gradually bring more back later.

I reckon this will benefit fuzzer evaluation as well: New bugs will help fuzzers identify what they missed in the past and avoid overfitting on finding the old ones.