Integrate with google/oss-fuzz for continuous fuzz testing?
silvergasp opened this issue · comments
Hey I'd like to suggest adding starlark-rust to google/oss-fuzz. If you aren't familiar with fuzz testing, here is a bit of a run down (from Wikipedia);
In programming and software development, fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program. The program is then monitored for exceptions such as crashes, failing built-in code assertions, or potential memory leaks.
Google offers a free continuous fuzzing service called OSS-fuzz. If starlark-rust is integrated into oss-fuzz, the (yet to be written) fuzz tests under starlark-rust will be built and then run once a day, to search for bugs and vulnerabilities in starlark-rust. This service can be integrated with the CI for starlark-rust, so that the fuzz tests are run for 10min or so for every pull request, preventing buggy code from being merged.
Here is an example of another FB project zstd that has been integrated into oss-fuzz https://github.com/google/oss-fuzz/blob/master/projects/zstd/project.yaml.
I've opened up a pull request to add a basic fuzz-testing harness here #64. If you are keen on adding starlark-rust to oss-fuzz I'd be happy to champion the integration :)
Other relevant issues:
Continuing on from #64 (comment), I've gone ahead and added a draft PR in OSS-fuzz google/oss-fuzz#9809. There are a couple of things that I need in order to continue with the PR.
- A primary contact email address. I'm guessing this will be either @ndmitchell or @stepancheg?
- A list of email addresses to add to the CC's. This will translate to the list of people that will get access to the dashboard and bug reports when they become available.
- #64 will need to be merged before I can integrate with oss-fuzz. For testing I've just pointed the draft PR at my personal fork.
- A developer here to comment/approve google/oss-fuzz#9809
Note that I can use the @meta.com emails from the git log, however for you to get full access to the oss-fuzz dashboard I'll need to add an email address to auto_css that is associated with a google account; You can read about why this is required here.
Happy to be primary contact with @stepancheg a CC. My email is ndmitchell -at- gmail.com which you can use and is a Google account (although I think Meta emails are actually also Google accounts, but I'd prefer my personal one anyway).
Sorted :)
I've now seen 5 reports:
- Two stack overflow processing large nested syntax.
- One timeout, which is easy enough to induce by writing a loop.
- One out of memory, which is easy enough to cause.
- One panic in printing a diagnostic. A good find - I'll have a patch for it out soon.
Thanks @silvergasp for doing this work, as the panic while printing diagnostic had been reported in the past, and I wasn't able to figure out how to fix it. With the reproducer it didn't take long.
See 5febddf for the fix that was produced.
Closing since we do interoperate. Nothing interesting showed up for a while, mostly stack overflow, but great to have that layer of safety.