eurecom-s3 / symcc

I am running a concolic solver benchmark on fuzzbench with symcc, symqemu, fuzzolic and eclipser:
https://www.fuzzbench.com/reports/experimental/2021-07-03-symbolic/index.html

I try to make this as fair as possible, last issue though is the runtime/timeout given to the solver, here symcc/symqemu. there it is not possible to set a value like it is possible for fuzzolic and eclipser.
what is the total maximum time that symcc/symqemu are running? So I can set this value to fuzzolic and eclipser for the next benchmark.

thank you!

Hi,

thanks for running the benchmark ! Very cool :)

I'm not sure which timeout is hit, AFAIK, there is one specified in the symcc helper, 90 seconds overall:

symcc/util/symcc_fuzzing_helper/src/symcc.rs

Line 28 in c4eacea

const TIMEOUT: u32 = 90;

And one in the QSym backend of 10 seconds per call to the solver here:
https://github.com/eurecom-s3/qsym/blob/d17a39d40dc3ea1d17262dd52607f8a6527dde10/qsym/pintool/solver.cpp#L11

There is also a 10s timeout for the calls to the solver in the simple backend here:

symcc/runtime/simple_backend/Runtime.cpp

Line 127 in c4eacea

Z3_set_param_value(cfg, "timeout", "10000"); // milliseconds

That would be better to make this configurable in one place, I'll mark this as a possible future enhancement.

ah thank you!
where do these three values comes from? Did you test them which gave the best results?

Well the QSYM paper mentions 5 minutes for overall timeout (runs not solver time it seems), and compares that to various values for Driller. But the code uses 10 seconds for solver time. Note also that Qsym does quite some magic on the queries before sending them to the solver, and that difference could be see when compared to SymCC simple backend.

So far, and as much as I can remember, there was no evaluation of the effect of tuning this solver timeout parameter, and I agree that this is definitely something interesting to look at !

To actually answer your question, I imagine we set the same value as QSym or fairness in comparison, and that this value was set so that most queries could be solved and only few that take very long be stopped. So I think this is mostly arbitrary.

Total max runtime