SBFT experiments are frozen

Question

SBFT experiments are frozen

addisoncrump opened this issue a year ago · comments

Addison Crump commented a year ago

It appears that the SBFT experiments are frozen:

Is this intentional? No results seem to have been added past 45 minutes.

jonathanmetzman · Answer 1 · Tue Mar 07 2023 23:56:25 GMT+0800 (China Standard Time)

I'll take a look, but sometimes there are bottlenecks saving results. Thanks for being on top of this.

Dongge Liu · Answer 2 · Wed Mar 08 2023 06:36:51 GMT+0800 (China Standard Time)

Hi @VTCAKAVSMoACE,

I just checked our google cloud logs of both experiments, and I am pretty sure they are still running fuzzer evaluations.
The report may seem frozen, but FuzzBench will automatically update the plot and statistics periodically.

jonathanmetzman · Answer 3 · Wed Mar 08 2023 06:48:05 GMT+0800 (China Standard Time)

the reason why "periodically" is so infrequent now is probably due to the size of this experiment

jonathanmetzman · Answer 4 · Thu Mar 09 2023 03:32:45 GMT+0800 (China Standard Time)

I think there's something going on with measurement that I need to fix

jonathanmetzman · Answer 5 · Sat Mar 11 2023 03:05:17 GMT+0800 (China Standard Time)

I manually fixed the issue. We may need to fix some of the results (some benchmarks should be excluded based on most fuzzers being incompatible): https://www.fuzzbench.com/reports/experimental/2023-03-06-sbft23-cov-manual/index.html

Addison Crump · Answer 6 · Mon Mar 13 2023 00:49:48 GMT+0800 (China Standard Time)

Awesome, thank you for the updates. Shall we close this now that final evals have been announced?

Dongge Liu · Answer 7 · Mon Mar 13 2023 07:18:51 GMT+0800 (China Standard Time)

Sure!
Thanks again for reporting this to us!