Experiments not happening since upgrade
jonathanmetzman opened this issue · comments
Reported here: #1571 (comment) and here #1573
partial fix: #1574
partial fix: #1578
CC @Alan32Liu
is this still a problem?
It seems to be alright now, as I can see the new experiments were created successfully.
But let's keep it for two days and see if the results are the same as expected.
There's another issue that you (partially?) fixed here: #1581
By making all of the bug-based benchmarks into coverage benchmarks they were getting built in coverage experiments.
What was the rationale for doing this btw?
I need to make these back into bug-based:
diff --git a/benchmarks/arrow_parquet-arrow-fuzz/benchmark.yaml b/benchmarks/arrow_parquet-arrow-fuzz/benchmark.yaml
diff --git a/benchmarks/aspell_aspell_fuzzer/benchmark.yaml b/benchmarks/aspell_aspell_fuzzer/benchmark.yaml
diff --git a/benchmarks/ffmpeg_ffmpeg_demuxer_fuzzer/benchmark.yaml b/benchmarks/ffmpeg_ffmpeg_demuxer_fuzzer/benchmark.yaml
diff --git a/benchmarks/file_magic_fuzzer/benchmark.yaml b/benchmarks/file_magic_fuzzer/benchmark.yaml
diff --git a/benchmarks/grok_grk_decompress_fuzzer/benchmark.yaml b/benchmarks/grok_grk_decompress_fuzzer/benchmark.yaml
diff --git a/benchmarks/libarchive_libarchive_fuzzer/benchmark.yaml b/benchmarks/libarchive_libarchive_fuzzer/benchmark.yaml
diff --git a/benchmarks/libgit2_objects_fuzzer/benchmark.yaml b/benchmarks/libgit2_objects_fuzzer/benchmark.yaml
--- a/benchmarks/libgit2_objects_fuzzer/benchmark.yaml
+++ b/benchmarks/libgit2_objects_fuzzer/benchmark.yaml
diff --git a/benchmarks/libhevc_hevc_dec_fuzzer/benchmark.yaml b/benchmarks/libhevc_hevc_dec_fuzzer/benchmark.yaml
diff --git a/benchmarks/libhtp_fuzz_htp/benchmark.yaml b/benchmarks/libhtp_fuzz_htp/benchmark.yaml
diff --git a/benchmarks/libxml2_libxml2_xml_reader_for_file_fuzzer/benchmark.yaml b/benchmarks/libxml2_libxml2_xml_reader_for_file_fuzzer/benchmark.yaml
diff --git a/benchmarks/matio_matio_fuzzer/benchmark.yaml b/benchmarks/matio_matio_fuzzer/benchmark.yaml
diff --git a/benchmarks/mruby-2018-05-23/benchmark.yaml b/benchmarks/mruby-2018-05-23/benchmark.yaml
diff --git a/benchmarks/muparser_set_eval_fuzzer/benchmark.yaml b/benchmarks/muparser_set_eval_fuzzer/benchmark.yaml
diff --git a/benchmarks/njs_njs_process_script_fuzzer/benchmark.yaml b/benchmarks/njs_njs_process_script_fuzzer/benchmark.yaml
diff --git a/benchmarks/openh264_decoder_fuzzer/benchmark.yaml b/benchmarks/openh264_decoder_fuzzer/benchmark.yaml
diff --git a/benchmarks/php_php-fuzz-execute/benchmark.yaml b/benchmarks/php_php-fuzz-execute/benchmark.yaml
diff --git a/benchmarks/php_php-fuzz-parser-2020-07-25/benchmark.yaml b/benchmarks/php_php-fuzz-parser-2020-07-25/benchmark.yaml
diff --git a/benchmarks/poppler_pdf_fuzzer/benchmark.yaml b/benchmarks/poppler_pdf_fuzzer/benchmark.yaml
diff --git a/benchmarks/proj4_standard_fuzzer/benchmark.yaml b/benchmarks/proj4_standard_fuzzer/benchmark.yaml
diff --git a/benchmarks/quickjs_eval-2020-01-05/benchmark.yaml b/benchmarks/quickjs_eval-2020-01-05/benchmark.yaml
diff --git a/benchmarks/stb_stbi_read_fuzzer/benchmark.yaml b/benchmarks/stb_stbi_read_fuzzer/benchmark.yaml
diff --git a/benchmarks/systemd_fuzz-varlink/benchmark.yaml b/benchmarks/systemd_fuzz-varlink/benchmark.yaml
diff --git a/benchmarks/usrsctp_fuzzer_connect/benchmark.yaml b/benchmarks/usrsctp_fuzzer_connect/benchmark.yaml
diff --git a/benchmarks/wireshark_fuzzshark_ip/benchmark.yaml b/benchmarks/wireshark_fuzzshark_ip/benchmark.yaml
diff --git a/benchmarks/zstd_stream_decompress/benchmark.yaml b/benchmarks/zstd_stream_decompress/benchmark.yaml
is this still a problem?
It was a problem as of my last comment. But it may be fixed by #1586