The scripts contained in this archive are used during the experiments, corpus preparation, triage, and plotting figures from experiment data.
Corpus Preparation
There are various script that are involved with the corpus preparation. The following two scripts are to ease the calling to manually calling afl-cmin or afl-showmap and passing individual parameter set. This is to be used in conjunction with the configuration files containing the environment parameters. There are some sample configuration files in "sample_env" Currently, the location of the configuration files is expected to be in the folder of ~/llvm_asan/
To obtain the cmin corpus and afl tuples for further processing, execute the following:
cmin.sh <initial corpus> <program>
showmaps.sh <initial corpus> <program>
where initial corpus is the directory of the initial corpus, and program is the program which will be used in fuzzing. It is limited by the configuration files, and currently it can take value from the set of {pdf, sox, wav, tiff409, ttf253, svg24020, and xml290}.
The output from showmaps.sh is converted using MoonBeam to be bitvectors as input to MoonLight. "moonlight_call.sh" is a wrapper to ease the calling of MoonLight. It takes as input the program and the weight type, either none which means unweighted, "size" which means MoonLight will use file size as the weight, and "time" which means MoonLight will use execution time as the weight. The file size weight is obtained directly from the file information, or we can use "get_size.sh". The execution time weight is obtained by first executing "check_timeout.sh", and then we parse the information about the execution time for each seed from the log. For legacy reason, the actual weight values are first combined using "combine_time_size.sh" and then formatted correctly to the format expected by MoonLight by using "get_moonlight_weight.sh".
get_size.sh <program>
check_timeout.sh <program> <initial corpus>
combine_time_size.sh <prefix to the output of get_size.sh and check_timeout.sh>
get_moonlight_weight.sh <program> <infix to the output of get_size.sh and check_timeout.sh>
moonlight_call.sh <program> [ "size" | "time" | "" ]
To produce coverage for minset, we reuse the bitvectors produced by MoonBeam, and create the archive with necessary information with "minset_coverage.sh".
minset_coverage.sh <program> <bitvectors> <output directory>
There are other scripts that are not directly used to produce the corpora in the MoonLight paper. check_weight.sh is used to extract information about the weight in the MoonLight solution. sort_coverage.sh and copy_sorted_corpus is used to get the (reverse) sorted corpora based on coverage.
check_weight.sh <solution file> <time weight> <file size weight>, the weights are the result of
get_moonlight_weight.sh
sort_coverage.sh <corpus directory>
copy_sorted_corpus.sh <source directory> <target directory> <result of sort_coverage.sh>
In summary, the procedure on getting the corpora in the MoonLight paper:
- Full : none
- CMIN : cmin.sh
- Minset : showmaps.sh -> MoonBeam -> minset_coverage.sh -> minset -> copy the files from Full
- ML-U : showmaps.sh -> MoonBeam -> MoonLight -> move_solution.py -> moonlight_call.sh
- ML-S and ML-T : showmaps.sh -> MoonBeam -> check_timeout.sh -> get_size.sh -> combine_time_size.sh -> get_moonlight_weight.sh -> moonlight_call.sh
- Random : create_random.sh
If you want to get the (reverse) sorted based on coverage, use sort_coverage.sh -> copy_sorted_corpus.sh from the AFL tuples of the seeds in the above corpora.
There are scripts to hardlink the original corpora and append empty seed into the corpus. We have not used this in the paper yet, but this is a work in progress for adding empty seed to other corpus to see if we have performance benefit. "add_empty_wrapper.sh" is the wrapper to the "add_empty_to_corpus.sh"
add_empty_wrapper.sh <program> <file extension>
add_empty_to_corpus.sh <original empty corpus> <original source corpus> <output directory> <file extension>
Configuration Files
The configuration files, files with suffix "_env" in env subfolder, contains the parameters which will be used during throughout the experiments. Mainly they specify the arguments to the binary, memory limit, timeout, and the location of extra libraries needed.
Fuzzing
These scripts are used as a trivial fuzzing experiment manager. fuzz.sh is the script which will load configuration file and run a fuzzing experiment accordingly. run.sh invokes a number fuzz.sh (limit permitting), and pause when the maximum number of concurrent fuzzing experiment is hit.
run.sh <priority> <distillation techniques> <program> <starting trial number>
fuzz.sh <program> <distillation technique> <trial number> <fuzzer number>
The other scripts are used for archiving, cleaning the queue, and check if the fuzzers are instantiated. Cleaning the queue is achieved by replacing the files in the queue with empty files. Checking the how many fuzzers are instantiated is achieved by counting how many "fuzzer_stats" exist in the fuzzing output directory.
archive.sh <program> <distillation techniques> <minimum trial number> <maximum trial number>
clean_queue.sh <distillation technique> <trial number>
check_fuzzers_sound.sh <distillation techniques> <minimum trial number> <maximum trial number>
Triage
The scripts for triaging specific targets are contained in their respective subdirectories. There are three broad vein on how to triage the crashes found in the fuzzing experiment for MoonLight.
- Get the patch for a binary and then check if it does not produce the crash anymore, e.g., FreeType and libxml.
- If the bug is specific enough, then we can use the output of ASAN to distinguish bugs, e.g., SoX on both MP3 and WAV.
- Using breakpoint on GDB to tell if a specific situation which will lead to bug is triggered, mainly on libtiff.
Most of the scripts uses the configuration files and get_core.sh, and the triage scripts are invoked in the same way. "get_core.sh" is a script that is invoked by the triagin script when testing whether a particular crash is triggered.
get_core.sh <program> <experiment directory> <fuzzer number> <crash id>
<program>_triage.sh <distillation techniques> <minimum trial> <maximum trial> > <program>_triage_result
Since libtiff is using GDB, there are additional GDB scripts accompanying the script to triage libtiff.
Plot
We have a lot of scripts for various plotting purposes, and we tried to label them using tags in the header of each script. Some scripts are used to aggregate the data from several fuzzers in an experiment, calculate the average, truncating data points to speed up plotting process, and plot figures. There are two scripts that bind these processes together: get_get_coverage.plot.sh to plot coverage, and generate_bug_plot_data.sh which plot bugs found overtime.
get_coverage_plot.sh <program> <time limit>
generate_bug_plot_data.sh <program> <time limit>
The time limit is used to cap the time of an experiment in the granularity of hours. The scripts expect a directory of fuzzing experiments (and "triage_output" subdirectory containing the bug class of a corresponding crash) in the format of _<time\ limit>h, e.g., sox_18h.
We also have script that plots the speedup plot in the paper, and also the distribution of the corpora in the 2D-space (projected using PCA), and plotting the number of executions allocated for each seed in the corpus (derivatives from a seed are counted as part of the seed). The speedup plot uses the by-product information produced by the scripts to plot bugs overtime as input.
plot_bug_stats_aux.sh <program> <output>
Rscript plot_corpus_distance.R --algnames <distillation techniques> --inputprefix <prefix to the input files>
--output <output> --title <plot header>
There is no wrapper script for plotting number of executions allocated for each seed. "plot_seed_stats_grouped.R" is the individual version of the "plot_seed_stats.R".
Rscript plot_seed_stats_grouped.R --input <input file> --outputprefix <prefix of the output file>
--title <plot header> --highlight <file containing seed to highlight> --limit <limit on the number of seeds to show>
With respect to the previous plotting functionalities, these are the related scripts:
- generate_plot_data.py : combine the coverage of 8 fuzzers from each fuzzing experiment, and also produce its average overtime over all of the experiments.
- time_to_find.py : combine triaged bugs from fuzzing experiments.
- dedup_average.py : produce the average bugs over time (output of time_to_find.py)
- calculate_AUC.R : produce the area under curve for the input data, either it is output from generate_plot_data.py or dedup_average.py
- get_display_data.R : truncate the data points that are not visible in the plot to reduce plot processing time. Takes input from the output of generate_plot_data.py or dedup_average.py
- plot.R : Main plotting script. Takes input data from the ouput of calculate_AUC.R and get_display_data.R.
- plot_utils.R : Some useful functions used in plot.R, mainly to split the plotting script file so that it is not too big and modularity / reusability.
- pca.py : Get the 2D PCA operators from the tuples in a corpus directory (Full).
- pca_transform.py : Project the tuples in a corpus directory according to the operators learned by pca.py
- plot_legend.R : Plot the legend for bug / coverage over time as separate file for modularity in the Latex file.
- csv_to_file.py : This script is used to setup the fuzzing directories with the bugs from triage result. This is because time_to_find.py expect the directory structure and crash IDs to be the same as the fuzzing experiment.
- fill_non_bugs.sh : This script pad the triage result with no bug ("") when a particular crash is not a bug. This is because sometimes people give me triage result of only interesting bugs and leave behind the non-bug crashes.
- count_execs.py : Trace the AFL queue back to their sources and gather the statistics, i.e., the number of executions allocated to this seed.
- count_execs_avg.py : Aggregate the result of "count_execs.py" in the form of mean and standard deviation.
Apart from the previously mentioned scripts, there are various auxilliary scripts:
- calculate_CV.py : Script to get the coefficient of variation on the experiments result. This is deprecated since we removed the tables from the paper.
- compare_result.sh : Simple script to help me eyeball the output of "calculate_AUC.R"
- convert_pdf_to_jpg.sh : Rasterize a vector image (PDF), mainly used for plotting the corpus distance. We would love to maintain the vector format, but then it will take forever to display due to the number of data points from full corpus, c'est la vie.
- get_bug_count.sh : Get the number of trials in the campaign (of a distillation) that hit a particular bug.
- get_CV_table.sh : The wrapper script to call "calculate_CV.py" and format it into the Latex format.
- get_vda_from_log.sh : Get the VD.A values from the log of executing "plot.R" with "--conclusion" enabled.
- produce_vda_table.sh : Format the output of "get_vda_from_log.sh" into Latex format.
- trace_source_main.sh : Wraps the other trace source scripts, used to get an idea of which seed produce which bugs.
- trace_source.py : Predecessor of "count_execs.py", mainly this is tracing a crash to its source(s).
- trace_source_setup.sh : Setup the fuzzing experiments to have crash directories which contains only a particular bug.