Changes to statistics reported by the LLVM static compiler on account of just basic block reordering.

This project aims to report incorrect/phantom statistics generated by the llvm static compiler. We wrote a LLVM IR pass that reorders basic blocks randomly in the hope to see if it will fire varied statistics.

Description

The experiment is as follows

We keep a global register (across different source files) of unique statistics seen so far

This is in stats_seen_so_far.txt

A script to diff stats and check if unique stats exist (diff_and_check.py)

This script reads the statistics files, diffs them and checks if there are any different stats from the global register.

A script (script.sh) that does the following job:

Take a source bitcode file, call is source.bc and compile it two times, with/without our reordering pass and store the statistics in both cases. If we see a diff in the statistics (using diff_and_check.py), then the script returns status interesting (exit 1) for that source file otherwise (exit 0).

A script that invokes llvm-reduce to reduce the input file

The parent.py script invokes llvm-reduce with script.sh to check if the input is interesting. If so, we save it. We also update the global register with new statistics that were seen and continue for all source files. In our experiment, the source files were named with integer values, for instance, 0.bc, 1.bc ... There is a dataset in samples

Structure description

Each numbered folder in INTERESTING_FILES is a unique source file. The folder contains the reduced part of the original bitcode file.

For instance, 1/59489066aba24d4321ea37e2bf9882233ce9374c83ffc490b09bbf23eda4b16f.bc

The llvm IR has the extension .ll For instance, 1/59489066aba24d4321ea37e2bf9882233ce9374c83ffc490b09bbf23eda4b16f.ll

original.bc -- The bitcode file compiled with opt using -O3

reordered.bc -- The bitcode file compiled with opt using -O3 and the random basic block permutation pass

original.ll -- The corresponding llvm IR

reordered.ll -- The corresponding llvm IR

original.stats -- The statistics by obtained compiling original.bc with llc using -O3

reordered.stats -- The statistics obtained by compiling reordered.bc with llc using -O3

set_difference.stats -- Stores the diff of the statistics

original.s -- assembly of orignal file

reordered.s -- assembly of reordered file

For example, the source file 1 has the following diff in set_difference.stats

pass - description: <before> vs <after>

branch-folder       - Number of branches optimized      :  1   vs Nil  
dagcombine          - Number of dag nodes combined      :  Nil vs 4

Dependencies

LLVM version with basic block reordering: https://github.com/vidsinghal/llvm-project/tree/shuffle_bb
The LLVM version is added as a submodule.
Make sure you checkout to shuffle_bb branch.
Build LLVM with -stats enabled.
A dataset of source files to test, we have around 1000 files in samples.

Executing program

Python3 parent.py runs the experiment.

Authors

Vidush Singhal

Johannes Doerfert

vidsinghal / BBReorderingExperiment