Running SATestAdd.py on open source benchmarks

[László Nagy]

I’ve written up some instructions below on how to run the $CLANG_SOURCES_DIR/utils/analyzer/SATestAdd.py script. We have a buildbot that uses this script to make sure we catch regressions on openssl and postgresql, as well as internal projects.

Instructions:

1. Create a directory to hold the benchmarks:

   $ mkdir clang-analyzer-tests-open-source

2. Download a benchmark (e.g., openssl from https://www.openssl.org/source/openssl-1.0.0s.tar.gz)

3. Untar it in the clang-analyzer-tests-open-source directory.

4. Add a run_static_analyzer.cmd file to the untarred project directory to tell SATestAdd.py how to build the project. For openssl, the contents of this file should be:

   ./config
   make clean
   make -j1

5. Add a cleanup_run_static_analyzer.sh file to the untarred project directory to tell SATestBuild.py how to clean up after building the project. For openssl, this is:

   make clean
   exit 0

6. Make sure you have both scan-build and the clang you want to analyze with in your path:

   $ export PATH=$LLVM_BUILD_DIR/Release+Asserts/bin:$CLANG_SOURCES_DIR/tools/scan-build/:$PATH

7. Run SATestAdd.py to add the benchmark to the projectMap (make sure you are in the directory you created in step 1.:

   $ python $CLANG_SOURCES_DIR/utils/analyzer/SATestAdd.py openssl-1.0.0s 1

Here the ‘1’ indicates that the project should be built with scan-build (a 0 or 2 instead of a 1 would indicate that it is a single-file benchmark — but these aren’t important for scan-build).
This will run the analyzer on the project to create a reference set of results and create a projectMap.csv file.

You should see something like:

--- Building project openssl-1.0.0s
  Build directory: /Volumes/Data/SATests/clang-analyzer-tests-open-source/openssl-1.0.0s.
Log file: /Volumes/Data/SATests/clang-analyzer-tests-open-source/openssl-1.0.0s/RefScanBuildResults/Logs/run_static_analyzer.log
Output directory: /Volumes/Data/SATests/clang-analyzer-tests-open-source/openssl-1.0.0s/RefScanBuildResults
  Executing: /Volumes/Data/SATests/clang-analyzer-tests-open-source/openssl-1.0.0s/cleanup_run_static_analyzer.sh
  Executing: scan-build --use-analyzer /usr/bin/clang -plist-html -o /Volumes/Data/SATests/clang-analyzer-tests-open-source/openssl-1.0.0s/RefScanBuildResults -enable-checker alpha.unix.SimpleStream,alpha.security.taint,cplusplus.NewDeleteLeaks,core,cplusplus,deadcode,security,unix,osx --keep-empty --override-compiler  ./config
  Executing: scan-build --use-analyzer /usr/bin/clang -plist-html -o /Volumes/Data/SATests/clang-analyzer-tests-open-source/openssl-1.0.0s/RefScanBuildResults -enable-checker alpha.unix.SimpleStream,alpha.security.taint,cplusplus.NewDeleteLeaks,core,cplusplus,deadcode,security,unix,osx --keep-empty --override-compiler  make clean -j6
  Executing: scan-build --use-analyzer /usr/bin/clang -plist-html -o /Volumes/Data/SATests/clang-analyzer-tests-open-source/openssl-1.0.0s/RefScanBuildResults -enable-checker alpha.unix.SimpleStream,alpha.security.taint,cplusplus.NewDeleteLeaks,core,cplusplus,deadcode,security,unix,osx --keep-empty --override-compiler  make -j1
  Executing: /Volumes/Data/SATests/clang-analyzer-tests-open-source/openssl-1.0.0s/cleanup_run_static_analyzer.sh
Build complete (time: 326.35). See the log for more details: /Volumes/Data/SATests/clang-analyzer-tests-open-source/openssl-1.0.0s/RefScanBuildResults/Logs/run_static_analyzer.log
Number of bug reports (non-empty plist files) produced: 87
Completed tests for project openssl-1.0.0s (time: 326.71).
Warning: Creating the Project Map file!!
The project map is updated:  /Volumes/Data/SATests/clang-analyzer-tests-open-source/projectMap.csv

This will create a RefScanBuildResults directory in the project directory with reference analysis results. (Note: you will get an error if you try to add the same project twice.)

It is important to us that scan-build-py works with these scripts (this is a indicator that other build-bot-style uses of scan-build out in the wild can be replaced with scan-build-py) and that it reports the same issues on benchmarks like openssl as the old scan-build does (including issues with multi-file paths reported in the .plist output).