akhikolla / RcppDeepState

RcppDeepState, a simple way to fuzz test code in Rcpp packages

RcppDeepState

RcppDeepState, a simple way to fuzz test compiled code in Rcpp packages. This package extends the DeepState framework to fully support Rcpp based packages.

Note: RcppDeepState is currently supported on Linux and macOS, with windows support in progress.

See Also

The RcppDeepState blog to know more about the working of RcppDeepState.

Dependencies

First, make sure to install the following dependencies on your local machine.

• CMake
• GCC and G++ with multilib support
• Python 3.6 (or newer)
• Setuptools

Use this command line to install the dependencies.

sudo apt-get install build-essential gcc-multilib g++-multilib cmake python3-setuptools libffi-dev z3

Installation

The RcppDeepState package can be installed from GitHub as follows:

install.packages("devtools")

devtools::install_github("akhikolla/RcppDeepState")

Functionalities

To test your package using RcppDeepState follow the steps below:

All these files generated are stored in inst/test files in your test package.

(a) deepstate_harness_compile_run: This function creates the TestHarnesses for all the functions in the test package with the corresponding makefiles. This function compiles and runs all those TestHarnesses that are created above and test your code for errors/bugs and stores the results in logfiles.

RcppDeepState::deepstate_harness_compile_run("~/R/RcppDeepState/inst/testpkgs/testSAN")

It gives a list of functions that are successfully compiled in the package:

>[1] "rcpp_read_out_of_bound"      "rcpp_use_after_deallocate" 
[3] "rcpp_use_after_free"         "rcpp_use_uninitialized"
[5] "rcpp_write_index_outofbound" "rcpp_zero_sized_array"

(b) deepstate_harness_analyze_pkg: This function analyzes each binary crash/fail file generated and provides a list of error messages and the inputs passed on to the function. The generated log files are stored in the respective crash file folder along with the inputs i.e inst/function/12abc.crash/valgrind_log

result = RcppDeepState::deepstate_harness_analyze_pkg("~/R/RcppDeepState/inst/testpkgs/testSAN")

The result contains a data table with three columns: binary.file,inputs,logtable

> head(result,2)
                                                                                                                                                                                         binaryfile
1: /home/akolla/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/rcpp_read_out_of_bound_output/0001957a365ef90344a992e32cc2d49d4aedf572.crash
2: /home/akolla/R/x86_64-pc-linux-gnu-library/3.6/RcppDeepState/testpkgs/testSAN/inst/testfiles/rcpp_read_out_of_bound/rcpp_read_out_of_bound_output/0001b796162c8cd4b00f4b7ccf165b55b566cfce.crash
      inputs          logtable
1: <list[1]> <data.table[2x5]>
2: <list[1]> <data.table[1x5]>

The inputs column contains all the inputs that are passed:

> result$inputs[[1]]
$rbound
[1] 437585945

The logtable has the data table with a list of errors:

> result$logtable[[1]]

      err.kind                message                     file.line
1: InvalidRead Invalid read of size 4 src/read_out_of_bound.cpp : 7
                                                         address.msg
1: Address 0x11eecf884 is not stack'd, malloc'd or (recently) free'd
            address.trace
1: No Address Trace found

Before testing your package using RcppDeepState, we need to make sure that RcppDeepState is working correctly. To do so please make sure to check if RcppDeepState::deepstate_fuzz_fun_analyze() produces the same results as expected.

For example, when we run the function rcpp_write_index_outofbound:

> fun_path <- file.path(path,"inst/testfiles/rcpp_write_index_outofbound") 
> seed_analyze<-deepstate_fuzz_fun_analyze(fun_path,1603403708,5)
> seed_analyze

       err.kind                 message                          file.line
1: InvalidWrite Invalid write of size 4 src/write_index_outofbound.cpp : 8
                                                        address.msg
1: Address 0x2707127c is not stack'd, malloc'd or (recently) free'd
            address.trace
1: No Address Trace found

Now RcppDeepState makes it easy to use RcppDeepState on Travis-CI.

deepstate_ci_setup: This function edits your .travis.yml file or creates one if it doesn't exist with the necessary packages and environment variables and now push your updated code to GitHub and check the Travis build for the test package for results.

RcppDeepState::deepstate_ci_setup(pathtotestpackage)