ETH Robustness Analyzer for Neural Networks (ERAN) is a state-of-the-art sound, precise, and scalable analyzer based on abstract interpretation for the complete and incomplete robustness verification of MNIST, CIFAR10, and ACAS Xu based networks. ERAN produces state-of-the-art precision and performance for both complete and incomplete verification. ERAN is developed at the SRI Lab, Department of Computer Science, ETH Zurich as part of the Safe AI project. The goal of ERAN is to automatically verify robustness of neural networks with feedforward, convolutional, and residual layers against input perturbations (e.g., L∞-norm attacks, geometric transformations, etc).
ERAN supports networks with ReLU, Sigmoid and Tanh activations and is sound under floating point arithmetic. It employs custom abstract domains which are specifically designed for the setting of neural networks and which aim to balance scalability and precision. Specifically, ERAN supports the following three analysis:
-
DeepZ [NIPS'18]: contains specialized abstract Zonotope transformers for handling ReLU, Sigmoid and Tanh activation functions.
-
DeepPoly [POPL'19]: based on a domain that combines floating point Polyhedra with Intervals.
-
RefineZono [ICLR'19]: combines DeepZ analysis with MILP and LP solvers for more precision.
All analysis are implemented using the ELINA library for numerical abstractions. More details can be found in the publications below.
Note that ERAN subsumes the first abstract interpretation based analyzer AI2, so if you aim to compare, please use ERAN as a baseline.
GNU C compiler, ELINA, Gurobi's Python interface,
python3.6 or higher, tensorflow 1.11 or higher, numpy.
Clone the ERAN repository via git as follows:
git clone https://github.com/eth-sri/ERAN.git
cd ERAN
The dependencies for ERAN can be installed step by step as follows (sudo rights might be required):
Install m4:
wget ftp://ftp.gnu.org/gnu/m4/m4-1.4.1.tar.gz
tar -xvzf m4-1.4.1.tar.gz
cd m4-1.4.1
./configure
make
make install
cp src/m4 /usr/bin
cd ..
rm m4-1.4.1.tar.gz
Install gmp:
wget https://gmplib.org/download/gmp/gmp-6.1.2.tar.xz
tar -xvf gmp-6.1.2.tar.xz
cd gmp-6.1.2
./configure --enable-cxx
make
make install
cd ..
rm gmp-6.1.2.tar.xz
Install mpfr:
wget https://www.mpfr.org/mpfr-current/mpfr-4.0.2.tar.xz
tar -xvf mpfr-4.0.1.tar.xz
cd mpfr-4.0.1
./configure
make
make install
cd ..
rm mpfr-4.0.1.tar.xz
Install ELINA:
git clone https://github.com/eth-sri/ELINA.git
cd ELINA
./configure
make
make install
cd ..
Install Gurobi:
wget https://packages.gurobi.com/8.1/gurobi8.1.0_linux64.tar.gz
tar -xvf gurobi8.1.0_linux64.tar.gz
cd gurobi810/linux64
python3 setup.py install
sudo cp lib/libgurobi81.so /usr/lib
cd ../../
Update environment variables:
export GUROBI_HOME="Current_directory/gurobi810/linux64"
export PATH="${PATH}:${GUROBI_HOME}/bin"
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib:${GUROBI_HOME}/lib
We also provide scripts that will install ELINA and all the necessary dependencies. One can run it as follows:
sudo ./install.sh
source gurobi_setup_path.sh
Note that to run ERAN with Gurobi one needs to obtain an academic license for gurobi from https://user.gurobi.com/download/licenses/free-academic.
To install the remaining python dependencies (numpy and tensorflow), type:
pip3 install -r requirements.txt
ERAN may not be compatible with older versions of tensorflow (we have tested ERAN with versions >= 1.11.0), so if you have an older version and want to keep it, then we recommend using the python virtual environment for installing tensorflow.
cd tf_verify
python3 . --netname <path to the network file> --epsilon <float between 0 and 1> --domain <deepzono/deeppoly/refinezono> --dataset <mnist/cifar10/acasxu> [optional] --complete <True/False> --timeout_lp <float> --timeout_milp <float> --use_area_heuristic <True/False>
-
<epsilon>: specifies bound for the L∞-norm based perturbation (default is 0). This parameter is not required for testing ACAS Xu networks. -
<use_area_heuristic>: specifies whether to use area heuristic for the ReLU approximation in DeepPoly -
Note that the residual layers are currently only supported with the DeepZ (called with deepzono) domain.
-
Refinezono refines the analysis results from the DeepZ domain using the approach in our ICLR'19 paper. The optional parameters timeout_lp and timeout_milp (default is 1 sec for both) specify the timeouts for the LP and MILP forumlations of the network respectively.
-
Since Refinezono uses timeout for the gurobi solver, the results will vary depending on the processor speeds.
-
Setting the parameter "complete" (default is False) to True will enable MILP based complete verification using the bounds provided by the specified domain. When complete verification fails, ERAN prints an adversarial image within the specified adversarial region along with the misclassified label and the correct label.
-
ERAN currently supports verifying only the property 9 for ACAS Xu as defined in [https://arxiv.org/pdf/1702.01135.pdf] (known to be hard). Support for other properties will be added soon.
python3 . --netname ../nets/pytorch/mnist/convBig__DiffAI.pyt --epsilon 0.1 --domain deepzono --dataset mnist
will evaluate the local robustness of the MNIST convolutional network (upto 35K neurons) with ReLU activation trained using DiffAI on the 100 MNIST test images. In the above setting, epsilon=0.1 and the domain used by our analyzer is the deepzono domain. Our analyzer will print the following:
-
'Verified' for an image when it can prove the robustness of the network and 'Failed' when it cannot. It will also print an error message when the network misclassifies an image.
-
the timing in seconds.
-
The ratio of images on which the network is robust versus the number of images on which it classifies correctly.
For the ACAS Xu networks, ERAN will output whether the property has been verified along with the timing.
-
Fast and Effective Robustness Certification.
Gagandeep Singh, Timon Gehr, Matthew Mirman, Markus Püschel, and Martin Vechev.
NIPS 2018.
-
An Abstract Domain for Certifying Neural Networks.
Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev.
POPL 2019.
-
Boosting Robustness Certification of Neural Networks.
Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev.
ICLR 2019.
We provide a number of pretrained MNIST and CIAFR10 defended and undefended feedforward and convolutional neural networks with ReLU, Sigmoid and Tanh activations trained with the PyTorch and TensorFlow frameworks. The adversarial training to obtain the defended networks is performed using PGD and DiffAI.
| Dataset | Model | Type | #units | #layers | Activation | Training Defense | Download |
|---|---|---|---|---|---|---|---|
| MNIST | 3x50 | fully connected | 160 | 3 | ReLU | None | ⬇️ |
| 3x100 | fully connected | 310 | 3 | ReLU | None | ⬇️ | |
| 5x100 | fully connected | 510 | 5 | ReLU | DiffAI | ⬇️ | |
| 6x100 | fully connected | 610 | 6 | ReLU | None | ⬇️ | |
| 9x100 | fully connected | 910 | 9 | ReLU | None | ⬇️ | |
| 6x200 | fully connected | 1,210 | 6 | ReLU | None | ⬇️ | |
| 9x200 | fully connected | 1,810 | 9 | ReLU | None | ⬇️ | |
| 6x500 | fully connected | 3,010 | 6 | ReLU | None | ⬇️ | |
| 6x500 | fully connected | 3,010 | 6 | ReLU | PGD ε=0.1 | ⬇️ | |
| 6x500 | fully connected | 3,010 | 6 | ReLU | PGD ε=0.3 | ⬇️ | |
| 6x500 | fully connected | 3,010 | 6 | Sigmoid | None | ⬇️ | |
| 6x500 | fully connected | 3,010 | 6 | Sigmoid | PGD ε=0.1 | ⬇️ | |
| 6x500 | fully connected | 3,010 | 6 | Sigmoid | PGD ε=0.3 | ⬇️ | |
| 6x500 | fully connected | 3,010 | 6 | Tanh | None | ⬇️ | |
| 6x500 | fully connected | 3,010 | 6 | Tanh | PGD ε=0.1 | ⬇️ | |
| 6x500 | fully connected | 3,010 | 6 | Tanh | PGD ε=0.3 | ⬇️ | |
| 4x1024 | fully connected | 4,106 | 4 | ReLU | None | ⬇️ | |
| ConvSmall | convolutional | 3,604 | 3 | ReLU | None | ⬇️ | |
| ConvSmall | convolutional | 3,604 | 3 | ReLU | PGD | ⬇️ | |
| ConvSmall | convolutional | 3,604 | 3 | ReLU | DiffAI | ⬇️ | |
| ConvMed | convolutional | 4,804 | 3 | ReLU | None | ⬇️ | |
| ConvMed | convolutional | 4,804 | 3 | ReLU | PGD ε=0.1 | ⬇️ | |
| ConvMed | convolutional | 4,804 | 3 | ReLU | PGD ε=0.3 | ⬇️ | |
| ConvMed | convolutional | 4,804 | 3 | Sigmoid | None | ⬇️ | |
| ConvMed | convolutional | 4,804 | 3 | Sigmoid | PGD ε=0.1 | ⬇️ | |
| ConvMed | convolutional | 4,804 | 3 | Sigmoid | PGD ε=0.3 | ⬇️ | |
| ConvMed | convolutional | 4,804 | 3 | Tanh | None | ⬇️ | |
| ConvMed | convolutional | 4,804 | 3 | Tanh | PGD ε=0.1 | ⬇️ | |
| ConvMed | convolutional | 4,804 | 3 | Tanh | PGD ε=0.3 | ⬇️ | |
| ConvMaxpool | convolutional | 13,798 | 9 | ReLU | None | ⬇️ | |
| ConvBig | convolutional | 34,688 | 6 | ReLU | DiffAI | ⬇️ | |
| ConvSuper | convolutional | 88,500 | 6 | ReLU | DiffAI | ⬇️ | |
| Skip | Residual | 71,650 | 9 | ReLU | DiffAI | ⬇️ | |
| CIFAR10 | 4x100 | fully connected | 410 | 4 | ReLU | None | ⬇️ |
| 6x100 | fully connected | 610 | 6 | ReLU | None | ⬇️ | |
| 9x200 | fully connected | 1,810 | 9 | ReLU | None | ⬇️ | |
| 6x500 | fully connected | 3,010 | 6 | ReLU | None | ⬇️ | |
| 6x500 | fully connected | 3,010 | 6 | ReLU | PGD ε=0.0078 | ⬇️ | |
| 6x500 | fully connected | 3,010 | 6 | ReLU | PGD ε=0.0313 | ⬇️ | |
| 6x500 | fully connected | 3,010 | 6 | Sigmoid | None | ⬇️ | |
| 6x500 | fully connected | 3,010 | 6 | Sigmoid | PGD ε=0.0078 | ⬇️ | |
| 6x500 | fully connected | 3,010 | 6 | Sigmoid | PGD ε=0.0313 | ⬇️ | |
| 6x500 | fully connected | 3,010 | 6 | Tanh | None | ⬇️ | |
| 6x500 | fully connected | 3,010 | 6 | Tanh | PGD ε=0.0078 | ⬇️ | |
| 6x500 | fully connected | 3,010 | 6 | Tanh | PGD ε=0.0313 | ⬇️ | |
| 7x1024 | fully connected | 7,178 | 7 | ReLU | None | ⬇️ | |
| ConvSmall | convolutional | 4,852 | 3 | ReLU | None | ⬇️ | |
| ConvSmall | convolutional | 4,852 | 3 | ReLU | PGD | ⬇️ | |
| ConvSmall | convolutional | 4,852 | 3 | ReLU | DiffAI | ⬇️ | |
| ConvMed | convolutional | 6,244 | 3 | ReLU | None | ⬇️ | |
| ConvMed | convolutional | 6,244 | 3 | ReLU | PGD ε=0.0078 | ⬇️ | |
| ConvMed | convolutional | 6,244 | 3 | ReLU | PGD ε=0.0313 | ⬇️ | |
| ConvMed | convolutional | 6,244 | 3 | Sigmoid | None | ⬇️ | |
| ConvMed | convolutional | 6,244 | 3 | Sigmoid | PGD ε=0.0078 | ⬇️ | |
| ConvMed | convolutional | 6,244 | 3 | Sigmoid | PGD ε=0.0313 | ⬇️ | |
| ConvMed | convolutional | 6,244 | 3 | Tanh | None | ⬇️ | |
| ConvMed | convolutional | 6,244 | 3 | Tanh | PGD ε=0.0078 | ⬇️ | |
| ConvMed | convolutional | 6,244 | 3 | Tanh | PGD ε=0.0313 | ⬇️ | |
| ConvMaxpool | convolutional | 53,938 | 9 | ReLU | None | ⬇️ | |
| ConvBig | convolutional | 62,464 | 6 | ReLU | DiffAI | ⬇️ |
We provide the first 100 images from the testset of both MNIST and CIFAR10 datasets in the 'data' folder. Our analyzer first verifies whether the neural network classifies an image correctly before performing robustness analysis. In the same folder, we also provide ACAS Xu networks and property specifications.
We ran our experiments for the feedforward networks on a 3.3 GHz 10 core Intel i9-7900X Skylake CPU with a main memory of 64 GB whereas our experiments for the convolutional networks were run on a 2.6 GHz 14 core Intel Xeon CPU E5-2690 with 512 GB of main memory. We first compare the precision and performance of DeepZ and DeepPoly vs Fast-Lin on the MNIST 6x100 network in single threaded mode. It can be seen that DeepZ has the same precision as Fast-Lin whereas DeepPoly is more precise while also being faster.
In the following, we compare the precision and performance of DeepZ and DeepPoly on a subset of the neural networks listed above in multi-threaded mode. In can be seen that DeepPoly is overall more precise than DeepZ but it is slower than DeepZ on the convolutional networks.
The table below compares the performance and precision of DeepZ and DeepPoly on our large networks trained with DiffAI.
| Dataset | Model | ε | % Verified Robustness | % Average Runtime (s) | ||
| DeepZ | DeepPoly | DeepZ | DeepPoly | |||
| MNIST | ConvBig | 0.1 | 97 | 97 | 5 | 50 |
| ConvBig | 0.2 | 79 | 78 | 7 | 61 | |
| ConvBig | 0.3 | 37 | 43 | 17 | 88 | |
| ConvSuper | 0.1 | 97 | 97 | 133 | 400 | |
| Skip | 0.1 | 95 | N/A | 29 | N/A | |
| CIFAR10 | ConvBig | 0.006 | 50 | 52 | 39 | 322 |
| ConvBig | 0.008 | 33 | 40 | 46 | 331 | |
More experimental results can be found in our papers.
-
Gagandeep Singh (lead contact) - gsingh@inf.ethz.ch
-
Adrian Hoffmann - adriahof@student.ethz.ch
- Copyright (c) 2018 Secure, Reliable, and Intelligent Systems Lab (SRI), Department of Computer Science ETH Zurich
- Licensed under the Apache License