This repository contains the code for reproducing the results of evaluating all algorithms under most of settings, of our BENCHMARKING ADVERSARIAL ROBUSTNESS.
- Python 3.5
- TensorFlow >= 1.3.0 (with GPU support)
- PyTorch 1.0.1
- tensorpack
realsafe
is a package that includes all attack and defense algorithms that are incorporated into our benchmark for evaluation.
benchmark-v1
and benchmark-v2
include all the benchmark codes against all defense algorithms mentioned in the paper. Specifically,
benchmark_v1
contains benchmark scripts by using DeepFool and all black-box attack algorithms we use, and benchmark_v2
is based on white-box algorithms.
Among them,
- [⬆ Effectiveness]:
_distoration
scripts are to get the accuracy vs. perturbation budget, we perform a line search followed by a binary search on pertubation to find its minimum value.
- [⬆ Efficiency]:
_iteration
scripts can obtain the accuracy vs. attack strength, where the attacks trength is defined as the number of iterations or model queries for different attack methods
Note that these two folders do not use the same interface, this will be fixed later.
cd benchmark-v1
python3 cifar_resnet56_iteration.py --method nes --goal ut --distance l_2 --xs PATH_CIFAR_XS --ys PATH_CIFAR_YS --ys-target PATH_CIFAR_YS_TARGETS --output CIFAR_OUTPUT.npy
cd benchmark-v2
python3 cifar_convex_distortion.py --method bim --goal t --distance-metric l_inf --xs PATH_CIFAR_XS --ys PATH_CIFAR_YS --ys-target PATH_CIFAR_YS_TARGETS -- output CIFAR_OUTPUT.npy
cd benchmark-v1
python3 imagenet_incv3_iteration.py --method boundary --goal ut --distance l_2 --start 0 --end 100 --output IMAGENET_OUT.npy
cd benchmark-v2
python3 imagenet_incv3_distortion.py --method mim --goal t --distance l_inf --output IMAGENET_OUT.npy