This Python package provides utility functions to create submissions for the Robust Vision Benchmark and scripts to automatically test and upload them. You might also want to have a look at Foolbox, our python toolbox to benchmark the robustness of machine learning models using a large set of adversarial attacks.
We test using Python 2.7, 3.5 and 3.6. Other Python versions might work as well, but we recommend using Python 3.5 or newer.
pip install robust-vision-benchmark
git clone https://github.com/bethgelab/robust-vision-benchmark.git
cd robust-vision-benchmark && python setup.py install && cd ..
Now, you need to copy data directory to the installation directory.
d=$(python -c "import os, robust_vision_benchmark as rvb; print(os.path.dirname(rvb.__file__))")
cp -r robust-vision-benchmark/data $d
A model submission consists of a Dockerfile, a script (e.g. Python) and data (e.g. network weights).
Create a Python script that turns your model into a Foolbox model using one of our wrappers for TensorFlow, PyTorch, Theano, Keras, Lasagne, MXNet and starts the model_server.
from robust_vision_benchmark import model_server
# create your model
# ...
# turn it into a Foolbox model
model = foolbox.models.SomeModel(...)
# start the server
mnist_model_server(model)
cifar_model_server(model, channel_order='RGB or BGR')
imagenet_model_server(model, channel_order='RGB or BGR', image_size=224)
# For CIFAR and Imagenet, the channel_order must be set to either 'RGB' or 'BGR'.
# For ImageNet, the image_size must be set to an integer (usually 224 vor VGG-like networks and 299 for inception-like networks).
Create a Dockerfile that installs all dependencies and starts the script.
FROM nvidia/cuda:8.0-cudnn5-devel-ubuntu16.04
RUN apt-get update ... && install ...
RUN pip3 install --no-cache-dir robust-vision-benchmark
...
COPY main.py main.py
CMD ["python3", "./main.py"]
For compatibility with our backend, please use a derivative of the nvidia/cuda:8.0 image (e.g. nvidia/cuda8.0-cudnn5-devel-ubuntu16.04) as the base image or contact us if you have special requirements.
Put the Dockerfile, script, data and other required files into a folder, e.g. model and run the following in your shell:
rvb-test-model model/
Note that you need to have nvidia-docker installed and in the path for this to run. You can find an example in examples/model/.
Once your model is ready for submission, upload it:
rvb-upload model/
Go to https://robust.vision/benchmark/participate and put the URL returned by the upload script into the Submission URL.
An attack submission consists of a Dockerfile and a script (e.g. Python).
Create a Python script that implements your attack and starts the attack_server.
from robust_vision_benchmark import attack_server
# implement your attack
def attack(a):
# ...
# start the server
attack_server(attack)
Create a Dockerfile that installs all dependencies and starts the script.
FROM python:3.6
RUN pip3 install --no-cache-dir robust-vision-benchmark
...
COPY main.py main.py
CMD ["python3", "./main.py"]
Put the Dockerfile, script and other required files into a folder, e.g. attack and run the following in your shell:
rvb-test-attack attack/
You can find an example in examples/attack/.
Once your attack is ready for submission, upload it:
rvb-upload attack/
Go to https://robust.vision/benchmark/participate and put the URL returned by the upload script into the Submission URL.