Transfer Learning using Adversarially Robust ImageNet Models

This repository contains the code and models necessary to replicate the results of our paper:

Do Adversarially Robust ImageNet Models Transfer Better?
Hadi Salman*, Andrew Ilyas*, Logan Engstrom, Ashish Kapoor, Aleksander Madry
Paper: https://arxiv.org/abs/2007.08489
Blog post: https://www.microsoft.com/en-us/research/blog/adversarial-robustness-as-a-prior-for-better-transfer-learning/

    @InProceedings{salman2020adversarially,
        title={Do Adversarially Robust ImageNet Models Transfer Better?},
        author={Hadi Salman and Andrew Ilyas and Logan Engstrom and Ashish Kapoor and Aleksander Madry},
        year={2020},
        booktitle={ArXiv preprint arXiv:2007.08489}
    }

Getting started

Our code relies on the MadryLab public robustness library, which will be automatically installed when you follow the instructions below.

Clone our repo: git clone https://github.com/microsoft/robust-models-transfer.git

Install dependencies:

conda create -n robust-transfer python=3.7
conda activate robust-transfer
pip install -r requirements.txt

Running transfer learning experiments

The entry point of our code is main.py (see the file for a full description of arguments).

1- Download one of the pretrained robust ImageNet models, say an L2-robust ResNet-18 with ε = 3. For a full list of models, see the section below!

mkdir pretrained-models & 
wget -O pretrained-models/resnet-18-l2-eps3.ckpt "https://huggingface.co/madrylab/robust-imagenet-models/resolve/main/resnet18_l2_eps3.ckpt"

2- Run the following script (best parameters for each dataset and architecture can be found here)

python src/main.py --arch resnet18 \
  --dataset cifar10 \
  --data /tmp \
  --out-dir outdir \
  --exp-name cifar10-transfer-demo \
  --epochs 150 \
  --lr 0.01 \
  --step-lr 30 \
  --batch-size 64 \
  --weight-decay 5e-4 \
  --adv-train 0 \
  --model-path pretrained-models/resnet-18-l2-eps3.ckpt
  --freeze-level -1

--freeze-level -> -1: full-network transfer | 4: fixed-feature transfer

3- That's it!

Datasets that we use (see our paper for citations)

aircraft (Download)
birds (Download)
caltech101 (Download)
caltech256 (Download)
cifar10 (Automatically downloaded when you run the code)
cifar100 (Automatically downloaded when you run the code)
dtd (Download)
flowers (Download)
food (Download)
pets (Download)
stanford_cars (Download)
SUN397 (Download)

To use any of these datasets in the code:

Download (click or use wget) and extract the desired dataset somewhere, e.g.
```
tar -xvf pets.tar
mkdir /tmp/datasets
mv pets /tmp/datasets
```

Add the dataset name and path as arguments , e.g.

python src/main.py --arch resnet18  ... --dataset pets --data /tmp/datasets/pets

Architectures

You can choose an architecture to use by simply passing it as arguments to the code e.g.

python src/main.py --arch resnet50 ...

The set of possible architectures is:

archs = [resnet18, 
         resnet50, 
         wide_resnet50_2, 
         wide_resnet50_4, 
         densenet,
         mnasnet,
         mobilenet,
         resnext50_32x4d,
         shufflenet,
         vgg16_bn
         ]

Download our robust ImageNet models

If you find our pretrained models useful, please consider citing our work.

Standard Accuracy of L2-Robust ImageNet Models

Model	ε=0	ε=0.01	ε=0.03	ε=0.05	ε=0.1	ε=0.25	ε=0.5	ε=1.0	ε=3.0	ε=5.0
ResNet-18	69.79	69.90	69.24	69.15	68.77	67.43	65.49	62.32	53.12	45.59
ResNet-50	75.80	75.68	75.76	75.59	74.78	74.14	73.16	70.43	62.83	56.13
Wide-ResNet-50-2	76.97	77.25	77.26	77.17	76.74	76.21	75.11	73.41	66.90	60.94
Wide-ResNet-50-4	77.91	78.02	77.87	77.77	77.64	77.10	76.52	75.51	69.67	65.20

Model	ε=0	ε=3
DenseNet	77.37	66.98
MNASNET	60.97	41.83
MobileNet-v2	65.26	50.40
ResNeXt50_32x4d	77.38	66.25
ShuffleNet	64.25	43.32
VGG16_bn	73.66	57.19

Standard Accuracy of Linf-Robust ImageNet Models

Model	ε=0.5/255	ε=1/255	ε=2/255	ε=4/255	ε=8/255
ResNet-18	66.13	63.46	59.63	52.49	42.11
ResNet-50	73.73	72.05	69.10	63.86	54.53
Wide-ResNet-50-2	75.82	74.65	72.35	68.41	60.82

We are hosting these models on HuggingFace too, check them out!

Maintainers

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

microsoft / robust-models-transfer