michaelaerni / adversarial_spatial

Investigating the robustness of state-of-the-art CNN architectures to simple spatial transformations.

Adversarial rotations and translations for CIFAR10

This repository contains code to train and evaluate CIFAR10 models against adversarially chosen rotations and translations (code for ImageNet at https://github.com/MadryLab/spatial-pytorch). It can be used to reproduce the main experiments of:

Exploring the Landscape of Spatial Robustness
Logan Engstrom*, Brandon Tran*, Dimitris Tsipras*, Ludwig Schmidt, Aleksander Mądry
ICML 2019
http://arxiv.org/abs/1712.02779

The main scipts to run are train.py and eval.py, which will train and evaluate a model respectively. Options are all included in config.json annotated below.

{
  "model": {
      "output_dir": "output/test",
      # padding mode, passed directly to tf.pad
      "pad_mode": "constant", 
      "filters": [16, 16, 32, 64],
      # size of image fed to classifier,set to 64 for black-canvas setting (no
      # information loss during rotation and translation)
      "pad_size": 32
  },

  "training": {
      "tf_random_seed": 557212,
      "np_random_seed": 993101,
      "max_num_training_steps": 80000,
      "num_output_steps": 100,
      "num_summary_steps": 100,
      "num_eval_steps": 500,
      "num_checkpoint_steps": 500,
      "batch_size": 128,
      "step_size_schedule": [[0, 0.1], [40000, 0.01], [60000, 0.001]],
      "momentum": 0.9,
      "weight_decay": 0.0002,
      # interleaves evaluation steps during training, useful for single GPU runs
      "eval_during_training": true,
      # include Linf and spatial attacks during training
      "adversarial_training": false,
      # use random left-right flip (see note below)
      "data_augmentation": true
  },

  "eval": {
      "num_eval_examples": 10000,
      "batch_size": 128,
      # useful for quickly computing standard accuracy if set to false
      "adversarial_eval": true
  },

  "attack": {
      # perform Linf-bounded PGD attack
      "use_linf": false,
      # perform adversarial rotations and translations
      "use_spatial": true,

      # parameters for PGD attacks
      "loss_function": "xent", # can also be set to "cw" for Carlini-Wagner
      "epsilon": 8.0,
      "num_steps": 5,
      "step_size": 2.0,
      "random_start": false,

      # parameters for spatial attack
      # can either be chosen using a few random tries or exhaustive grid search
      "spatial_method": "random", # or "grid"
      "spatial_limits": [3, 3, 30], # trans_x pix, trans_y pix, rotation degrees
      "random_tries": 10, # if method is random choose the worst of x tries
      "grid_granularity": [5, 5, 31] # controls how many points are in the grid
  },

  "data": { "data_path": "/scratch/datasets/cifar10" }
}

Data augmentation only included random left-right flips. Standard CIFAR10 augmentation (+-2 pixel crops) can be achieved by setting adversarial_training: true, spatial_method: random, random_tries: 1, spatial_limits: [2, 2, 0].