Code for the paper "On the Impact of Hard Adversarial Instances on Overfitting in Adversarial Training"
python >= 3.7
pytorch >= 1.3
torchvision >= 0.4
We rewrite the data loader for the dataset SVHN and CIFAR10 under the folder dataset
.
This will enable us to use a subset of the training set (Section 4.1) or to use additional data for training (Section 6.3).
util/attack.py
and util/data_parser.py
define different attack algorithms and data loaders used in this paper.
To load extra data, you should download the corresponding data and put it under extradata/cifar10
/extradata/cifar10-c
/extradata/svhn
.
run/train_normal.py
is the main script for training, in which we save the loss value of each instance in each epoch. This information will be used to calculate the instance difficulty.
run/attack_normal.py
is the main script for evaluation under different attacks. Especially, we use the AutoAttack library (Github Link), the state-of-the-art attacker, for evaluation.
run/train_fast.py
is the main script for fast adversarial training with adaptive use of easy and hard instances.
run/tune_normal.py
is the main script for finetuning a pretrained model with additional data.
- Adversarially train a RN18 model against PGD on CIFAR10.
python run/train_normal.py --valid_ratio 0.02 --epoch_num 100 --model_type resnet --out_folder OUTPUT --model_name 100epoch_resnet --optim name=sgd,lr=1e-1,momentum=0.9,weight_decay=5e-4 --attack name=pgd,step_size=2,threshold=8,iter_num=10,order=-1 --gpu $GPU_ID$ --lr_schedule name=jump,min_jump_pt=50,jump_freq=25,start_v=0.1,power=0.1
- Obtain the difficulty function of PGD adversarial training on a RN18 model.
python analyze/sort_instance.py --valid_ratio 0.02 --min_epoch 10 --json_file OUTPUT/100epoch_resnet.json --metric loss --out_file OUTPUT/100epoch_resnet_sortby_loss.json
- PGD adversairl training using the easiest/hardest 10000 training instances.
python run/train_normal.py --valid_ratio 0.02 --per_file OUTPUT/100epoch_resnet_sortby_loss.json --subset num=10000,mode=easy --epoch_num 200 --out_folder OUTPUT --model_name 200epoch_resnet_easy10k --optim name=sgd,lr=1e-1,momentum=0.9,weight_decay=5e-4 --lr_schedule name=jump,min_jump_pt=100,jump_freq=50,start_v=0.1,power=0.1 --attack name=pgd,step_size=2,threshold=8,iter_num=10,order=-1 --gpu $GPU_ID$
python run/train_normal.py --valid_ratio 0.02 --per_file OUTPUT/100epoch_resnet_sortby_loss.json --subset num=10000,mode=hard --epoch_num 200 --out_folder OUTPUT --model_name 200epoch_resnet_hard10k --optim name=sgd,lr=1e-1,momentum=0.9,weight_decay=5e-4 --lr_schedule name=jump,min_jump_pt=100,jump_freq=50,start_v=0.1,power=0.1 --attack name=pgd,step_size=2,threshold=8,iter_num=10,order=-1 --gpu $GPU_ID$
- Using ATTA-based fast adversarial training to train a CIFAR10 model using adaptive target or reweighting.
python run/train_fast.py --aug_policy crop,vflip --epoch_num 38 --model_type wrn --out_folder OUTPUT/adv_sat_fast --model_name rho0.9_beta0.1 --optim name=sgd,lr=1e-1,momentum=0.9,weight_decay=5e-4 --lr_schedule name=jump,min_jump_pt=30,jump_freq=6,start_v=0.1,power=0.1 --threshold 8 --step_size 4 --delta_reset 1000 --rho 0.9 --beta 0.1 --warmup 5 --test_attack name=pgd,step_size=2,threshold=8,iter_num=10,order=-1 --gpu $GPU_ID$ --loss_param name=ce
python run/train_fast.py --aug_policy crop,vflip --epoch_num 38 --model_type wrn --out_folder OUTPUT/adv_sat_fast --model_name rho0.9_beta1.0_warmup5_ce_nat-prob_rw --optim name=sgd,lr=1e-1,momentum=0.9,weight_decay=5e-4 --lr_schedule name=jump,min_jump_pt=30,jump_freq=6,start_v=0.1,power=0.1 --threshold 8 --step_size 4 --delta_reset 1000 --rho 0.9 --beta 1.0 --warmup 5 --warmup_rw 5 --test_attack name=pgd,step_size=2,threshold=8,iter_num=10,order=-1 --gpu $GPU_ID$ --loss_param name=ce --batch_size 128 --reweight nat_prob
- Finetune a pretrained CIFAR10 model using re-weighting and KL-regularization for 1 epoch. To disable re-weighting, replace the last hyper-parameter with
gamma=6,ada_weight=0
. To disable KL-regularization, replace the last hyper-parameter withgamma=0
.
python run/tune_normal.py --valid_ratio 0.02 --model_type wrn --valid_freq 500 --plus_prop 0.5 --batch_size 128 --epoch_num 1 --model2load PRETRAINED.ckpt --out_folder OUTPUT --model_name kl_reweighed_1epoch --optim name=sgd,lr=1e-3,momentum=0.9,weight_decay=5e-4 --lr_schedule name=const,start_v=0.001 --attack name=pgd,step_size=2,threshold=8,iter_num=10,order=-1 --gpu $GPU_ID$ --tune_mode kl --tune_params gamma=6
- Evaluate the robustness of a model by different attacks.
python run/attack_normal.py --dataset cifar10 --model_type wrn --model2load MODEL.ckpt --out_file output_apgd100-ce.json --gpu $GPU_ID$ --attack name=apgd,threshold=8,iter_num=100,order=-1,rho=0.75,loss_type=ce
python run/attack_normal.py --dataset cifar10 --model_type wrn --model2load MODEL.ckpt --out_file output_apgd100-dlr.json --gpu $GPU_ID$ --attack name=apgd,threshold=8,iter_num=100,order=-1,rho=0.75,loss_type=dlr
python run/attack_normal.py --dataset cifar10 --model_type wrn --model2load MODEL.ckpt --out_file output_square5000.json --gpu $GPU_ID$ --attack name=square,threshold=8,iter_num=5000,order=-1,window_size_factor=0
@article{liu2021impact,
title={On the impact of hard adversarial instances on overfitting in adversarial training},
author={Liu, Chen and Huang, Zhichao and Salzmann, Mathieu and Zhang, Tong and S{\"u}sstrunk, Sabine},
journal={arXiv preprint arXiv:2112.07324},
year={2021}
}