Hadisalman / smoothing-adversarial

Code for our NeurIPS 2019 *spotlight* "Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers"

Home Page:https://arxiv.org/abs/1906.04584

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

unexpected results from pretrained models (CIFAR-10)

vietvo89 opened this issue · comments

Hi, I got some troubles with pretrained models on CIFAR-10. I cloned your repo and downloaded pretrained models. I then carried out evaluation of some pretrained models on CIFAR-10 testset. I got both expected and unexpected results even though I load the same dataset with the same transformation as well as smoothed+based classifier.

When loading pretrained models as below, I got unexpected results e.g. clean accuracy is more or less than 30%:

path = "../smoothing-adversarial/pretrained_models/cifar10/PGD_10steps/eps_64/cifar10/resnet110/noise_0.12/checkpoint.pth.tar"

or

path = "../smoothing-adversarial/pretrained_models/cifar10/PGD_10steps_multiNoiseSamples/2-multitrain/eps_64/cifar10/resnet110/noise_0.12/checkpoint.pth.tar"

BUT, when loading pretrained models as below, I got expected results e.g. clean accuracy is from 85% to 90%:

path = "../smoothing-adversarial/pretrained_models/cifar10/finetune_cifar_from_imagenetPGD2steps/PGD_10steps_30epochs_multinoise/2-multitrain/eps_64/cifar10/resnet110/noise_0.12/checkpoint.pth.tar"

or

path = "../smoothing-adversarial/pretrained_models/cifar10/finetune_cifar_from_imagenetPGD2steps/PGD_10steps_30epochs_multinoise/8-multitrain/eps_64/cifar10/resnet110/noise_0.12/checkpoint.pth.tar"

or

path = "../smoothing-adversarial/pretrained_models/cifar10/self_training/PGD_10steps/weight_0.1/eps_64/cifar10/resnet110/noise_0.12/checkpoint.pth.tar"

I do not know what's wrong. My code is as below:

sigma = 0.12
N = 1000
batch = 200
dataset = "cifar10"
alpha = 0.001

checkpoint = torch.load(path)
base_classifier = get_architecture(checkpoint["arch"], dataset)
base_classifier.load_state_dict(checkpoint['state_dict'])
base_classifier.eval()
smoothed_classifier = Smooth(base_classifier, get_num_classes(dataset), sigma)

Thanks

@vietvo89 how did you solve this?

nvm found it #3