unexpected results from pretrained models (CIFAR-10)
vietvo89 opened this issue · comments
Hi, I got some troubles with pretrained models on CIFAR-10. I cloned your repo and downloaded pretrained models. I then carried out evaluation of some pretrained models on CIFAR-10 testset. I got both expected and unexpected results even though I load the same dataset with the same transformation as well as smoothed+based classifier.
When loading pretrained models as below, I got unexpected results e.g. clean accuracy is more or less than 30%:
path = "../smoothing-adversarial/pretrained_models/cifar10/PGD_10steps/eps_64/cifar10/resnet110/noise_0.12/checkpoint.pth.tar"
or
path = "../smoothing-adversarial/pretrained_models/cifar10/PGD_10steps_multiNoiseSamples/2-multitrain/eps_64/cifar10/resnet110/noise_0.12/checkpoint.pth.tar"
BUT, when loading pretrained models as below, I got expected results e.g. clean accuracy is from 85% to 90%:
path = "../smoothing-adversarial/pretrained_models/cifar10/finetune_cifar_from_imagenetPGD2steps/PGD_10steps_30epochs_multinoise/2-multitrain/eps_64/cifar10/resnet110/noise_0.12/checkpoint.pth.tar"
or
path = "../smoothing-adversarial/pretrained_models/cifar10/finetune_cifar_from_imagenetPGD2steps/PGD_10steps_30epochs_multinoise/8-multitrain/eps_64/cifar10/resnet110/noise_0.12/checkpoint.pth.tar"
or
path = "../smoothing-adversarial/pretrained_models/cifar10/self_training/PGD_10steps/weight_0.1/eps_64/cifar10/resnet110/noise_0.12/checkpoint.pth.tar"
I do not know what's wrong. My code is as below:
sigma = 0.12
N = 1000
batch = 200
dataset = "cifar10"
alpha = 0.001
checkpoint = torch.load(path)
base_classifier = get_architecture(checkpoint["arch"], dataset)
base_classifier.load_state_dict(checkpoint['state_dict'])
base_classifier.eval()
smoothed_classifier = Smooth(base_classifier, get_num_classes(dataset), sigma)
Thanks
