PGD/BIM implementation is incorrect
carlini opened this issue · comments
The PGD (and BIM) implementation in this repository is significantly less effective than as reported in prior work. In Table XIV PGD (or BIM) appears to succeed 82.4% (or 75.6%) of the time. When I run the code in the repository, I get a very similar result: 82.5% (or 74.2%).
This should be somewhat surprising given that prior work reports PGD and BIM succeeds nearly 100% of the time with the same distortion bound of 0.3. See for example Figure 4 of Madry et al. (2018), or Table IV of Carlini & Wagner (2017). Indeed, when I put a loop around my FGSM call (using the approach discussed in #3) I reach 100% attack success rate with both BIM and PGD.
I have not investigated the cause of this discrepancy further.
It is deeply concerning that now I have checked five results (FGSM/PGD/BIM/JSMA/PAT) and all of them have issues (#3 / this issue / #14 / #4). Did you cross-check the results of your attacks with any other libraries?
Fixed in d4e1181 in defining the model for both MNIST and CIFAR10, though it is suggested by PyTorch officially (https://github.com/pytorch/examples/blob/master/mnist/main.py).
Nothing needs to be changed in our implementation of BIM and PGD.
After retraining the model for MNIST and attacking, the misclassification rates of BIM and PGD at eps=0.3 on MNIST are both 100%.