Issue with replication yml/training file
rtaori opened this issue · comments
Hi,
I downloaded the models from the provided link and am specifically looking at the imagenet models. Are the resnet50 models in the replication directory similar to the ones from https://arxiv.org/pdf/1902.02918.pdf?
If so, is there a reason why the yml file says the 0 noise model was trained with 0.12 noise:
jobs:
- name: imagenet/resnet50/noise_0.00
sku: G4
sku_count: 1
command:
- python code/train.py imagenet resnet50 ./ --batch 256 --workers 16 --noise 0.12
id: application_1554838020656_3420
If the model was trained with 0.12 noise, could one with 0.00 noise be provided?
Hi @rtaori !
Thanks for checking our repo! You are right, these models are supposed to be similar to those in https://arxiv.org/pdf/1902.02918.pdf. We reran them on our machines just to make sure our comparison's are fair (same cuda/packages versions etc.)
Indeed, the job you pointed to should have --noise 0.0 instead of --noise 0.12, nice catch. Therefore the corresponding trained model is not actually trained with zero noise (whereas it should be). Anyway, we didn't actually use any of the models trained with zero noise in our paper as they are irrelevant for randomized smoothing, and hence they are not needed for replicating the results of our paper.
In any case, note that zero noise trained models are essentially models trained using standard training, and I am currently training the zero noise model for you. Will ping you here when I upload the model and update the repo. You can as well use the model provided by https://github.com/locuslab/smoothing in the meantime, since I am basically using their code to generate the replication results.
Please let me know if you have further questions!
Ah I see, that makes sense. Thanks a lot for the clarification!