This repository contains the datasets and some code for the paper Benchmarking Neural Network Robustness to Common Corruptions and Perturbations (ICLR 2019) by Dan Hendrycks and Thomas Dietterich.
Requires Python 3+ and PyTorch 0.3+. For evaluation, please download the data from the links below.
Download ImageNet-C here. (Mirror.)
Download Tiny ImageNet-C here. (Mirror.)
Tiny ImageNet-C has 200 classes with images of size 64x64, while ImageNet-C has all 1000 classes where each image is the standard size. For even quicker experimentation, there is CIFAR-10-C, but improvements on CIFAR-10-C may be much less indicative of ImageNet-C improvements. Evaluation using the JPEGs above is strongly prefered to computing the corruptions in memory, so that evaluation is deterministic and consistent.
ImageNet-C Robustness with a ResNet-50 Backbone
| Method | Reference | mCE |
|---|---|---|
| Stylized ImageNet Data Augmentation | Geirhos et al. (ICLR 2019) | 69.3% |
| ResNet-50 Baseline | 76.7% |
Other backbones can obtain better results. For example, a vanilla ResNeXt-101 has an mCE of 62.2%.
Submit a pull request if you beat the state-of-the-art on ImageNet-C.
ImageNet-P sequences are MP4s not GIFs. The spatter perturbation sequence is a validation sequence.
Download Tiny ImageNet-P here. (Mirror.)
ImageNet-P Perturbation Robustness with a ResNet-50 Backbone
| Method | Reference | mFR | mT5D |
|---|---|---|---|
| Low Pass Filter Pooling (bin-5) | Zhang (ICML 2019) | 51.2% | 71.9% |
| ResNet-50 Baseline | 58.0% | 78.4% |
Submit a pull request if you beat the state-of-the-art on ImageNet-P.
If you find this useful in your research, please consider citing:
@article{hendrycks2019robustness,
title={Benchmarking Neural Network Robustness to Common Corruptions and Perturbations},
author={Dan Hendrycks and Thomas Dietterich},
journal={Proceedings of the International Conference on Learning Representations},
year={2019}
}
Part of the code was contributed by Tom Brown.
