pandaupc / BREEDS-Benchmarks

Creating BREEDS Sub-population shift Benchmarks

This repository contains the code and data for our paper:

BREEDS: Benchmarks for Subpopulation Shift
Shibani Santurkar*, Dimitris Tsipras*, Aleksander Madry
Paper: https://arxiv.org/abs/2008.04859

    @InProceedings{santurkar2020breeds,
        title={BREEDS: Benchmarks for Subpopulation Shift},
        author={Shibani Santurkar and Dimitris Tsipras and Aleksander Madry},
        year={2020},
        booktitle={ArXiv preprint arXiv:2008.04859}
    }

Getting started

Our code relies on the MadryLab public robustness library, which will be automatically installed when you follow the instructions below.

1. Clone our repo: git clone https://github.com/MadryLab/BREEDS-Benchmarks.git

2. Install dependencies:

   conda create -n breeds-benchmarks python=3.7 pip
   conda activate breeds-benchmarks
   pip install -r requirements.txt
   conda install pygraphviz
   

3. Download the ImageNet dataset.

That's it! Now you can create new BREEDS datasets, or look closer at the ones we use in our paper using this notebook. You can also find a detailed walkthrough of this code in this documentation. In order to train models using these datasets, you can use simply use existing code from the robustness library.

Maintainers

• Shibani Santurkar
• Dimitris Tsipras
• Aleksander Madry