This repository is the official implementation of the NeurIPS 2020 paper PEP: Parameter Ensembling by Perturbation.
To install requirements:
pip install -r requirements.txt
-
Change the project root in the
helpers/settings.py
file to the directory that you want to run experiments. -
Run
settings.py
file to create the folder structure for this project:cd helpers python settings.py
which should create the following folder structure:
├── intermediate │ ├── data │ │ ├── arrays │ │ └── sheets │ └── models └── raw
-
Download the preprocessed ImageNet data (center cropped) that we used in this project from the following link: Download Link
-
copy the folders imagenet_224 and imagenet_299 to the
[project_root]/intermediate/data/arrays/
folder. These are numpy arrays containing 50,000 validation images from ILSVRC2012. They are cropped and resized to sizes of 224 and 299 which are the sizes to our the ImageNet pre-trained networks that we are going to use.
Optionally, instead of running steps 3 and 4 you can create these arrays
by runnin the code in preprocessing
folder.
First, you have to download the ILSVRC2012_devkit_t12
form ImageNet website, put the JPEG images in [project_root]/raw/imagenet_validation/
and then run the following codes for pre-processing:
cd preprocessing
python 1_resize_crop_images.py
python 2_preprocess_y_val.py
Running preporcessing code requires OpenCV
installation:
pip install opencv
Installing opencv on latest versions of conda can be tricky, so we recommend to download the arrays (step 3 and 4 above) if you could not install OpenCV.
To reproduce the effect of PEP on pre-trained inception-V3 (Figure 1 in
the manuscript), first run 1_perturb_inception.py
inside ImageNet/1_figure1/
cd Imagenet/1_figure1
python 1_pertrub_inception.py
Then open run jupyter notebook and run the cells inside 2_draw_chart.ipynb
.
The result is the following figure.
For each of the pretrained networks run the code inside ImageNet/2_optimal_sigma
folder:
cd ImageNet/2_optimal_sigma
python densenet121.py
python densenet169.py
python inceptionv3.py
python resnet50.py
python vgg16.py
python vgg19.py
For each of the pretrained networks run the code inside ImageNet/3_PEP
folder:
cd ImageNet/3_PEP
python densenet121.py
python densenet169.py
python inceptionv3.py
python resnet50.py
python vgg16.py
python vgg19.py
To find the optimal temperature for pre-trained models
run ts.py
inside ImageNet/4_temp_scaling\
cd Imagenet/4_temp_scaling
python ts.py
To compare baselines with PEP, and temperature scaling (Table 1 of the manuscript)
run the notebooks inside ImageNet/5_analysis
. The notebooks also
draw reliability diagrams.
To reproduce the results of Table 2
run the notebooks inside MNIST_CIFAR10/1_Table2
.
To reproduce the realationship between overfitting and PPE, first run the following scripts:
cd MNIST_CIFAR10/2_overfitting
python overfitting_mnist_mlp.py
python overfitting_mnist_cnn.py
python overfitting_cifar_cnn.py
Then open run jupyter notebook and run the cells inside overfitting figures.ipynb
.