OOD_milestone_implement

This is Pytorch implementation of the OOD experiments with pytorch-template in the following milestone papers:

[MSP] A Baseline for Detecting Missclassified and Out-of-Distribution Examples in Neural Networks | [ICLR2017] [paper]
[ODIN] Enhancing The Reliability of Out-of-Distribution Image Detection in Neural Networks | [ICLR2018] [paper]
[Mahalanobis] A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks | [NIPS2018] [paper]
[OE] Deep Anomaly Detection with Outlier Exposure | [ICLR2019] [paper]
[Energy] Energy-based Out-of-distribution Detection | [NIPS2020] [paper]

Installation & requirement

The current version of the code has been tested with python 3.6.9 on an Ubuntu 18.04 OS with the following versions of Pytorch and Torchvision:

pytorch 1.7.1
torchvision 0.8.2

Further Python-packages used are listed in requirements.txt. Assuming Python and pip are set up, these packages can be installed using:

pip install -r requirements.txt

Folder Structure

OOD_milestone_implement/
│
├── data_loader/
│   └── data_loaders.py - full training dataloader
│   └── in_data_loaders.py - in-distribution eval dataloader
│   └── out_data_loaders.py - out-of-distribution eval dataloader
│   └── svhn_data_loaders.py - svhb dataset data loader
├── datasets/ - put standard or your dataset
├── detector/ - package for evaluation
│   └── detector.py
├── model/ - models, losses, and metrics
│   ├── densenet.py
│   ├── mahalanobis.py - metrics for mahalanobis method
│   ├── metric.py - metrics of confidence score and evaluation
│   └── wideresnet.py
├── runs/ - tensorboard log folder, it will be updated
├── trainer/ - trainers
│   └── trainer.py/ - full training src
├── utils/ - small utility functions
│   ├── mahalanobis_lib.py - function for mahalanobis
│   └── select_svhn_data.py - function for select svhn data file
├── eval_ood_detection.py - **main script to OOD evaluation**
├── generate_mahalanobis_hyperparam.py - Get mahalanobis's hyper parameters
├── license
├── requirements.txt
└── train.py - **main script to start training**

Running custom experiments

The main options of this script are:

--in_dataset: choose in distribution dataset (CIFAR-10|CIFAR-100|SVHN)
--model_arch: choose model architecture (densenet|wideresnet)
--method: choose confidence scoring method (msp|odin|mahalanobis)

How to run?

If you want to full-train

python3 train.py --in_dataset CIFAR-10 --model_arch densenet --epochs 100 --name experiment_1 --tensorboard

If you want to evaluate OOD score

python3 eval_ood_detection.py --in_dataset CIFAR-10 --method msp --name eval_msp

Result sample

Natural OOD
nat_in vs. nat_out
in_distribution: CIFAR-10
out_distribution: All
Model Name: test_odin

 OOD detection method: odin
 FPR    DTERR  AUROC  AUIN   AUOUT
 17.20   9.00  95.51  96.12  94.30

License

This project is licensed under the Apache-2.0 License. See LICENSE for more details

Reference

Project structure: https://github.com/jfc43/informative-outlier-mining
Odin: https://github.com/facebookresearch/odin
Mahalanobis: https://github.com/pokaxpoka/deep_Mahalanobis_detector

yunseokddi / OOD_milestone_implement