y0ast / DUE

Code for "On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty".

DUE and SNGP

Deterministic Uncertainty Estimation

This repo contains the official code for On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty.

Spectral Normalized GP

It also contains an implementation of Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness (or SNGP), for easy comparison. We only implement the exact predictive covariance version, which is both simpler and performs better than the momentum based scheme.

Run Example

Make sure the dependencies listed in environment.yml are available and run:

python train_due.py

which will automatically download the dataset (CIFAR10 by default), and start training. There are several command line flags available for changing the hyper-parameters.

A model trained using the defaults is available from here.

A regression example is implemented in toy_regression.ipynb.

If you want to train SNGP, simply add the flag (and adjust the learning rate):

python train_due.py --sngp --learning_rate 0.05

Library

The repository is split into a reusable library and utils only used for the specific training script. You can install the library part using pip:

pip install --upgrade git+https://github.com/y0ast/DUE.git

Alternatively you can just copy over the components you want!

Citation

If you use this repository, please cite:

@article{van2021on,
  title={On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty},
  author={van Amersfoort, Joost and Smith, Lewis and Jesson, Andrew and Key, Oscar and Gal, Yarin},
  journal={arXiv preprint arXiv:2102.11409},
  year={2021}
}

If you use the SNGP model, then please cite the original paper:

@article{liu2020simple,
  title={Simple and principled uncertainty estimation with deterministic deep learning via distance awareness},
  author={Liu, Jeremiah and Lin, Zi and Padhy, Shreyas and Tran, Dustin and Bedrax Weiss, Tania and Lakshminarayanan, Balaji},
  journal={Advances in Neural Information Processing Systems},
  volume={33},
  pages={7498--7512},
  year={2020}
}

Acknowledgements

Thanks to Lewis for the RFF implementation, John for testing DUE, Jishnu for evaluating SNGP, and Jeremiah for checking SNGP.