Denoising Diffusion Probabilistic Models (WIP)
PyTorch implementation of "Denoising Diffusion Probabilistic Models" (DPPM) and DPPM improvements from "Improved Denoising Diffusion Probabilistic Models".
The original paper can be found here.
OpenAI released a (claimed) improvement upon DDPM, which is incorporated in this repo. Their paper can be found here.
My aim is to provide an implementation that incorporates elements from both papers whilst remaining (relatively) simple to understand.
Currently a work-in-progress. Stay tuned!
Installation
TODO: pip install instructions
TODO: build instructions
Usage
DDPM Model
TODO: instructions on using the model as a standalone module
Training
TODO: instructions on using the training script
Generation
TODO: instructions on using sample generation script
Modifications
TODO: note any deviations from the original works
Samples
TODO: add some nice samples from the trained model
Checkpoints
TODO: add trained checkpoints
TODO:
- Forward diffusion
- Linear noise scheduling
- Cosine noise scheduling
- (Other schedules?)
- Tractable computations for loss calculation
- Implement UNet architecture
- Main model structure
- Multi-headed self-attention at certain resolutions
- Improved time embedding injection (Appendix A)
- EMA parameter updates
- Learning variance (Section 3.1)
- Class conditioning
- Training script
- Main script
- Simple loss computation
- Hybrid loss computation
- Auxiliary denoising loss (from D3PM)
- Sample generation
- Misc
- Nice README :)
- Nice docstrings
- Nice Samples
- PyPi library
- Trained Checkpoints
References
Denoising Diffusion Probabilistic Models
Jonathan Ho, Ajay Jain, Pieter Abbeel
@misc{ho2020denoising,
title={Denoising Diffusion Probabilistic Models},
author={Jonathan Ho and Ajay Jain and Pieter Abbeel},
year={2020},
eprint={2006.11239},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
Improved Denoising Diffusion Probabilistic Models
Alex Nichol, Prafulla Dhariwal
@misc{nichol2021improved,
title={Improved Denoising Diffusion Probabilistic Models},
author={Alex Nichol and Prafulla Dhariwal},
year={2021},
eprint={2102.09672},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
Structured Denoising Diffusion Models in Discrete State-Spaces
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, Rianne van den Berg
@misc{austin2021structured,
title={Structured Denoising Diffusion Models in Discrete State-Spaces},
author={Jacob Austin and Daniel D. Johnson and Jonathan Ho and Daniel Tarlow and Rianne van den Berg},
year={2021},
eprint={2107.03006},
archivePrefix={arXiv},
primaryClass={cs.LG}
}