Zeke's repositories
deep-learning-dynamics-paper-list
This is a list of peer-reviewed representative papers on deep learning dynamics (optimization dynamics of neural networks). The success of deep learning attributes to both network architecture and stochastic optimization. Thus, deep learning dynamics play an essentially important role in theoretical foundation of deep learning.
adaptive-inertia-adai
[ICML 2022, Oral] The PyTorch Implementation of Adaptive Inertia Methods. The algorithms are based on our paper: "Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum".
stable-weight-decay-regularization
[NeurIPS 2023] The PyTorch Implementation of Scheduled (Stable) Weight Decay.
artificial-neural-variability-for-deep-learning
[Neural Computation, MIT Press] The PyTorch Implementation of Variable Optimizers/ Neural Variable Risk Minimization proposed in our Neural Computation paper: Artificial Neural Variability for Deep Learning: On overfitting, Noise Memorization, and Catastrophic Forgetting.
Positive-Negative-Momentum
[ICML 2021] The official PyTorch Implementations of Positive-Negative Momentum Optimizers.