efficient-deep-learning model-compression neural-network-pruning pruning-at-initialization

Awesome Pruning at Initialization

This repo aims to provide a comprehensive survey of recent papers on Neural Network Pruning at Initialization (PaI). Feel free to add any paper you think is pertinent!

Survey accepted in IJCAI'22 Survey Track:
Code Base

2018

2018-ICLR-Deep rewiring: Training very sparse deep networks
2018-ICML-Stronger generalization bounds for deep nets via a compression approach
2018-Nature Communications-Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science

2019

2019-ICLR-Snip: Single-shot network pruning based on connection sensitivity
2019-ICLR-The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks [Best Paper!] [Code 1] [Code 2]
2019-ICML-Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization
2019-NIPS-Deconstructing lottery tickets: Zeros, signs, and the supermask
2019-NIPS-One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers [Code, Code Report]
2019.7-Sparse networks from scratch: Faster training without losing performance [ICLR'20 rejected version] [Code]

2020

2020-ICLR-Picking Winning Tickets Before Training By Preserving Gradient Flow [Code]
2020-ICLR-Playing the lottery with rewards and multiple languages: Lottery tickets in RL and NLP
2020-ICLR-Drawing early-bird tickets: Towards more efficient training of deep networks (Spotlight)
2020-ICLR-The Early Phase of Neural Network Training
2020-ICLR-A signal propagation perspective for pruning neural networks at initialization
2020-ICLR-Comparing Rewinding and Fine-tuning in Neural Network Pruning [Code] (Oral)
2020-ICLR-Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers
2020-ICLR reject-Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win
2020-CVPR-What's hidden in a randomly weighted neural network?
2020-ICML-Proving the Lottery Ticket Hypothesis: Pruning is All You Need
2020-ICML-Rigging the Lottery: Making All Tickets Winners [Code]
2020-ICML-Linear Mode Connectivity and the Lottery Ticket Hypothesis
2020-ICML-Finding trainable sparse networks through neural tangent transfer
2020-ICML-Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection [Code]
2020-NIPS-Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot
2020-NIPS-Logarithmic Pruning is All You Need
2020-NIPS-Winning the Lottery with Continuous Sparsification
2020-NIPS-Movement Pruning: Adaptive Sparsity by Fine-Tuning
2020-NIPS-The Lottery Ticket Hypothesis for Pre-trained BERT Networks
2020-NIPS-Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficient
2020-Neural Computing and Applications-Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware
2020-NIPS-Train-by-Reconnect: Decoupling Locations of Weights from their Values
2020-NIPS-Pruning neural networks without any data by iteratively conserving synaptic flow
2020.2-Calibrate and Prune: Improving Reliability of Lottery Tickets Through Prediction Calibration
2020.9-The Hardware Lottery
2020.10-The Sooner The Better: Investigating Structure of Early Winning Lottery Tickets

2021

2021-ICLR-Pruning Neural Networks at Initialization: Why Are We Missing the Mark?
2021-ICLR-Long Live the Lottery- The Existence of Winning Tickets in Lifelong Learning
2021-ICLR-Robust Pruning at Initialization
2021-ICLR-Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network [PyTorch Code]
2021-ICLR-Layer-adaptive Sparsity for the Magnitude-based Pruning [Code]
2021-ICLR-Progressive skeletonization: Trimming more fat from a network at initialization
2021-CVPR-The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
2021-ICML-Lottery Ticket Implies Accuracy Degradation, Is It a Desirable Phenomenon?
2021-ICML-A Unified Lottery Ticket Hypothesis for Graph Neural Networks
2021-ICML-Do we actually need dense over-parameterization? in-time over-parameterization in sparse training
2021-ICML-Efficient Lottery Ticket Finding: Less Data is More
2021-NIPS-The Elastic Lottery Ticket Hypothesis [Code]
2021-NIPS-Training Neural Networks with Fixed Sparse Masks [Code]
2021-NIPS-Validating the Lottery Ticket Hypothesis with Inertial Manifold Theory [Code] (Code link seems invalid now...)
2021-NIPS-Sparse Training via Boosting Pruning Plasticity with Neuroregeneration [Code]
2021-NIPS-Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?
2021-ACL-Parameter-Efficient Transfer Learning with Diff Pruning
2021.6-Why is Pruning at Initialization Immune to Reinitializing and Shuffling
2021.7-Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsity

2022

About

[IJCAI'22 Survey] Recent Advances on Neural Network Pruning at Initialization.

efficient-deep-learning model-compression neural-network-pruning pruning-at-initialization

MIT License