Awesome ML Model Compression 
An awesome style list that curates the best machine learning model compression and acceleration research papers, articles, tutorials, libraries, tools and more. PRs are welcome!
Contents
Papers
General
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- Model compression as constrained optimization, with application to neural nets. Part I: general framework
- Model compression as constrained optimization, with application to neural nets. Part II: quantization
Architecture
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- MobileNetV2: Inverted Residuals and Linear Bottlenecks: Mobile Networks for Classification, Detection and Segmentation
- Xception: Deep Learning with Depthwise Separable Convolutions
- ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
- SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
- Fast YOLO: A Fast You Only Look Once System for Real-time Embedded Object Detection in Video
- AddressNet: Shift-based Primitives for Efficient Convolutional Neural Networks
- ResNeXt: Aggregated Residual Transformations for Deep Neural Networks
- ResBinNet: Residual Binary Neural Network
- Residual Attention Network for Image Classification
- Squeezedet: Unified, small, low power fully convolutional neural networks
- SEP-Nets: Small and Effective Pattern Networks
- Dynamic Capacity Networks
- Learning Infinite Layer Networks Without the Kernel Trick
- Efficient Sparse-Winograd Convolutional Neural Networks
- DSD: Dense-Sparse-Dense Training for Deep Neural Networks
- Coordinating Filters for Faster Deep Neural Networks
- Deep Networks with Stochastic Depth
Quantization
- Quantized Convolutional Neural Networks for Mobile Devices
- Towards the Limit of Network Quantization
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- Compressing Deep Convolutional Networks using Vector Quantization
- Trained Ternary Quantization
- The ZipML Framework for Training Models with End-to-End Low Precision: The Cans, the Cannots, and a Little Bit of Deep Learning
- ShiftCNN: Generalized Low-Precision Architecture for Inference of Convolutional Neural Networks
- Deep Learning with Low Precision by Half-wave Gaussian Quantization
- Loss-aware Binarization of Deep Networks
- Quantize weights and activations in Recurrent Neural Networks
- Fixed-Point Performance Analysis of Recurrent Neural Networks
Binarization
- Binarized Convolutional Neural Networks with Separable Filters for Efficient Hardware Acceleration
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- Local Binary Convolutional Neural Networks
- XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
Pruning
- Faster CNNs with Direct Sparse Convolutions and Guided Pruning
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- Pruning Convolutional Neural Networks for Resource Efficient Inference
- Pruning Filters for Efficient ConvNets
- Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning
- Learning to Prune: Exploring the Frontier of Fast and Accurate Parsing
- Fine-Pruning: Joint Fine-Tuning and Compression of a Convolutional Network with Bayesian Optimization
- Learning both Weights and Connections for Efficient Neural Networks
- ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
- Data-Driven Sparse Structure Selection for Deep Neural Networks
- Soft Weight-Sharing for Neural Network Compression
- Dynamic Network Surgery for Efficient DNNs
- Channel pruning for accelerating very deep neural networks
- AMC: AutoML for model compression and acceleration on mobile devices
- ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGA
Distillation
- Distilling the Knowledge in a Neural Network
- Deep Model Compression: Distilling Knowledge from Noisy Teachers
- Learning Efficient Object Detection Models with Knowledge Distillation
- Data-Free Knowledge Distillation For Deep Neural Networks
- Knowledge Projection for Effective Design of Thinner and Faster Deep Neural Networks
- Moonshine: Distilling with Cheap Convolutions
- Model Distillation with Knowledge Transfer from Face Classification to Alignment and Verification
- Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
- Sequence-Level Knowledge Distillation
- Learning Loss for Knowledge Distillation with Conditional Adversarial Networks
- Dark knowledge
- DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer
- FitNets: Hints for Thin Deep Nets
- MobileID: Face Model Compression by Distilling Knowledge from Neurons
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
Low Rank Approximation
- Speeding up convolutional neural networks with low rank expansions
- Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
- Convolutional neural networks with low-rank regularization
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Accelerating Very Deep Convolutional Networks for Classification and Detection
- Efficient and Accurate Approximations of Nonlinear Convolutional Networks
Articles
Content published on the Web.
Howtos
Assorted
- Why the Future of Machine Learning is Tiny
- Deep Learning Model Compression for Image Analysis: Methods and Architectures
Reference
Blogs
Tools
Libraries
- TensorFlow Model Optimization Toolkit. Accompanied blog post, TensorFlow Model Optimization Toolkit — Pruning API
Frameworks
Videos
Talks
Training & tutorials
License
To the extent possible under law, Cedric Chee has waived all copyright and related or neighboring rights to this work.