강의 주제: TinyML and Efficient Deep Learning Computing
Instructor : Song Han(Associate Professor, MIT EECS)
Fall 2023([schedule] | [youtube]) | Fall 2022([schedule] | [youtube])
-
효율적인 추론 방법 공부
딥러닝 연산에 있어서 효율성을 높일 수 있는 알고리즘을 공부한다.
-
제한된 성능에서의 딥러닝 모델 구성
디바이스의 제약에 맞춘 효율적인 딥러닝 모델을 구성한다.
-
latency, storage, energy
Memory-Related(#parameters, model size, #activations), Computation(MACs, FLOP)
-
Pruning Granularity, Pruning Critertion
Unstructured/Structured pruning(Fine-grained/Pattern-based/Vector-level/Kernel-level/Channel-level)
Pruning Criterion: Magnitude(L1-norm, L2-norm), Sensitivity and Saliency(SNIP), Loss Change(First-Order, Second-Order Taylor Expansion)
Data-Aware Pruning Criterion: Average Percentage of Zero(APoZ), Reconstruction Error, Entropy
-
Automatic Pruning, Lottery Ticket Hypothesis
Finding Pruning Ratio: Reinforcement Learning based, Rule based, Regularization based, Meta-Learning based
Lottery Ticket Hypothesis(Winning Ticket, Iterative Magnitude Pruning, Scaling Limitation)
Pruning at Initialization(Connection Sensitivity, Gradient Flow)
-
System & Hardware Support for Sparsity
EIE(CSC format: relative index, column pointer)
M:N Sparsity
-
Basic Concepts of Quantization
Numeric Data Types: Integer, Fixed-Point, Floating-Point(IEEE FP32/FP16, BF16, NVIDIA FP8), INT4 and FP4
Uniform vs Non-uniform quantization, Symmetric vs Asymmetric quantization
-
Vector Quantization, Linear Quantization
Vector Quantization(VQ): Deep Compression(iterative pruning, retrain codebook, Huffman encoding), Product Quantization(PQ): AND THE BIT GOES DOWN
Linear Quantization: Zero point, Scaling Factor, Quantization Error(clip error, round error), Linear Quantized Matrix Multiplization(FC layer, Conv layer)
-
Weight Quantiztion: Per-Tensor Activation Per-Channel Activation, Group Quantization(Per-Vector, MX), Weight Equalization, Adative Rounding
Activation Quantization: During training(EMA), Calibration(Min-Max, KL-divergence, Mean Squared Error)
Bias Correction, Zero-Shot Quantization(ZeroQ)
-
Quantization-Aware Training, Low bit-width quantization
Fake quantization, Straight-Through Estimator
Binary Quantization(Deterministic, Stochastic, XNOR-Net), Ternary Quantization
-
Neural Architecture Search: basic concepts & manually-designed neural networks
input stem, stage, head
AlexNet, VGGNet, SqueezeNet(global average pooling, fire module, pointwise convolution), ResNet50(bottleneck block, residual learning), ResNeXt(grouped convolution)
MobileNet(depthwise-separable convolution, width/resolution multiplier), MobileNetV2(inverted bottleneck block), ShuffleNet(channel shuffle), SENet(squeeze-and-excitation block), MobileNetV3(redesigning expensive layers, h-swish)
-
Neural Architecture Search: Search Space
Search Space: Macro, Chain-Structured, Cell-based(NASNet), Hierarchical(Auto-DeepLab, NAS-FPN)
design search space: Cumulative Error Distribution, FLOPs distribution, zero-cost proxy
-
Neural Architecture Search: Performance Estimation & Hardware-Aware NAS
Weight Inheritance, HyperNetwork, Weight Sharing(super-network, sub-network)
Performance Estimation Heuristics: Zen-NAS, GradSign
Hardware-Aware NAS(ProxylessNAS, HAT), One-Shot NAS(Once-for-All)
-
Knowledge Distillation(distillation loss, softmax temperature)
What to Match?: intermediate weights, features(attention maps), sparsity pattern, relational information
Distillation Scheme: Offline Distillation, Online Distillation, Self-Distillation
-
Applications: Object Detection, Semantic Segmentation, GAN, NLP
Tiny Neural Network: NetAug
-
Microcontroller, MCUNet(TinyNAS, TinyEngine), automated search space optimization(weight/resolution multiplier), resource-constrained model specialization(Once-for-All)
MCUNetV2: patch-based inference, network redistribution, joint automated search for optimization, MCUNetV2 architecture(VWW dataset inference)
RNNPool, MicroNets(MOPs & latency/energy consumption relationship)
-
memory hierarchy of MCU, data layout(NCHW, NHWC, CHWN)
TinyEngine: Loop Unrolling, Loop Reordering, Loop Tiling, SIMD programming, Im2col, In-place depthwise convolution, appropriate data layout(pointwise, depthwise convolution), Winograd convolution
-
2D CNNs for Video Understanding, 3D CNNs for Video Understanding(I3D), Temporal Shift Module(TSM)
Other Efficient Methods: Kernel Decomposition, Multi-Scale Modeling, Neural Architecture Search(X3D), Skipping Redundant Frames/Clips, Utilizing Spatial Redundancy
-
Generative Adversarial Networks (GANs)
GANs(Generator, Discriminator), Conditional/Unconditional GANs, Difficulties in GANs
Compress Generator(GAN Compression), Dynamic Cost GANs(Anycost GANs), Data-Efficient GANs(Differentiable Augmenatation)
-
NLP Task(Discriminative, Generative), Pre-Transformer Era(RNN, LSTM, CNN)
Transformer: Tokenizer, Embedding, Multi-Head Attention, Feed-Forward Network, Layer Normalization(Pre-Norm, Post-Norm), Positional Encoding
-
Encoder-Decoder(T5), Encoder-only(BERT), Decoder-only(GPT), Relative Positional Encoding, KV cache optimization, Gated Linear Unit
[ slides ]
(Part I)
(Part II)
(Part III)
Design for Microcontrollers
Lecture 12: Paper Reading Presentation
Lecture 24: Final Project Presentation
Lecture 25: Final Project Presentation