EverestRs / TinyML_and_Efficient_DLC

TinyML and Efficient Deep Learning Computing | MIT 6.S965 Fall 2022

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

TinyML and Efficient Deep Learning Computing

강의 주제: TinyML and Efficient Deep Learning Computing
Instructor : Song Han(Associate Professor, MIT EECS)
Fall 2023([schedule] | [youtube]) | Fall 2022([schedule] | [youtube])

💡 목표

  • 효율적인 추론 방법 공부

    딥러닝 연산에 있어서 효율성을 높일 수 있는 알고리즘을 공부한다.

  • 제한된 성능에서의 딥러닝 모델 구성

    디바이스의 제약에 맞춘 효율적인 딥러닝 모델을 구성한다.


🚩 정리한 문서 목록

📖 Basics of Deep Learning

  • Efficiency Metrics

    latency, storage, energy

    Memory-Related(#parameters, model size, #activations), Computation(MACs, FLOP)

📔 Efficient Inference

  • Pruning Granularity, Pruning Critertion

    Unstructured/Structured pruning(Fine-grained/Pattern-based/Vector-level/Kernel-level/Channel-level)

    Pruning Criterion: Magnitude(L1-norm, L2-norm), Sensitivity and Saliency(SNIP), Loss Change(First-Order, Second-Order Taylor Expansion)

    Data-Aware Pruning Criterion: Average Percentage of Zero(APoZ), Reconstruction Error, Entropy

  • Automatic Pruning, Lottery Ticket Hypothesis

    Finding Pruning Ratio: Reinforcement Learning based, Rule based, Regularization based, Meta-Learning based

    Lottery Ticket Hypothesis(Winning Ticket, Iterative Magnitude Pruning, Scaling Limitation)

    Pruning at Initialization(Connection Sensitivity, Gradient Flow)

  • System & Hardware Support for Sparsity

    EIE(CSC format: relative index, column pointer)

    M:N Sparsity


  • Basic Concepts of Quantization

    Numeric Data Types: Integer, Fixed-Point, Floating-Point(IEEE FP32/FP16, BF16, NVIDIA FP8), INT4 and FP4

    Uniform vs Non-uniform quantization, Symmetric vs Asymmetric quantization

  • Vector Quantization, Linear Quantization

    Vector Quantization(VQ): Deep Compression(iterative pruning, retrain codebook, Huffman encoding), Product Quantization(PQ): AND THE BIT GOES DOWN

    Linear Quantization: Zero point, Scaling Factor, Quantization Error(clip error, round error), Linear Quantized Matrix Multiplization(FC layer, Conv layer)

  • Post Training Quantization

    Weight Quantiztion: Per-Tensor Activation Per-Channel Activation, Group Quantization(Per-Vector, MX), Weight Equalization, Adative Rounding

    Activation Quantization: During training(EMA), Calibration(Min-Max, KL-divergence, Mean Squared Error)

    Bias Correction, Zero-Shot Quantization(ZeroQ)

  • Quantization-Aware Training, Low bit-width quantization

    Fake quantization, Straight-Through Estimator

    Binary Quantization(Deterministic, Stochastic, XNOR-Net), Ternary Quantization


  • Neural Architecture Search: basic concepts & manually-designed neural networks

    input stem, stage, head

    AlexNet, VGGNet, SqueezeNet(global average pooling, fire module, pointwise convolution), ResNet50(bottleneck block, residual learning), ResNeXt(grouped convolution)

    MobileNet(depthwise-separable convolution, width/resolution multiplier), MobileNetV2(inverted bottleneck block), ShuffleNet(channel shuffle), SENet(squeeze-and-excitation block), MobileNetV3(redesigning expensive layers, h-swish)

  • Neural Architecture Search: Search Space

    Search Space: Macro, Chain-Structured, Cell-based(NASNet), Hierarchical(Auto-DeepLab, NAS-FPN)

    design search space: Cumulative Error Distribution, FLOPs distribution, zero-cost proxy

  • Neural Architecture Search: Performance Estimation & Hardware-Aware NAS

    Weight Inheritance, HyperNetwork, Weight Sharing(super-network, sub-network)

    Performance Estimation Heuristics: Zen-NAS, GradSign

    Hardware-Aware NAS(ProxylessNAS, HAT), One-Shot NAS(Once-for-All)


  • Knowledge Distillation

    Knowledge Distillation(distillation loss, softmax temperature)

    What to Match?: intermediate weights, features(attention maps), sparsity pattern, relational information

    Distillation Scheme: Offline Distillation, Online Distillation, Self-Distillation

  • Distillation for Applications

    Applications: Object Detection, Semantic Segmentation, GAN, NLP

    Tiny Neural Network: NetAug


  • MCUNet

    Microcontroller, MCUNet(TinyNAS, TinyEngine), automated search space optimization(weight/resolution multiplier), resource-constrained model specialization(Once-for-All)

    MCUNetV2: patch-based inference, network redistribution, joint automated search for optimization, MCUNetV2 architecture(VWW dataset inference)

    RNNPool, MicroNets(MOPs & latency/energy consumption relationship)

⚙️ Efficient Training and System Support

  • TinyEngine

    memory hierarchy of MCU, data layout(NCHW, NHWC, CHWN)

    TinyEngine: Loop Unrolling, Loop Reordering, Loop Tiling, SIMD programming, Im2col, In-place depthwise convolution, appropriate data layout(pointwise, depthwise convolution), Winograd convolution


🔧 Application-Specific Optimizations

  • Efficient Video Understanding

    2D CNNs for Video Understanding, 3D CNNs for Video Understanding(I3D), Temporal Shift Module(TSM)

    Other Efficient Methods: Kernel Decomposition, Multi-Scale Modeling, Neural Architecture Search(X3D), Skipping Redundant Frames/Clips, Utilizing Spatial Redundancy

  • Generative Adversarial Networks (GANs)

    GANs(Generator, Discriminator), Conditional/Unconditional GANs, Difficulties in GANs

    Compress Generator(GAN Compression), Dynamic Cost GANs(Anycost GANs), Data-Efficient GANs(Differentiable Augmenatation)


  • Transformer

    NLP Task(Discriminative, Generative), Pre-Transformer Era(RNN, LSTM, CNN)

    Transformer: Tokenizer, Embedding, Multi-Head Attention, Feed-Forward Network, Layer Normalization(Pre-Norm, Post-Norm), Positional Encoding

  • Transformer Design Variants

    Encoder-Decoder(T5), Encoder-only(BERT), Decoder-only(GPT), Relative Positional Encoding, KV cache optimization, Gated Linear Unit


🔍 Schedule (Fall 2022)

Lecture 1: Introduction

[ slides ]

Lecture 2: Basics of Deep Learning

[ slides | video ]


Efficient Inference


Lecture 3: Pruning and Sparsity (Part I)

[ slides | video ]

Lecture 4: Pruning and Sparsity (Part II)

[ slides | video ]

Lecture 5: Quantization (Part I)

[ slides | video ]

Lecture 6: Quantization (Part II)

[ slides | video ]

Lecture 7: Neural Architecture Search

(Part I)

[ slides | video ]

Lecture 8: Neural Architecture Search

(Part II)

[ slides | video ]

Lecture 9: Neural Architecture Search

(Part III)

[ slides | video ]

Lecture 10: Knowledge Distillation

[ slides | video ]

Lecture 11: MCUNet - Tiny Neural Network

Design for Microcontrollers

[ slides | video ]

Lecture 12: Paper Reading Presentation


Efficient Training and System Support


Lecture 13: Distributed Training and Gradient Compression (Part I)

[ slides | video ]

Lecture 14: Distributed Training and Gradient Compression (Part II)

[ slides | video ]

Lecture 15: On-Device Training and Transfer Learning (Part I)

[ slides | video ]

Lecture 16: On-Device Training and Transfer Learning (Part II)

[ slides | video ]

Lecture 17: TinyEngine - Efficient Training and Inference on Microcontrollers

[ slides | video ]


Application-Specific Optimizations


Lecture 18: Efficient Point Cloud Recognition

[ slides | video ]

Lecture 19: Efficient Video Understanding and GANs

[ slides | video ]

Lecture 20: Efficient Transformers

[ slides | video ]


Quantum ML


Lecture 21: Basics of Quantum Computing

[ slides | video ]

Lecture 22: Quantum Machine Learning

[ slides | video ]

Lecture 23: Noise Robust Quantum ML

[ slides | video ]

Lecture 24: Final Project Presentation

Lecture 25: Final Project Presentation

Lecture 26: Course Summary & Guest Lecture

[ slides | video ]

About

TinyML and Efficient Deep Learning Computing | MIT 6.S965 Fall 2022