brain-bzh/efficient-deep-learning

Repository for the course "Efficient Deep Learning" at IMT Atlantique

Course organisation / Syllabus

Here is a detailed schedule, session by session:

Introduction / Refresher on Deep Learning
1. General Intro - Why do we need to optimize deep learning ? Introduction of the MicroNet Challenge.
2. Course - Deep Learning and Transfer Learning.
3. Practical session - introduction to PyTorch
4. Short project - exploring hyper parameters on a fixed architecture
5. Presentation of the Challenge Objectives and Rules (Long Project)
Data Augmentation, Regularization
1. Short evaluation on Deep Learning Essentials and CNN layers
2. Course - Data Augmentation, Regularization and Self Supervised Learning
3. Practical Session - Data Augmentation
4. Short project - exploring hyper parameters on a fixed architecture
Quantization
1. Short evaluation on Data Augmentation
2. Students presentations of short project - exploring hyper parameters on a fixed architecture
3. Course - Quantifying Deep neural networks
4. Practical session - quantification on a small convolutional network
5. Long project 1 - Challenge
Pruning
1. Short evaluation on Quantization
2. Course - Pruning Deep neural networks
3. Practical session - pruning on a small convolutional network.
4. Long project 2 - Challenge
Factorization
1. Short evaluation on Pruning
2. Students presentations on current work on Challenge
3. Course - Factorizing Deep neural networks
4. Long Project 3 - Challenge
Distillation
1. Short evaluation on Factorization
2. Course - Distillation of knowledge and features between neural networks
3. Long Project 4 - Challenge
Embedded Software and Hardware for Deep Learning
1. Long evaluation on Distillation and previous courses
2. Course - Embedded Software and Hardware for Deep Learning
3. Long Project 5 - Challenge
Final Session
1. Students presentations - Challenge final results
2. Students feedback

What is expected for the Long Project

Short version : Exploration of the accuracy / complexity tradeoff

Long version : this course is mostly based on the long project, and you have a lot of freedom, which we expect you to use. The overarching goal is to explore the tradeoff between the performances of neural networks (= Accuracy on the test set) and complexity. Complexity can be either computational complexity (number of arithmetic operations), or memory complexity (memory size necessary to use the network).

We encourage students to get creative and test combinations of the various ideas that we present. Starting from the end of Session 1, you already have enough knowledge to explore the tradeoff between architecture, number of parameters, and accuracy. Then, we study new notions that open new avenues to explore this tradeoff : quantization, pruning, factorization, distillation. In session 7, you'll have a deeper insight on how to thing about specific software or hardware architecture in order to fully exploit all the optimizations that can be done.

Evaluation in this course

There are short written evaluations during the first 10 minutes of each session starting from session 2. Don't be late!

For the final session, we ask you to prepare a 15 minutes presentation, that will be followed by 5 Minutes of question.

What we expect for the presentations :

Explain your strategy to explore the complexity / accuracy tradeoff. We will judge whether you understood the goal, and whether the proposed strategy follows a rigourous approach.
The clarity of your exposition and quality of your support (slides)