dolittle007 / Deep-Learning-Experiments

Notes and experiments to understand deep learning concepts

Deep Learning Lecture Notes and Experiments

Code samples have links to other repo that I maintain (Advanced Deep Learning with Keras book) or contribute (Keras)

2022 Version

Welcome to the 2022 version of Deep Learning course. We made major changes in the coverage and delivery of this course to reflect the recent advances in the field.

What is new in 2022 version:

1. Emphasis on tools to use and deploy deep learning models. In the past, we learn how to build and train models to perform certain tasks. However, often times we want to use a pre-trained model for immediate deployment. testing or demonstration. Hence, we will use tools such as huggingface, gradio and streamlit in our discussions.

2. Emphasis on understanding deep learning building blocks. The ability to build, train and test models is important. However, when we want to optimize and deploy a deep learning model on a new hardware or run it on production, we need an in-depth understanding of the code implementation of our algorithms. Hence, there will be emphasis on low-level algorithms and their code implementations.

3. Emphasis on practical applications. Deep learning can do a lot more than recognition. Hence, we will highlight practical applications in vision (detection, segmentation), speech (ASR, TTS) and text (sentiment, summarization).

4. Various levels of abstraction. We will present deep learning concepts from low-level numpy and einops, to mid-level framework such as PyTorch, and to high-level APIs such as huggingface, gradio and streamlit. This enables us to use deep learning principles depending on the problem constraints.

5. Emphasis on individual presentation of assignments, machine exercises and projects. Online learning is hard. To maximize student learning, this course focuses on exchange of ideas to ensure individual student progress.

Coverage:

1. Deep Learning Toolkit - Anaconda, venv, VSCode, Python, Numpy, Einops, PyTorch, Timm, HuggingFace, Gradio, Streamlit, Colab, Deepnote, Kaggle, etc.

• Overview: PDF, YouTube
• Development Environment and Code Editor: PDF, YouTube
• Python: PDF, YouTube

2. Datasets - collection, labelling, loading, splitting, feeding
3. Supervised Learning
4. Building blocks - MLPs, CNNs, RNNs, Transformers
5. Backpropagation, Optimization and Regularization
6. Unsupervised Learning
7. AutoEncoders and Variational AutoEncoders
8. Practical Applications

2020 Version

So much have changed since this course was offerred. Hence, it is time to revise. I will keep the original lecture notes at the bottom. They will no longer be maintained. I am introducing 2020 version. Big changes that will happen are as follows:

1. Review of Machine Learning - Frustrated with the lack of depth in the ML part, I decided to develop a new course - Foundations of Machine Learning. Before studying DL, a good grasp of ML is of paramount importance. Without ML, it is harder to understand DL and to move it forward.

2. Lecture Notes w/ Less Clutter - Prior to this version, my lecture notes have too much text. In the 2020 version, I am trying to focus more on the key concepts while carefully explaining during lecture the idea behind these concepts. The lecture notes are closely coupled with sample implementations. This enables us to quickly move from concepts to actual code implementations.

Lecture Notes and Experiments

0. Course Roadmap

• Deep Learning
• Limits of Deep Learning

1. Multilayer Perceptron (MLP)

• Lecture Notes
• Experiments:
  • Linear Regression
  • MNIST Sampler
  • MNIST Classifier

2. Convolutional Neural Network (CNN)

• Lecture Notes
• Experiments:
  • CNN MNIST Classifier
  • CNN MNIST Classifier using Functional API
  • Y Network using Functional API
• Deep CNN
  • ResNet
  • DenseNet

3. Recurrent Neural Network (RNN)

• Lecture Notes
• Experiments:
  • RNN MNIST Classifier
  • LSTM MNIST Classifier

4. Transformer

• Lecture Notes
• Experiments:
  • Transformer MNIST Classifier

5. Regularizer

• Lecture Notes
• Experiments:
  • MLP on MNIST no Regularizer
  • MLP on MNIST with L2
  • MLP on MNIST with Dropout
  • MLP on MNIST with Data Augmentation

5. Optimizer

• Lecture Notes

6. AutoEncoder

• Lecture Notes
• Experiments:
  • AutoEncoder
  • Denoising AutoEncoder
  • Colorization AutoEncoder

7. Normalization

• Lecture Notes

8. Generative Adversarial Network (GAN)

• GAN
• Improved GAN
• Disentangled GAN
• Cross-Domain GAN
• Experiments: DCGAN , CGAN , WGAN , LSGAN , ACGAN , InfoGAN , CycleGAN

9. Variational AutoEncoder (VAE)

• Lecture Notes
• Experiments:
  • VAE MLP
  • VAE CNN
  • CVAE

10. Object Detection

• Lecture Notes
• Experiments:
  • SSD

11. Object Segmentation

• Lecture Notes
• Experiments:
  • FCN w/ PSPNet

12. Deep Reinforcement Learning (DRL)

• Experiments:
  • Q-Learning
  • Q-Learning on FrozenLake-v0
  • DQN and DDQN on CartPole-v0

13. Policy Gradient Methods

• Experiments:
  • REINFORCE. REINFORCE w/ Baseline, Actor-Critic, and A2C

Star, Fork, Cite

If you find this work useful, please give it a star, fork, or cite:

@misc{atienza2020dl,
  title={Deep Learning Lecture Notes},
  author={Atienza, Rowel},
  year={2020},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/roatienza/Deep-Learning-Experiments}},
}

Lecture Notes (Old - will no longer be maintained)

0. Course Roadmap

• Deep Learning
• Limits of Deep Learning

1. Background Materials

• Linear Algebra
• Probability
• Numerical Computation

2. Machine Learning Basics

• Concepts, Capacity, Estimators, Linear Regression
• MLE, Bayesian, Other ML Algorithms
• Stochastic Gradient Descent, etc

3. Deep Neural Networks

• Deep Feedforward Neural Networks, Cost, Output, Hidden Units
• Back Propagation
• PyTorch Sample Code
  • Backprop on a single unit MLP
• Keras Sample Code
  • Overview
  • MLP on Linear Model
  • MNIST Sampler
  • MLP on MNIST

4. Regularization

• Keras Sample Code
  • MLP on MNIST no Regularizer
  • MLP on MNIST with L2
  • MLP on MNIST with Dropout
  • MLP on MNIST with Data Augmentation

5. Optimization

6. Convolutional Neural Networks (CNN)

• Keras Sample Code
  • CNN on MNIST
  • CNN on MNIST using Functional API
  • CNN on MNIST Siamese Network

7. Deep Networks

• Keras Sample Code
  • DenseNet
  • ResNet

8. Embeddings

• Keras Sample Code
  • Simple Embedding and Sentiment Analysis
  • Glove Embedding

9. Recurrent Neural Networks, LSTM, GRU

• Keras Sample Code
  • SimpleRNN on MNIST
  • CuDNNLSTM on MNIST

10. AutoEncoders

• Keras Sample Code
  • AutoEncoder
  • Denoising AutoEncoder
  • Colorization AutoEncoder

11. Generative Adversarial Networks (GAN)

• Keras Sample Code
  • DCGAN
  • CGAN

11a. Improved GANs

• Keras Sample Code
  • WGAN
  • LSGAN
  • ACGAN

11b. Disentangled GAN

• InfoGAN

11c. Cross-Domain GAN

• CycleGAN

12. Variational Autoencoder (VAE)

• Keras Sample Code
  • VAE MLP
  • VAE CNN
  • CVAE

13. Deep Reinforcement Learning (DRL)

• Keras Sample Code
  • Q-Learning
  • Q-Learning on FrozenLake-v0
  • DQN and DDQN on CartPole-v0

14. Policy Gradient Methods

• Keras Sample Code
  • REINFORCE. REINFORCE w/ Baseline, Actor-Critic, and A2C