usbser / WangShuSen-DeepLearningNote

CS583: Deep Learning

1. Machine learning basics. This part briefly introduces the fundamental ML problems-- regression, classification, dimensionality reduction, and clustering-- and the traditional ML models and numerical algorithms for solving the problems.

   • ML basics [slides-1] [slides-2].

   • Regression [slides-1] [slides-2].

   • Classification.

     • Logistic regression [slides] [lecture note].

     • SVM [slides].

     • Softmax classifier [slides].

     • KNN classifier [slides].

   • Regularizations [slides-1] [slides-2].

   • Clustering [slides].

   • Dimensionality reduction [slides-1] [slides-2] [lecture note].

   • Scientific computing libraries. [slides].

2. Neural network basics. This part covers the multilayer perceptron, backpropagation, and deep learning libraries, with focus on Keras.

   • Multilayer perceptron and backpropagation [slides] [lecture note].

   • Keras [slides].

   • Further reading:

     • [activation functions]

     • [parameter initialization]

     • [optimization algorithms]

3. Convolutional neural networks (CNNs). This part is focused on CNNs and its application to computer vision problems.

   • CNN basics [slides].

   • Tricks for improving test accuracy [slides].

   • Feature scaling and batch normalization [slides].

   • Advanced topics on CNNs [slides].

   • Popular CNN architectures [slides].

   • Further reading:

     • [style transfer (Section 8.1, Chollet's book)]

     • [visualize CNN (Section 5.4, Chollet's book)]

4. Recurrent neural networks (RNNs). This part introduces RNNs and its applications in natural language processing (NLP).

   • Categorical feature processing [slides] [video (Chinese)].

   • Text processing and word embedding [slides] [video (Chinese)].

   • RNN basics [slides] [video (Chinese)].

   • LSTM [slides] [reference] [video (Chinese)].

   • Making RNNs more effective [slides] [video (Chinese)].

   • Text generation [slides] [video (Chinese)].

   • Machine translation [slides] [video (Chinese)].

   • Attention [slides] [video (English)] [video (Chinese)] [reference].

   • Self-attention [slides] [video (English)] [video (Chinese)].

   • Image caption generation [slides] [reference].

5. Transformer Models.

   • Transformer (1/2): attention without RNN [slides] [video (English)] [video (Chinese)].

   • Transformer (2/2): from shallow to deep [slides] [video (English)] [video (Chinese)] [reference].

   • BERT: pre-training Transformer [slides] [video (English)] [video (Chinese)] [reference].

   • Vision Transformer (ViT) [slides] [video (English)] [video (Chinese)].

6. Autoencoders. This part introduces autoencoders for dimensionality reduction and image generation.

   • Autoencoder for dimensionality reduction [slides].

   • Variational Autoencoders (VAEs) for image generation [slides].

7. Generative Adversarial Networks (GANs).

   • DC-GAN [slides].

8. Deep Reinforcement Learning.

   • Reinforcement learning basics [slides] [lecture note] [video (Chinese)].

   • Value-based learning [slides] [video (Chinese)].

   • Policy-based learning [slides] [video (Chinese)].

   • Actor-critic methods [slides] [video (Chinese)].

   • AlphaGo and Monte Carlo tree search [slides] [video (Chinese)].

9. Parallel Computing.

   • Basics and MapReduce [slides] [lecture note] [video (Chinese)].

   • Parameter server and decentralized network [slides] [video (Chinese)].

   • TensorFlow's mirrored strategy and ring all-reduce [slides] [video (Chinese)].

   • Federated learning [slides] [video (Chinese)].

10. Adversarial Robustness. This part introduces how to attack neural networks using adversarial examples and how to defend from the attack.

    • Data evasion attack and defense [slides] [lecture note].

    • Data poisoning attack [slides] [video (Chinese)].

    • Further reading: [Adversarial Robustness - Theory and Practice].

11. Meta Learning.

    • Few-shot learning: basic concepts [slides] [video (English)] [video (Chinese)].

    • Siamese network [slides] [video (English)] [video (Chinese)].

    • Pretraining + fine tuning [slides] [video (English)] [video (Chinese)].

12. Neural Architecture Search (NAS).

    • Basics [slides] [video (Chinese)].

    • RNN + Reinforcement Learning: [slides] [video (Chinese)].

    • Differentiable NAS: [slides] [video (Chinese)].