xashru / deep-learning-distilled

Notes on some important deep learning topics and paper summaries

deep-learning-distilled

Notes on some important deep learning topics and paper summaries.
Feel free to contribute.
Documents have corresponding latex file you can edit.

Notes added so far

• Information, Entropy, Cross-Entropy: ML perspective: Basic of information theory, why log is used to represent information. entropy, cross entropy, KL divergence, likelihood, why cross entropy loss is used in machine learning.
• Gradient Descent Optimizations: Three gradient descent varients, challenges with vanilla gradient descent, momentum, Nesterov accelerated gradient, Adagrad, RMSprop, Adadelta, Adam.
• Common activation functions used in neural net: why we need to use activation function, desirable properties of activation, sigmoid, tanh, relu, prelu, elu.
• Why ReLU(instead of sigmoid/tanh)?: why ReLU is better suited for deep learning compared to sigmoid or tanh, some of the potential problems with ReLU and how to mitigate them.

Paper Summary

1. How transferable are features in deep neural networks?
2. Learning and transferring mid-Level image representations using convolutional neural networks
3. Distilling the Knowledge in a Neural Network
4. Sequence to Sequence Learning with Neural Networks
5. Distributed Representations of Sentences and Documents
6. VGG
7. ResNet
8. Deep Sparse Rectifier Neural Networks
9. Network in Network
10. GoogLeNet
11. MobileNets
12. AlexNet
13. Inception-V2
14. Inception-V4
15. Dropout
16. Efficient Estimation of Word Representations in Vector Space
17. A Convolutional Neural Network for Modelling Sentences
18. Effective Approaches to Attention-based Neural Machine Translation
19. Neural Machine Translation by Jointly Learning to Align and Translate
20. Large-scale Video Classification with Convolutional Neural Networks
21. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
22. Curriculum Learning
23. Maxout Networks
24. Visualizing and Understanding Convolutional Networks
25. R-CNN
26. Fast R-CNN
27. Faster R-CNN
28. SSD: Single Shot MultiBox Detector

Notes to add

• Regularization
• Different types of Losses
• Activation functions: swish