Shivanshu-Gupta / Deep-Learning-Papers-Reading-Roadmap

Deep Learning papers reading roadmap for anyone who are eager to learn this amazing tech!

1. Deep Learning Papers Reading Roadmap

If you are a newcomer to the Deep Learning area, the first question you may have is "Which paper should I start reading from?" Here is a reading roadmap of Deep Learning papers!

The roadmap is constructed in accordance with the following four guidelines:

• From outline to detail
• From old to state-of-the-art
• from generic to specific areas
• focus on state-of-the-art

You will find many papers that are quite new but really worth reading.

I would continue adding papers to this roadmap.

---

1.1. Deep Learning History and Basics

1.1.1. Book

• Deep learning. Bengio, Yoshua, Ian J. Goodfellow, and Aaron Courville. [html] ⭐⭐⭐⭐⭐
  • Deep Learning Bible, you can read this book while reading following papers.

1.1.2. Survey

• Deep learning. LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. [pdf] ⭐⭐⭐⭐⭐
  • Three Giants' Survey

1.1.3. Deep Belief Network(DBN)(Milestone of Deep Learning Eve)

• A fast learning algorithm for deep belief nets. Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. [pdf] ⭐⭐⭐

  • Deep Learning Eve

• Reducing the dimensionality of data with neural networks. Hinton, Geoffrey E., and Ruslan R. Salakhutdinov. [pdf] ⭐⭐⭐

  • Milestone, Show the promise of deep learning

1.1.4. ImageNet Evolution（Deep Learning broke out from here）

• Imagenet classification with deep convolutional neural networks. Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. [pdf] ⭐⭐⭐⭐⭐

  • AlexNet, Deep Learning Breakthrough

• Very deep convolutional networks for large-scale image recognition. Simonyan, Karen, and Andrew Zisserman. [pdf] ⭐⭐⭐

  • VGGNet,Neural Networks become very deep!

• Going deeper with convolutions. Szegedy, Christian, et al. [pdf] ⭐⭐⭐

  • GoogLeNet

• Deep residual learning for image recognition. He, Kaiming, et al. [pdf] ⭐⭐⭐⭐⭐

  • ResNet,Very very deep networks, CVPR best paper

1.1.5. Speech Recognition Evolution

• Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. Hinton, Geoffrey, et al. [pdf]⭐⭐⭐⭐

  • Breakthrough in speech recognition

• Speech recognition with deep recurrent neural networks. Graves, Alex, Abdel-rahman Mohamed, and Geoffrey Hinton. [pdf]⭐⭐⭐ - RNN

• Towards End-To-End Speech Recognition with Recurrent Neural Networks. Graves, Alex, and Navdeep Jaitly. [pdf]⭐⭐⭐

• Fast and accurate recurrent neural network acoustic models for speech recognition. Sak, Haşim, et al. [pdf] ⭐⭐⭐

  • Google Speech Recognition System

• Deep speech 2: End-to-end speech recognition in english and mandarin. Amodei, Dario, et al. [pdf] ⭐⭐⭐⭐

  • Baidu Speech Recognition System

• Achieving Human Parity in Conversational Speech Recognition. W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu, G. Zweig [pdf] ⭐⭐⭐⭐

  • State-of-the-art in speech recognition, Microsoft

After reading above papers, you will have a basic understanding of the Deep Learning history, the basic architectures of Deep Learning model(including CNN, RNN, LSTM) and how deep learning can be applied to image and speech recognition issues. The following papers will take you in-depth understanding of the Deep Learning method, Deep Learning in different areas of application and the frontiers. I suggest that you can choose the following papers based on your interests and research direction.

1.2. Deep Learning Method

1.2.1. Model

• Improving neural networks by preventing co-adaptation of feature detectors. Hinton, Geoffrey E., et al. [pdf] ⭐⭐⭐

  • Dropout

• Dropout: a simple way to prevent neural networks from overfitting. Srivastava, Nitish, et al. [pdf] ⭐⭐⭐

• Batch normalization: Accelerating deep network training by reducing internal covariate shift. Ioffe, Sergey, and Christian Szegedy. [pdf] ⭐⭐⭐⭐

  • An outstanding Work in 2015

• Layer normalization. Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. [pdf] ⭐⭐⭐⭐

  • Update of Batch Normalization

• Binarized Neural Networks: Training Neural Networks with Weights and Activations Constrained to+ 1 or−1. Courbariaux, Matthieu, et al. [pdf] ⭐⭐⭐

  • New Model,Fast

• Decoupled neural interfaces using synthetic gradients. Jaderberg, Max, et al. [pdf] ⭐⭐⭐⭐⭐

  • Innovation of Training Method,Amazing Work

• Net2net: Accelerating learning via knowledge transfer. Chen, Tianqi, Ian Goodfellow, and Jonathon Shlens. [pdf] ⭐⭐⭐

  • Modify previously trained network to reduce training epochs

• Network Morphism. Wei, Tao, et al. [pdf] ⭐⭐⭐

  • Modify previously trained network to reduce training epochs

1.2.2. Optimization

• On the importance of initialization and momentum in deep learning. Sutskever, Ilya, et al. [pdf] ⭐⭐

  • Momentum optimizer

• Adam: A method for stochastic optimization. Kingma, Diederik, and Jimmy Ba. [pdf] ⭐⭐⭐

  • Maybe used most often currently

• Learning to learn by gradient descent by gradient descent. Andrychowicz, Marcin, et al. [pdf] ⭐⭐⭐⭐⭐

  • Neural Optimizer,Amazing Work

• Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. Han, Song, Huizi Mao, and William J. Dally. [pdf] ⭐⭐⭐⭐⭐

  • ICLR best paper, new direction to make NN running fast,DeePhi Tech Startup

• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 1MB model size. Iandola, Forrest N., et al. [pdf] ⭐⭐⭐⭐

  • Also a new direction to optimize NN,DeePhi Tech Startup

1.2.3. Unsupervised Learning / Deep Generative Model

• Building high-level features using large scale unsupervised learning. Le, Quoc V. [pdf] ⭐⭐⭐⭐

  • Milestone, Andrew Ng, Google Brain Project, Cat

• Auto-encoding variational bayes. Kingma, Diederik P., and Max Welling. [pdf] ⭐⭐⭐⭐

  • VAE

• Generative adversarial nets. Goodfellow, Ian, et al. [pdf] ⭐⭐⭐⭐⭐

  • GAN,super cool idea

• Unsupervised representation learning with deep convolutional generative adversarial networks. Radford, Alec, Luke Metz, and Soumith Chintala. [pdf] ⭐⭐⭐⭐

  • DCGAN

• DRAW: A recurrent neural network for image generation. Gregor, Karol, et al. [pdf] ⭐⭐⭐⭐⭐

  • VAE with attention, outstanding work

• Pixel recurrent neural networks. Oord, Aaron van den, Nal Kalchbrenner, and Koray Kavukcuoglu. [pdf] ⭐⭐⭐⭐

  • PixelRNN

• Conditional image generation with PixelCNN decoders. Oord, Aaron van den, et al. [pdf] ⭐⭐⭐⭐

  • PixelCNN

1.2.4. RNN / Sequence-to-Sequence Model

• Generating sequences with recurrent neural networks. Graves, Alex. [pdf] ⭐⭐⭐⭐

  • LSTM, very nice generating result, show the power of RNN

• Learning phrase representations using RNN encoder-decoder for statistical machine translation. Cho, Kyunghyun, et al. [pdf] ⭐⭐⭐⭐

  • First Seq-to-Seq Paper

• Sequence to sequence learning with neural networks. Sutskever, Ilya, Oriol Vinyals, and Quoc V. Le. [pdf] ⭐⭐⭐⭐⭐

  • Outstanding Work

• Neural Machine Translation by Jointly Learning to Align and Translate. Bahdanau, Dzmitry, KyungHyun Cho, and Yoshua Bengio. [pdf] ⭐⭐⭐⭐

• A neural conversational model. Vinyals, Oriol, and Quoc Le. [pdf] ⭐⭐⭐

  • Seq-to-Seq on Chatbot

1.2.5. Neural Turing Machine

• Neural turing machines. Graves, Alex, Greg Wayne, and Ivo Danihelka. [pdf] ⭐⭐⭐⭐⭐

  • Basic Prototype of Future Computer

• Reinforcement learning neural Turing machines. Zaremba, Wojciech, and Ilya Sutskever. [pdf] ⭐⭐⭐

• Memory networks. Weston, Jason, Sumit Chopra, and Antoine Bordes. [pdf] ⭐⭐⭐

• End-to-end memory networks. Sukhbaatar, Sainbayar, Jason Weston, and Rob Fergus. [pdf] ⭐⭐⭐⭐

• Pointer networks. Vinyals, Oriol, Meire Fortunato, and Navdeep Jaitly. [pdf] ⭐⭐⭐⭐

• Hybrid computing using a neural network with dynamic external memory. Graves, Alex, et al. [pdf] ⭐⭐⭐⭐⭐

  • Milestone,combine above papers' ideas

1.2.6. Deep Reinforcement Learning

• Playing atari with deep reinforcement learning. Mnih, Volodymyr, et al. [pdf]) ⭐⭐⭐⭐

  • First Paper named deep reinforcement learning

• Human-level control through deep reinforcement learning. Mnih, Volodymyr, et al. [pdf] ⭐⭐⭐⭐⭐

  • Milestone

• Dueling network architectures for deep reinforcement learning. Wang, Ziyu, Nando de Freitas, and Marc Lanctot. [pdf] ⭐⭐⭐⭐

  • ICLR best paper,great idea

• Asynchronous methods for deep reinforcement learning. Mnih, Volodymyr, et al. [pdf] ⭐⭐⭐⭐⭐

  • State-of-the-art method

• Continuous control with deep reinforcement learning. Lillicrap, Timothy P., et al. [pdf] ⭐⭐⭐⭐

  • DDPG

• Continuous Deep Q-Learning with Model-based Acceleration. Gu, Shixiang, et al. [pdf] ⭐⭐⭐⭐

  • NAF

• Trust region policy optimization. Schulman, John, et al. [pdf] ⭐⭐⭐⭐

  • TRPO

• Mastering the game of Go with deep neural networks and tree search. Silver, David, et al. [pdf] ⭐⭐⭐⭐⭐

  • AlphaGo

1.2.7. Deep Transfer Learning / Lifelong Learning / especially for RL

• Deep Learning of Representations for Unsupervised and Transfer Learning. Bengio, Yoshua. [pdf] ⭐⭐⭐

  • A Tutorial

• Lifelong Machine Learning Systems: Beyond Learning Algorithms. Silver, Daniel L., Qiang Yang, and Lianghao Li. [pdf] ⭐⭐⭐

  • A brief discussion about lifelong learning

• Distilling the knowledge in a neural network. Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. [pdf] ⭐⭐⭐⭐

  • Godfather's Work

• Policy distillation. Rusu, Andrei A., et al. [pdf] ⭐⭐⭐

  • RL domain

• Actor-mimic: Deep multitask and transfer reinforcement learning. Parisotto, Emilio, Jimmy Lei Ba, and Ruslan Salakhutdinov. [pdf] ⭐⭐⭐

  • RL domain

• Progressive neural networks. Rusu, Andrei A., et al. [pdf] ⭐⭐⭐⭐⭐

  • Outstanding Work, A novel idea

1.2.8. One Shot Deep Learning

• Human-level concept learning through probabilistic program induction. Lake, Brenden M., Ruslan Salakhutdinov, and Joshua B. Tenenbaum. [pdf] ⭐⭐⭐⭐⭐

  • No Deep Learning,but worth reading

• Siamese Neural Networks for One-shot Image Recognition. Koch, Gregory, Richard Zemel, and Ruslan Salakhutdinov. [pdf] ⭐⭐⭐

• One-shot Learning with Memory-Augmented Neural Networks. Santoro, Adam, et al. [pdf] ⭐⭐⭐⭐

  • A basic step to one shot learning

• Matching Networks for One Shot Learning. Vinyals, Oriol, et al. [pdf] ⭐⭐⭐

• Low-shot visual object recognition. Hariharan, Bharath, and Ross Girshick. [pdf] ⭐⭐⭐⭐

  • A step to large data

1.3. Applications

1.3.1. Natural Language Processing (NLP)

• Joint Learning of Words and Meaning Representations for Open-Text Semantic Parsing. Antoine Bordes, et al. [pdf] ⭐⭐⭐⭐
• Distributed representations of words and phrases and their compositionality. Mikolov, et al. [pdf] ⭐⭐⭐
  • word2vec
• “Sequence to sequence learning with neural networks. Sutskever, et al. [pdf] ⭐⭐⭐
• “Ask Me Anything: Dynamic Memory Networks for Natural Language Processing. Ankit Kumar, et al. [pdf] ⭐⭐⭐⭐
• Character-Aware Neural Language Models. Yoon Kim, et al. [pdf] ⭐⭐⭐⭐
• Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. Jason Weston, et al. [pdf] ⭐⭐⭐
  • bAbI tasks
• Teaching Machines to Read and Comprehend. Karl Moritz Hermann, et al. [pdf] ⭐⭐
  • CNN/DailyMail cloze style questions
• Very Deep Convolutional Networks for Natural Language Processing. Alexis Conneau, et al. [pdf] ⭐⭐⭐
  • state-of-the-art in text classification
• Bag of Tricks for Efficient Text Classification. Armand Joulin, et al. [pdf] ⭐⭐⭐
  • slightly worse than state-of-the-art, but a lot faster

1.3.2. Object Detection

• Deep neural networks for object detection. Szegedy, Christian, Alexander Toshev, and Dumitru Erhan. [pdf] ⭐⭐⭐

• Rich feature hierarchies for accurate object detection and semantic segmentation. Girshick, Ross, et al. [pdf] ⭐⭐⭐⭐⭐

  • RCNN

• Spatial pyramid pooling in deep convolutional networks for visual recognition. He, Kaiming, et al. [pdf] ⭐⭐⭐⭐

  • SPPNet

• Fast r-cnn. Girshick, Ross. [pdf] ⭐⭐⭐⭐

• Faster R-CNN: Towards real-time object detection with region proposal networks. Ren, Shaoqing, et al. [pdf] ⭐⭐⭐⭐

• You only look once: Unified, real-time object detection. Redmon, Joseph, et al. [pdf] ⭐⭐⭐⭐⭐

  • YOLO,Oustanding Work, really practical

• SSD: Single Shot MultiBox Detector. Liu, Wei, et al. [pdf] ⭐⭐⭐

• R-FCN: Object Detection via Region-based Fully Convolutional Networks. Dai, Jifeng, et al. [pdf] ⭐⭐⭐⭐

• Mask R-CNN. He, Gkioxari, et al. [pdf] ⭐⭐⭐⭐

1.3.3. Visual Tracking

• Learning a deep compact image representation for visual tracking. Wang, Naiyan, and Dit-Yan Yeung. [pdf] ⭐⭐⭐

  • First Paper to do visual tracking using Deep Learning,DLT Tracker

• Transferring rich feature hierarchies for robust visual tracking. Wang, Naiyan, et al. [pdf] ⭐⭐⭐⭐

  • SO-DLT

• Visual tracking with fully convolutional networks. Wang, Lijun, et al. [pdf] ⭐⭐⭐⭐

  • FCNT

• Learning to Track at 100 FPS with Deep Regression Networks. Held, David, Sebastian Thrun, and Silvio Savarese. [pdf] ⭐⭐⭐⭐

  • GOTURN,Really fast as a deep learning method,but still far behind un-deep-learning methods

• Fully-Convolutional Siamese Networks for Object Tracking. Bertinetto, Luca, et al. [pdf] ⭐⭐⭐⭐

  • SiameseFC,New state-of-the-art for real-time object tracking

• Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking. Martin Danelljan, Andreas Robinson, Fahad Khan, Michael Felsberg. [pdf] ⭐⭐⭐⭐

  • C-COT

• Modeling and Propagating CNNs in a Tree Structure for Visual Tracking. Nam, Hyeonseob, Mooyeol Baek, and Bohyung Han. [pdf] ⭐⭐⭐⭐

  • VOT2016 Winner,TCNN

1.3.4. Image Caption

• Every picture tells a story: Generating sentences from images. Farhadi,Ali,etal. [pdf] ⭐⭐⭐

• Baby talk: Understanding and generating image descriptions. Kulkarni, Girish, et al. [pdf]⭐⭐⭐⭐

• Show and tell: A neural image caption generator. Vinyals, Oriol, et al. [pdf]⭐⭐⭐

• Long-term recurrent convolutional networks for visual recognition and description. Donahue, Jeff, et al. [pdf]

• Deep visual-semantic alignments for generating image descriptions. Karpathy, Andrej, and Li Fei-Fei. [pdf]⭐⭐⭐⭐⭐

• Deep fragment embeddings for bidirectional image sentence mapping. Karpathy, Andrej, Armand Joulin, and Fei Fei F. Li. [pdf]⭐⭐⭐⭐

• From captions to visual concepts and back. Fang, Hao, et al. [pdf]⭐⭐⭐⭐⭐

• Learning a recurrent visual representation for image caption generation. Chen, Xinlei, and C. Lawrence Zitnick. [pdf]⭐⭐⭐⭐

• Deep captioning with multimodal recurrent neural networks (m-rnn). Mao, Junhua, et al. [pdf]⭐⭐⭐

• Show, attend and tell: Neural image caption generation with visual attention. Xu, Kelvin, et al. [pdf]⭐⭐⭐⭐⭐

1.3.5. Machine Translation

Some milestone papers are listed in RNN / Seq-to-Seq topic.

• Addressing the rare word problem in neural machine translation. Luong, Minh-Thang, et al. [pdf] ⭐⭐⭐⭐

• Neural Machine Translation of Rare Words with Subword Units. Sennrich, et al. [pdf]⭐⭐⭐

• Effective approaches to attention-based neural machine translation. Luong, Minh-Thang, Hieu Pham, and Christopher D. Manning. [pdf] ⭐⭐⭐⭐

• A Character-Level Decoder without Explicit Segmentation for Neural Machine Translation. Chung, et al. [pdf]⭐⭐

• Fully Character-Level Neural Machine Translation without Explicit Segmentation. Lee, et al. [pdf]⭐⭐⭐⭐⭐

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. Wu, Schuster, Chen, Le, et al. [pdf] ⭐⭐⭐⭐

  • Milestone

1.3.6. Robotics

• Evolving large-scale neural networks for vision-based reinforcement learning. Koutník, Jan, et al. [pdf] ⭐⭐⭐

• End-to-end training of deep visuomotor policies. Levine, Sergey, et al. [pdf] ⭐⭐⭐⭐⭐

• Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. Pinto, Lerrel, and Abhinav Gupta. [pdf] ⭐⭐⭐

• Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection. Levine, Sergey, et al. [pdf] ⭐⭐⭐⭐

• Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning. Zhu, Yuke, et al. [pdf] ⭐⭐⭐⭐

• Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Search. Yahya, Ali, et al. [pdf] ⭐⭐⭐⭐

• Deep Reinforcement Learning for Robotic Manipulation. Gu, Shixiang, et al. [pdf] ⭐⭐⭐⭐

• Sim-to-Real Robot Learning from Pixels with Progressive Nets. A Rusu, M Vecerik, Thomas Rothörl, N Heess, R Pascanu, R Hadsell [pdf] ⭐⭐⭐⭐

• Learning to navigate in complex environments. Mirowski, Piotr, et al. [pdf] ⭐⭐⭐⭐

1.3.7. Art

• Inceptionism: Going Deeper into Neural Networks. Mordvintsev, Alexander; Olah, Christopher; Tyka, Mike (2015.pdf). [html]

  • Deep Dream :star::star::star::star:

• A neural algorithm of artistic style. Gatys, Leon A., Alexander S. Ecker, and Matthias Bethge. [pdf] ⭐⭐⭐⭐⭐

  • Outstanding Work, most successful method currently

• Generative Visual Manipulation on the Natural Image Manifold. Zhu, Jun-Yan, et al. [pdf] ⭐⭐⭐⭐

  • iGAN

• Semantic Style Transfer and Turning Two-Bit Doodles into Fine Artworks. Champandard, Alex J. [pdf] ⭐⭐⭐⭐

  • Neural Doodle

• Colorful Image Colorization. Zhang, Richard, Phillip Isola, and Alexei A. Efros. [pdf] ⭐⭐⭐⭐

• Perceptual losses for real-time style transfer and super-resolution. Johnson, Justin, Alexandre Alahi, and Li Fei-Fei. [pdf] ⭐⭐⭐⭐

• A learned representation for artistic style. Vincent Dumoulin, Jonathon Shlens and Manjunath Kudlur. [pdf] ⭐⭐⭐⭐

• Controlling Perceptual Factors in Neural Style Transfer. Gatys, Leon and Ecker, et al. [pdf]⭐⭐⭐⭐

  • control style transfer over spatial location,colour information and across spatial scale

• Texture Networks: Feed-forward Synthesis of Textures and Stylized Images. Ulyanov, Dmitry and Lebedev, Vadim, et al. [pdf] ⭐⭐⭐⭐

  • texture generation and style transfer

1.3.8. Object Segmentation

• Fully convolutional networks for semantic segmentation. J. Long, E. Shelhamer, and T. Darrell. [pdf] ⭐⭐⭐⭐⭐

• Semantic image segmentation with deep convolutional nets and fully connected crfs. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. [pdf] ⭐⭐⭐⭐⭐

• Learning to segment object candidates.. Pinheiro, P.O., Collobert, R., Dollar, P. [pdf] ⭐⭐⭐⭐

• Instance-aware semantic segmentation via multi-task network cascades. Dai, J., He, K., Sun, J. [pdf] ⭐⭐⭐

• Instance-sensitive Fully Convolutional Networks. Dai, J., He, K., Sun, J. [pdf] ⭐⭐⭐