wmathor / nlp-tutorial

Natural Language Processing Tutorial for Deep Learning Researchers

nlp-tutorial

nlp-tutorial is a tutorial for who is studying NLP(Natural Language Processing) using Pytorch. Most of the models in NLP were implemented with less than 100 lines of code.(except comments or blank lines)

Dependencies

• Python 3.6+
• Pytorch 1.2.0+

Curriculum - (Example Purpose)

1. Basic Embedding Model

• 1-1. NNLM(Neural Network Language Model) - Predict Next Word
  • Paper - A Neural Probabilistic Language Model(2003)
  • Colab - NNLM_Torch.ipynb
  • bilibili - https://www.bilibili.com/video/BV1AT4y1J7bv/
• 1-2. Word2Vec(Skip-gram) - Embedding Words and Show Graph
  • Paper - Distributed Representations of Words and Phrases and their Compositionality(2013)
  • Colab - Word2Vec_Torch(Softmax).ipynb
  • bilibili - https://www.bilibili.com/video/BV14z4y19777/
• 1-3. FastText(Application Level) - Sentence Classification
  • Paper - Bag of Tricks for Efficient Text Classification(2016)
  • Colab - FastText.ipynb

2. CNN(Convolutional Neural Network)

• 2-1. TextCNN - Binary Sentiment Classification
  • Paper - Convolutional Neural Networks for Sentence Classification(2014)
  • Colab - TextCNN_Torch.ipynb
  • bilibili - https://www.bilibili.com/video/BV1ip4y1U735/

3. RNN(Recurrent Neural Network)

• 3-1. TextRNN - Predict Next Step
  • Paper - Finding Structure in Time(1990)
  • Colab - TextRNN_Torch.ipynb
  • bilibili - https://www.bilibili.com/video/BV1iK4y147ff/
• 3-2. TextLSTM - Autocomplete
  • Paper - LONG SHORT-TERM MEMORY(1997)
  • Colab - TextLSTM_Torch.ipynb
• 3-3. Bi-LSTM - Predict Next Word in Long Sentence
  • Colab - Bi_LSTM_Torch.ipynb
  • bilibili - https://www.bilibili.com/video/BV1tf4y117hA/

4. Attention Mechanism

• 4-1. Seq2Seq - Change Word
  • Paper - Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation(2014)
  • Colab - Seq2Seq_Torch.ipynb
  • bilibili - https://www.bilibili.com/video/BV1Q5411W7zz/
• 4-2. Seq2Seq with Attention - Translate
  • Paper - Neural Machine Translation by Jointly Learning to Align and Translate(2014)
  • Colab - Seq2Seq(Attention)_Torch.ipynb
  • bilibili - https://www.bilibili.com/video/BV1op4y1U7ag/
• 4-3. Bi-LSTM with Attention - Binary Sentiment Classification
  • Colab - Bi_LSTM(Attention)_Torch.ipynb

5. Model based on Transformer

• 5-1. The Transformer - Translate
  • Paper - Attention Is All You Need(2017)
  • Colab - Transformer_Torch.ipynb
  • bilibili - https://www.bilibili.com/video/BV1mk4y1q7eK
• 5-2. BERT - Classification Next Sentence & Predict Masked Tokens
  • Paper - BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding(2018)
  • Colab - BERT_Torch.ipynb
  • bilibili - https://www.bilibili.com/video/BV11p4y1i7AN

Model	Example
NNLM	Predict Next Word
Word2Vec(Softmax)	Embedding Words and Show Graph
TextCNN	Sentence Classification
TextRNN	Predict Next Step
TextLSTM	Autocomplete
Bi-LSTM	Predict Next Word in Long Sentence
Seq2Seq	Change Word
Seq2Seq with Attention	Translate
Bi-LSTM with Attention	Binary Sentiment Classification
Transformer	Translate
Greedy Decoder Transformer	Translate
BERT	how to train

Author

• Tae Hwan Jung(Jeff Jung) @graykode，modify by wmathor
• Author Email : [e-mail]
• Acknowledgements to mojitok as NLP Research Internship.