cnamuangtoun / rnn-examples-chapwit

Various sequence modelling examples using PyTorch.

Practical, short, batched, gpu-ready, self-contained implementations of various NLP tasks using Pytorch 1.3.0

Data

Names of different nationalities are taken from Pytorch tutorial: https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html Thai names: https://hearnames.com/pronunciations/thai-names/thai-surnames Translation pairs: https://www.manythings.org/anki/

Name classifiers.

1. name_classifier_rnn_onehot.py

• Classify texts using RNN with one hot encoded vectors as inputs.

2. name_classifier_rnn_embedding.py

• Classify names using RNN with dense embeddings.

3. name_classifier_cnn_onehot.py Convolutional Neural Networks for Sentence Classification

• Classify texts using CONV1D layers.
• Using one hot encoded vectors.

Name generators.

1. name_generator.py

• Taking name prefixes and matching it with the next letter. Easy to understand implementation.

2. name_generator_optimized.py

• Targets are mapped to input packed sequence order. Uses less samples, therefore is faster and memory efficient.

Sentence to sequence translators.

1. seq2seq_onehot.py Sequence to Sequence Learning with Neural Networks

• Latest hidden state of the encoder RNN is passed as initial state of the decoder RNN.
• This implementation is using teacher forcing and one hot encoded vectors.

2. seq2seq_unforced_onehot.py Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks

• Latest hidden state of the encoder RNN is passed as initial state of the decoder RNN.
• Tunable teacher forcing for the decoder allows to use model outputs during training.
• Using one hot encoded vectors.

3. seq2seq_propagated_onehot.py Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation

• Latest hidden state of the encoder RNN is passed as initial state of the decoder RNN as well as additional context to be combined with every hidden state and output.
• Using teacher forcing and one hot encoded vectors.

4. seq2seq_attn_onehot.py Neural Machine Translation by Jointly Learning to Align and TranslateEffective Approaches to Attention-based Neural Machine Translation

• All hidden states of the encoder RNN is passed to the decoder RNN.
• Decoder RNN choses what hidden state to use when generating output, therefore removing the bottleneck introduced by encoder output vector size.
• Using one hot encoded vectors.