abhishek-niranjan / tensorflow2-rnn-tutorials

Tensorflow 2.0 tutorials for RNN based architectures for textual problems

The tutorial notebooks in this repository are aimed to help beginners program basic recurrent neural networks(RNNs) for textual problems in tensorflow-2.

Prerequisite

• Understanding of basic textual processing methods, i.e familiarity with tokenization, etc.
• Functioning of basic RNN, GRU, LSTM cells
• Fundamentals of RNN based Encoder-Decoder Architectures
• Attention mechanism, mainly Bahdanau Attention and Luong Attention
• A basic idea of beam search algorithm.

Contents

1. utils directory contains helper classes and functions.

   • utils/dataset.py contains NMTDataset class which creates training and validation tf.data.Dataset splits and also returns input and target side tokenizers (tf.keras.preprocessing.text.Tokenizer). The working of utils/dataset.py have been explained in first notebook on text-processing notebook
   • utils/attention.py contains BahdanauAttention and LuongAttention class. These attention mechanisms have also been explained in [encoder-decoder with attention notebooks](lesson 4 and lesson 5)

2. tutorial-notebooks directory contains all the jupyter notebooks.

   • Notebook-1 (text-processing): explains how to use tf.keras.preprocessing module to preprocess textual corpus and prepare tf.data.Dataset objects.
   • Notebook-2 (embedding-and-classification): Fundamentals of a many-to-one recurrent neural network and how to program it in tensorflow-2.0
   • Notebook-3 (encoder-decoder): We build a encoder-decoder architecture with tf.keras.layers.GRU as the base recurrent layer in both encoder and decoder.
   • Notebook-4 (Bahdanau Attention): We describe the Bahdanau Attention mechanism and how it is incorporated in an encoder-decoder architecture.
   • Notebook-5 (Luong Attention): We describe Luong Attention and how is it incorporated in an encoder-decoder systm.
   • Notebook-6 (Stacked Encoder-Decoder): We build stacked(more than one layer) encoder with Bi-directional LSTM layers and stacked decoder with Unidirectional LSTM layers.
   • Notebook-7 (GoogleNMT): We program the architecture described in Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. The encoder and decoder stacks contain a total of 8 layers and residual connections from 3^rd layer onwards.
   • Notebook-8 (tf-addons): We learn how to use tf.addons.seq2seq module and how to use BasicDecoder and BeamSearchDecoder class for inference step.