sushant097 / Nepali-Poet-Generation-with-char-RNN-model

This project is the implementation of Andrej Karpathy implementation of chracter RNN model in Keras. This generates the Nepali poet train on Laxmi Prasad Devkota Poems.

Nepali Poem Generation using char-rnn model

Inspired By Andrej Karpathy work.

Description: This project use the char-rnn model where

• layer of LSTM =3 with units=256
• Dropout Probability = 0.2
• Final Dense layer with units = Number of vocabs=90

Here i use stateful=true in each LSTM layer since it help to maintain long term dependencies in each sequence. If stateful=false you can use whole file at a once to train.

BATCH_SIZE = 128
# to read batches created
def read_batches():
    cur_index = 0
    steps_per_epoch = len(x) // BATCH_SIZE  # x is whole text data in a vector
    while True:
        if(cur_index + BATCH_SIZE) > len(x):
            cur_index = 0
        batchRange = range(cur_index, cur_index + BATCH_SIZE)
        X = np.asarray([x[i,:,:] for i in batchRange])
        Y = np.asarray([y[i, :] for i in batchRange])
        cur_index += BATCH_SIZE
        yield X, Y

The following 3 files are used mainly as:

generate.py ==> Used to generate the predicted new text of the given length i.e in this case is Nepali Poem.
model.py ==> Used to create the model of the Network
train.py  ==> Used to train the model. 
Here due to `stateful=true` in lSTM, it should be follow as train in Batch.

This project you can also use with English text Generation. For eg: Joke large text > 1MB data to generate the new Joke text based on Char-RNN model.

The Model Summary in Keras is shown below:

Layer (type)                 Output Shape              Param #
=================================================================
lstm_1 (LSTM)                (128, 60, 256)            355328
_________________________________________________________________
dropout_1 (Dropout)          (128, 60, 256)            0
_________________________________________________________________
lstm_2 (LSTM)                (128, 60, 256)            525312
_________________________________________________________________
dropout_2 (Dropout)          (128, 60, 256)            0
_________________________________________________________________
lstm_3 (LSTM)                (128, 256)                525312
_________________________________________________________________
dropout_3 (Dropout)          (128, 256)                0
_________________________________________________________________
dense_1 (Dense)              (128, 90)                 23130
=================================================================
Total params: 1,429,082
Trainable params: 1,429,082
Non-trainable params: 0

You can also use Embedding layer on top of LSTM as embedding every character as Dense Layer to improve accuracy but not much seem as improvement. Like this:

 model.add(Embedding(vocab_size, units, batch_input_shape=(batch_size, seq_len)))

The char-RNN model is here used to generate the Nepali Poem written by Adikabi "Laxmi Prasad Devkota" who is pioneer in Nepali Literature.

• n_vocabs = number of unique characters appear in the whole text i.e 90
• seq_length = sentence length to be taken set as 60
• Batch_size = the number of sentence text that is pass to the model at once i.e set as 128.

Predicted Text

की
खरिद प्यार मासुकी !
ए गर्त ! यौनप्यासकी !
ए पुत्तली विनाश
 रेशम चुल्यो !
बादल भन्छौ तिमी नै बिरामी,
आत्मा बस्छन् किन ? प्राण !
निश्चल, स्वर्ग भै सारा !

नेपालीको कालो कुस्की, त्यो प्राण !
के तिम्रो सुनको र म पापी भन्छन् ।
बिरामी


कसरी गइन् ती मेरी आमा, तिम्लाई छाडेर,
कसरी गइन् बिचरीलाई
सुनको बारी,

मुनाको बाटो बुझेका आँखा रसाई बिरामी,
कुइरी दिदी ! चीतनले भन्छिन्, “दैवले हुँदैन,

म आएँ आए समय निदाइ सुनको रामा ।

कलिलो बस्छौ सारा भएको बेलामा बल्दछ,
हे मेरी आमा ! म हाम्रा मेरी ! म आएँ

See on the predicted.txt file for more

LOSS AND Accuracy Overview

The model is run for only 21 epoch take me 5 hours in 940MX 2GB Nvidia GPU. The loss and accuracy untill 21 epochs is: Loss:1.364 , Accuracy=61%. Train it for more than 40 epochs as loss is nearly 0.20 that can give perfect output of the model.

Tips to Increase Accuracy of the Model

• The dataset should be well formated and arranged.
• Increase Size of dataset minimum >1MB of text .
• Use More Layer of LSTM to deeper understanding but aware of model overfitting.
• Use more Epoch untill loss is very minimum.