Training is very slow

[Kevi Aday]

Hi Skuldur,

I am using your training script to train with my own midi files. I don't know why, but the training is going very slowly. It takes about 25 minutes to train only one epoch. I'm confused because i have a brand new GTX 1070 with drivers installed including geforce experience, cuda 9.0 (because tensorflow is incompatible with version 9.2 for some reason), and cudNN. With another training script (it's for generating text) it took only 1-2 minutes per epoch, but its model only had 1 LSTM so that might also be relatively slow.
How long did it take you to train with your midi files, with what GPU?
If you can help, it would be greatly appreciated.

Thanks,
Kevaday

[Kevi Aday]

Also, I tried using your text generating LSTM but it's also very slow on my GPU, it takes about 1 hour to do one epoch. Shouldn't my GPU be a lot faster at training?

[mehrzeller]

you can use CuDNNLSTM instead :
from keras.layers import CuDNNLSTM
instead of
from keras.layers import LSTM
don't forget to change all your LSTM statements in the source code

[Kevi Aday]

Thanks mehrzeller,
actually, it's a coincidence because I was just looking at the keras recurrent layers site and just saw that there's such thing
but thanks anyways

[victor-felicitas]

For new Keras versions specifying CuDNNLSTM is obsolete.
Training is slow because with recurrent_dropout >0 CUDNN can not be used.

From Keras documentation:
"The requirements to use the cuDNN implementation are:
activation == tanh
recurrent_activation == sigmoid
recurrent_dropout == 0
unroll is False
use_bias is True
Inputs, if use masking, are strictly right-padded.
Eager execution is enabled in the outermost context."

I suggest changing the model by adding dropout layer instead, after each LSTM layer