Gated Orthogonal Recurrent Unit
This model combines gating mechanism and orthogonal RNN approach. It solves forgetting problem and long-term dependency.
If you find this work useful, please cite arXiv:1706.02761.
requires TensorFlow 1.2.0
To use GORU in your model, simply copy goru.py.
Then you can use GORU in the same way you use built-in LSTM:
from goru import GORUCell
cell = GORUCell(hidden_size, capacity, fft)
Args:
hidden_size:Integer.capacity:Optional.Integer. Only works for tunable style.fft:Optional.Bool. IfTrue, GORU is set to FFT style. Default isTrue.
We put two examples toy tasks: copying task and denoise task
python copying_task.py --model GORU
python denoise_task.py --model GORU