Can replace gru with transformer?
hegc opened this issue · comments
Hi, yes, both are suitable for sequence modelling.
I replaced gru with transformer, but the model did not converge...If you have time to try, please share it, thank you.
Hi, when using transformer, I suggest that you could carefully tune the warm-up stage such as peak learning rate and number of steps.