Inconsistency between PyTorch and JAX implementation
aboros98 opened this issue · comments
Hello!
In the PyTorch implementation, in the MLP, exact GeLU is used as a gating function.
In the JAX version, the approximate gelu is used.
Could you please clarify which version is the correct one?
I see this PR is fixing it, will help land it soon.
#37
PR is checked in. Closing this.