LiyuanLucasLiu / Transformer-Clinic

Understanding the Difficulty of Training Transformers

https://arxiv.org/abs/2004.08249

LiyuanLucasLiu/Transformer-Clinic Issues

Position of residual connection in PreLN architecture is wrong
Closed a year ago1
How to get the beta_{i,j} for each residual branch?
Updated 2 years ago
Admin for 100L-100L model？
Closed 2 years ago1
Ensemble models
Closed 2 years ago
How to add Radam to fairseq ?
Closed 3 years ago1
argdict
Closed 3 years ago1
Reimplement Admin in new fairseq but get bad valid loss
Closed 3 years ago
Question about the adaptive optimizer
Closed 3 years ago1
Difference of implementation from the original paper
Closed 3 years ago1
`RuntimeError: expected scalar type Float but found Half` during the eval step
Closed 3 years ago5
Scripts for Post-LN in Figure 10?
Closed 3 years ago1
Is wmt14en-fr.sh missing in pre-process dir?
Closed 3 years ago1
wmt_en_de admin: Function 'SoftmaxBackward' returned nan values in its 0th output.
Closed 3 years ago8
tmp_weight is not defined
Closed 3 years ago4
IWSLT'14 Results
Closed 4 years ago1
Post-LN with 12-12 is trained ok, but 12-3 diverge
Closed 4 years ago9
How to make sure that only performing one step forward pass in profiling phase?
Closed 4 years ago1
is "tmp_weight" in transformer_layer.py useless?
Closed 4 years ago3
Details of total batch size
Closed 4 years ago1
Do the embedding layer's layernorm parameters need to be reparameterized accordingly?
Closed 4 years ago1
Can I use a pre-trained model to initialize the model?
Closed 4 years ago1
Is the "attention_ratio_change" and "fc_ratio_change" trainable or not?
Closed 4 years ago2
whta's the meaning of 'adaptive-scale' argument?
Closed 4 years ago1