RuntimeError: Expected x1.dtype() == cos.dtype() to be true, but got false. (Could this error message be improved? If so, please report an enhancement request to PyTorch.)
JerryDaHeLian opened this issue · comments
When I Pre-train LLaMA, there is a error:
File "/usr/local/lib/python3.8/dist-packages/torch/autograd/function.py", line 551, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/home/xxx/TinyLlama/lit_gpt/fused_rotary_embedding.py", line 39, in forward
rotary_emb.apply_rotary(
RuntimeError: Expected x1.dtype() == cos.dtype() to be true, but got false. (Could this error message be improved? If so, please report an enhancement request to PyTorch.)
Who can help me? 3q!
I have solved it! Look at follows:
The bug stem form
Line 130 in da71ade
I managed to change it :
from transformers.utils import is_torch_bf16_gpu_available
dtype=torch.bfloat16 if is_torch_bf16_gpu_available() else torch.float16,