The mathematical trick in self-attention, why it returns false for torch.allclose(xbow, xbow2)?

[Ryan]

Hi
I noticed that the result of torch.allclose(xbow, xbow2), torch.allclose(xbow, xbow3) are all false when running the Collab example gpt-dev.ipynb in The mathematical trick in self-attention section. Here is what I got, has anyone encountered the same issue?

[Arwa]

@Ryan-ZL-Lin You can adjust the relative tolerance for less strict comparison. the default value is 1e-05 in PyTorch 2.2

This snippet will output True

torch.allclose(xbow, xbow2, rtol= 1e-04) # default 1e-05