Is the model code properly embedding the input tokens?

[ey-cai]

Lines 173-174 in models.py say:

# Will use fixed sin-cos embedding:
self.pos_embed = nn.Parameter(torch.zeros(1, num_patches, hidden_size), requires_grad=False)

This seems to be initializing all of the positional embeddings to 0, rather than using any of the 2d_sincos_pos_embed functions defined later in the script. Is this intentional?