Giters

labmlai / annotated_deep_learning_paper_implementations

🧑‍🏫 60 Implementations/tutorials of deep learning papers with side-by-side notes 📝; including transformers (original, xl, switch, feedback, vit, ...), optimizers (adam, adabelief, sophia, ...), gans(cyclegan, stylegan2, ...), 🎮 reinforcement learning (ppo, dqn), capsnet, distillation, ... 🧠

Home Page:https://nn.labml.ai

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Question about value_pe

Youngea opened this issue · comments

commented

In the file, rope/value_pe/init.py
image

It seems the value embedding is rotated twice in line 230 and line 234.
From my understanding, only one rotation is enough.
Please let me know whether I misunderstood this.
Thanks.

commented

Thanks. It's a bug

commented

Fixed it here 09d0937