Giters

luyug / Condenser

EMNLP 2021 - Pre-training architectures for dense retrieval

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

What is exactly is CONdition of Condensor?

hmthanh opened this issue · comments

commented

As title, I already read paper but still confused about the Condition of CLS token.
What is exactly is CONdition of Condensor?

commented

If you take a closer look at the design of the head Transformer layers, the MLM prediction has an explicit conditioning/dependency on the CLS vector from the backbone LM.