What's the difference between unilm1.2-base-uncased and unilm2-base-uncased ?
blackskylh opened this issue · comments
I'm using UniLMv2, but I don't know the difference between unilm1.2-base-uncased and unilm2-base-uncased. Is the corpus used for pretrain different?
These two models are described in the paper https://arxiv.org/pdf/2002.12804.pdf .
unilm2 = unilm1.2 + rel_pos (see Table 2, 3 in the paper)