Giters

huawei-noah / Pretrained-Language-Model

Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Ternary BERT: initialize student model seems not loading

NancySunnn opened this issue · comments

commented

Hi everyone, I'm trying to apply the TernaryBERT code with sts-b task. I first train the BERT with sts-b for 3 epochs(follow the paper). And then got below message when I code the student model:

image

It seems like the weights name in student model are with _clip_val, which is not corresponds to the teacher model it's self. I'm wondering if anyone also get this message.

The other question I have is from the paper, the ternary bert seems like have two student model one is full-precision and the other is quantized. But from the code, there is only one student model. Is there any part the student model store the full precious weights?

Thanks!