Ternary BERT: initialize student model seems not loading
NancySunnn opened this issue · comments
Hi everyone, I'm trying to apply the TernaryBERT code with sts-b task. I first train the BERT with sts-b for 3 epochs(follow the paper). And then got below message when I code the student model:
It seems like the weights name in student model are with _clip_val, which is not corresponds to the teacher model it's self. I'm wondering if anyone also get this message.
The other question I have is from the paper, the ternary bert seems like have two student model one is full-precision and the other is quantized. But from the code, there is only one student model. Is there any part the student model store the full precious weights?
Thanks!