AstraZeneca / KAZU

Fast, world class biomedical NER

Home Page:https://AstraZeneca.github.io/KAZU/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Delete legacy layer in TinyBern2 model

RichJackson opened this issue · comments

original comment from @wonjininfo

@RichJackson reported warning when we load the model:

Some weights of the model checkpoint at .../kazu/kazu_model_pack_public-v0.0.25/tinybern were not used when initializing BertForTokenClassification: ['fit_dense.weight', 'fit_dense.bias']
- This IS expected if you are initializing BertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

I think, this layer (fit_dense) comes from legacy code and will not affect performance or functionality.

Related code:
https://github.com/AstraZeneca/KAZU/blob/main/kazu/modelling/distillation/tiny_transformers.py#L56-L57

Todo: I will first double-check it and will delete fit_dense layer!