Fine-tuning for classification
zas97 opened this issue · comments
Hello,
do you plan on adding the code to fine-tune your model in a classification task (like RVL-CDIP)?
PS : Nice work, splitting the layout and the language models is a really good idea!
Hi,
in fact, the code of classification task is very similar to that of NER task, since you only need to label [CLS] token with the corresponding classification category and label other tokens with the "ignore" flag (for example, -100 for torch.nn.CrossEntropyLoss).
Yes it seems like an option.
However, in the run_funsd.py script would replacing the AutoModelForTokenClassification with AutoModelForSequenceClassification and adapting the dataset to the classification task work?
No, it wouldn't work, since there is no LiLTRobertaLikeForSequenceClassification implemented in https://github.com/jpWang/LiLT/blob/main/LiLTfinetune/models/LiLTRobertaLike/modeling_LiLTRobertaLike.py for now. You can implement it as you wish.
Or, if you want to implement classification task based on run_funsd.py, you can focus on https://github.com/jpWang/LiLT/blob/main/examples/run_funsd.py#L180-L208, and give the classification category id for [CLS] token to label_ids, and give -100 for other tokens to label_ids.