TriTrain
Tri-Train: Automatic Pre-Fine Tuning between Pre-Training and Fine-Tuning for SciNER Authors: Qingkai Zeng (ND), Wenhao Yu (ND), Mengxia Yu (ND), Tianwen Jiang (HIT), Tim Weninger (ND), Meng Jiang (ND)
This paper proposed a novel framework to introduce a “pre-fine tuning” step between pre-training and fine-tuning to enhance the NER related tasks on scientific domain.
Requirements
A detailed dependencies list can be found in requirements.txt
and can be installed by:
pip install -r requirements.txt
Example of distant corpus
There are two files for labeled distant corpus. One is train.txt
which contains the sentences used in pre-fine-tuining. The other one is train.dist
which contains the position of the automatically labeled concepts.
train.txt
: We used neural network as a machine learning method.train.dist
: 0 0 1 1 0 0 1 1 0 0
Pre-Fine-tune a BERT model
cd language_model
sh run_lm.sh
Fine-tuning a NER model
Please put the BERT-based model after pre-fine-tuning in the ./NER_model/PFT
and update the parameters in ./NER_model/configs
.
cd NER_model
sh train_local.sh
Reference
@inproceedings{zeng2020tri,
title={Tri-Train: Automatic Pre-Fine Tuning between Pre-Training and Fine-Tuning for SciNER},
author={Zeng, Qingkai and Yu, Wenhao and Yu, Mengxia and Jiang, Tianwen and Weninger, Tim and Jiang, Meng},
booktitle={Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings},
pages={4778--4787},
year={2020}
}