JingYannn / CatastrophicForgetting-BERT

CF problem in BERT

CatastrophicForgetting-BERT

CF problem in BERT. Support contiual fine-tuning BERT on sequence classification tasks(ColA, MRPC and RTE) and token classification tasks(CoNLL-2003, UD).

Compare the prediction results of the first task after continual fine-tuning

Experiment1 : NER(10000) -> CoLA -> MRPC

Accuracy Matrix

CoNLL-2003	CoLA	MRPC
0.9883139	-0.0207027	0.3112745
0.9802894	0.5550197	0.4338235
0.9614164	0.3715264	0.8578431

Transfer Matrix

CoNLL-2003	CoLA	MRPC
0.	-0.5757224	-0.5465686
-0.0080245	0.	-0.4240196
-0.0268975	-0.1834933	0.

Explanation:

CoNLL-2003 's label lists: "O", "B-MISC", "I-MISC", "B-PER", "I-PER", "B-ORG", "I-ORG", "B-LOC", "I-LOC";

Frequency of each label in dev set: [42746, 922, 346, 1839, 1304, 1341, 751, 1837, 257]

avg. sequence length = 15.797846153846153, select the first 10000 sequences from the train set to do fine-tuning

• cnt1 : how many times is the label correctly predicted in the first time? [42568, 836, 286, 1812, 1287, 1258, 682, 1781, 233]

• cnt2 : how many times is the label correctly predicted in the second time? [42537, 712, 240, 1176, 1168, 1144, 670, 1516, 199]

• cnt3 : how many times is the label correctly predicted in the first time but uncorrectly in the second time? [107, 140, 50, 637, 120, 119, 28, 267, 34], [0.00, 0.15, 0.14, 0.35, 0.09, 0.09, 0.04, 0.15, 0.13]

• cnt4 : how many times is the label correctly predicted in the second time but uncorrectly in the first time? [76, 16, 4, 1, 1, 5, 16, 2, 0]

Experiment2: CoLA -> MRPC -> NER

Accuracy Matrix

CoLA	MRPC	CoNLL-2003
0.5699305	0.6838235	0.2488363
0.5528801	0.8455882	0.3384493
0.3818496	0.7647059	0.9872816

Transfer Matrix

CoLA	MRPC	CoNLL-2003
0.	-0.1617647	-0.7384453
-0.0170504	0.	-0.6488323
-0.1880809	-0.0808823	0.

CoLA's label list: label 0(ungrammatical), label 1(grammatical)

In CoLA's dev set, there are 1043 samples, in which 322 with label 0 and 721 with label 1.

• cnt1 : sequences predicted correctly in the first time => 860

• cnt2 : sequences predicted correctly in the second time => 789

• cnt3 : sequences predicted correctly in the first time but predicted uncorrectly in the second time (forget) => 118

  • cnt3_0 : how many times the label '0' are forgot? => 117, 0.36
  • cnt3_1 : how many times the label '1' are forgot? => 1, 0.01

• cnt4 : sequences predicted correctly in the second time but predicted uncorrectly in the first time => 47

  • cnt4_0 : how many times is label '0' are learned? => 0
  • cnt4_1 : how many times is label '1' are learned? => 47, 0.07

Experiment3: CoLA -> MRPC -> UD(POS)

Accuracy Matrix

CoLA	MRPC	UD
0.578641	0.6838235	0.0393673
0.5548848	0.8235294	0.0428605
0.314600	0.7328431	0.9685219

Transfer Matrix

CoLA	MRPC	UD
0.	-0.1397059	-0.9291546
-0.0237568	0.	-0.9256614
-0.2640415	-0.0906863	0.

CoLA's label list: label 0(ungrammatical), label 1(grammatical)

In CoLA's dev set, there are 1043 samples, in which 322 with label 0 and 721 with label 1.

• cnt1 : sequences predicted correctly in the first time => 864

• cnt2 : sequences predicted correctly in the second time => 772

• cnt3 : sequences predicted correctly in the first time but predicted uncorrectly in the second time (forget) => 139

  • cnt3_0 : how many times the label '0' are forgot? => 119, 0.37
  • cnt3_1 : how many times the label '1' are forgot? => 20, 0.03

• cnt4 : sequences predicted correctly in the second time but predicted uncorrectly in the first time => 47

  • cnt4_0 : how many times is label '0' are learned? => 11, 0.03
  • cnt4_1 : how many times is label '1' are learned? => 36, 0.05

Experiment4 : NER -> MNLI

Result Matrix

CoNLL-2003 (F1)	MNLI (ACC)
0.9473067	0.3552725
0.6580307	0.8366786

Transfer Matrix

CoNLL-2003 (F1)	MNLI (ACC)
0.	-0.4814061
-0.289276	0.

Explanation:

CoNLL-2003 's label lists: "O", "B-MISC", "I-MISC", "B-PER", "I-PER", "B-ORG", "I-ORG", "B-LOC", "I-LOC";

Frequency of each label in dev set: [42746, 922, 346, 1839, 1304, 1341, 751, 1837, 257]

avg. sequence length = 15.797846153846153, select all sequences from the train set to do fine-tuning

• cnt1 : how many times is the label correctly predicted in the first time? [42589, 834, 291, 1812, 1289, 1257, 690, 1786, 236]

• cnt2 : how many times is the label correctly predicted in the second time? [39261, 743, 273, 1755, 1281, 1139, 586, 1701, 101]

• cnt3 : how many times is the label correctly predicted in the first time but uncorrectly in the second time? [3356, 117, 34, 67, 13, 132, 116, 105, 136], [0.0785, 0.1269, 0.0983, 0.0364, 0.01, 0.0984, 0.1545, 0.0572, 0.5292]

• cnt4 : how many times is the label correctly predicted in the second time but uncorrectly in the first time? [28, 26, 16, 10, 5, 14, 12, 20, 1]

Package

• python 3.6.9
• PyTorch 1.7.0+cu101
• Transformers 4.1.0.dev0
• conllu 4.2.1
• seqeval 1.2.2
• tqdm 4.41.1

Code

• run.py: the main code to run the training-evaluation code.

• processors.py: the code to process text data.

• utils.py: some code including formating time and converting arguments to dict.

• arguments.py: some code to parse arguments.

Run the Code

Please use run_cl.sh to fine-tune BERT on several tasks sequentially. The order of tasks and the hyperparameters setting of each task are defined in order1.json. The arguments are introduced in arguments.py.

TODO