Bug in the flat-ner pre-processing
yhcc opened this issue · comments
Hi, I found a bug in your pre-processing code.
I think it should be
span_labels.append((start, len(sentence_tags) -1 , sentence_tags[-1].split("-")[-1]))This bug will cause the number of entities decreases. In conll2003, each split has the following number of entities
train: 23499
dev: 5942
test: 5648
Because of this bug, it will decrease to
train: 22781
dev: 5796
test: 5465
I am not sure whether this will have an effect on the performance. I tried to run the corrected version by the configuration given in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/master/log/en_conll03.txt , but in Telsa V100 (16G memory) it goes OOM. Therefore I have to use 'batch_size=16, gradient_accumulation_steps=2' to simulate, but the performance is far behind the reported one. Is there any trick to restrict the memory used or I need even bigger memory?
BTW, the following are my outputs when I use ''batch_size=16, gradient_accumulation_steps=2'', I delete some of the middle outputs to make it shorter. In conll2003, usually (like LSTM+CRF) the F1 on the development set will be more than 3% higher than the test set (Besides, usually the F1 in the development set of BiLSTM+CRF version will achieve 95.0, but with 91.4 test set F1), and the following results are consistent with this. But in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/master/log/en_conll03.txt , the gap is about 1.4% . Is there any possibility that you may use a corrected version of conll2003 like in those in https://github.com/pfliu-nlp/Named-Entity-Recognition-NER-Papers/tree/master/ner_dataset ?
{
"bert_frozen": "false",
"hidden_size": 768,
"hidden_dropout_prob": 0.2,
"classifier_sign": "multi_nonlinear",
"clip_grad": 1,
"bert_config": {
"attention_probs_dropout_prob": 0.1,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"max_position_embeddings": 512,
"num_attention_heads": 12,
"num_hidden_layers": 12,
"type_vocab_size": 2,
"vocab_size": 28996
},
"config_path": "/path/to/mrc_for_flat_nested_ner/config/en_bert_base_cased.json",
"data_dir": "/home/ubuntu/exps/TransformerNER/data/conll2003/",
"bert_model": "/path/to/bert-base-uncased/",
"task_name": null,
"max_seq_length": 150,
"train_batch_size": 16,
"dev_batch_size": 32,
"test_batch_size": 32,
"checkpoint": 600,
"learning_rate": 3e-05,
"num_train_epochs": 6,
"warmup_proportion": -1.0,
"local_rank": -1,
"gradient_accumulation_steps": 2,
"seed": 2333,
"export_model": false,
"output_dir": ".",
"data_sign": "conll03",
"weight_start": 1.0,
"weight_end": 1.0,
"weight_span": 1.0,
"entity_sign": "flat",
"n_gpu": 1,
"dropout": 0.2,
"entity_threshold": 0.5,
"data_cache": false
}
--------------------
current data_sign: conll03
====================
loading train data ... ...
56164
56164 train data loaded
====================
loading dev data ... ...
13000
13000 dev data loaded
====================
loading test data ... ...
13812
13812 test data loaded
######################################################################
EPOCH: 0
------------------------------
current training loss is :
0.016730422154068947
............................................................
DEV: loss, acc, precision, recall, f1
0.011 0.6385 0.4431 0.117 0.1851
SAVED model path is :
./bert_finetune_model_0_600.bin
............................................................
TEST: loss, acc, precision, recall, f1
0.0104 0.652 0.4137 0.0996 0.1606
------------------------------
------------------------------
current training loss is :
0.002673722803592682
............................................................
DEV: loss, acc, precision, recall, f1
0.0083 0.7126 0.6186 0.4184 0.4992
SAVED model path is :
./bert_finetune_model_0_1200.bin
............................................................
TEST: loss, acc, precision, recall, f1
0.0093 0.7037 0.57 0.4091 0.4763
------------------------------
------------------------------
current training loss is :
0.0007827995577827096
............................................................
............................................................
DEV: loss, acc, precision, recall, f1
0.0049 0.8653 0.8349 0.7296 0.7787
SAVED model path is :
./bert_finetune_model_0_3000.bin
............................................................
TEST: loss, acc, precision, recall, f1
0.0053 0.8561 0.8078 0.7175 0.76
------------------------------
------------------------------
current training loss is :
0.006401577964425087
............................................................
DEV: loss, acc, precision, recall, f1
0.0047 0.7522 0.6744 0.337 0.4494
------------------------------
------------------------------
current training loss is :
0.00014035690401215106
............................................................
............................................................
DEV: loss, acc, precision, recall, f1
0.0031 0.8934 0.8443 0.7814 0.8116
SAVED model path is :
./bert_finetune_model_0_6600.bin
............................................................
TEST: loss, acc, precision, recall, f1
0.0038 0.8791 0.8159 0.7416 0.777
------------------------------
######################################################################
EPOCH: 1
current learning rate 2.8499999999999998e-05
current learning rate 2.8499999999999998e-05
------------------------------
current training loss is :
0.018947187811136246
............................................................
DEV: loss, acc, precision, recall, f1
0.0028 0.8717 0.8382 0.7353 0.7834
------------------------------
------------------------------
current training loss is : [386/1723]
5.050747859058902e-05
............................................................
DEV: loss, acc, precision, recall, f1
0.0026 0.9438 0.9256 0.9055 0.9154
SAVED model path is :
./bert_finetune_model_1_1200.bin
............................................................
TEST: loss, acc, precision, recall, f1
0.0039 0.9084 0.8754 0.8581 0.8667
------------------------------
------------------------------
current training loss is :
9.175582090392709e-05
............................................................
DEV: loss, acc, precision, recall, f1
0.0029 0.9237 0.896 0.8175 0.8549
------------------------------
------------------------------
current training loss is :
0.0015588540118187666
............................................................
DEV: loss, acc, precision, recall, f1
0.0035 0.9103 0.8982 0.8307 0.8631
------------------------------
------------------------------
current training loss is :
0.00026114226784557104
............................................................
DEV: loss, acc, precision, recall, f1
0.0028 0.8382 0.8883 0.5875 0.7072
------------------------------
------------------------------
current training loss is :
0.00016665547445882112
............................................................
DEV: loss, acc, precision, recall, f1
0.0028 0.9406 0.9147 0.8547 0.8837
------------------------------
######################################################################
current learning rate 2.7074999999999997e-05
current learning rate 2.7074999999999997e-05
------------------------------
current training loss is :
0.002201023744419217
............................................................
............................................................
DEV: loss, acc, precision, recall, f1
0.0032 0.9519 0.9355 0.9003 0.9175
SAVED model path is :
./bert_finetune_model_2_2400.bin
............................................................
TEST: loss, acc, precision, recall, f1
0.0043 0.9279 0.9094 0.8656 0.8869
------------------------------
######################################################################
EPOCH: 3
current learning rate 2.5721249999999997e-05
current learning rate 2.5721249999999997e-05
------------------------------
current training loss is :
4.75841952720657e-05
............................................................
DEV: loss, acc, precision, recall, f1
0.0031 0.9186 0.8595 0.8149 0.8366
------------------------------
------------------------------
current training loss is :
1.2278333088033833e-05
............................................................
DEV: loss, acc, precision, recall, f1
0.0028 0.9512 0.9316 0.9067 0.9189
SAVED model path is :
./bert_finetune_model_3_1200.bin
............................................................
TEST: loss, acc, precision, recall, f1
0.004 0.9248 0.9031 0.8724 0.8875
------------------------------
------------------------------
current training loss is :
7.809271482983604e-05
current training loss is :
4.4565200369106606e-05
............................................................
DEV: loss, acc, precision, recall, f1
0.0057 0.9165 0.8801 0.8271 0.8528
------------------------------
------------------------------
current training loss is :
0.0001912666339194402
............................................................
DEV: loss, acc, precision, recall, f1
0.0037 0.9392 0.9026 0.8773 0.8898
------------------------------
------------------------------
current training loss is :
2.610034789540805e-05
............................................................
DEV: loss, acc, precision, recall, f1
0.0036 0.938 0.924 0.8722 0.8973
######################################################################
EPOCH: 4
current learning rate 2.4435187499999995e-05
current learning rate 2.4435187499999995e-05
------------------------------
current training loss is :
6.310163007583469e-05
............................................................
DEV: loss, acc, precision, recall, f1
0.0035 0.9354 0.8781 0.8549 0.8663
------------------------------
------------------------------
current training loss is :
6.9439424805750605e-06
............................................................
TEST: loss, acc, precision, recall, f1
0.0049 0.9332 0.916 0.8687 0.8917
------------------------------
------------------------------
current training loss is :
6.0556856624316424e-05
------------------------------
------------------------------
current training loss is :
4.879345578956418e-05
............................................................
DEV: loss, acc, precision, recall, f1
0.0037 0.9415 0.9147 0.8654 0.8894
------------------------------
######################################################################
EPOCH: 5
current learning rate 2.3213428124999993e-05
current learning rate 2.3213428124999993e-05
------------------------------
current training loss is :
0.00025477795861661434
............................................................
DEV: loss, acc, precision, recall, f1
0.0038 0.9279 0.86 0.8468 0.8533
------------------------------
------------------------------
current training loss is :
9.94959282252239e-06
............................................................
DEV: loss, acc, precision, recall, f1
0.0032 0.9525 0.9208 0.9046 0.9126
............................................................
DEV: loss, acc, precision, recall, f1
0.0029 0.9423 0.8931 0.8811 0.8871
------------------------------
------------------------------
current training loss is :
0.0022768585477024317
............................................................
DEV: loss, acc, precision, recall, f1
0.0035 0.9508 0.9247 0.892 0.9081
-----------------------------*-
=&==&==&==&==&==&==&==&==&==&==&==&==&==&==&=
Best DEV : overall best loss, acc, precision, recall, f1
0.0034 0.9566 0.939 0.9091 0.9238
scores on TEST when Best DEV:loss, acc, precision, recall, f1
0.0049 0.9332 0.916 0.8687 0.8917
=&==&==&==&==&==&==&==&==&==&==&==&==&==&==&=
I add the following line in
to make sure the loop will not stuckidx += 1
Thanks for asking!
I will check the code and preprocessed datasets.
And I will reply later.
Thanks！
Is there any progress for this bug ?
Sorry for the delay.
I am dealing with an emergency.
The final reply will be given by the 16th.
Many many thanks! Hope you all the best.
Thank you for your reply. Sorry for the pressure, the 16th is totally fine.
Any update?
BTW, the following are my outputs when I use ''batch_size=16, gradient_accumulation_steps=2'', I delete some of the middle outputs to make it shorter. In conll2003, usually (like LSTM+CRF) the F1 on the development set will be more than 3% higher than the test set (Besides, usually the F1 in the development set of BiLSTM+CRF version will achieve 95.0, but with 91.4 test set F1), and the following results are consistent with this. But in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/master/log/en_conll03.txt , the gap is about 1.4% . Is there any possibility that you may use a corrected version of conll2003 like in those in https://github.com/pfliu-nlp/Named-Entity-Recognition-NER-Papers/tree/master/ner_dataset ?
{
"bert_frozen": "false",
"hidden_size": 768,
"hidden_dropout_prob": 0.2,
"classifier_sign": "multi_nonlinear",
"clip_grad": 1,
"bert_config": {
"attention_probs_dropout_prob": 0.1,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"max_position_embeddings": 512,
"num_attention_heads": 12,
"num_hidden_layers": 12,
"type_vocab_size": 2,
"vocab_size": 28996
},
"config_path": "/path/to/mrc_for_flat_nested_ner/config/en_bert_base_cased.json",
"data_dir": "/home/ubuntu/exps/TransformerNER/data/conll2003/",
"bert_model": "/path/to/bert-base-uncased/",
"task_name": null,
"max_seq_length": 150,
"train_batch_size": 16,
"dev_batch_size": 32,
"test_batch_size": 32,
"checkpoint": 600,
"learning_rate": 3e-05,
"num_train_epochs": 6,
"warmup_proportion": -1.0,
"local_rank": -1,
"gradient_accumulation_steps": 2,
"seed": 2333,
"export_model": false,
"output_dir": ".",
"data_sign": "conll03",
"weight_start": 1.0,
"weight_end": 1.0,
"weight_span": 1.0,
"entity_sign": "flat",
"n_gpu": 1,
"dropout": 0.2,
"entity_threshold": 0.5,
"data_cache": false
}
--------------------
current data_sign: conll03
====================
loading train data ... ...
56164
56164 train data loaded
====================
loading dev data ... ...
13000
13000 dev data loaded
====================
loading test data ... ...
13812
13812 test data loaded
######################################################################
EPOCH: 0
------------------------------ current training loss is : 0.016730422154068947 ............................................................ DEV: loss, acc, precision, recall, f1 0.011 0.6385 0.4431 0.117 0.1851 SAVED model path is : ./bert_finetune_model_0_600.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0104 0.652 0.4137 0.0996 0.1606 ------------------------------
------------------------------ current training loss is : 0.002673722803592682 ............................................................ DEV: loss, acc, precision, recall, f1 0.0083 0.7126 0.6186 0.4184 0.4992 SAVED model path is : ./bert_finetune_model_0_1200.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0093 0.7037 0.57 0.4091 0.4763 ------------------------------
------------------------------ current training loss is : 0.0007827995577827096 ............................................................ ............................................................ DEV: loss, acc, precision, recall, f1 0.0049 0.8653 0.8349 0.7296 0.7787 SAVED model path is : ./bert_finetune_model_0_3000.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0053 0.8561 0.8078 0.7175 0.76 ------------------------------
------------------------------ current training loss is : 0.006401577964425087 ............................................................ DEV: loss, acc, precision, recall, f1 0.0047 0.7522 0.6744 0.337 0.4494 ------------------------------
------------------------------ current training loss is : 0.00014035690401215106 ............................................................ ............................................................ DEV: loss, acc, precision, recall, f1 0.0031 0.8934 0.8443 0.7814 0.8116 SAVED model path is : ./bert_finetune_model_0_6600.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0038 0.8791 0.8159 0.7416 0.777 ------------------------------
######################################################################
EPOCH: 1
current learning rate 2.8499999999999998e-05
current learning rate 2.8499999999999998e-05
------------------------------ current training loss is : 0.018947187811136246 ............................................................ DEV: loss, acc, precision, recall, f1 0.0028 0.8717 0.8382 0.7353 0.7834 ------------------------------
------------------------------ current training loss is : [386/1723] 5.050747859058902e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0026 0.9438 0.9256 0.9055 0.9154 SAVED model path is : ./bert_finetune_model_1_1200.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0039 0.9084 0.8754 0.8581 0.8667 ------------------------------
------------------------------ current training loss is : 9.175582090392709e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0029 0.9237 0.896 0.8175 0.8549 ------------------------------
------------------------------ current training loss is : 0.0015588540118187666 ............................................................ DEV: loss, acc, precision, recall, f1 0.0035 0.9103 0.8982 0.8307 0.8631 ------------------------------
------------------------------ current training loss is : 0.00026114226784557104 ............................................................ DEV: loss, acc, precision, recall, f1 0.0028 0.8382 0.8883 0.5875 0.7072 ------------------------------
------------------------------ current training loss is : 0.00016665547445882112 ............................................................ DEV: loss, acc, precision, recall, f1 0.0028 0.9406 0.9147 0.8547 0.8837 ------------------------------
######################################################################
current learning rate 2.7074999999999997e-05
current learning rate 2.7074999999999997e-05
------------------------------ current training loss is : 0.002201023744419217 ............................................................ ............................................................ DEV: loss, acc, precision, recall, f1 0.0032 0.9519 0.9355 0.9003 0.9175 SAVED model path is : ./bert_finetune_model_2_2400.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0043 0.9279 0.9094 0.8656 0.8869 ------------------------------
######################################################################
EPOCH: 3
current learning rate 2.5721249999999997e-05
current learning rate 2.5721249999999997e-05
------------------------------ current training loss is : 4.75841952720657e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0031 0.9186 0.8595 0.8149 0.8366 ------------------------------
------------------------------ current training loss is : 1.2278333088033833e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0028 0.9512 0.9316 0.9067 0.9189 SAVED model path is : ./bert_finetune_model_3_1200.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.004 0.9248 0.9031 0.8724 0.8875 ------------------------------
------------------------------ current training loss is : 7.809271482983604e-05 current training loss is : 4.4565200369106606e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0057 0.9165 0.8801 0.8271 0.8528 ------------------------------
------------------------------ current training loss is : 0.0001912666339194402 ............................................................ DEV: loss, acc, precision, recall, f1 0.0037 0.9392 0.9026 0.8773 0.8898 ------------------------------
------------------------------ current training loss is : 2.610034789540805e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0036 0.938 0.924 0.8722 0.8973 ###################################################################### EPOCH: 4 current learning rate 2.4435187499999995e-05 current learning rate 2.4435187499999995e-05 ------------------------------
current training loss is :
6.310163007583469e-05
............................................................
DEV: loss, acc, precision, recall, f1
0.0035 0.9354 0.8781 0.8549 0.8663
------------------------------ ------------------------------
current training loss is :
6.9439424805750605e-06
............................................................
TEST: loss, acc, precision, recall, f1
0.0049 0.9332 0.916 0.8687 0.8917
------------------------------ ------------------------------
current training loss is :
6.0556856624316424e-05
------------------------------ ------------------------------
current training loss is :
4.879345578956418e-05
............................................................
DEV: loss, acc, precision, recall, f1
0.0037 0.9415 0.9147 0.8654 0.8894
------------------------------ ###################################################################### EPOCH: 5 current learning rate 2.3213428124999993e-05 current learning rate 2.3213428124999993e-05 ------------------------------
current training loss is :
0.00025477795861661434
............................................................
DEV: loss, acc, precision, recall, f1
0.0038 0.9279 0.86 0.8468 0.8533
------------------------------ ------------------------------
current training loss is :
9.94959282252239e-06
............................................................
DEV: loss, acc, precision, recall, f1
0.0032 0.9525 0.9208 0.9046 0.9126
............................................................
DEV: loss, acc, precision, recall, f1
0.0029 0.9423 0.8931 0.8811 0.8871
------------------------------ ------------------------------
current training loss is :
0.0022768585477024317
............................................................
DEV: loss, acc, precision, recall, f1
0.0035 0.9508 0.9247 0.892 0.9081
-----------------------------*-
=&==&==&==&==&==&==&==&==&==&==&==&==&==&==&=
Best DEV : overall best loss, acc, precision, recall, f1
0.0034 0.9566 0.939 0.9091 0.9238
scores on TEST when Best DEV:loss, acc, precision, recall, f1
0.0049 0.9332 0.916 0.8687 0.8917
=&==&==&==&==&==&==&==&==&==&==&==&==&==&==&=I add the following line in
to make sure the loop will not stuck
idx += 1
@yhcc What you mean by the corrected version of the ConLL2003 dataset?
@kamalkraj Since the original conll2003 data has some wrong annotation, some papers (as some listed in https://github.com/pfliu-nlp/Named-Entity-Recognition-NER-Papers/tree/master/ner_dataset ) have tried to revise these annotation errors, and released the revised dataset. If the revised data is used, usually the performance will be higher than previous work.
@yhcc Thanks. Are you able to reproduce the results for the ConLL2003 dataset?
No, I tried to use the code in this repo(revised the bug as post here), and get the results as I have post before. I also tried to rewrite this code from scratch, and got similar results as I post. And my log (based on their code) is quite different from their optimization log in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/master/log/en_conll03.txt , and I am not sure whether I did something wrong or the bug caused the difference.
We will get back in one day or two. Sincerely sorry for the delay. We were busy with the corefQA release, the other repo.
Looking forward to your reply
BTW, the following are my outputs when I use ''batch_size=16, gradient_accumulation_steps=2'', I delete some of the middle outputs to make it shorter. In conll2003, usually (like LSTM+CRF) the F1 on the development set will be more than 3% higher than the test set (Besides, usually the F1 in the development set of BiLSTM+CRF version will achieve 95.0, but with 91.4 test set F1), and the following results are consistent with this. But in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/master/log/en_conll03.txt , the gap is about 1.4% . Is there any possibility that you may use a corrected version of conll2003 like in those in https://github.com/pfliu-nlp/Named-Entity-Recognition-NER-Papers/tree/master/ner_dataset ?
{
"bert_frozen": "false",
"hidden_size": 768,
"hidden_dropout_prob": 0.2,
"classifier_sign": "multi_nonlinear",
"clip_grad": 1,
"bert_config": {
"attention_probs_dropout_prob": 0.1,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"initializer_range": 0.02,
"intermediate_size": 3072,
"max_position_embeddings": 512,
"num_attention_heads": 12,
"num_hidden_layers": 12,
"type_vocab_size": 2,
"vocab_size": 28996
},
"config_path": "/path/to/mrc_for_flat_nested_ner/config/en_bert_base_cased.json",
"data_dir": "/home/ubuntu/exps/TransformerNER/data/conll2003/",
"bert_model": "/path/to/bert-base-uncased/",
"task_name": null,
"max_seq_length": 150,
"train_batch_size": 16,
"dev_batch_size": 32,
"test_batch_size": 32,
"checkpoint": 600,
"learning_rate": 3e-05,
"num_train_epochs": 6,
"warmup_proportion": -1.0,
"local_rank": -1,
"gradient_accumulation_steps": 2,
"seed": 2333,
"export_model": false,
"output_dir": ".",
"data_sign": "conll03",
"weight_start": 1.0,
"weight_end": 1.0,
"weight_span": 1.0,
"entity_sign": "flat",
"n_gpu": 1,
"dropout": 0.2,
"entity_threshold": 0.5,
"data_cache": false
}
--------------------
current data_sign: conll03
====================
loading train data ... ...
56164
56164 train data loaded
====================
loading dev data ... ...
13000
13000 dev data loaded
====================
loading test data ... ...
13812
13812 test data loaded
######################################################################
EPOCH: 0
------------------------------ current training loss is : 0.016730422154068947 ............................................................ DEV: loss, acc, precision, recall, f1 0.011 0.6385 0.4431 0.117 0.1851 SAVED model path is : ./bert_finetune_model_0_600.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0104 0.652 0.4137 0.0996 0.1606 ------------------------------
------------------------------ current training loss is : 0.002673722803592682 ............................................................ DEV: loss, acc, precision, recall, f1 0.0083 0.7126 0.6186 0.4184 0.4992 SAVED model path is : ./bert_finetune_model_0_1200.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0093 0.7037 0.57 0.4091 0.4763 ------------------------------
------------------------------ current training loss is : 0.0007827995577827096 ............................................................ ............................................................ DEV: loss, acc, precision, recall, f1 0.0049 0.8653 0.8349 0.7296 0.7787 SAVED model path is : ./bert_finetune_model_0_3000.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0053 0.8561 0.8078 0.7175 0.76 ------------------------------
------------------------------ current training loss is : 0.006401577964425087 ............................................................ DEV: loss, acc, precision, recall, f1 0.0047 0.7522 0.6744 0.337 0.4494 ------------------------------
------------------------------ current training loss is : 0.00014035690401215106 ............................................................ ............................................................ DEV: loss, acc, precision, recall, f1 0.0031 0.8934 0.8443 0.7814 0.8116 SAVED model path is : ./bert_finetune_model_0_6600.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0038 0.8791 0.8159 0.7416 0.777 ------------------------------
######################################################################
EPOCH: 1
current learning rate 2.8499999999999998e-05
current learning rate 2.8499999999999998e-05
------------------------------ current training loss is : 0.018947187811136246 ............................................................ DEV: loss, acc, precision, recall, f1 0.0028 0.8717 0.8382 0.7353 0.7834 ------------------------------
------------------------------ current training loss is : [386/1723] 5.050747859058902e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0026 0.9438 0.9256 0.9055 0.9154 SAVED model path is : ./bert_finetune_model_1_1200.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0039 0.9084 0.8754 0.8581 0.8667 ------------------------------
------------------------------ current training loss is : 9.175582090392709e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0029 0.9237 0.896 0.8175 0.8549 ------------------------------
------------------------------ current training loss is : 0.0015588540118187666 ............................................................ DEV: loss, acc, precision, recall, f1 0.0035 0.9103 0.8982 0.8307 0.8631 ------------------------------
------------------------------ current training loss is : 0.00026114226784557104 ............................................................ DEV: loss, acc, precision, recall, f1 0.0028 0.8382 0.8883 0.5875 0.7072 ------------------------------
------------------------------ current training loss is : 0.00016665547445882112 ............................................................ DEV: loss, acc, precision, recall, f1 0.0028 0.9406 0.9147 0.8547 0.8837 ------------------------------
######################################################################
current learning rate 2.7074999999999997e-05
current learning rate 2.7074999999999997e-05
------------------------------ current training loss is : 0.002201023744419217 ............................................................ ............................................................ DEV: loss, acc, precision, recall, f1 0.0032 0.9519 0.9355 0.9003 0.9175 SAVED model path is : ./bert_finetune_model_2_2400.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.0043 0.9279 0.9094 0.8656 0.8869 ------------------------------
######################################################################
EPOCH: 3
current learning rate 2.5721249999999997e-05
current learning rate 2.5721249999999997e-05
------------------------------ current training loss is : 4.75841952720657e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0031 0.9186 0.8595 0.8149 0.8366 ------------------------------
------------------------------ current training loss is : 1.2278333088033833e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0028 0.9512 0.9316 0.9067 0.9189 SAVED model path is : ./bert_finetune_model_3_1200.bin ............................................................ TEST: loss, acc, precision, recall, f1 0.004 0.9248 0.9031 0.8724 0.8875 ------------------------------
------------------------------ current training loss is : 7.809271482983604e-05 current training loss is : 4.4565200369106606e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0057 0.9165 0.8801 0.8271 0.8528 ------------------------------
------------------------------ current training loss is : 0.0001912666339194402 ............................................................ DEV: loss, acc, precision, recall, f1 0.0037 0.9392 0.9026 0.8773 0.8898 ------------------------------
------------------------------ current training loss is : 2.610034789540805e-05 ............................................................ DEV: loss, acc, precision, recall, f1 0.0036 0.938 0.924 0.8722 0.8973 ###################################################################### EPOCH: 4 current learning rate 2.4435187499999995e-05 current learning rate 2.4435187499999995e-05 ------------------------------
current training loss is :
6.310163007583469e-05
............................................................
DEV: loss, acc, precision, recall, f1
0.0035 0.9354 0.8781 0.8549 0.8663
------------------------------ ------------------------------
current training loss is :
6.9439424805750605e-06
............................................................
TEST: loss, acc, precision, recall, f1
0.0049 0.9332 0.916 0.8687 0.8917
------------------------------ ------------------------------
current training loss is :
6.0556856624316424e-05
------------------------------ ------------------------------
current training loss is :
4.879345578956418e-05
............................................................
DEV: loss, acc, precision, recall, f1
0.0037 0.9415 0.9147 0.8654 0.8894
------------------------------ ###################################################################### EPOCH: 5 current learning rate 2.3213428124999993e-05 current learning rate 2.3213428124999993e-05 ------------------------------
current training loss is :
0.00025477795861661434
............................................................
DEV: loss, acc, precision, recall, f1
0.0038 0.9279 0.86 0.8468 0.8533
------------------------------ ------------------------------
current training loss is :
9.94959282252239e-06
............................................................
DEV: loss, acc, precision, recall, f1
0.0032 0.9525 0.9208 0.9046 0.9126
............................................................
DEV: loss, acc, precision, recall, f1
0.0029 0.9423 0.8931 0.8811 0.8871
------------------------------ ------------------------------
current training loss is :
0.0022768585477024317
............................................................
DEV: loss, acc, precision, recall, f1
0.0035 0.9508 0.9247 0.892 0.9081
-----------------------------*-
=&==&==&==&==&==&==&==&==&==&==&==&==&==&==&=
Best DEV : overall best loss, acc, precision, recall, f1
0.0034 0.9566 0.939 0.9091 0.9238
scores on TEST when Best DEV:loss, acc, precision, recall, f1
0.0049 0.9332 0.916 0.8687 0.8917
=&==&==&==&==&==&==&==&==&==&==&==&==&==&==&=I add the following line in
to make sure the loop will not stuck
idx += 1
Thanks for pointing out my mistake.
Yes, this is a bug introduced when I was trying to clean my codebase.
I fixed it in the commit (f80ed26). Please pull the latest repo.
Hi, I found a bug in your pre-processing code.
I think it should be
span_labels.append((start, len(sentence_tags) -1 , sentence_tags[-1].split("-")[-1]))This bug will cause the number of entities decreases. In conll2003, each split has the following number of entities
train: 23499
dev: 5942
test: 5648
Because of this bug, it will decrease to
train: 22781
dev: 5796
test: 5465I am not sure whether this will have an effect on the performance. I tried to run the corrected version by the configuration given in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/master/log/en_conll03.txt , but in Telsa V100 (16G memory) it goes OOM. Therefore I have to use 'batch_size=16, gradient_accumulation_steps=2' to simulate, but the performance is far behind the reported one. Is there any trick to restrict the memory used or I need even bigger memory?
Thanks for pointing out my mistake.
Yes, this is a bug introduced when I was trying to clean my codebase.
I fixed it in the commit (f80ed26).
No, I tried to use the code in this repo(revised the bug as post here), and get the results as I have post before. I also tried to rewrite this code from scratch, and got similar results as I post. And my log (based on their code) is quite different from their optimization log in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/master/log/en_conll03.txt , and I am not sure whether I did something wrong or the bug caused the difference.
We used the official released CoNLL-2003 train/dev/test sets for experiments.
No, I tried to use the code in this repo(revised the bug as post here), and get the results as I have post before. I also tried to rewrite this code from scratch, and got similar results as I post. And my log (based on their code) is quite different from their optimization log in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/master/log/en_conll03.txt , and I am not sure whether I did something wrong or the bug caused the difference.
Since we update the codebase, I run the experiments with the code (f80ed26) and will update the log for CoNLL-2003 tomorrow. Many thanks.
I replicated the experiment result on English CoNLL03 with the latest code (f80ed26).
And configurations can be found in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/master/log/en_conll03.txt.
Please contact me if you have any further questions.
Many thanks and sorry for the late reply.
Thanks for your reply. BERT-large can achieve 93+ makes sense.
Did the results in the English Ontonotes also use the BERT-large model ?
Did the results in the English Ontonotes also use the BERT-large model ?
Sorry for the late reply.
Yes. The result of the English Ontonotes is obtained by finetuning the BERT-Large model.
I replicated the experiment result on English CoNLL03 with the latest code (f80ed26).
And configurations can be found in https://github.com/ShannonAI/mrc-for-flat-nested-ner/blob/master/log/en_conll03.txt.Please contact me if you have any further questions.
Many thanks and sorry for the late reply.
I was not able to reproduce the results reported in the ACL paper for the CoNLL 2003. Would it be possible to share the reproduce script for this dataset also? Thanks.