Giters

Riccorl / transformer-srl

Reimplementation of a BERT based model (Shi et al, 2019), currently the state-of-the-art for English SRL. This model implements also predicate disambiguation.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Pre-trained model for allennlp >= 2.0

Suneal opened this issue · comments

commented

The current pre-trained model uses v2.4.6, which requires allennlp<1.3,>=1.2.

Has anybody successfully trained the model for the new version of allennnlp? Would really appreciate the help.

commented

@Suneal I remember encountering similar errors last time I tried.

@Riccorl would it be possible to adapt the model to Huggingface hub: https://huggingface.co/models
That would ease training and inference a lot! Thanks!

commented

Hi @Riccorl,
I obtained the LDC dataset and used this blog to process and obtain *.gold_conll from *.gold_skel files.

I was able to train the model using bert_base_span.jsonnet config, and the latest version of transformer_srl.

However, after even after 14 epoch, I do not get 86% F1 for argument labeling. I mention this because I clearly have some examples where the srl_bert_base_conll2012.tar.gz model you provided works but mine fails.

Do you have any idea what might have gone wrong while training using new version of transformer_srl? Would appreciate the help.

This is the result I have right now. Thank you!


{
  "best_epoch": 13,
  "peak_worker_0_memory_MB": 7488.73828125,
  "peak_gpu_0_memory_MB": 10836.53125,
  "training_duration": "8:30:54.848019",
  "training_start_epoch": 0,
  "training_epochs": 13,
  "epoch": 13,
  "training_precision_role": 0.0,
  "training_recall_role": 0.0,
  "training_f1_role": 0.0,
  "training_precision_frame": 0.999747097492218,
  "training_recall_frame": 0.999747097492218,
  "training_fscore_frame": 0.999747097492218,
  "training_loss": 0.014607782365404236,
  "training_worker_0_memory_MB": 7488.73828125,
  "training_gpu_0_memory_MB": 10836.53125,
  "validation_precision_role": 0.8351303971951395,
  "validation_recall_role": 0.8383515741767513,
  "validation_f1_role": 0.8367378855684983,
  "validation_precision_frame": 0.9546420574188232,
  "validation_recall_frame": 0.9546420574188232,
  "validation_fscore_frame": 0.9546419978141785,
  "validation_loss": 0.44069189061090164,
  "best_validation_precision_role": 0.8351303971951395,
  "best_validation_recall_role": 0.8383515741767513,
  "best_validation_f1_role": 0.8367378855684983,
  "best_validation_precision_frame": 0.9546420574188232,
  "best_validation_recall_frame": 0.9546420574188232,
  "best_validation_fscore_frame": 0.9546419978141785,
  "best_validation_loss": 0.44069189061090164
}
commented

Also, I noticed that for the new model, there are issues with handling sentence with repeated verb. Here's an example for the sentence "I can eat an apple but I will not eat an orange." The SRL result completely messes up with the Verb tag.

{ "verbs": [ { "verb": "can", "description": "[ARG0: I] [ARGM-MOD: can] [ARG1: eat an apple] but I will not eat an orange .", "tags": [ "B-ARG0", "B-ARGM-MOD", "B-ARG1", "I-ARG1", "I-ARG1", "O", "O", "O", "O", "O", "O", "O", "O" ], "frame": "can.01", "frame_score": 0.26360154151916504, "lemma": "can" }, { "verb": "eat", "description": "I can eat an apple but I [ARGM-MOD: will] [ARGM-NEG: not] eat an orange .", "tags": [ "O", "O", "O", "O", "O", "O", "O", "B-ARGM-MOD", "B-ARGM-NEG", "O", "O", "O", "O" ], "frame": "eat.01", "frame_score": 0.9999997615814209, "lemma": "eat" }, { "verb": "will", "description": "I can eat an apple but I [go.04: will] not eat an orange .", "tags": [ "O", "O", "O", "O", "O", "O", "O", "B-V", "O", "O", "O", "O", "O" ], "frame": "go.04", "frame_score": 0.3325953483581543, "lemma": "will" }, { "verb": "eat", "description": "I [ARGM-MOD: can] eat an apple but [ARG0: I] [ARGM-MOD: will] [ARGM-NEG: not] eat an orange .", "tags": [ "O", "B-ARGM-MOD", "O", "O", "O", "O", "B-ARG0", "B-ARGM-MOD", "B-ARGM-NEG", "O", "O", "O", "O" ], "frame": "eat.01", "frame_score": 0.9999984502792358, "lemma": "eat" } ], "words": [ "I", "can", "eat", "an", "apple", "but", "I", "will", "not", "eat", "an", "orange", "." ] }

I am not sure if this is a bug for the latest version of tranformer_srl or the newly trained model.

commented

I was able to train the model by reverting to 2.4.6 and making few changes to make the code compatible with the new allennlp.

commented

@Suneal What F1 score did you achieve? Again 86%?
Again, would it be possible to adapt the model to (make it compatible with) Huggingface hub using their implementation of transformers: https://huggingface.co/models

commented

@logicReasoner I will have to look into it. Currently, I am using a model for a completely different task. I will definitely let you know if I get to making it compatible with huggingface.

commented