deepset-ai / FARM

Question
Hello,

I trained an NER model on my own dataset (about 30k labeled sentences) and the evaluation looks very promising (Test set has about 93% accuracy on 18 different entity types). However, whenever I run inference, i always get an empty predictions list. I tried many examples from my train and test set and never got any prediction.

This is not the case when I use the model that was trained with your ner example. Here inference works fine. I used the debugger to compare the processed data of your example with mine and it looks the same (key is "text" with the correct "ner_label" list).

I was able to replicate the problem by creating a simplified dataset with 25 examples and a single entity type. I overfit the model on this dataset to 100% accuracy and try to run inference on an example from the training data. Here is how the data format looks like:

-DOCSTART-	O
Ich	O
bin	O
7	number
.	O

-DOCSTART-	O
Mein	O
Alter	O
ist	O
12	number
.	O
. . .

When I run:

    basic_texts = [
        {"text": "Mein Alter ist 12 ."},
    ]
    model = Inferencer.load(save_dir)
    result = model.inference_from_dicts(dicts=basic_texts)
   print(result)

I get:

   [{'task': 'ner', 'task_name': 'ner', 'predictions': [[]]}]

The Code I use for training is here.
The only thing that is different from your working example is the dataset, the label_list, the delimiter and num_epochs. I also tried changing the delimiter and renaming the labels to fit to your example but it did not help.

Here is a full dump of the training process that shows correctly looking training samples, 100% accuracy on test set, followed by empty inference prediction on an example from the training set.

Here is the simplified dataset I used to reproduce the problem.

I hope I am not overlooking something obvious here. I you have any idea for what else I could look into to fix this, it would be very much appreciated!

I came across this issue as well a couple months ago when using FARM in my Windows work environment, ended up not using NER because of this

Hi @DieseKartoffel and @FabriceEll I can have a look at this problem but it will take some time(~2 weeks). Maybe this test case helps if you look into it yourselves. The test case uses another model, which also works:

FARM/test/test_conversion.py

Line 156 in 81d1b31

model = "dslim/bert-base-NER"

Could you please try loading the inferencer with inferencer = Inferencer.load(save_dir, task_type="ner", num_processes=0)? If the inferencer is the problem, maybe you can find out whether the storing of the model and loading of the inferencer does work correctly?

Hello Julian,
thank you for looking into this! Hopefully we can find the problem soon. I didn't know about the conversion test so I used the code from there. I also initialized the inferencer with task type and without multiprocessing (tried linux and windows) but never got any predictions.
However I found that the prediction head returns the correct probabilites for my inference example! At the corresponding word positions all_probs contain high values at the correct label index. So they must be lost somewhere along the way from there.

FARM/farm/modeling/prediction_head.py

Line 672 in 81d1b31

return all_probs

EDIT:
I looked further into it and finally found the place where the predictions are lost! The function formatted_preds of the ner prediction head has alle the correct data (correct word spans, correct labels from label_list and their probabilites) in preds_seq, spans_seq, probs_seq.
The data gets passed to convert_iob_to_simple_tags which returns the empty lists. The function contains string comparisons to "B-" and "I-" which I do not use in my dataset.

FARM/farm/modeling/prediction_head.py

Line 708 in 81d1b31

    
           tags, spans_seq, tag_probs = convert_iob_to_simple_tags(preds_seq, spans_seq, probs_seq)

It seems as Farm is currently not compatible with datasets that use their own entity naming convention. I want to try and fix this and get a PR ready if possible. Can you give me some context about what convert_iob_to_simple_tags is supposed to do exactly?

@DieseKartoffel Great insights! You identified the problem. IOB stands for Inside–Outside–Beginning and is a tagging format. FARM uses the IOB2 Format: https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging) B marks the first token (beginning) of an entity, I marks a token within an entity and O is a token outside of an entity. Example:

Alex B-PER
is O
going O
to O
Los B-LOC
Angeles I-LOC
in O
California B-LOC

convert_iob_to_simple_tags converts the IOB2 format to a format without IOB markers (plain text with spans).
By the way, FARM also has a function that can convert IOB to IOB2 format:

FARM/farm/data_handler/utils.py

Line 275 in 81d1b31

def _convertIOB1_to_IOB2(tags: List[str]):

Very much looking forward to your PR!

closed by #828

NER Inference always returns empty list