doccano / spacy-partial-tagger

A simple library for training named entity recognition model from partially annotated data

spacy-partial-tagger

This is a library to build a CRF tagger for a partially annotated dataset in spaCy. You can build your own NER tagger only from dictionary. The algorithm of this tagger is based on Effland and Collins. (2021).

Overview

Dataset Preparation

Prepare spaCy binary format file to train your tagger. If you are not familiar with spaCy binary format, see this page.

You can prepare your own dataset with spaCy's entity ruler as follows:

import spacy
from spacy.tokens import DocBin


nlp = spacy.blank("en")

patterns = [{"label": "LOC", "pattern": "Tokyo"}, {"label": "LOC", "pattern": "Japan"}]
ruler = nlp.add_pipe("entity_ruler")
ruler.add_patterns(patterns)

doc = nlp("Tokyo is the capital of Japan.")

doc_bin = DocBin()
doc_bin.add(doc)

# Replace /path/to/data.spacy with your own path
doc_bin.to_disk("/path/to/data.spacy")

Training

Train your tagger as follows:

python -m spacy train config.cfg --output outputs --paths.train /path/to/train.spacy --paths.dev /path/to/dev.spacy --gpu-id 0

This library is implemented as a trainable component in spaCy, so you could control the training setting via spaCy's configuration system. We provide you the default configuration file here. Or you could setup your own. If you are not familiar with spaCy's config file format, please check the documentation.

Don't forget to replace /path/to/train.spacy and /path/to/dev.spacy with your own.

Evaluation

Evaluate your tagger as follows:

python -m spacy evaluate outputs/model-best /path/to/test.spacy --gpu-id 0

Don't forget to replace /path/to/test.spacy with your own.

Installation

pip install spacy-partial-tagger

If you use M1 Mac, you might have problems installing fugashi. In that case, please try brew install mecab before the installation.

References

• Thomas Effland and Michael Collins. 2021. Partially Supervised Named Entity Recognition via the Expected Entity Ratio Loss. Transactions of the Association for Computational Linguistics, 9:1320–1335.