Mottl / ru_punkt

Russian language support for NLTK's PunktSentenceTokenizer

ru_punkt

Russian language support for NLTK's PunktSentenceTokenizer

ru_punkt is a part of nltk_data since 2019-07-04

Instalation

1. Install NLTK python package:

pip install nltk

2. Download punkt data:

import nltk
nltk.download('punkt')

Usage

import nltk

text = "Ай да А.С. Пушкин! Ай да сукин сын!"
print("Before:", nltk.sent_tokenize(text))
print("After:", nltk.sent_tokenize(text, language="russian"))

Output:

Before: ['Ай да А.С.', 'Пушкин!', 'Ай да сукин сын!']
After: ['Ай да А.С. Пушкин!', 'Ай да сукин сын!']

Training data

Data for sentence tokenization was taken from 3 sources:
– Articles from Russian Wikipedia (about 1 million sentences);
– Common Russian abbreviations from Russian orthographic dictionary, edited by V. V. Lopatin;
– Generated names initials.

Implementation notes

After some research it was found that the single params.abbrev_types performs better than together with params.collocations and params.ortho_content, so the latter were removed from the trained tokenizer.