nikitakit / self-attentive-parser

High-accuracy NLP parser with models for 11 languages.

https://parser.kitaev.io/

Avoid downloading of nltk 'punkt' tokenizer

duichwer opened this issue 5 years ago · comments

duichwer commented 5 years ago

Shouldn't the parameter preserve_line=True being added to the call of nltk.word_tokenize since there should be only one sentence everytime?

self-attentive-parser/benepar/nltk_plugin.py

Line 89 in 1ee43a8

sentence = nltk.word_tokenize(sentence, self._tokenizer_lang)