Feature: Add simple offline tokenizer
ndabAP opened this issue · comments
Basically a different NLP type. Based on white space separator.
Difficulty increased a lot, since a tokenizer must return also the part of speech now.
Possible solution: Offline tokenizer uses dictionary.