StephenWattam / unitok

Fork of Lexical Computing's unitok tokeniser that works with python3

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

unitok

Fork of Lexical Computing's unitok tokeniser that works with python3 and is usable easily as a pip module.

License

Licensed, as per the original, as MPL2. See LICENSE for the full text.

Acknowledgements

Many thanks to lexical computing, Jan Pomikalek, Jan Michelfeit and Vit Suchomel for writing one of the best tokenisers out there.

About

Fork of Lexical Computing's unitok tokeniser that works with python3

License:Mozilla Public License 2.0


Languages

Language:Python 100.0%