Giters
arbox
/
tokenizer
A simple tokenizer in Ruby for NLP tasks.
Geek Repo:
Geek Repo
Github PK Tool:
Github PK Tool
Stargazers:
45
Watchers:
5
Issues:
4
Forks:
11
arbox/tokenizer Issues
Initialization options are ignored and additional pre and post splitters could be utilized
Updated
7 years ago
french words that contains single quote get broken down
Updated
7 years ago
Comments count
1
URLs tokenzing individual characters in the URL
Updated
7 years ago
Comments count
1
Consider using #chunk.
Closed
8 years ago
Comments count
1