winkjs / wink-tokenizer

Multilingual tokenizer that automatically tags each token with its type

http://winkjs.org/wink-tokenizer/

winkjs/wink-tokenizer Issues

Incorrect tokenization of words with a hyphen like "time-consuming"
Closed 2 years ago1
Function slows down when I use more than 2 custom regex expressions for tagging
Closed 2 years ago2
Unexpected output in custom regex
Closed 4 years ago2
RangeError: Maximum call stack size exceeded in wink-tokenizer.js
Closed 4 years ago1
Incorrect handling of composite emojis
Closed 4 years ago7
question on sanskrit
Closed 5 years ago5
Add `startsAt` and `endsAt` to the result
Closed 5 years ago2
Allow non-http url match
Closed 5 years ago1
Incorrect tokenization of don't"
Closed 5 years ago1
Incorrect tokenization of don't
Closed 5 years ago1
Words in double quotes are not tokenized
Closed 5 years ago3
detect ordinal as a separately
Closed 6 years ago
improve number tokenization
Closed 6 years ago
update dev dependencies
Closed 6 years ago
add support for devanagari script
Closed 6 years ago
handle CamelCase in #hashtags
Closed 6 years ago
add multilingual support
Closed 6 years ago
use symbol char instead of "s" in finger printing
Closed 6 years ago
tokenize special characters as "symbol"
Closed 6 years ago
tokenize "&" as word
Closed 6 years ago1
change "homepage" in package.json to winkjs.org
Closed 6 years ago1
update copyright notice year in License Header
Closed 6 years ago
add more currency symbols
Closed 6 years ago
add support for tokenizing words like Zoës, Nestlé and Résumé
Closed 6 years ago1