fnl / syntok

Text tokenization and sentence segmentation (segtok v2)

fnl/syntok Issues

German ordinal numbers lead to over splitting
Updated 2 years ago8
Missed abbreviations (Min., Sen.)
Closed 2 years ago1
(Remove?) Support for Windows builds
Updated 2 years ago1
Undersplitting sentence ending in URL
Updated 2 years ago
Under-splitting on "set." and "ago."
Closed 2 years ago
Oversplitting on "No. X" abbreviations
Closed 2 years ago1
Parenthesis at the end of input cause IndexError
Closed 2 years ago4
git tag for 1.2.2
Closed 5 years ago1
Splitting on single \n for sentence tokenization
Closed 4 years ago2
Issues with paragraph identification
Closed 5 years ago2
git tag for 1.3.1
Closed 4 years ago1
Do not segment inside parenthesis
Closed 3 years ago2
Bug in not-contraction handling code
Closed 3 years ago5
Segmenting sentences at colons
Updated 4 years ago6
Benchmark against pragmatic segmenter
Updated 4 years ago4
Uppercase letters in tokens
Closed 5 years ago4
Best way to get sentence spans
Updated 5 years ago4
Wrong offset with nonword-prefix
Closed 5 years ago2
There's no direct way of getting offset of a sentence w.r.t. a document.
Closed 5 years ago1
A large diff between syntok and segtok on Web of Science datasets
Closed 5 years ago3
Splitting two words joined by ellipsis (...) and no spaces into different tokens
Closed 5 years ago2