tokenizer

There are 8 repositories under tokenizer topic.

theseer / tokenizer
A small library for converting tokenized PHP source code into XML (and potentially other formats)
php tokenizer xml
Language:PHP 5113
Chevrotain / chevrotain
Parser Building Toolkit for JavaScript
grammars javascript lexer open-source parser-library parsing tokenizer typescript
Language:TypeScript 2387
natasha
natasha / natasha
Solves basic Russian NLP tasks, API for lower level Natasha projects
embeddings morphology ner nlp python russian sentence-segmentation syntax tokenizer visualization
Language:Python 1148
roshan-research / hazm
Persian NLP Toolkit
dependency-parser embeddings farsi lemmatization natural-language-processing nlp normalization persian persian-nlp pos-tagging python text-processing tokenizer
Language:Python 1095
lovit / soynlp
한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.
korean-nlp korean-text-processing nlp postagging tokenizer word-extraction
Language:Python 900
no-context / moo
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
javascript lexer regexp tokenizer
Language:JavaScript 800
ikawaha / kagome
Self-contained Japanese Morphological Analyzer written in pure Go
hacktoberfest japanese japanese-language korean morphological-analysis nlp-library pos-tagging segmentation tokenizer
Language:Go 789
mathewsanders / Mustard
🌭 Mustard is a Swift library for tokenizing strings when splitting by whitespace doesn't cut it.
substrings swift tokenizer
Language:Swift 690
BLKSerene / Wordless
An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation
corpus corpus-analysis corpus-statistics corpus-linguistics corpus-tools corpus-processing literature translation tokenizer tagger lemmatizer stopword corpus-search dependency-parser linguistics
Language:Python 665
cbaziotis / ekphrasis
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
nlp nlp-library semeval spell-corrector spelling-correction text-processing text-segmentation tokenization tokenizer word-normalization word-segmentation
Language:Python 656
SmoothNLP
smoothnlp / SmoothNLP
专注于可解释的NLP技术 An NLP Toolset With A Focus on Explainable Inference
depedency-parsing nlp nlp-pipeline postagging python tokenizer
Language:Java 618
open-korean-text / open-korean-text
Open Korean Text Processor - An Open-source Korean Text Processor
korean korean-text-processing natural-language-processing text-processing tokenizer korean-tokenizer
Language:Scala 594
jflex-de / jflex
The fast scanner generator for Java™ with full Unicode support
bazel-rules cup dfa dfa-minimization flex grammar java lexer lexer-generator lexical-analyzer maven-plugin nfa parsing regexp scanner scanner-generator tokenizer yacc
Language:Java 573
tiktokenizer
dqbd / tiktokenizer
Online playground for OpenAPI tokenizers
chatgpt nextjs openai t3-stack tiktoken tokenizer
Language:TypeScript 540
glayzzle / php-parser
:herb: NodeJS PHP Parser - extract AST or tokens
ast development javascript lexer parser php php-ast php-parser static-code-analysis tokenizer
Language:JavaScript 513
alasdairforsythe / tokenmonster
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
text-tokenization tokenisation tokenization tokenize tokenizer tokenizing vocabulary vocabulary-builder vocabulary-generator
Language:Go 493
wangfenjin / simple
支持中文和拼音的 SQLite fts5 全文搜索扩展｜ A SQLite3 fts5 tokenizer which supports Chinese and PinYin
chinese cpp14 fts fts5 pinyin sqlite sqlite3 sqlite3-fts5 tokenizer
Language:C++ 477
hplt-project / sacremoses
Python port of Moses tokenizer, truecaser and normalizer
machine-translation nlp tokenizer
Language:Python 476
lydell / js-tokens
Tiny JavaScript tokenizer.
ecmascript javascript regex tokenizer
Language:JavaScript 476
lionsoul2014 / friso
High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.
c chinese-tokenizer chinese-word-segmentation cjk-tokenizer full-text-search japanese-tokenizer korean-tokenizer php-tokenizer tokenizer
Language:C 473
CogComp / cogcomp-nlp
CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, transliteration, verb-sense, and more.
big-data cogcomp data-mining dependency-parsing lemmatization lemmatizer named-entity-recognition natural-language-processing natural-language-understanding ner nlp parts-of-speech-tagging pos pos-tagging relation-extraction similarity tokenizer transliteration
Language:Java 470
neurosnap / sentences
A multilingual command line sentence tokenizer in Golang
cli sentence-tokenizer sentences tokenizer
Language:Go 418
timtadh / lexmachine
Lex machinary for go.
dfa go lex lexer lexical-analysis-engines lexical-analysis-framework nfa regular-expression tokenizer
Language:Go 400
nagisa
taishi-i / nagisa
A Japanese tokenizer based on recurrent neural networks
dynet word-segmentation pos-tagging japanese nlp-library sequence-labeling natural-language-processing nlp tokenizer
Language:Python 373
niieani / gpt-tokenizer
JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4. Port of OpenAI's tiktoken with additional features.
bpe decoder encoder gpt-2 gpt-3 gpt-4 machine-learning openai tokenizer
Language:TypeScript 372
ku-nlp / jumanpp
Juman++ (a Morphological Analyzer Toolkit)
cjk japanese juman morphological-analyser morphological-analysis nlp part-of-speech-tagger pos-tagger pos-tagging tokenizer word-segmentation
Language:C++ 365
fugashi
polm / fugashi
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
cython-wrapper japanese mecab nlp tokenizer
Language:C++ 363
lindera-morphology / lindera
A multilingual morphological analysis library.
analyzer library morphological multilingual tokenizer
Language:Rust 348
vscode-blockman
leodevbro / vscode-blockman
VSCode extension to highlight nested code blocks
abstract-syntax-tree ast highlight-blocks indentation parser tokenizer vscode-api vscode-blockman vscode-extension
Language:TypeScript 341
daac-tools / vibrato
🎤 vibrato: Viterbi-based accelerated tokenizer
japanese morphological-analysis nlp rust segmentation tokenization tokenizer
Language:Rust 293
belladoreai / llama-tokenizer-js
JS tokenizer for LLaMA
javascript llama llm tokenizer
Language:JavaScript 292
bitextor
bitextor / bitextor
Bitextor generates translation memories from multilingual websites
document-aligner apertium dictionaries crawler wget hunalign sentence-segmentation tokenizer bicleaner tmx warc corpus-tools corpus-processing corpus-generator parallel-corpora machine-translation neural-machine-translation statistical-machine-translation bitextor bleualign
Language:Python 277
guillaume-be / rust-tokenizers
Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
deep-learning rust-lang tokenizer transformer
Language:Rust 273
artitw / text2text
Text2Text: Crosslingual NLP/G toolkit
backtranslation chatgpt cross-lingual data-augmentation embeddings information-retrieval levenshtein-distance llm multi-lingual natural-language-generation natural-language-processing nlp question-answering question-generation search summarization tf-idf tokenizer transformers translator
Language:Python 272
OpenNMT / Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
bpe cpp icu machine-translation natural-language-processing python sentencepiece tokenization tokenizer unicode
Language:C++ 260
daac-tools / vaporetto
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
analyzer japanese morphological-analysis nlp rust segmentation tokenization tokenizer
Language:Rust 213