tokenizer

There are 14 repositories under tokenizer topic.

theseer / tokenizer
A small library for converting tokenized PHP source code into XML (and potentially other formats)
php tokenizer xml
Language:PHP 5199
Chevrotain / chevrotain
Parser Building Toolkit for JavaScript
grammars javascript lexer open-source parser-library parsing tokenizer typescript
Language:TypeScript 2698
tiktokenizer
dqbd / tiktokenizer
Online playground for OpenAPI tokenizers
chatgpt nextjs openai t3-stack tiktoken tokenizer
Language:TypeScript 1393
roshan-research / hazm
Persian NLP Toolkit
natural-language-processing python persian nlp persian-nlp dependency-parser embeddings text-processing tokenizer lemmatization farsi normalization pos-tagging
Language:Python 1340
natasha
natasha / natasha
Solves basic Russian NLP tasks, API for lower level Natasha projects
nlp russian tokenizer sentence-segmentation embeddings morphology syntax ner visualization python
Language:Python 1289
lovit / soynlp
한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.
korean-nlp korean-text-processing nlp tokenizer postagging word-extraction
Language:Python 981
ikawaha / kagome
Self-contained Japanese Morphological Analyzer written in pure Go
hacktoberfest japanese japanese-language korean morphological-analysis nlp-library pos-tagging segmentation tokenizer
Language:Go 910
no-context / moo
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
javascript lexer regexp tokenizer
Language:JavaScript 871
wangfenjin / simple
支持中文和拼音的 SQLite fts5 全文搜索扩展｜ A SQLite3 fts5 tokenizer which supports Chinese and PinYin
sqlite3 tokenizer fts5 chinese pinyin sqlite fts cpp14 sqlite3-fts5
Language:C++ 747
BLKSerene / Wordless
An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation
corpus corpus-analysis corpus-statistics corpus-linguistics corpus-tools corpus-processing literature translation tokenizer tagger lemmatizer stopword corpus-search dependency-parser linguistics
Language:Python 742
risesoft-y9 / Data-Labeling
数据标注是一款专门对文本数据进行处理和标注的工具，通过简化快捷的文本标注流程和动态的算法反馈，支持用户快速标注关键词并能通过算法持续减少人工标注的成本和时间。数据标注的过程先由人工标注构建基础，再由自动标注反哺人工标注，最后由人工标注进行纠偏，从而大幅度提高标注的精准度和高效性。数据标注需要依赖开源的数字底座进行人员岗位管控。
chinese data-annotation-tools data-annotations docker elasticsearch java nacos springboot2 tokenizer tokenizer-parser vue3
Language:Java 690
mathewsanders / Mustard
🌭 Mustard is a Swift library for tokenizing strings when splitting by whitespace doesn't cut it.
swift substrings tokenizer
Language:Swift 687
cbaziotis / ekphrasis
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
nlp text-processing nlp-library spelling-correction tokenizer tokenization word-segmentation word-normalization spell-corrector text-segmentation semeval
Language:Python 672
niieani / gpt-tokenizer
The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.
bpe gpt-2 gpt-3 machine-learning gpt-4 tokenizer decoder encoder openai gpt-4o gpt-o1 gpt-5
Language:TypeScript 669
open-korean-text / open-korean-text
Open Korean Text Processor - An Open-source Korean Text Processor
korean korean-text-processing natural-language-processing text-processing tokenizer korean-tokenizer
Language:Scala 649
jflex-de / jflex
The fast scanner generator for Java™ with full Unicode support
scanner java lexical-analyzer lexer-generator dfa dfa-minimization grammar flex nfa lexer parsing maven-plugin bazel-rules cup tokenizer yacc scanner-generator regexp
Language:Java 624
SmoothNLP
smoothnlp / SmoothNLP
专注于可解释的NLP技术 An NLP Toolset With A Focus on Explainable Inference
nlp tokenizer postagging depedency-parsing python nlp-pipeline
Language:Java 621
therealoliver / Deepdive-llama3-from-scratch
Achieve the llama3 inference step-by-step, grasp the core concepts, master the process derivation, implement the code.
inference kv-cache llama llms attention attention-mechanism gpt language-model llm-configuration mask multi-head-attention positional-encoding residuals rms rms-norm rope swiglu tokenizer transformer rotary-position-encoding
Language:Jupyter Notebook 609
alasdairforsythe / tokenmonster
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
tokenisation tokenization tokenize tokenizer tokenizing vocabulary vocabulary-builder text-tokenization vocabulary-generator
Language:Go 604
glayzzle / php-parser
:herb: NodeJS PHP Parser - extract AST or tokens
php-parser ast lexer javascript parser tokenizer static-code-analysis development php php-ast
Language:JavaScript 554
lindera / lindera
A multilingual morphological analysis library.
analyzer library morphological multilingual tokenizer
Language:Rust 539
lydell / js-tokens
Tiny JavaScript tokenizer.
regex javascript tokenizer ecmascript
Language:JavaScript 530
lionsoul2014 / friso
High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.
c tokenizer chinese-tokenizer php-tokenizer full-text-search chinese-word-segmentation japanese-tokenizer korean-tokenizer cjk-tokenizer
Language:C 504
hplt-project / sacremoses
Python port of Moses tokenizer, truecaser and normalizer
machine-translation nlp tokenizer
Language:Python 495
vscode-blockman
leodevbro / vscode-blockman
VSCode extension to highlight nested code blocks
vscode-extension parser tokenizer ast abstract-syntax-tree vscode-api indentation highlight-blocks vscode-blockman
Language:TypeScript 490
fugashi
polm / fugashi
A Cython MeCab wrapper for fast, pythonic Japanese tokenization and morphological analysis.
japanese mecab tokenizer cython-wrapper nlp
Language:C++ 484
CogComp / cogcomp-nlp
CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, transliteration, verb-sense, and more.
natural-language-processing nlp big-data data-mining cogcomp lemmatizer lemmatization tokenizer transliteration pos pos-tagging parts-of-speech-tagging ner named-entity-recognition relation-extraction dependency-parsing similarity natural-language-understanding
Language:Java 479
flash-tokenizer
NLPOptimize / flash-tokenizer
EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING
deep-learning nlp python berttokenizer flash wordpiece wordpiece-tokenization cpp cpp17 tokenizer bert huggingface pybind11 trie
Language:C++ 467
neurosnap / sentences
A multilingual command line sentence tokenizer in Golang
sentence-tokenizer tokenizer cli sentences
Language:Go 458
FoundationVision / UniTok
[NeurIPS 2025 Spotlight] A Unified Tokenizer for Visual Generation and Understanding
autoregressive-models generative generative-ai generative-model image-generation image-tokenizer large-language-models text-to-image tokenizer
Language:Python 440
timtadh / lexmachine
Lex machinary for go.
go regular-expression nfa lexer tokenizer lexical-analysis-engines lexical-analysis-framework dfa lex
Language:Go 410
nagisa
taishi-i / nagisa
A Japanese tokenizer based on recurrent neural networks
dynet word-segmentation pos-tagging japanese nlp-library sequence-labeling natural-language-processing nlp tokenizer
Language:Python 408
ku-nlp / jumanpp
Juman++ (a Morphological Analyzer Toolkit)
juman nlp japanese morphological-analysis morphological-analyser pos-tagging pos-tagger part-of-speech-tagger word-segmentation cjk tokenizer
Language:C++ 399
daac-tools / vibrato
🎤 vibrato: Viterbi-based accelerated tokenizer
japanese morphological-analysis nlp rust segmentation tokenization tokenizer
Language:Rust 384
belladoreai / llama-tokenizer-js
JS tokenizer for LLaMA 1 and 2
javascript llama llm tokenizer
Language:JavaScript 360
zurawiki / tiktoken-rs
Ready-made tokenizer library for working with GPT and tiktoken
bpe openai rust tokenizer
Language:Rust 346

tokenizer

theseer / tokenizer

Chevrotain / chevrotain

dqbd / tiktokenizer

roshan-research / hazm

natasha / natasha

lovit / soynlp

ikawaha / kagome

no-context / moo

wangfenjin / simple

BLKSerene / Wordless

risesoft-y9 / Data-Labeling

mathewsanders / Mustard

cbaziotis / ekphrasis

niieani / gpt-tokenizer

open-korean-text / open-korean-text

jflex-de / jflex

smoothnlp / SmoothNLP

therealoliver / Deepdive-llama3-from-scratch

alasdairforsythe / tokenmonster

glayzzle / php-parser

lindera / lindera

lydell / js-tokens

lionsoul2014 / friso

hplt-project / sacremoses

leodevbro / vscode-blockman

polm / fugashi

CogComp / cogcomp-nlp

NLPOptimize / flash-tokenizer

neurosnap / sentences

FoundationVision / UniTok

timtadh / lexmachine

taishi-i / nagisa

ku-nlp / jumanpp

daac-tools / vibrato

belladoreai / llama-tokenizer-js

zurawiki / tiktoken-rs