tokenization

There are 23 repositories under tokenization topic.

explosion / spaCy
💫 Industrial-strength Natural Language Processing (NLP) in Python
natural-language-processing data-science machine-learning python cython nlp artificial-intelligence ai spacy nlp-library neural-network neural-networks deep-learning named-entity-recognition entity-linking text-classification tokenization
Language:Python 32784
AgentOps-AI / tokencost
Easy token price estimates for 400+ LLMs. TokenOps.
analytics claude large-language-models llm observability openai price price-tracker token tokenization
Language:Python 1834
NVIDIA / Cosmos-Tokenizer
A suite of image and video neural tokenizers
diffusion tokenization transformers
Language:Jupyter Notebook 1679
lunasec-io / lunasec
LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTrace GitHub App: https://github.com/marketplace/lunatrace-by-lunasec/
tokenization web-security compliance security soc2 pci-dss gdpr zero-trust devsecops log4shell dependency-analysis scanning cybersecurity security-tools scanning-tool cve-scanning sbom sbom-generator continuous-delivery software-composition-analysis
Language:TypeScript 1456
databunker
securitybunker / databunker
Secure Vault for Customer PII/PHI/PCI/KYC Records
gdpr pii piidata security privacy privacy-by-design user-consent ccpa compliance legaltech anonymization data-anonymization database encryption vault tokenization application-server passportjs data-protection secure-storage
Language:Go 1338
RavenProject / Ravencoin
Ravencoin Core integration/staging tree
raven asset token tokenization ravencoin bitcoin blockchain
Language:C 1111
VKCOM / YouTokenToMe
Unsupervised text tokenizer focused on computational efficiency
natural-language-processing word-segmentation nlp bpe tokenization
Language:C++ 971
AmoDinho / datacamp-python-data-science-track
All the slides, accompanying code and exercises all stored in this repo. 🎈
python machinelearning data-science datacamp datacamp-course datacamp-exercises datacamp-projects datacamp-python datacamp-machine-learning datacamp-solutions-python pandas neural-network neural-networks natural-language-processing nlp bokeh scikit-learn tokenization datascience
Language:Python 872
explosion / spacy-streamlit
👑 spaCy building blocks and visualizers for Streamlit apps
spacy nlp natural-language-processing visualizers dependency-parsing part-of-speech-tagging named-entity-recognition ner streamlit visualizer text-classification word-vectors tokenization machine-learning
Language:Python 845
nlp-uoregon / trankit
Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing
nlp natural-language-processing pytorch language-model xlm-roberta machine-learning deeplearning artificial-intelligence universal-dependencies multilingual adapters sentence-segmentation tokenization part-of-speech-tagging morphological-tagging dependency-parsing lemmatization
Language:Python 771
cbaziotis / ekphrasis
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
nlp text-processing nlp-library spelling-correction tokenizer tokenization word-segmentation word-normalization spell-corrector text-segmentation semeval
Language:Python 672
alasdairforsythe / tokenmonster
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
tokenisation tokenization tokenize tokenizer tokenizing vocabulary vocabulary-builder text-tokenization vocabulary-generator
Language:Go 604
adobe / NLP-Cube
Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing
embeddings parse nlp-cube language-pipeline tokenization sentence-splitting part-of-speech-tagger lemmatization dependency-parser dependency-parsing universal-dependencies machine-translation information-extraction
Language:HTML 561
yooper / php-text-analysis
PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language
nlp php tokenization php-text-analysis php-language text-analysis
Language:PHP 533
blockchain-bike-rental
alejoacosta74 / blockchain-bike-rental
Solidity based "BIKE RENTAL SHOP" on Ethereum network.
ether solidity ethereum tokenization truffle-framework
Language:JavaScript 431
WorksApplications / sudachi.rs
Sudachi in Rust 🦀 and new generation of SudachiPy
sudachi nlp-libary tokenization morphological-analysis segmentation pos-tagging rust python
Language:Rust 394
daac-tools / vibrato
🎤 vibrato: Viterbi-based accelerated tokenizer
nlp rust japanese morphological-analysis tokenizer segmentation tokenization
Language:Rust 384
macmade / ClangKit
ClangKit provides an Objective-C frontend to LibClang. Source tokenization, diagnostics and fix-its are actually implemented.
clang llvm parsing tokenization diagnostics source code syntax-highlighting static-analysis c c-plus-plus objective-c
Language:C 366
SaberaTalukder / TOTEM
The official code 👩‍💻 for - TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis
foundation-models representation-learning time-series time-series-analysis time-series-anomaly-detection time-series-forecasting time-series-foundation-model time-series-imputation tokenization
Language:Python 350
OpenNMT / Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
bpe cpp icu machine-translation natural-language-processing python sentencepiece tokenization tokenizer unicode
Language:C++ 322
FoundationVision / OmniTokenizer
[NeurIPS 2024]OmniTokenizer: one model and one weight for image-video joint tokenization.
auto-regressive-model image-generation tokenization vae video-generation vqvae
Language:Python 316
razdel
natasha / razdel
Rule-based token, sentence segmentation for Russian language
nlp python russian sentence-boundary-detection sentence-segmentation tokenization
Language:Python 275
zjukg / MyGO
[Paper][AAAI 2025] (MyGO)Tokenization, Fusion, and Augmentation: Towards Fine-grained Multi-modal Entity Representation
contrastive-learning knowledge-graph-completion multi-modal multi-modal-fusion multi-modal-knowledge-graph mygo tokenization
Language:Python 262
CodeChain-io / codechain
CodeChain's official implementation in Rust.
blockchain asset rust tokenization digital-securities
Language:Rust 259
janlukasschroeder / nlp-cheat-sheet-python
NLP Cheat Sheet, Python, spacy, LexNPL, NLTK, tokenization, stemming, sentence detection, named entity recognition
cheat-sheet dependency-parsing introduction lemmatization lexnlp machine-learning named-entity-recognition nlp nltk pos-tagging python sentence-similarity spacy spacy-nlp spans starter-kit tokenization
Language:Jupyter Notebook 249
daac-tools / vaporetto
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
rust tokenizer nlp japanese analyzer tokenization morphological-analysis segmentation
Language:Rust 247
SmartTokenLabs / TokenScript
TokenScript schema, specs and paper
tokens xml web3 mobile cryptography security tokenization tokenisation blockchain
Language:JavaScript 247
ImadSaddik / Train_Your_Language_Model_Course
Train a language model to chat like you using your personal conversations from WhatsApp, Telegram, Signal, or other platforms.
bpe llm personae-cloning tokenization training-llms transformer transformer-evolution
Language:Jupyter Notebook 226
milaan9 / Python_Natural_Language_Processing
This repository consists of a complete guide on natural language processing (NLP) in Python where we'll learn various techniques for implementing NLP including parsing & text processing and understand how to use NLP for text feature engineering.
nlp tokenization stemming lemmatization stopwords partofspeech-tagger named-entity-recognition sentence-segmentation bag-of-words termfrequency inversedocumentfrequency tf-idf vocabulary-matching python4datascience python4everybody tutor-milaan9 ipython-notebook
Language:Jupyter Notebook 202
adbar / simplemma
Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
corpus-tools language-detection language-identification lemmatiser lemmatization lemmatizer low-resource-nlp morphological-analysis nlp tokenization tokenizer wordlist
Language:Python 178
cohere-ai / magikarp
Code for the paper "Fishing for Magikarp"
large-language-models tokenization
Language:Python 172
GlitchedPolygons / l8w8jwt
Minimal, OpenSSL-less and super lightweight JWT library written in C.
jwt token tokenization token-based-authentication ecc cryptography auth authentication authorization c performance
Language:C 168
gautierdag / bpeasy
Fast bare-bones BPE for modern tokenizer training
bpe tokenization tokenizer
Language:Python 164
rth / vtext
Simple NLP in Rust with Python bindings
nlp information-retrieval tokenization bag-of-words tf-idf
Language:Rust 153
THUDM / icetk
A unified tokenization tool for Images, Chinese and English.
tokenization transformer
Language:Python 153
LongRoPE
jshuadvd / LongRoPE
Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
ai artificial-intelligence attention-is-all-you-need attention-mechanisms deep-learning gpt language-model llm machine-learning natural natural-language-inference natural-language-processing natural-language-procressing natural-language-understanding nlp tokenization tokenizers transformer-architecture transformers
Language:Python 152