tokenization

There are 19 repositories under tokenization topic.

explosion / spaCy
💫 Industrial-strength Natural Language Processing (NLP) in Python
ai artificial-intelligence cython data-science deep-learning entity-linking machine-learning named-entity-recognition natural-language-processing neural-network neural-networks nlp nlp-library python spacy text-classification tokenization
Language:Python 29339
lunasec-io / lunasec
LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTrace GitHub App: https://github.com/marketplace/lunatrace-by-lunasec/
tokenization web-security compliance security soc2 pci-dss gdpr zero-trust devsecops log4shell dependency-analysis scanning cybersecurity security-tools scanning-tool cve-scanning sbom sbom-generator continuous-delivery software-composition-analysis
Language:TypeScript 1424
AgentOps-AI / tokencost
Easy token price estimates for 400+ LLMs. TokenOps.
analytics claude large-language-models llm observability openai price price-tracker token tokenization
Language:Python 1302
databunker
securitybunker / databunker
Secure SDK/vault for personal records/PII built to comply with GDPR
gdpr pii piidata security privacy privacy-by-design user-consent ccpa compliance legaltech anonymization data-anonymization database encryption vault tokenization application-server passportjs data-protection secure-storage
Language:Go 1229
RavenProject / Ravencoin
Ravencoin Core integration/staging tree
raven asset token tokenization ravencoin bitcoin blockchain
Language:C 1077
VKCOM / YouTokenToMe
Unsupervised text tokenizer focused on computational efficiency
bpe natural-language-processing nlp tokenization word-segmentation
Language:C++ 950
explosion / spacy-streamlit
👑 spaCy building blocks and visualizers for Streamlit apps
dependency-parsing machine-learning named-entity-recognition natural-language-processing ner nlp part-of-speech-tagging spacy streamlit text-classification tokenization visualizer visualizers word-vectors
Language:Python 782
AmoDinho / datacamp-python-data-science-track
All the slides, accompanying code and exercises all stored in this repo. 🎈
bokeh data-science datacamp datacamp-course datacamp-exercises datacamp-machine-learning datacamp-projects datacamp-python datacamp-solutions-python datascience machinelearning natural-language-processing neural-network neural-networks nlp pandas python scikit-learn tokenization
Language:Python 774
nlp-uoregon / trankit
Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing
nlp natural-language-processing pytorch language-model xlm-roberta machine-learning deeplearning artificial-intelligence universal-dependencies multilingual adapters sentence-segmentation tokenization part-of-speech-tagging morphological-tagging dependency-parsing lemmatization
Language:Python 717
cbaziotis / ekphrasis
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
nlp text-processing nlp-library spelling-correction tokenizer tokenization word-segmentation word-normalization spell-corrector text-segmentation semeval
Language:Python 660
adobe / NLP-Cube
Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing
embeddings parse nlp-cube language-pipeline tokenization sentence-splitting part-of-speech-tagger lemmatization dependency-parser dependency-parsing universal-dependencies machine-translation information-extraction
Language:HTML 553
alasdairforsythe / tokenmonster
Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript
text-tokenization tokenisation tokenization tokenize tokenizer tokenizing vocabulary vocabulary-builder vocabulary-generator
Language:Go 531
yooper / php-text-analysis
PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language
nlp php tokenization php-text-analysis php-language text-analysis
Language:PHP 517
macmade / ClangKit
ClangKit provides an Objective-C frontend to LibClang. Source tokenization, diagnostics and fix-its are actually implemented.
clang llvm parsing tokenization diagnostics source code syntax-highlighting static-analysis c c-plus-plus objective-c
Language:C 359
daac-tools / vibrato
🎤 vibrato: Viterbi-based accelerated tokenizer
japanese morphological-analysis nlp rust segmentation tokenization tokenizer
Language:Rust 311
WorksApplications / sudachi.rs
Sudachi in Rust 🦀 and new generation of SudachiPy
morphological-analysis nlp-libary pos-tagging python rust segmentation sudachi tokenization
Language:Rust 290
OpenNMT / Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
tokenizer sentencepiece natural-language-processing machine-translation bpe unicode tokenization icu python cpp
Language:C++ 271
CodeChain-io / codechain
CodeChain's official implementation in Rust.
blockchain asset rust tokenization digital-securities
Language:Rust 258
razdel
natasha / razdel
Rule-based token, sentence segmentation for Russian language
sentence-segmentation tokenization sentence-boundary-detection python nlp russian
Language:Python 248
SmartTokenLabs / TokenScript
TokenScript schema, specs and paper
tokens xml web3 mobile cryptography security tokenization tokenisation blockchain
Language:JavaScript 244
daac-tools / vaporetto
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
analyzer japanese morphological-analysis nlp rust segmentation tokenization tokenizer
Language:Rust 223
zjukg / MyGO
[Paper][Preprint 2024] MyGO: Discrete Modality Information as Fine-Grained Tokens for Multi-modal Knowledge Graph Completion
contrastive-learning knowledge-graph-completion multi-modal multi-modal-fusion multi-modal-knowledge-graph mygo tokenization
Language:Python 200
janlukasschroeder / nlp-cheat-sheet-python
NLP Cheat Sheet, Python, spacy, LexNPL, NLTK, tokenization, stemming, sentence detection, named entity recognition
spacy spacy-nlp cheat-sheet nlp python starter-kit introduction machine-learning tokenization lemmatization spans sentence-similarity pos-tagging dependency-parsing named-entity-recognition nltk lexnlp
Language:Jupyter Notebook 199
FoundationVision / OmniTokenizer
OmniTokenizer: one model and one weight for image-video joint tokenization.
auto-regressive-model image-generation tokenization vae video-generation vqvae
Language:Python 193
milaan9 / Python_Natural_Language_Processing
This repository consists of a complete guide on natural language processing (NLP) in Python where we'll learn various techniques for implementing NLP including parsing & text processing and understand how to use NLP for text feature engineering.
nlp tokenization stemming lemmatization stopwords partofspeech-tagger named-entity-recognition sentence-segmentation bag-of-words termfrequency inversedocumentfrequency tf-idf vocabulary-matching python4datascience python4everybody tutor-milaan9 ipython-notebook
Language:Jupyter Notebook 190
THUDM / icetk
A unified tokenization tool for Images, Chinese and English.
tokenization transformer
Language:Python 147
rth / vtext
Simple NLP in Rust with Python bindings
nlp information-retrieval tokenization bag-of-words tf-idf
Language:Rust 146
GlitchedPolygons / l8w8jwt
Minimal, OpenSSL-less and super lightweight JWT library written in C.
auth authentication authorization c cryptography ecc jwt performance token token-based-authentication tokenization
Language:C 136
adbar / simplemma
Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
nlp lemmatization lemmatizer lemmatiser tokenization wordlist morphological-analysis corpus-tools tokenizer low-resource-nlp language-detection language-identification
Language:Python 134
gautierdag / bpeasy
Fast bare-bones BPE for modern tokenizer training
bpe tokenization tokenizer
Language:Python 129
lucidrains / charformer-pytorch
Implementation of the GBST block from the Charformer paper, in Pytorch
artificial-intelligence deep-learning tokenization transformer
Language:Python 119
LongRoPE
jshuadvd / LongRoPE
Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
ai artificial-intelligence attention-is-all-you-need attention-mechanisms deep-learning gpt language-model llm machine-learning natural natural-language-inference natural-language-processing natural-language-procressing natural-language-understanding nlp tokenization tokenizers transformer-architecture transformers
Language:Python 104
lima
aymara / lima
The Libre Multilingual Analyzer, a Natural Language Processing (NLP) C++ toolkit.
natural-language-processing multilingual powerful free-software linux windows machine-learning nlp python cpp deep-learning information-extraction data-science entity-extraction ai neural-network named-entity-recognition nlp-library relation-extraction tokenization
Language:C++ 103
bminixhofer / zett
Code for Zero-Shot Tokenizer Transfer
language-model llm llms multilingual tokenization transfer-learning
Language:Python 101
mit-ccc / TweebankNLP
[LREC 2022] An off-the-shelf pre-trained Tweet NLP Toolkit (NER, tokenization, lemmatization, POS tagging, dependency parsing) + Tweebank-NER dataset
nlp-toolkit ner tokenization lemmatization dependency-parser pos-tagging named-entity-recognition text-annotation twitter-nlp tweet-analysis machine-learning natural-language-processing
Language:Python 100
ash-01xor / bpe.c
Simple Byte pair Encoding mechanism used for tokenization process . written purely in C
llm tokenization
Language:C 98