sentencepiece

There are 0 repository under sentencepiece topic.

niedev / RTranslator
Open source real-time translation app for Android that runs locally
translator bluetooth-le realtime-translator android-app onnx onnxruntime sentencepiece transformers translation nllb whisper mobile-app android offline
Language:C++ 9154
OpenNMT / Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
tokenizer sentencepiece natural-language-processing machine-translation bpe unicode tokenization icu python cpp
Language:C++ 316
himkt / konoha
🌿 An easy-to-use Japanese Text Processing tool, which makes it possible to switch tokenizers with small changes of code.
nlp text-processing mecab kytea sudachi sentencepiece natural-language-processing japanese janome
Language:Python 254
taishan1994 / sentencepiece_chinese_bpe
使用sentencepiece中BPE训练中文词表，并在transformers中进行使用。
sentencepiece tokenization chinese-vocab
Language:Python 117
lingvanex-mt / models
Free and open source pre-trained translation models, including Kurdish, Samoan, Xhosa, Lao, Corsican, Cebuano, Galician, Russian, Belarusian and Yoruba.
cebuano corsican ctranslate2 galician kurdish language-localization lao machine-translation multilingual neural-networks nlp samoan sentencepiece translate translation translator xhosa yoruba
87
dhpollack / huggingface_libtorch
Minimal example of using a traced huggingface transformers model with libtorch
pytorch libtorch transformers-library nlp cpp sentencepiece albert
Language:C++ 35
kitoken
Systemcluster / kitoken
Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.
bpe nlp sentencepiece tokenizer unigram word-segmentation nodejs python rust web
Language:Rust 33
nguyenvulebinh / vietnamese-roberta
A Robustly Optimized BERT Pretraining Approach for Vietnamese
vietnamese pretrained-models natural-language-processing roberta bert bert-embeddings pytorch fairseq sentencepiece vietnamese-nlp transformer
Language:Python 32
eliben / go-sentencepiece
Go implementation of the SentencePiece tokenizer
encoding go golang language-model llm sentencepiece tokenization
Language:Go 27
bnosac / sentencepiece
R package for Byte Pair Encoding / Unigram modelling based on Sentencepiece
sentencepiece byte word-segmentation natural-language-processing
Language:C++ 25
Andras7 / gpt2-pytorch
Extremely simple and understandable GPT2 implementation with minor tweaks
gpt2 mixed-precision pytorch sentencepiece transformers
Language:Python 21
danieldk / sentencepiece
Rust binding for the sentencepiece library
sentencepiece rust
Language:Rust 20
stephantul / piecelearn
Learning BPE embeddings by first learning a segmentation model and then training word2vec
bpe wordpiece sentencepiece embeddings word2vec
Language:Python 19
sctg-development / sentencepiece-js
sentencepiece port to webassembly with browser compatibility
ai sentencepiece tokenizer
Language:TypeScript 13
to-aoki / my-pytorch-bert
BERT implementation of PyTorch
nlp bert pytorch japanese-language pytorch-bert sentencepiece albert
Language:Python 11
Masao-Taketani / japanese_text_classification
To investigate various DNN text classifiers including MLP, CNN, RNN, BERT approaches.
text-recognition natural-language-processing text-classification sentencepiece mecab deep-learning japanese
Language:Jupyter Notebook 9
jkrukowski / swift-sentencepiece
Use SentencePiece in Swift for tokenization and detokenization.
sentencepiece tokenization
Language:Swift 8
NishantkSingh0 / Generative-Language-Model
Trained Decoder only model on large BookCorpus Dataset. First time!
attention-mechanism numpy os pandas sentencepiece tensorflow transformer-architecture book-corpus
Language:Jupyter Notebook 7
wang1ang / SentencePieceWrapper
sentencepiece C# wrapper
sentencepiece csharp wrapper
Language:C++ 5
leliuga / datrin
dataset, train, inference
dataset flax inference jax safetensors train sentencepiece
Language:Python 4
smafjal / bengali_tokenizer
Bengali language Tokenizer (SentencePiece)
sentencepiece bengali bengali-natural-language-processing tokenizer bengali-tokenizer unsupervised-learning
Language:Python 4
kgarg8 / NMT-RNN
NMT with RNN Models: (1) in Vanilla style, (2) with Sentencepiece, (3) using Pre-trained models from FairSeq
pytorch fairseq rnn machine-translation sentencepiece
Language:Python 2
twinnydotdev / toxe
SentencePiece tokenizer for cross-encoders
artificial-intelligence crossencoder machine-learning sentencepiece tokenizer twinny
Language:JavaScript 2
arusl / anlp_nlp2021_d3-1
This repository contains codes related to the experiments in "An Experimental Evaluation of Japanese Tokenizers for Sentiment-Based Text Classification" presented at https://www.anlp.jp/nlp2021/. Authors: Andre Rusli and Makoto Shishido (Tokyo Denki University).
natural-language-processing japanese-tokenizer text-classification mecab sentencepiece sudachipy
Language:Jupyter Notebook 1
Doarakko / vector-text-similarity-search
Search for similar documents using Elasticsearch and BERT.
bert elasticsearch similarity-search japanese sentencepiece
Language:Jupyter Notebook 1
evan176 / sentencepiecego
sentencepiece golang
Language:Go 1
sftblw / spm_jamo_tsv
korean sentencepiece
Language:JavaScript 1
Sid911 / sentencepiece_dart
Sentencepiece Dart is a wrapper for Google's Sentencepiece C++ library modified
flutter-plugin sentencepiece dart preprocessing nautral-language-processing
Language:C++ 1
burcgokden / Sentencepiece-Tokenizer-Wrapper-for-PLDR-LLM
A framework for building Sentencepiece tokenizer from a dataset
deep-learning large-language-models llm machine-learning natural-language-processing pldr-llm sentencepiece tokenizer transformer keras tensorflow bpe unigram
Language:Python 0
ReshiAdavan / Thoth
industry standard tokenizer purposed for large-scale language models (GPT, Claude, Llama, etc.)
python tokenizer bytepairencoding gpt-2 gpt-4 llama2 sentencepiece tiktoken rust natural-language-processing
Language:Python 0
Abhigyan126 / SentencePiece-Tokenisation
A python and rust implementation of SentencePiece (A language-independent subword tokeniser and de-tokeniser developed by Google)
python rust sentencepiece tokeizer tokeniser sentence-piece-tokenizer
Language:Rust
amadews23 / TransformersExamples
huggingface-transformers sentencepiece tokenizer transformer translate
Language:Python
anthonywu / sentencepiece
Temp fork to provide Python 3.13 macOS wheels ahead of official project releases
python313 sentencepiece
Language:C++
burcgokden / SentencePiece-Tokenizer-Wrapper-for-PLDR-LLM-KVG-cache
SentencePiece Tokenizer Wrapper implementation for PLDR-LLM with KV cache and G-cache
bpe deep-learning large-language-models llm machine-learning natural-language-processing pldr-llm pytorch sentencepiece tokenizer transformer unigram
Language:Python
mahdertesf / SentencePiece-and-Byte-Pair-Encoding-BPE-Implementation
This repository provides a hands-on exploration of SentencePiece tokenization and Byte-Pair Encoding (BPE) .The code demonstrates data preprocessing steps like NFKC normalization and lossless tokenization, followed by a practical implementation of the BPE algorithm from scratch.
nlp byte-pair-encoding sentencepiece tokenizaiton
Language:Jupyter Notebook
rafael-vasconcellos / sugoi-v4-space
A huggingface space for Sugoi V4
ai api ctranslate2 deep-learning huggingface machine-learning natural-language-processing nlp translation flask sentence-piece-tokenizer sentencepiece backend server-side
Language:Python

sentencepiece

niedev / RTranslator

OpenNMT / Tokenizer

himkt / konoha

taishan1994 / sentencepiece_chinese_bpe

lingvanex-mt / models

dhpollack / huggingface_libtorch

Systemcluster / kitoken

nguyenvulebinh / vietnamese-roberta

eliben / go-sentencepiece

bnosac / sentencepiece

Andras7 / gpt2-pytorch

danieldk / sentencepiece

stephantul / piecelearn

sctg-development / sentencepiece-js

to-aoki / my-pytorch-bert

Masao-Taketani / japanese_text_classification

jkrukowski / swift-sentencepiece

NishantkSingh0 / Generative-Language-Model

wang1ang / SentencePieceWrapper

leliuga / datrin

smafjal / bengali_tokenizer

kgarg8 / NMT-RNN

twinnydotdev / toxe

arusl / anlp_nlp2021_d3-1

Doarakko / vector-text-similarity-search

evan176 / sentencepiecego

sftblw / spm_jamo_tsv

Sid911 / sentencepiece_dart

burcgokden / Sentencepiece-Tokenizer-Wrapper-for-PLDR-LLM

ReshiAdavan / Thoth

Abhigyan126 / SentencePiece-Tokenisation

amadews23 / TransformersExamples

anthonywu / sentencepiece

burcgokden / SentencePiece-Tokenizer-Wrapper-for-PLDR-LLM-KVG-cache

mahdertesf / SentencePiece-and-Byte-Pair-Encoding-BPE-Implementation

rafael-vasconcellos / sugoi-v4-space