google / sentencepiece

Unsupervised text tokenizer for Neural Network-based text generation.

google/sentencepiece Issues

Build sentencepiece with mingw
Closed 8 days ago1
Tokenize at the word level without spacers nor joiners
Closed 11 days ago2
Treat Hawaiian Glottal stop as consonant, not punctuation
Closed 25 days ago4
No make file found while build and install the Python wrapper
Closed a month ago2
Is GGUF supported?
Closed a month ago1
Windows pip Dependancy Installation Error
Closed 2 months ago2
Support for Windows Python 3.12.2
Closed 2 months ago
Error when running this command: pip install 'transformers[tf-cpu]' on mac
Closed 2 months ago1
Getting requirements to build wheel did not run successfully.
Closed 3 months ago5
Any api for setting user defined symbols?
Closed 2 months ago1
Inconsistent result between py and cpp
Closed 2 months ago1
Only Pretokenization
Closed 2 months ago3
pip subprocess to install build dependencies did not run successfully. │ exit code: 1
Updated 2 months ago1
coredump when build with CXXFLAGS `-Wp,-D_GLIBCXX_ASSERTIONS`
Updated 2 months ago
High frequency token segmented into letter sequence when input is a tsv file
Updated 3 months ago3
coredump when build with CXXFLAG `-Wp,-D_GLIBCXX_ASSERTIONS`
Closed 3 months ago4
Allow whitespace-only pieces
Updated 3 months ago
entry points return non-zero exit code (at least for `--help`)
Closed 3 months ago2
Adding New Argument min_freq to SentencePieceTrainer.train in Python
Closed 3 months ago2
Sequence of byte '<0x09>' as token
Closed 3 months ago1
TSV for NFC normalization
Closed 3 months ago1
Many tests fail
Closed 3 months ago2
HELP NEEDED Mask Token in SentencePiece tokenizer HELP NEEDED
Closed 3 months ago1
crash in absl::~Flag when used in python along with PyTorch
Closed 3 months ago2
Unable to install sentencepiece on Python 3.12
Closed 3 months ago3
Segmentation fault (core dumped)
Closed 3 months ago2
Error while installing the library "sentence-transformers" which has dependency on "sentencepiece"
Closed 3 months ago11
error while installing sentencepiece python 3.12.2
Closed 3 months ago2
Tips for Termux installation
Updated 3 months ago7
Not found google.protobuf packages
Closed 3 months ago1
Merging tokenizers issue
Closed 4 months ago4
RuntimeError
Closed 4 months ago1
Official support for Android compilation in Release/Assets
Closed 4 months ago1
How to use EncodeAsIds for text that contains `<s>/</s>'
Closed 6 months ago3
Trainer incorrectly concatenating input files
Closed 4 months ago2
Same oov count while using different vocab size
Closed 4 months ago2
Evaluate Profile-Guided Optimization (PGO)
Updated 4 months ago
Extract & modify the merge rules from the .model file of a SentencePiece BPE model
Closed 4 months ago1
How to safely extend vocabulary?
Closed 4 months ago3
FileNotFoundError: [WinError 2]
Closed 5 months ago2
Ensure workflows run with minimal permissions
Closed 5 months ago
c++ API compilation problem
Closed 5 months ago1
latency 100ms
Closed 5 months ago1
Error when compiling for AMD with HIP C++ compiler (`hipcc`)
Closed 5 months ago
ImportError: cannot import name 'SentencePieceProcessor' from 'sentencepiece' (unknown location)
Closed 6 months ago2
[Question] Is sentence splitting required or optional?
Closed 6 months ago1
What is the appropriate vocabulary length setting?
Closed 6 months ago1
Does bpe support num_threads parameter?
Closed 6 months ago2
Hash-pin GitHub Actions used in workflows, keep them updated with Dependabot
Closed 6 months ago
[Question] How does encoding work?
Closed 6 months ago2