ankane / tokenizers-ruby

Fast state-of-the-art tokenizers for Ruby

ankane/tokenizers-ruby Issues

Add optional punctuation cleanup during decoding - clean_up_tokenization equivalent
Closed a month ago5
Clarify how to load tokenizers from files
Closed 5 months ago1
Error when loading tokenizers gem
Closed 7 months ago5
Issues when deploying to Ubuntu 20.04
Closed a year ago6
Cannot install tokenizer 0.3.2
Closed a year ago5
error when using in docker alpine
Closed a year ago5
Issue with CharBPETokenizer and pile_tokenizer.json
Closed a year ago3
Ruby 2.7 didn't have JSON.load_file method
Closed a year ago1
Using tiktoken
Closed a year ago1
Unable to run Rails when gem is installed
Closed a year ago6
More flexible handling of special tokens
Closed a year ago6
Version 0.2 relying on an old version of libssl
Closed a year ago4
Support to Ruby 3.2.0 (release 0.2.1)
Closed a year ago3
I get an error when installing
Closed 2 years ago5
Error compiling
Closed 2 years ago1