bytepairencoding

There are 0 repository under bytepairencoding topic.

• AnimaVR / TOKENIZER-BytePairEncoderDecoder-ModelTrainer-CSharp

  Actual C Sharp Byte Pair Encoder that works. Use bin folder or add your own data to be able to train your own model, this model is then used to encode into train.bin and val.bin binary files to use to train an LLM or similar.

  aiartificial-intelligencebytepairencodingllm
  Language:C# 2

• deepanprabhu / fastbpe

  Java library implementing Byte-Pair Encoding Tokenization

  bpebytepairencodingllm
  Language:Java 2

• vatsalsaglani / BytePairEncoding

  A python package to build a corpus vocabulary using the byte pair methodology and also a tokenizer to tokenize input texts based on the built vocab.

  vocabularyvocabulary-buildertokenizerbytepairencodingbpesubword-unitssubwordtokenizationsubwordtokensnatural-language-processingnatural-language-understandingnlpnlp-library
  Language:Python 1

• mohsenfayyaz / nlp-course-ut

  Natural Language Processing course assignments @ University of Tehran

  bytepairencodinglstmnaive-bayesnatural-language-processingrnntokenizationtransformer
  Language:Jupyter Notebook 0

• ReshiAdavan / Thoth

  An Industry Standard Tokenizer, purposed for large-scale language models like OpenAI's GPT Series.

  pythontokenizerbytepairencodinggpt-2gpt-4llama2sentencepiecetiktokenrustnatural-language-processing
  Language:Python 0

• shivendrra / tokenizers

  self made byte-pair-encoding tokenizer

  bpe-tokenizerbytepairencodingllmtokenizationtokenizer
  Language:Python 0

• JunhoKim94 / Transformer

  This repository is reimplementation of Transformer model which was introduced in 2017 NeurIPS paper "Attention is all you need"

  transformerbytepairencodingwmt-15self-attention
  Language:Python