deepanprabhu / fastbpe

Java library implementing Byte-Pair Encoding Tokenization

fastbpe

Java implementation of Neural Machine Translation of Rare Words with Subword Units

fastbpe implements byte pair encoding tokenization in java. The library is supposed to be

• simple,
• fast,
• handle large volumes of text - minimum 1 gb,
• flawless with tests,
• ready for alpha testing soon.

References,

• https://towardsdatascience.com/byte-pair-encoding-subword-based-tokenization-algorithm-77828a70bee0
• https://leimao.github.io/blog/Byte-Pair-Encoding/
• https://huggingface.co/learn/nlp-course/chapter6/5?fw=pt#tokenization-algorithm