juanpablogd / NETokenizer

NETokenizer

Provides an implementation of https://github.com/huggingface/tokenizers in .net

Usage

1. Build Tokenizer

cargo build --lib

add -r flag to Release

2. import tokenizer.dll to .net Project