tamuhey / tokenizations

Robust and Fast tokenizations alignment library for Rust and Python https://tamuhey.github.io/tokenizations/

⚠ MOVED

This repository was moved to https://github.com/explosion/tokenizations, and now maintained by Explosion, spaCy's authors.

Robust and Fast tokenizations alignment library for Rust and Python

Demo: demo
Rust document: docs.rs
Blog post: How to calculate the alignment between BERT and spaCy tokens effectively and robustly