daac-tools / vibrato

🎤 vibrato: Viterbi-based accelerated tokenizer

Home Page:https://docs.rs/vibrato

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Remove lifetime parameter from `Worker`

vbkaisetsu opened this issue · comments

Currently, the Worker struct is defined as follows:

struct Worker<'a> {
    ...
}

where 'a is a lifetime parameter of the Tokenizer. By this definition, the Worker can refer to the Tokenizer automatically for every tokenization.

This definition causes a problem when creating wrappers for other programming languages that use garbage collectors (GC).
The above definition means that the Tokenizer cannot be removed when the Worker is alive, but there is no way to impose this constraint on the GC.

To solve this problem, we need to remove the lifetime parameter and give the Worker struct to the Tokenizer for every tokenization.