Remove lifetime parameter from `Worker`
vbkaisetsu opened this issue · comments
Koichi Akabe commented
Currently, the Worker
struct is defined as follows:
struct Worker<'a> {
...
}
where 'a
is a lifetime parameter of the Tokenizer
. By this definition, the Worker
can refer to the Tokenizer
automatically for every tokenization.
This definition causes a problem when creating wrappers for other programming languages that use garbage collectors (GC).
The above definition means that the Tokenizer
cannot be removed when the Worker
is alive, but there is no way to impose this constraint on the GC.
To solve this problem, we need to remove the lifetime parameter and give the Worker
struct to the Tokenizer
for every tokenization.