Systemcluster / sentencepiece-model

SentencePiece model parser

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

sentencepiece-model

Crates.io Docs.rs

SentencePiece model parser generated from the SentencePiece protobuf definition.

use sentencepiece_model::SentencePieceModel;

let model = SentencePieceModel::from_file("tests/t5-spiece.model")?;
assert_eq!(model.pieces.len(), 32000);
assert_eq!(model.trainer()?.unk_id(), 2);

About

SentencePiece model parser

License:BSD 2-Clause "Simplified" License


Languages

Language:Rust 100.0%