awkrail / tinysegmenter

Japanese tiny tokenizer written in Rust

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Tinysegmenter

Japanese tiny segmenter written in Rust.

Usage

Run cargo run with a Japanese sentence like this:

cargo run 私の名前は西村です

The output would be expected as:

私 の 名前 は 西村 です

About

Japanese tiny tokenizer written in Rust

License:BSD 3-Clause "New" or "Revised" License


Languages

Language:Rust 72.7%Language:Python 27.3%