sorenmulli / tokens-per-word

How many tokens per word using given tokenizer and given dataset?

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

How many tokens per word using given tokenizer and given dataset?

$ pip install .

$ cat raw-sentences.txt | tokens-per-word -t "north/t5_base_scand3M"
# or (if you have csvkit/csvtools)
$ csvcut -c "text-column" tabular-dataset.csv | tokens-per-word -t "north/t5_base_scand3M"
# or (if you jave jq)
$ cat line-formatted-dataset.jsonl | jq .text-field | tokens-per-word -t "north/t5_base_scand3M"

About

How many tokens per word using given tokenizer and given dataset?


Languages

Language:Python 100.0%