TalkBank / utterance-tokenizer

model to tokenize CHAT utterances

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

utterance-tokenizer

A NER-like BERT model trainable/trained on TalkBank data that performs the task of utterance tokenization. This is used to add punctuation and tokenization to ADR outputs in the pipeline to process raw audio data into usable transcripts for TalkBank.

Utterance tokenization is notably different from sentence tokenization; for the definition of a speech utterance, refer to the CHAT spec.

Refer to the batchalign repository for the usage of a trained model

About

model to tokenize CHAT utterances


Languages

Language:Python 100.0%