(简体中文|English)

FunASR: A Fundamental End-to-End Speech Recognition Toolkit

FunASR hopes to build a bridge between academic research and industrial applications on speech recognition. By supporting the training & finetuning of the industrial-grade speech recognition model, researchers and developers can conduct research and production of speech recognition models more conveniently, and promote the development of speech recognition ecology. ASR for Fun！

Highlights

FunASR is a fundamental speech recognition toolkit that offers a variety of features, including speech recognition (ASR), Voice Activity Detection (VAD), Punctuation Restoration, Language Models, Speaker Verification, Speaker Diarization and multi-talker ASR. FunASR provides convenient scripts and tutorials, supporting inference and fine-tuning of pre-trained models.
We have released a vast collection of academic and industrial pretrained models on the ModelScope and huggingface, which can be accessed through our Model Zoo. The representative Paraformer-large, a non-autoregressive end-to-end speech recognition model, has the advantages of high accuracy, high efficiency, and convenient deployment, supporting the rapid construction of speech recognition services. For more details on service deployment, please refer to the service deployment document.

What's new:

2023/11/08: The offline file transcription service 3.0 (CPU) of Mandarin has been released, adding punctuation large model, Ngram language model, and wfst hot words. For detailed information, please refer to docs.
2023/10/17: The offline file transcription service (CPU) of English has been released. For more details, please refer to (docs).
2023/10/13: SlideSpeech: A large scale multi-modal audio-visual corpus with a significant amount of real-time synchronized slides.
2023/10/10: The ASR-SpeakersDiarization combined pipeline Paraformer-VAD-SPK is now released. Experience the model to get recognition results with speaker information.
2023/10/07: FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec.
2023/09/01: The offline file transcription service 2.0 (CPU) of Mandarin has been released, with added support for ffmpeg, timestamp, and hotword models. For more details, please refer to (docs).
2023/08/07: The real-time transcription service (CPU) of Mandarin has been released. For more details, please refer to (docs).
2023/07/17: BAT is released, which is a low-latency and low-memory-consumption RNN-T model. For more details, please refer to (BAT).
2023/06/26: ASRU2023 Multi-Channel Multi-Party Meeting Transcription Challenge 2.0 completed the competition and announced the results. For more details, please refer to (M2MeT2.0).

Installation

Please ref to installation docs

Model Zoo

FunASR has open-sourced a large number of pre-trained models on industrial data. You are free to use, copy, modify, and share FunASR models under the Model License Agreement. Below are some representative models, for more models please refer to the Model Zoo.

(Note: 🤗 represents the Huggingface model zoo link, ⭐ represents the ModelScope model zoo link)

Model Name	Task Details	Training Date	Parameters
paraformer-zh (⭐ 🤗 )	speech recognition, with timestamps, non-streaming	60000 hours, Mandarin	220M
paraformer-zh-spk ( ⭐ 🤗 )	speech recognition with speaker diarization, with timestamps, non-streaming	60000 hours, Mandarin	220M
paraformer-zh-online ( ⭐ 🤗 )	speech recognition, streaming	60000 hours, Mandarin	220M
paraformer-en ( ⭐ 🤗 )	speech recognition, with timestamps, non-streaming	50000 hours, English	220M
paraformer-en-spk (🤗 ⭐ )	speech recognition with speaker diarization, non-streaming	50000 hours, English	220M
conformer-en ( ⭐ 🤗 )	speech recognition, non-streaming	50000 hours, English	220M
ct-punc ( ⭐ 🤗 )	punctuation restoration	100M, Mandarin and English	1.1G
fsmn-vad ( ⭐ 🤗 )	voice activity detection	5000 hours, Mandarin and English	0.4M
fa-zh ( ⭐ 🤗 )	timestamp prediction	5000 hours, Mandarin	38M

Quick Start

Quick start for new users（tutorial）

FunASR supports inference and fine-tuning of models trained on industrial data for tens of thousands of hours. For more details, please refer to modelscope_egs. It also supports training and fine-tuning of models on academic standard datasets. For more information, please refer to egs.

Below is a quick start tutorial. Test audio files (Mandarin, English).

Speech Recognition (Non-streaming)

from funasr import infer

p = infer(model="paraformer-zh", vad_model="fsmn-vad", punc_model="ct-punc", model_hub="ms")

res = p("asr_example_zh.wav", batch_size_token=5000)
print(res)

Note: model_hub: represents the model repository, ms stands for selecting ModelScope download, hf stands for selecting Huggingface download.

Speech Recognition (Streaming)

from funasr import infer

p = infer(model="paraformer-zh-streaming", model_hub="ms")

chunk_size = [0, 10, 5] #[0, 10, 5] 600ms, [0, 8, 4] 480ms
param_dict = {"cache": dict(), "is_final": False, "chunk_size": chunk_size, "encoder_chunk_look_back": 4, "decoder_chunk_look_back": 1}

import torchaudio
speech = torchaudio.load("asr_example_zh.wav")[0][0]
speech_length = speech.shape[0]

stride_size = chunk_size[1] * 960
sample_offset = 0
for sample_offset in range(0, speech_length, min(stride_size, speech_length - sample_offset)):
    param_dict["is_final"] = True if sample_offset + stride_size >= speech_length - 1 else False
    input = speech[sample_offset: sample_offset + stride_size]
    rec_result = p(input=input, param_dict=param_dict)
    print(rec_result)

Note: chunk_size is the configuration for streaming latency. [0,10,5] indicates that the real-time display granularity is 10*60=600ms, and the lookahead information is 5*60=300ms. Each inference input is 600ms (sample points are 16000*0.6=960), and the output is the corresponding text. For the last speech segment input, is_final=True needs to be set to force the output of the last word.

Quick start for new users can be found in docs

Deployment Service

FunASR supports deploying pre-trained or further fine-tuned models for service. Currently, it supports the following types of service deployment:

File transcription service, Mandarin, CPU version, done
The real-time transcription service, Mandarin (CPU), done
File transcription service, English, CPU version, done
File transcription service, Mandarin, GPU version, in progress
and more.

For more detailed information, please refer to the service deployment documentation.

Community Communication

If you encounter problems in use, you can directly raise Issues on the github page.

You can also scan the following DingTalk group or WeChat group QR code to join the community group for communication and discussion.

DingTalk group	WeChat group

Contributors

The contributors can be found in contributors list

License

This project is licensed under the The MIT License. FunASR also contains various third-party components and some code modified from other repos under other open source licenses. The use of pretraining model is subject to model license

Citations

@inproceedings{gao2023funasr,
  author={Zhifu Gao and Zerui Li and Jiaming Wang and Haoneng Luo and Xian Shi and Mengzhe Chen and Yabin Li and Lingyun Zuo and Zhihao Du and Zhangyu Xiao and Shiliang Zhang},
  title={FunASR: A Fundamental End-to-End Speech Recognition Toolkit},
  year={2023},
  booktitle={INTERSPEECH},
}
@inproceedings{An2023bat,
  author={Keyu An and Xian Shi and Shiliang Zhang},
  title={BAT: Boundary aware transducer for memory-efficient and low-latency ASR},
  year={2023},
  booktitle={INTERSPEECH},
}
@inproceedings{gao22b_interspeech,
  author={Zhifu Gao and ShiLiang Zhang and Ian McLoughlin and Zhijie Yan},
  title={{Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition}},
  year=2022,
  booktitle={Proc. Interspeech 2022},
  pages={2063--2067},
  doi={10.21437/Interspeech.2022-9996}
}

lyblsgo / FunASR