MonTTS: A Real-time and High-fidelity Mongolian TTS Model with Pure Non-autoregressive Mechanism (MonTTS:完全非自回归的实时、高保真蒙古语语音合成模型)
This is an implementation of the following paper.
MonTTS: A Real-time and High-fidelity Mongolian TTS Model with Pure Non-autoregressive Mechanism. Journal of Chinese Information Processing (中文信息学报). 2022
Rui Liu, Shiyin Kang, Guanglai Gao, Jindong Li, Feilong Bao.
This project uses conda
to manage all the dependencies, you should install anaconda if you have not done so.
# Clone the repo
git clone https://github.com/ttslr/MonTTS.git
cd $PROJECT_ROOT_DIR
conda env create -f Environment/environment.yaml
conda activate montts
Prepare our MonSpeech dataset in the following format:
|- MonSpeech/
| |- metadata.csv
| |- wavs/
| |- file1.wav
| |- ...
Where metadata.csv
has the following format: id|transcription
. This is a ljspeech-like format.
Here are some speech samples from our MonSpeech.
You can find MonSpeech in the Links section.
The preprocessing has two steps:
- Preprocess audio features
- Convert characters to IDs
- Compute mel spectrograms
- Normalize mel spectrograms to [-1, 1] range
- Split the dataset into train and validation
- Compute the mean and standard deviation of multiple features from the training split
- Standardize mel spectrogram based on computed statistics
To reproduce the steps above:
tensorflow-tts-preprocess --rootdir /home/rui/MonSpeech --outdir ./dump_mon --config preprocess/mon_preprocess.yaml --dataset mon
tensorflow-tts-normalize --rootdir ./dump_mon --outdir ./dump_mon --config preprocess/mon_preprocess.yaml --dataset mon
Based on the script train_fastspeech2.py
.
This example code show you how to train MonTTS from scratch with Tensorflow 2 based on custom training loop and tf.function.
Here is an example command line to training MonTTS from scratch:
CUDA_VISIBLE_DEVICES=0 python examples/fastspeech2/train_fastspeech2.py \
--train-dir ./dump_mon/train/ \
--dev-dir ./dump_mon/valid/ \
--outdir ./examples/fastspeech2/exp/train.fastspeech2-mon.v1/ \
--config ./examples/fastspeech2/conf/fastspeech2.v1.yaml \
--use-norm 1 \
--f0-stat ./dump_mon/stats_f0.npy \
--energy-stat ./dump_mon/stats_energy.npy \
--mixed_precision 1 \
--resume ""
IF you want to use MultiGPU to training you can replace CUDA_VISIBLE_DEVICES=0
by CUDA_VISIBLE_DEVICES=0,1,2,3
for example. You also need to tune the batch_size
for each GPU (in config file) by yourself to maximize the performance. Note that MultiGPU now support for Training but not yet support for Decode.
In case you want to resume the training progress, please following below example command line:
--resume ./examples/fastspeech2/exp/train.fastspeech2-mon.v1/checkpoints/ckpt-100000
If you want to finetune a model, use --pretrained
like this with your model filename
--pretrained pretrained.h5
First, you need training generator with only stft loss:
CUDA_VISIBLE_DEVICES=0 python examples/hifigan/train_hifigan.py \
--train-dir ./dump_mon/train/ \
--dev-dir ./dump_mon/valid/ \
--outdir ./examples/hifigan/exp/train.hifigan-mon.v1/ \
--config ./examples/hifigan/conf/hifigan.v1.yaml \
--use-norm 1 \
--generator_mixed_precision 1 \
--resume ""
Then resume and start training generator + discriminator:
CUDA_VISIBLE_DEVICES=0 python examples/hifigan/train_hifigan.py \
--train-dir ./dump_mon/train/ \
--dev-dir ./dump_mon/valid/ \
--outdir ./examples/hifigan/exp/train.hifigan-mon.v1/ \
--config ./examples/hifigan/conf/hifigan.v1.yaml \
--use-norm 1 \
--resume ./examples/hifigan/exp/train.hifigan-mon.v1/checkpoints/ckpt-100000
You should find a dir log
in all of your output dirs, that is the LOG_DIR
you should use below.
tensorboard --logdir=${LOG_DIR}
For example, you can follow below example command lines to access the tensrobords to check the training progress:
tensorboard --logdir examples/fastspeech2/exp/train.fastspeech2-mon-exp1
tensorboard --logdir examples/hifigan/exp/train.hifigan-mon.v1
You can follow below example command line to generate synthesized speeh for given text in 'dump_mon/inference.txt' using Griffin-Lim and trained HiFi-GAN vocoder:
CUDA_VISIBLE_DEVICES=1 python examples/fastspeech2/inference_fastspeech2-mon.py \
--outdir prediction/mon_inference_fastspeech2 \
--infile dump_mon/inference.txt \
--tts_ckpt examples/fastspeech2/exp/train.fastspeech2-mon.v1/checkpoints/model-200000.h5 \
--vocoder_ckpt examples/hifigan/exp/train.hifigan-mon.v1/checkpoints/generator-420000.h5 \
--stats_path dump_mon/stats.npy \
--dataset_config preprocess/mon_preprocess.yaml \
--tts_config examples/fastspeech2/conf/fastspeech2.v1.yaml \
--vocoder_config examples/hifigan/conf/hifigan.v1.yaml \
--lan_json dump_mon/mon_mapper.json
You can find pre-trained models in the Links section.
The synthesized speech will save to prediction/mon_inference_fastspeech2
folder.
You can also run demo_server.py
to build a online demo.
python examples/fastspeech2/demo_server.py
Note that you need to point your browser at localhost:9000 and then type what you want to synthesize.
- Please contact the Author to access the MonSpeech corpus and Pre-trained models
- Demo: link to synthesized audio samples
Rui Liu
E-mail: liurui_imu@163.com
Please kindly cite the following paper if you use this code repository in your work,
@inproceedings{liu2021montts,
title={MonTTS: A Real-time and High-fidelity Mongolian TTS Model with Complete Non-autoregressive Mechanism (in Chinese)},
author={Rui, Liu and Shiyin, Kang and Jingdong, Li amd Feilong, Bao and Guanglai, Gao},
booktitle={JOURNAL OF CHINESE INFORMATION PROCESSING (中文信息学报)},
year={2021}
}
Tensorflow-TTS: https://github.com/TensorSpeech/TensorFlowTTS
This work is released under MIT License.