Real-Time State-of-the-art Speech Synthesis for Tensorflow 2
- 2020/12/02 (NEW!) Support German TTS with Thorsten dataset. See the Colab. Thanks thorstenMueller and monatis.
- 2020/11/24 (NEW!) Add HiFi-GAN vocoder. See here
- 2020/11/19 (NEW!) Add Multi-GPU gradient accumulator. See here
- 2020/08/23 Add Parallel WaveGAN tensorflow implementation. See here
- 2020/08/23 Add MBMelGAN G + ParallelWaveGAN G example. See here
- 2020/08/20 Add C++ inference code. Thank @ZDisket. See here
- 2020/08/18 Update new base processor. Add AutoProcessor and pretrained processor json file
- 2020/08/14 Support Chinese TTS. Pls see the colab. Thank @azraelkuan
- 2020/08/05 Support Korean TTS. Pls see the colab. Thank @crux153
- 2020/07/17 Support MultiGPU for all Trainer
- 2020/07/05 Support Convert Tacotron-2, FastSpeech to Tflite. Pls see the colab. Thank @jaeyoo from the TFlite team for his support
- 2020/06/20 FastSpeech2 implementation with Tensorflow is supported.
- 2020/06/07 Multi-band MelGAN (MB MelGAN) implementation with Tensorflow is supported
- High performance on Speech Synthesis.
- Be able to fine-tune on other languages.
- Fast, Scalable, and Reliable.
- Suitable for deployment.
- Easy to implement a new model, based-on abstract class.
- Mixed precision to speed-up training if possible.
- Support Single/Multi GPU gradient Accumulate.
- Support both Single/Multi GPU in base trainer class.
- TFlite conversion for all supported models.
- Android example.
- Support many languages (currently, we support Chinese, Korean, English.)
- Support C++ inference.
- Support Convert weight for some models from PyTorch to TensorFlow to accelerate speed.
This repository is tested on Ubuntu 18.04 with:
- Python 3.7+
- Cuda 10.1
- CuDNN 7.6.5
- Tensorflow 2.2/2.3
- Tensorflow Addons >= 0.10.0
Different Tensorflow version should be working but not tested yet. This repo will try to work with the latest stable TensorFlow version. We recommend you install TensorFlow 2.3.0 to training in case you want to use MultiGPU.
$ pip install TensorFlowTTS
Examples are included in the repository but are not shipped with the framework. Therefore, to run the latest version of examples, you need to install the source below.
$ git clone https://github.com/TensorSpeech/TensorFlowTTS.git $ cd TensorFlowTTS $ pip install .
If you want to upgrade the repository and its dependencies:
$ git pull $ pip install --upgrade .
Supported Model architectures
TensorFlowTTS currently provides the following architectures:
- MelGAN released with the paper MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis by Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brebisson, Yoshua Bengio, Aaron Courville.
- Tacotron-2 released with the paper Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions by Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ Skerry-Ryan, Rif A. Saurous, Yannis Agiomyrgiannakis, Yonghui Wu.
- FastSpeech released with the paper FastSpeech: Fast, Robust, and Controllable Text to Speech by Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu.
- Multi-band MelGAN released with the paper Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech by Geng Yang, Shan Yang, Kai Liu, Peng Fang, Wei Chen, Lei Xie.
- FastSpeech2 released with the paper FastSpeech 2: Fast and High-Quality End-to-End Text to Speech by Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu.
- Parallel WaveGAN released with the paper Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram by Ryuichi Yamamoto, Eunwoo Song, Jae-Min Kim.
- HiFi-GAN released with the paper HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis by Jungil Kong, Jaehyeon Kim, Jaekyoung Bae.
We are also implementing some techniques to improve quality and convergence speed from the following papers:
- Guided Attention Loss released with the paper Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention by Hideyuki Tachibana, Katsuya Uenoyama, Shunsuke Aihara.
Prepare a dataset in the following format:
|- [NAME_DATASET]/ | |- metadata.csv | |- wavs/ | |- file1.wav | |- ...
metadata.csv has the following format:
id|transcription. This is a ljspeech-like format; you can ignore preprocessing steps if you have other format datasets.
NAME_DATASET should be
[ljspeech/kss/baker/libritts] for example.
The preprocessing has two steps:
- Preprocess audio features
- Convert characters to IDs
- Compute mel spectrograms
- Normalize mel spectrograms to [-1, 1] range
- Split the dataset into train and validation
- Compute the mean and standard deviation of multiple features from the training split
- Standardize mel spectrogram based on computed statistics
To reproduce the steps above:
tensorflow-tts-preprocess --rootdir ./[ljspeech/kss/baker/libritts/thorsten] --outdir ./dump_[ljspeech/kss/baker/libritts/thorsten] --config preprocess/[ljspeech/kss/baker/thorsten]_preprocess.yaml --dataset [ljspeech/kss/baker/libritts/thorsten] tensorflow-tts-normalize --rootdir ./dump_[ljspeech/kss/baker/libritts/thorsten] --outdir ./dump_[ljspeech/kss/baker/libritts/thorsten] --config preprocess/[ljspeech/kss/baker/libritts/thorsten]_preprocess.yaml --dataset [ljspeech/kss/baker/libritts/thorsten]
Note: To run
libritts preprocessing, please first read the instruction in examples/fastspeech2_libritts. We need to reformat it first before run preprocessing.
After preprocessing, the structure of the project folder should be:
|- [NAME_DATASET]/ | |- metadata.csv | |- wav/ | |- file1.wav | |- ... |- dump_[ljspeech/kss/baker/libritts/thorsten]/ | |- train/ | |- ids/ | |- LJ001-0001-ids.npy | |- ... | |- raw-feats/ | |- LJ001-0001-raw-feats.npy | |- ... | |- raw-f0/ | |- LJ001-0001-raw-f0.npy | |- ... | |- raw-energies/ | |- LJ001-0001-raw-energy.npy | |- ... | |- norm-feats/ | |- LJ001-0001-norm-feats.npy | |- ... | |- wavs/ | |- LJ001-0001-wave.npy | |- ... | |- valid/ | |- ids/ | |- LJ001-0009-ids.npy | |- ... | |- raw-feats/ | |- LJ001-0009-raw-feats.npy | |- ... | |- raw-f0/ | |- LJ001-0001-raw-f0.npy | |- ... | |- raw-energies/ | |- LJ001-0001-raw-energy.npy | |- ... | |- norm-feats/ | |- LJ001-0009-norm-feats.npy | |- ... | |- wavs/ | |- LJ001-0009-wave.npy | |- ... | |- stats.npy | |- stats_f0.npy | |- stats_energy.npy | |- train_utt_ids.npy | |- valid_utt_ids.npy |- examples/ | |- melgan/ | |- fastspeech/ | |- tacotron2/ | ...
stats.npycontains the mean and std from the training split mel spectrograms
stats_energy.npycontains the mean and std of energy values from the training split
stats_f0.npycontains the mean and std of F0 values in the training split
valid_utt_ids.npycontains training and validation utterances IDs respectively
We use suffix (
wave) for each input type.
- This preprocessing step is based on ESPnet so you can combine all models here with other models from ESPnet repository.
- Regardless of how your dataset is formatted, the final structure of the
dumpfolder SHOULD follow the above structure to be able to use the training script, or you can modify it by yourself
To know how to train model from scratch or fine-tune with other datasets/languages, please see detail at example directory.
- For Tacotron-2 tutorial, pls see examples/tacotron2
- For FastSpeech tutorial, pls see examples/fastspeech
- For FastSpeech2 tutorial, pls see examples/fastspeech2
- For FastSpeech2 + MFA tutorial, pls see examples/fastspeech2_libritts
- For MelGAN tutorial, pls see examples/melgan
- For MelGAN + STFT Loss tutorial, pls see examples/melgan.stft
- For Multiband-MelGAN tutorial, pls see examples/multiband_melgan
- For Parallel WaveGAN tutorial, pls see examples/parallel_wavegan
- For Multiband-MelGAN Generator + Parallel WaveGAN Discriminator tutorial, pls see examples/multiband_pwgan
- For HiFi-GAN tutorial, pls see examples/hifigan
Abstract Class Explaination
Abstract DataLoader Tensorflow-based dataset
A detail implementation of abstract dataset class from tensorflow_tts/dataset/abstract_dataset. There are some functions you need overide and understand:
- get_args: This function return argumentation for generator class, normally is utt_ids.
- generator: This function have an inputs from get_args function and return a inputs for models. Note that we return a dictionary for all generator functions with the keys that exactly match with the model's parameters because base_trainer will use model(**batch) to do forward step.
- get_output_dtypes: This function need return dtypes for each element from generator function.
- get_len_dataset: Return len of datasets, normaly is len(utt_ids).
- A pipeline of creating dataset should be: cache -> shuffle -> map_fn -> get_batch -> prefetch.
- If you do shuffle before cache, the dataset won't shuffle when it re-iterate over datasets.
- You should apply map_fn to make each element return from generator function have the same length before getting batch and feed it into a model.
Abstract Trainer Class
A detail implementation of base_trainer from tensorflow_tts/trainer/base_trainer.py. It include Seq2SeqBasedTrainer and GanBasedTrainer inherit from BasedTrainer. All trainer support both single/multi GPU. There a some functions you MUST overide when implement new_trainer:
- compile: This function aim to define a models, and losses.
- generate_and_save_intermediate_result: This function will save intermediate result such as: plot alignment, save audio generated, plot mel-spectrogram ...
- compute_per_example_losses: This function will compute per_example_loss for model, note that all element of the loss MUST has shape [batch_size].
All models on this repo are trained based-on GanBasedTrainer (see train_melgan.py, train_melgan_stft.py, train_multiband_melgan.py) and Seq2SeqBasedTrainer (see train_tacotron2.py, train_fastspeech.py).
import numpy as np import soundfile as sf import yaml import tensorflow as tf from tensorflow_tts.inference import AutoConfig from tensorflow_tts.inference import TFAutoModel from tensorflow_tts.inference import AutoProcessor # initialize fastspeech model. fs_config = AutoConfig.from_pretrained('./examples/fastspeech/conf/fastspeech.v1.yaml') fastspeech = TFAutoModel.from_pretrained( config=fs_config, pretrained_path="./examples/fastspeech/pretrained/model-195000.h5" ) # initialize melgan model melgan_config = AutoConfig.from_pretrained('./examples/melgan/conf/melgan.v1.yaml') melgan = TFAutoModel.from_pretrained( config=melgan_config, pretrained_path="./examples/melgan/checkpoint/generator-1500000.h5" ) # inference processor = AutoProcessor.from_pretrained(pretrained_path="./test/files/ljspeech_mapper.json") ids = processor.text_to_sequence("Recent research at Harvard has shown meditating for as little as 8 weeks, can actually increase the grey matter in the parts of the brain responsible for emotional regulation, and learning.") ids = tf.expand_dims(ids, 0) # fastspeech inference masked_mel_before, masked_mel_after, duration_outputs = fastspeech.inference( ids, speaker_ids=tf.zeros(shape=[tf.shape(ids)], dtype=tf.int32), speed_ratios=tf.constant([1.0], dtype=tf.float32) ) # melgan inference audio_before = melgan.inference(masked_mel_before)[0, :, 0] audio_after = melgan.inference(masked_mel_after)[0, :, 0] # save to file sf.write('./audio_before.wav', audio_before, 22050, "PCM_16") sf.write('./audio_after.wav', audio_after, 22050, "PCM_16")
Minh Nguyen Quan Anh: email@example.com, erogol: firstname.lastname@example.org, Kuan Chen: email@example.com, Dawid Kobus: firstname.lastname@example.org, Takuya Ebata: email@example.com, Trinh Le Quang: firstname.lastname@example.org, Yunchao He: email@example.com, Alejandro Miguel Velasquez: firstname.lastname@example.org
Overall, Almost models here are licensed under the Apache 2.0 for all countries in the world, except in Viet Nam this framework cannot be used for production in any way without permission from TensorFlowTTS's Authors. There is an exception, Tacotron-2 can be used with any purpose. If you are Vietnamese and want to use this framework for production, you Must contact us in advance.