This project is a part of Mozilla Common Voice. TTS aims a deep learning based Text2Speech engine, low in cost and high in quality. To begin with, you can hear a sample generated voice from here.
TTS includes two different model implementations which are based on Tacotron and Tacotron2. Tacotron is smaller, efficient and easier to train but Tacotron2 provides better results, especially when it is combined with a Neural vocoder. Therefore, choose depending on your project requirements.
If you are new, you can also find here a brief post about TTS architectures and their comparisons.
Highly recommended to use miniconda for easier installation.
- python>=3.6
- pytorch>=0.4.1
- librosa
- tensorboard
- tensorboardX
- matplotlib
- unidecode
Install TTS using setup.py. It will install all of the requirements automatically and make TTS available to all the python environment as an ordinary python module.
python setup.py develop
Or you can use requirements.txt to install the requirements only.
pip install -r requirements.txt
A barebone Dockerfile exists at the root of the project, which should let you quickly setup the environment. By default, it will start the server and let you query it. Make sure to use nvidia-docker to use your GPUs. Make sure you follow the instructions in the server README before you build your image so that the server can find the model within the image.
docker build -t mozilla-tts .
nvidia-docker run -it --rm -p 5002:5002 mozilla-tts
Check out here to compare the samples (except the first) below.
| Models | Dataset | Commit | Audio Sample | Details |
|---|---|---|---|---|
| Tacotron-iter-62410 | LJSpeech | 99d56f7 | link | First model with plain Tacotron implementation. |
| Tacotron-iter-170K | LJSpeech | e00bc66 | link | More stable and longer trained model. |
| Tacotron-iter-270K | LJSpeech | 256ed63 | link | Stop-Token prediction is added, to detect end of speech. |
| Tacotron-iter-120K | LJSpeech | bf7590 | link | Better for longer sentences |
| Tacotron-iter-108K | TWEB | 2810d57 | link | mozilla#22 |
| Tacotron-iter-185K | LJSpeech | db7f3d3 | link | link |
| Tacotron2-iter-260K | LJSpeech | 824c091 | soundcloud | link |
Below you see Tacotron model state after 16K iterations with batch-size 32 with LJSpeech dataset.
"Recent research at Harvard has shown meditating for as little as 8 weeks can actually increase the grey matter in the parts of the brain responsible for emotional regulation and learning."
Audio examples: https://soundcloud.com/user-565970875
The most time-consuming part is the vocoder algorithm (Griffin-Lim) which runs on CPU. By setting its number of iterations lower, you might have faster execution with a small loss of quality. Some of the experimental values are below.
Sentence: "It took me quite a long time to develop a voice, and now that I have it I'm not going to be silent."
Audio length is approximately 6 secs.
| Time (secs) | System | # GL iters | Model |
|---|---|---|---|
| 2.00 | GTX1080Ti | 30 | Tacotron |
| 3.01 | GTX1080Ti | 60 | Tacotron |
| 3.57 | CPU | 60 | Tacotron |
| 5.27 | GTX1080Ti | 60 | Tacotron2 |
| 6.50 | CPU | 60 | Tacotron2 |
TTS provides a generic dataloder easy to use for new datasets. You need to write an preprocessor function to integrade your own dataset.Check datasets/preprocess.py to see some examples. After the function, you need to set dataset field in config.json. Do not forget other data related fields too.
Some of the open-sourced datasets that we successfully applied TTS, are linked below.
Here you can find a CoLab notebook for a hands-on example, training LJSpeech. Or you can manually follow the guideline below.
To start with, split metadata.csv into train and validation subsets respectively metadata_train.csv and metadata_val.csv. Note that for text-to-speech, validation performance might be misleading since the loss value does not directly measure the voice quality to the human ear and it also does not measure the attention module performance. Therefore, running the model with new sentences and listenning the results is the best way to go.
shuf metadata.csv > metadata_shuf.csv
head -n 12000 metadata_shuf.csv > metadata_train.csv
tail -n 1100 metadata_shuf.csv > metadata_val.csv
To train a new model, you need to define your own config.json file (check the example) and call with the command below. You also set the model architecture in config.json.
train.py --config_path config.json
To fine-tune a model, use --restore_path.
train.py --config_path config.json --restore_path /path/to/your/model.pth.tar
For multi-GPU training use distribute.py. It enables process based multi-GPU training where each process uses a single GPU.
CUDA_VISIBLE_DEVICES="0,1,4" distribute.py --config_path config.json
Each run creates a new output folder and config.json is copied under this folder.
In case of any error or intercepted execution, if there is no checkpoint yet under the output folder, the whole folder is going to be removed.
You can also enjoy Tensorboard, if you point Tensorboard argument--logdir to the experiment folder.
Best way to test your network is to use Notebooks under notebooks folder.
-
Discourse Forums - If your question is not addressed in the Wiki, the Discourse Forums is the next place to look. They contain conversations on General Topics, Using TTS, and TTS Development.
-
Issues - Finally, if all else fails, you can open an issue in our repo.
- Implement the model.
- Generate human-like speech on LJSpeech dataset.
- Generate human-like speech on a different dataset (Nancy) (TWEB).
- Train TTS with r=1 successfully.
- Enable process based distributed training. Similar to (https://github.com/fastai/imagenet-fast/).
- Adapting Neural Vocoder. TTS works with (https://github.com/erogol/WaveRNN)
- Multi-speaker embedding.
- Model optimization (model export, prunning etc.)
- https://github.com/keithito/tacotron (Dataset and Test processing)
- https://github.com/r9y9/tacotron_pytorch (Initial Tacotron architecture)