A lighter-weight (perhaps!) Text-to-Speech for Chinese/Mandarin synthesize, inspired by Tacotron & FastSpeech2 & RefineGAN.
It is also my shitty graduation design project, just a toy, so lower your expectations :)
Since TransTacoS
is implemented in tensorflow while RefineGAN
in torch respectively, you could separate them by creating virtual envs, but they are likely not to conflict, thus you could try to put all these together:
- install
tensorflow-gpu==1.14.0 tensorboard==1.14.0
followinghttps://tensorflow.google.cn/install/pip
- install
torch==1.8.0+cu1xx torchaudio==0.8.0
followinghttps://pytorch.org/
, wherecu1xx
is your cuda version - run
pip install -r requirements.txt
for the rest dependencies
- download and unzip the open source dataset DataBaker
- other dataset requires user-defined preprocessor, please refer to
transtacos/dataset/__skel__.py
- check path configs in all
Makefile
cd transtacos & make preprocess
to prepare acoustic features (linear/mel/f0/c0/zcr)cd transtacos & make train
to train TransTacoScd retunegan & make finetune
to train RetuneGAN using preprocessed linear spectrograms (rather than from raw wave)
- check port configs in all
Makefile
cd transtacos & make server
to start TransTacoS headless HTTP server (default at port 5105)cd retunegan & make server
to start RetuneGAN headless HTTP server (default at port 5104)python app.py
to start the WebUI app (default at port 5103)- point your browser to
http://localhost:5103
, now have a try!
- TransTacoS :=
- embed: self-designed G2P solution (syl4), employ prodosy marks as linguistic feature
- for G2P solution, I tried char(seq), phoneme, pinyin (CV), CVC, VCV, CVVC ...
- de facto, they merely influence mel_loss, but effect that how controllable the pronounciation is
- so I design syl4 to split phoneme against tone (just small improvement)
- I predict prodosy marks by simple CNN rather than RNN, it's not enough reasonable though..
- for G2P solution, I tried char(seq), phoneme, pinyin (CV), CVC, VCV, CVVC ...
- encoder: modified from FastSpeech2
- I use multi-head self-DotAttn with GFFW (gated-FFW) for backbone transform
- f0/c0 feature is quantilized to embed, then fused in with cross-DotAttn
- decoder: inherited from Tacotron
- this RNN+LSA decoder is really complicated, I dare not to touch :(
- posnet: self-designed simple MLP and grouped-Linear
- I found simple Linear layer with growing depth leads to lower mel_loss than Conv1d
- thus realy DO NOT understand why Tacotron2 even FastSpeech2 still use a postnet directly strech n_channels from 80 (n_mel) to 1025 (n_freq), this is surely tooooo hard to compensate information loss
- embed: self-designed G2P solution (syl4), employ prodosy marks as linguistic feature
Frankly speaking, TransTacoS didn't improve any thing profoundly from Tacotron, but I just found that shallower network leads to lower mel_loss, so maybe simple embed+decoder is already enough :(
- To predict
f0/sp/ap
features so that we can use WORLD vocoder- It seems
sp
is OK, because it resembles mel very much - but
ap
requires to be carefully normalized, and accuratef0
is even harder to predict - audio quality of WORLD sounds worse than Griffin-Lim at times :(
- It seems
- To predict spectrograms' magnitude part together with phase part, so that we can directly vocode using pure
istft
- It should be hard to optimize losses on phase, especially on conjunction points between mel frames
- but these guys claim that they did it: iSTFTNet
- To predict
f0
anddyn
so that vocoder might benefits- I don't know how to separate them from mel, because so far I must regularize decoder's output to be mel (for the sake of teacher force)
- When I remove the mel loss, I found that the RNN decoder became lazy to learn, and the align model also not work
- mel is even more quantized that linear, to abstract
f0
anddyn
from only mel seems not the reasonable
- To extract duration info from the soft-aligned alignment map, so that we can further train a non-autogressive FastSpeech2 to speed up inference
- just like FastSpeech and DeepSinger did
- but FastSpeech reported that extracted duration is not enough accurate, yet I could not fully understand and reproduce DeepSinger's DP algorithm for duration extraction
- RetuneGAN :=
- preprocess
- extract
reference wav
using Griffin-Lim - extract
u/v mask
by hand-tuned zrc/c0 threshold (for Split-G only)
- extract
- generators
UNet-G
(encoder-decoder generator): modified from RefineGAN, we use the output of Griffin-Lim as reference wav, rather than an F0/C0-guided hand-crafted speech templateSplit-G
(split u/v generator): self-designed, inspired by Multi-Band MelGAN, but I found the generated quality is holy shit :(ResStack
borrowed from MelGANResBlock
modified from HiFiGAN
- discriminators
MSD
(multi scale discriminator): borrowed from MelGAN, I think it's good for plosive consonantsMPD
(multi period discriminator): borrowed from HiFiGAN, I take it as a multiple MSDs' stack-upMTD
(multi stft discriminator): modified from UnivNet, it has two work modes depending on its input (MPSD seems better indeed ...)MPSD
(multi parameter spectrogram discriminator): like in UnivNet, but we let it judge both phase part and magnitude partPHD
(phase discriminator): self-designed, care more about phase, sincel_mstft
has already regulated magnitude- input of MPSD is
[(mag_real, phase_real), (mag_fake, phase_fake)]
, thus distinguishes real/fake stft data - input of PHD is
[(mag_real, phase_real), (mag_real, phase_fake)]
, thus ONLY distinguishes real/fake phase
- input of MPSD is
- losses
l_adv
(adversarial loss): modified from HiFiGAN, but relativizedl_fm
(feature map loss): borrowed from MelGANl_mstft
(multi stft loss): modified from Parallel WaveGAN, but we calculate mel_loss rather than linear_lossl_env
(envlope loss): borrowed from RefineGANl_dyn
(dynamic loss): self-designed, inspired byl_env
l_sm
(strip mirror loss): self-designed, but might hurts audio quality :(
- preprocess
Oh my dude, it's really a biggy feng-he monster :(
- To divide consonant part against vowel part in time domain, then use two generator to generate them separately
- I found this will bring breakups in conjunction points, thus audio sounds noisy, yet mstft loss will be more unstable
- To shallow fuse the reference wav with half-decoded mel, rather than the
encode-merge-decode
UNet architecture- I found this will make the generator lazy to learn, even overfit to train set
- might because waveform is far from audio semantics but near representation, so an encoder is necessary to extract semantical info
- NOTE: the weight of mstft loss should NOT be too overwhelming in front of adversarial loss (like in HiFiGAN)
- adversarial loss leads to more clear plosives (
b/p/g/k/d/t
), whilemstft loss
contributes little to consonants hop_length
in stft is much larger thatstride
in discriminators, thus mstft loss is usually more coarse than adversarial loss in time domain
- adversarial loss leads to more clear plosives (
Codes referred to:
Ideas plagiarized from:
- Tacotron: Towards End-to-End Speech Synthesis
- FastSpeech: Fast, Robust and Controllable Text to Speech
- FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
- MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
- Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech
- HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
- UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation
- RefineGAN: Universally Generating Waveform Better than Ground Truth with Highly Accurate Pitch and Intensity Responses
code release kept under the MIT license, greatest thanks all the authors!! :)
by Armit 2022/02/15 2022/05/25