Deep-learning-based Text-to-speech-TTS Papers and resources
Various Text-to-speech (TTS) papers and resources based on Deep-learning
Data
[Melspectrogram]
- Speech Technology: A Practical Introduction, Topic: Spectrogram, Cepstrum and Mel-Frequency Analysis (K. Prahallad., CMU, slide, video)
Mel-spectrogram Generator
[Autoregressive]
RNN
- Char2Wav (J. Soleto et. al., Feb. 2017., Motreal., paper)
- Tacotron (Y. Wang et. al., Mar. 2017., Google, arxiv)
- Tacotron 2 (J. Shen et.al., Dec. 2017., Google, arxiv)
CNN
- Deep Voice 3 (W. Ping et. al., Oct. 2017., Baidu, arxiv)
- Deep Convolutional Text-to-speech (H. Tachibana et. al., Oct. 2017., arxiv)
Transformer
- Transformer TTS (N. Li et. al., Sep. 2018., Microsoft, arxiv)
[Non-autoregressive]
CNN
- ParaNet (K. Peng et. al., May. 2019., Baidu, arxiv)
Transformer
- Fast Speech (Y. Ren et. al., May. 2019., Microsoft, arxiv)
- Align TTS (Z. Zeng et. al., Mar. 2020., Ping An Tech., arxiv)
- Fast Speech 2 (Y. Ren et. al., Jun. 2020., Microsoft, arxiv)
[Graph Neural Networks]
- Graph TTS (A. Sun et. al., Mar. 2020., Ping An Tech., arxiv)
[Attention Improvement]
- Monotonic Attention (C. Raffel et. al., Jun. 2017., Google Brain, arxiv)
- Monotonic Chunkwise Attention (C.C. Chiu et. al., Dec. 2017., Google Brain, arxiv)
- Stepwise Monotonic Attention (M. He et. al., Jun. 2019., Microsoft, arxiv)
- Location-relative Attention Mechanisms for Robust Long-form Speech Synthesis (E. Battenberg et. al., Oct. 2019., Google, arxiv)
[Training Algorithm]
- A New GAN-based Training Algorithm (H. Guo. et. al., Apr. 2019., Microsoft, arxiv)
[Data-Efficient]
- Semi-supervised Training for Improving Data Efficiency in End-to-end Speech Synthesis (Y.A. Chung et. al., Aug. 2018., MIT & Google, arxiv)
- Sample Efficient Adaptive Text-to-Speech (Y. Chen., Jan. 2019., DeepMind & Google, arxiv)
Neural Vocoder
[Autoregressive Model]
- WaveNet (A. V. Oord et. al., Sep. 2016., Deep Mind, arxiv)
- SampleRNN (S. Mehri et. al., Dec. 2016, Montreal, arxiv)
[Inverse Autoregressive Flow Model]
- Parallel WaveNet (A. V. Oord et. al., Nov. 2017., Deep Mind, arxiv)
- ClariNet (W. Ping et. al., Jul. 2018., Baidu, arxiv)
- WaveGlow (R. Prenger et. al., Nov. 2018., NVIDIA, arxiv)
- FlowWaveNet (S. Kim et. al., Nov. 2018., SNU, arxiv)
[Generative Adversarial Network]
- WaveGAN (C.Donahue et. al., Feb. 2018., UCSD, arxiv)
- GAN-TTS (M. Binkowski, Sep. 2019., Google, arxiv)
- Parallel WaveGAN (R. Yamamoto et. al., Oct. 2019., Naver, arxiv)
Style Modeling
[Style Token]
- Uncovering Latent Style Factors for Expressive Speech Synthesis (Y. Wang et. al., Nov. 2017., Google, arxiv)
- GST Tacotron (Y. Wang et. al., Mar. 2018., Google, arxiv)
- TP-GST Tacotron (D. Santon et al., Aug. 2018., Google, arxiv)
[Generative Adversarial Network]
- TTS-GAN (S. Ma et. al., Apr. 2019., Microsoft, paper)
- Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cross-Language Voice Cloning (Y. Zhang et. al., Jul. 2019., Google, arxiv)
- Disentangling Correlated Speaker and Noise for Speech Synthesis via Data Augmentation and Adversarial Factorization (W. N. Hsu et. al., Sep. 2019., Google, paper)
[Mutual Information]
- Unsupervised Style and Content Separation by Minimizing Mutual Information for Speech Synthesis (T.Y. Hu et. al., Mar. 2020., CMU & Apple, arxiv)
Dataset
[English]
ASR
- LibriSoeech dataset (paper, download) (V. Panayotov et al., 2015)
- total 2484 speakers, 1000+ hours
- VocCeleb1 dataset (paper download) (A. Nagrani et al., 2017)
- 151,516 utterances, 1251 speakers, 352 hours
- VoxCeleb2 dataset (paper download) (J. S> Chung et al., 2018)
- 1,128,246 utterances, 6112 speakers, 2442 hours
TTS
- LJSpeech dataset download (Keith Ito and Linda Johnson, 2017)
- single female speaker, 13100 samples, approximately 24 hours
- CSTR VCTK Corpus (download (C. Veaux et. al.))
- 109 english speakers with various accents, 400 speeches per speaker
- Blizzard dataset download
[Korean]
ASR
- 한국어 및 영어 음향모델 훈련용 음성 데이터 (download) (ETRI)
- (Korean speech) 50 speakers * 100 speeches/speaker (total 5,000 speech samples)
- (English speech pronounciated by Korean) 50 speakers * 100 speech (total 5,000 speech samples)
- 음성인터페이스 개발을 위한 어린이 음성 데이터 (download) (ETRI)
- 50 speakers * 100 speeches/speaker * 3 environments (total 16,200 speech samples)
- speaker info: elementary school students (from 1st to 4th grade)
- recorded from IPhone5, Samsung GalaxyS4, and microphones
- ClovaCall datset (paper download) (Naver Corp.)
- 140000+ speeches, 211+ hours of noisy and clean speech
- KSponSpeech (download) (ETRI)
- 2000 speakers, 1000+ hours, various topics(life, shopping, hobby, weather, etc..)
- Korean Read Speech Corpus (download) (국립국어원)
- 8 speakers, 120+ hours
- Zeroth-Korean (download) (Lucas Jo and Wonkyum Lee)
- 115 speakers, 52.8 hours, 22720 utterances in total
- Pansori TED x KR Corpus (paper download) (Y. Choi and B. Lee)
- 3 hours, 41 speakers
TTS
- Korean Single Speaker (KSS) Speech Dataset (download) (K. Park)
- single female speaker, 12853 samples, 12+ hours
- 감정 음성합성 데이터셋 (download) ((주) 아크릴)
- single female speaker, 7 emotion (neutral, sad, fear, happy, angry, disgusting, surprise), total 22,000 samples (about 3,000 samples per emotion)
- EmotionTTS-Open-DB dataset (download) (KAIST and (주) 셀바스AI)
- single-speaker, multi-speaker, and multi-speaker-multi-emotion dataset
- 카이스트 오디오북 데이터셋 (download) (KAIST)
- 58559 speeches, 72+ hours, 13 speakers
- various reading materials (news, novel, etc..)