entn-at / CapSpeech

CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech

🧢 CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech

📄 Paper  |  🌐 Project Page  |  🗂 Dataset  |  🤗 Models  |  🚀 Live Demo

Introduction

🧢 CapSpeech comprises over 10 million machine-annotated audio-caption pairs and nearly 0.36 million human-annotated audio-caption pairs. CapSpeech provides a new benchmark including these tasks:

1. CapTTS: style-captioned TTS

2. CapTTS-SE: text-to-speech synthesis with sound effects

3. AccCapTTS: accent-captioned TTS

4. EmoCapTTS: emotion-captioned TTS

5. AgentTTS: text-to-speech synthesis for chat agent

capspeech.mp4

Usage

⚡ Quick Start

Explore CapSpeech directly in your browser — no installation needed.

• 🚀 Live Demo: 🤗 Spaces

🛠️ Local Deployment

Install and Run CapSpeech locally.

• 💿 Installation & Usage: 📄 Instrucitons

Development

Please refer to the following documents to prepare the data, train the model, and evaluate its performance.

• Data Preparation
• Training
• Evaluation

Main Contributors

• Helin Wang at Johns Hopkins University
• Jiarui Hai at Johns Hopkins University

Citation

If you find this work useful, please consider contributing to this repo and cite this work:

@misc{wang2025capspeechenablingdownstreamapplications,
      title={CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech}, 
      author={Helin Wang and Jiarui Hai and Dading Chong and Karan Thakkar and Tiantian Feng and Dongchao Yang and Junhyeok Lee and Laureano Moro Velazquez and Jesus Villalba and Zengyi Qin and Shrikanth Narayanan and Mounya Elhiali and Najim Dehak},
      year={2025},
      eprint={2506.02863},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2506.02863}, 
}

License

All datasets, listening samples, source code, pretrained checkpoints, and the evaluation toolkit are licensed under the Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).
See the LICENSE file for details.

Acknowledgements

This implementation is based on Parler-TTS, F5-TTS, SSR-Speech, Data-Speech, EzAudio, and Vox-Profile. We appreciate their awesome work.

🌟 Like This Project?

If you find this repo helpful or interesting, consider dropping a ⭐ — it really helps and means a lot!