Exploring "linear attention" for text-to-speech.
It predicts audio codec "à la" MusicGen : delayed residual vector quantizers so that we do not need multiple models.
Featuring RWKV, Mamba, Gated Linear Attention.
Compared to other LM TTS model :
- Can be easily pretrained and finetuned on midrange GPUs.
- Tiny memory footprint.
- Trained on long context (up to 2000 tokens : ~27s).
Model | #Params | Dataset | Checkpoint | Steps | Note |
---|---|---|---|---|---|
GLA | 60M, 130M | Librilight-medium | Download | 300k | GPU inference only |
Mamba | 60M | Librilight-medium | Download | 300k | GPU inference only |
RWKV v6 | 60M | LibriTTS | Download | 150k | GPU inference only |
Following the linear complexity LM you choose, follow respective instructions first:
- For Mamba check the official repo.
- For GLA/RWKV inference check flash-linear-attention.
- For RWKV training check RWKV-LM
Download configuration and weights above, then check Inference.ipynb
.
- Fix RWKV6 inference and/or switch to FLA implem.
- Provide a Datamodule for training (lhotse might also work well).
- Implement CFG.
- Scale up.
- The RWKV authors and the community around for carrying high-level truly opensource research.
- @SmerkyG for making my life easy at testing cutting edge language model.
- @lucidrains for its huge codebase.
- @sustcsonglin who made GLA and FLA.
- @harrisonvanderbyl fixing RWKV inference.
@software{lemerle2024linaspeech,
title = {LinaSpeech: Exploring "linear attention" for text-to-speech.},
author = {Lemerle, Théodor},
url = {https://github.com/theodorblackbird/lina-speech},
month = april,
year = {2024}
}
This work is performed in the Analysis/Synthesis team of the STMS Laboratory at IRCAM, and is part of the following project: ANR Exovoices
![](https://private-user-images.githubusercontent.com/1331899/330370394-7391b3c2-ec9a-431e-a090-f2ac5f55026b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjEyMTIxODgsIm5iZiI6MTcyMTIxMTg4OCwicGF0aCI6Ii8xMzMxODk5LzMzMDM3MDM5NC03MzkxYjNjMi1lYzlhLTQzMWUtYTA5MC1mMmFjNWY1NTAyNmIucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDcxNyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MTdUMTAyNDQ4WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NjA2ZGVhNWI2YjRhZjJlYjY5MmZjZjUxMTgwMjhhODdiYTYwMzg2MTgyODE3ZDMxYjg3MGVjMzYwOWYzODlhYyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.99k3P9WTVupgILQ604rOf7rbeEjnhKxsqvqDPOM1V4I)
![](https://private-user-images.githubusercontent.com/1331899/330370291-74cc1ade-be95-4087-9cc1-83af6d7a54be.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjEyMTIxODgsIm5iZiI6MTcyMTIxMTg4OCwicGF0aCI6Ii8xMzMxODk5LzMzMDM3MDI5MS03NGNjMWFkZS1iZTk1LTQwODctOWNjMS04M2FmNmQ3YTU0YmUucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDcxNyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MTdUMTAyNDQ4WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YmE1MTgzNWFiMGI1MTFjNzlkNWNjZjViZjlmOTQ2ODZiYTQ0YWU3NjA1ZmViNDA3MWNlNDAwODRhMmUyNTQyYiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.5GUv_FKwCqen_O4pNj77hRm1uKYB7m4Mo6O5zs71wA4)
![](https://private-user-images.githubusercontent.com/1331899/330370469-fc0ae259-26ae-451b-8893-80471255479d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjEyMTIxODgsIm5iZiI6MTcyMTIxMTg4OCwicGF0aCI6Ii8xMzMxODk5LzMzMDM3MDQ2OS1mYzBhZTI1OS0yNmFlLTQ1MWItODg5My04MDQ3MTI1NTQ3OWQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDcxNyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MTdUMTAyNDQ4WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NmQ2OTA3NzY5ZWY0MWMwZTliN2VlNzhlZGMzYzhjMjYzZDJkNjQ4YjE1ZWE5YWFiMDc5MGEzMWJlZGIyZTUyMCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.3ov-62ZyNcu7sRQY7IwPSNVBRVFdpuJPWH6IZmy_W_w)