Theehawau / TalkTrain

Champion Project for Alibaba x GITEX Hackathon 2023

TalkTrain

     

Hawau Olamide Toyin ^1,2   Kane Lindsay¹ 

¹ Metaverse Lab, MBZUAI   ² Speech Lab, MBZUAI  

GITEX AI InnovateFest 2023 Powered by Alibaba Cloud

TalkTrain is an AI-powered public speaking and presentation practise application. Make your speech to our virtual assistant, and you will be provided with useful metrics, as well as questions extrapolated from your own presentation.

TalkTrain is an entry for the GITEX AI InnovateFest 2023 Hackathon, powered by Alibaba Cloud. Watch project presentation on YouTube.

(Coming SOON!!!) A step-by-step guide to build TalkTrain from scratch using Alibaba Cloud services

Features

• gradio_app.py - WebUI interface code and entry point
• utils.py - Helper functions for webUI event listeners, speech metrics calculation and Automatic-Speech-Recognition (ASR)
• config.json - Configuration for tokens, prompts etc
• tts.py - Helper functions for Text-To-Speech (TTS)
• generate_questions.py - Helper functions for Question-Generation (QG)
• /SadTalker/inference.py - Helper function for Face Animation (FA)

Environment

We developed TalkTrain in Ubuntu 20.04 OS with python version 3.10

Install Instructions

1. Clone this repository

git clone https://github.com/Theehawau/TalkTrain.git
cd TalkTrain

2. Install the necessary packages onto your machine with apt install or similar. portaudio19-dev python3-all-dev

3. Create an environment with Python 3.10

conda create -n TalkTrain python=3.10

3. In the environment, install packages in requirements.txt using pip.

conda activate TalkTrain
pip install -r requirements.txt

4. Please download the SadTalker weights from this Google drive for Face Animation(FA): https://drive.google.com/drive/folders/1UZxnS41k7QuseRqANcStSFKNXamUNtT_?usp=drive_link and place the folders in SadTalker.

5. Setup Alibaba cloud services and configure tokens in config and utils files:

   • Tongyi Qianwen LLM
   • Intelligent Spech Services

If you have ubuntu OS:

You can install all requirements with

bash install.sh 

Running Instructions

WebUI demo

conda activate TalkTrain
cd TalkTrain
gradio gradio_app.py

Test Pipeline

For testing the QG -> TTS -> FA pipeline, run

python llm+tts+avatar_example.py 

Issues, Limitations

You need an Alibaba cloud account and services initiated to test this out. A recorded demo is available on YouTube.

If you run into QS error , setting this environment variable solves this.

export QT_QPA_PLATFORM=offscreen 

Acknowledgements

TalkTrain builds on existing technologies. We are grateful for training session provided by Alibaba Cloud that facilitated using their platform and their consistent support through the development of this project.

• Question Generation(QG): Alibaba Tongyi Qianwen LLM
• Automatic Speech Recognition (ASR): OpenAI whisper
• TTS: Alibaba Intelligent Speech Interaction
• Avatar Animation: SadTalker