nasim-alamdari / RealTime-Custom-Keyword-Spotting

Implementation and Deployment of Multilingual Custom Keyword Spotting Running in Real-time on an Edge Device.

Real-time Multilingual Custom Keyword Spotting

• Built a lightweight few-shot keyword spotting (FS-KWS) for personalization of keyword spotting or wake-word detection running in real-time on an edge device.
• The customization is achieved by recording audio from the user for less than 30 seconds, followed by enhancement and segmentation of the speaker's audio clip to 1-second speech audio files via deep learning models [1] and [2]. Segmented audio files are then used to fine-tune and customize an efficientnet-B0-based multilingual keyword spotting model through few-shot learning. The baseline model that we used in this study is based on Harvard-edge[3] work.
• Our model operates on short audio chunks (1 second) but at a much higher temporal resolution (every 25 ms).
• Average processing time for each 1-second audio chunk observed to be 35 ms.

Solution Architecture

Component Setup - step by step

1. Clone the Project

git clone https://github.com/nasim-alamdari/RealTime-Custom-Keyword-Spotting.git
cd RealTime-Custom-Keyword-Spotting

2. Import and install relevant libraries to your Python project.

conda create --name kws
conda activate kws
pip install -r requirement.txt

3. Regarding installing Pyaudio on M1 Mac or EC2 :

# for M1 Mac:
brew install portaudio
pip install PyAudio

# for EC2 Ubuntu:
sudo apt install libasound-dev portaudio19-dev libportaudio2 libportaudiocpp0 ffmpeg libav-tools
pip install PyAudio

4. Enjoy real-time custom keyword spotting inside your streamlit app! 🎈

cd Code
streamlit run streamlit_rltime_app.py

5. To Deploy App on AWS EC2 Cloud ☁️🤖

Follow Steps 1-3, then:

sudo apt-get install tmux
cd Code

tmux new -s sess_streamlit
streamlit run streamlit_ec2.py

tmux new -s sess_fastapi
uvicorn main_ec2:app --reload --workers 1 --host 0.0.0.0 --port 8000

Keyword Spotting App on AWS EC2:

Using Streamlit and FastAPI Model Serving

Demo Link: YouTube🎥

Real-Time Deployment via Streamlit:

The real-time version of Streamlit App should be run locally.

Demo Link: YouTube🎥

Presentation Link: YouTube🎥

Contributors:

Nasim Alamdari and Christos Magganas

Feel free to reach out to us in case you have any questions!
Pls consider leaving a star ☆ with this repository to show your support.