Cross-Caps / STFADE

Spectral Temporal Feature intergAtion in Deep acoustic modEls

STFADE 🎤

Spectral Temporal Feature integration in Deep acoustic models

Loss Landscapes of different models at a glance

Gradient Maps for different models at a glance

What's New?

• (05/1/2021) Trained Depthwise Seprable Convolution (DSC) based vanilla Contextnet over Librispeech Dataset http://arxiv.org/abs/2005.03191
• (05/6/2021) Implemented Low rank Convolution (LRC) in ContextNet with both spectrogram and raw audio input.
• (05/16/2021) Generated Loss Landscapes for the trained models, see demo_loss
• (05/20/2020) Trained a LRC based deepnet with wave input Low rank decomposition model
• (06/7/2020) Generated Integrated Gradients for trained models Keras Integrated Gradients documentation
• (20/7/2020) Added demo COLAB notebooks and animation for visualization

Table of Contents

• What's New?
• Table of Contents
• Publications
• Installation
• Training & Testing Steps
• Visualisation Loss Landscapes and Gradient Maps
• References & Credits
• Contact

Publications

• ContextNet (Reference)

• Raw Waveform Based CNN Through Low-Rank Spectro-Temporal Decoupling (Reference)

  Low rank spectro-temporal decoupling implementation in this project

  

Installation

git clone https://github.com/Cross-Caps/STFADE.git
cd STFADE
python setup.py build
python setup.py install

Training & Testing Steps

1. Define config YAML file, see the config.yml files in the contextnent folder for reference (you can copy and modify values such as parameters, paths, etc.. to match your local machine configuration)
2. Download your corpus (a.k.a datasets) and run download_links.shscripts folder to download files For more detail, see datasets. Note: Make sure your data contain only characters in your language, for example, english has a to z and '. Do not use cache if your dataset size is not fit in the RAM.
3. run create_transcripts_from_data.sh from scripts folder to generate .tsv files(the format in which the input is given is .tsv). Librispeech has been used in this work.
4. For training, see train.py files in the contextnet folder to see the options
5. For testing, see test.py files in the contextnet folder to see the options.

Visualisation Loss Landscapes and Gradient Maps

1. Loss Landscapes

    cd contextnet/contextnet_visualisation/loss_landscape_visualisation
    python generate_lists.py   
    python plot_loss.py
    python video_create.py
   

   Loss Landscape: Demo Notebook

   For Loss Landscape, go to loss video

2. Gradient Maps

    cd contextnet/contextnet_visualisation/gradient_visualisation
    python integrated_grad_vis.py
    python plot_gradients.py
    python video_create.py
   

   Gradient Visualisation: Demo Notebook

   For Gradient Maps, go to gradients videos

References & Credits

1. TensorFlowASR
2. Loss landscape visualisation
3. Keras Integrated Gradients

Contact

Vaibhav Singh ([e-mail])

Dr. Vinayak Abrol ([e-mail])