liyuejul / multimodal-decoding

Code associated with the paper titled "A high-performance neuroprosthesis for speech decoding and avatar control" , published in Nature in 2023.

multimodal-decoding

Code for our paper: "A high-performance neuroprosthesis for speech decoding and avatar control" (https://www.nature.com/articles/s41586-023-06443-4)

To explore the code for each modality, check out the corresponding folder!

Text decoding

text folder - train a model on the 1024-word-General sentence set and evaluate it on completely unseen sentences!

Synthesis

synthesis - train a model using HuBERT units for speech synthesis from brain activity

Avatar

avatar - train a model on orofacial movements

Thank you for your interest in our work!