LifangHe / mica-deep-mcca

Deep Multiset Canonical Correlation Analysis - An extension of CCA to multiple datasets

Multimodal Representation Learning using Deep Multiset Canonical Correlation Analysis

Paper HERE

In this work, we propose an extension for CCA to model more than 2 modalities: Multiset CCA. With deep learning framework we show that MCCA can be used to model large and complex datasets.

Keras==2.2.2
pandas==0.19.2
matplotlib==2.1.0
scipy==1.1.0
numpy==1.15.1
h5py==2.8.0
Theano==1.0.2+2.gc449c8699
seaborn==0.9.0
scikit_learn==0.20.0

The paper was submitted to ICASSP 2019. A draft version can be found here

Detailed experimental results can be found here in the wiki page

Download the noisy MNIST and noisy Bangla datasets, or from here. And process them using the scipts in this repo. For downloading synthetic data, send a request to this email.

After the data is downloaded, follow the comments in the DeepMCCA.py script. The instructions must be self explanatory.