Author: @RobertTLange [Last Update: August 2021]
This repository contains a solution approach to the Algonauts Challenge 2021, in which one had to predict fMRI-recorded voxel activity across different ROIs/subjects and for different small video snippet stimuli.
You can read my challenge report with details on exploration and final submission here.
Short solution description: The final submission is based on a SimCLR-v2 contrastive pre-trained ResNet-50, which was fine-tuned on supervised labels of ImageNet. The layer-wise features extracted from the videos were PCA-dimensionality reduced down to 50 features. Afterwards, we run a Bayesian Optimization (BO) Loop to tune the hyperparameters of a PLS regression encoding model. The BO procedure is layer-, subject- and ROI-specific and uses 10-fold cross-validation. Afterwards, we select the best performing encoding model and retrain on all video-voxel datapoints. The full pipeline is depicted below:
Install all required dependencies (in a clean conda environment):
./setup.sh
Generate activations from different neural network architectures (e.g., AlexNet, VGG, ResNets, VOneNetworks or SimCLR-v2):
python run_features.py
Note that in order to load the network checkpoint you will previously have to download and convert the pre-trained model checkpoints:
cd feature_extraction/simclr_v2_model
python download.py
python convert.py
Afterwards, you can run a Bayesian Optimization loop to tune (50 iterations) for Subject 1 and ROI V1 with SimCLR-v2 activations, PCA-50 features and PLS regression encoding by executing:
python run_bayes_opt.py -config_fname configs/train/base_config.json
Running every single configuration sequentially can take a long time. We parallelize the general Encoding-10-Fold-CV-BayesOpt pipeline over subjects and ROIs. In order to do so efficiently we rely on the MLE-Toolbox. It provides a framework for logging and scheduling experiments on either Slurm, OpenGridEngine or Google Cloud Platform. A grid search over subjects and ROIs can then be launched via:
mle run configs/cluster/base_roi.yaml
Finally, if you would like to run the Auto-Sklearn v.1.0 pipeline for a subject, you can do so via
python run_autosklearn.py -config_fname configs/train/roi/auto_resnet50.json
jupyter lab notebooks/inspect_experiments.ipynb
jupyter lab notebooks/manual_submit.ipynb
This repository makes use of several modified open source repositories. We include their respective license in the respective subdirectories. The code repositories include the following:
SimCLR-v2-PyTorchby Separius - GPL-3.0 License.barlowtwinsby facebookresearch - MIT License.vonenetby dicarlolab - GPL-3.0 License.
Finally, we are thankful to the timm package creators for providing easily accessible model checkpoints and the algonauts challenge organizers for putting together this challenge!