Candidate Models for Brain-Score: Which Artificial Neural Network is most Brain-Like?
Candidate models to evaluate on brain measurements, i.e. neural and behavioral recordings. Brain recordings are packaged in Brain-Score.
Quick setup
pip install "candidate_models @ git+https://github.com/brain-score/candidate_models"
The above command will not install ML frameworks such as Pytorch, TensorFlow or Keras.
You can install them yourself or using the following commands (in a conda environment):
- Pytorch:
conda install pytorch torchvision -c pytorch
- Keras:
conda install keras
- TensorFlow:
conda install tensorflow
. To use the predefined TensorFlow models, you will have to install the TF-slim library. See here for quick instructions.
Usage
PYTHONPATH=. python candidate_models --model alexnet
During first-time use, ImageNet validation images (9.8 GB) will be downloaded, so give it a couple of minutes.
See the examples for more elaborate examples.
Environment variables
Environment variables are prefixed with CM_
for this framework.
Environment variables from brain-score
and model-tools might also be useful.
Variable | Description |
---|---|
CM_HOME | path to framework root |
CM_TSLIM_WEIGHTS_DIR | path to stored weights for TensorFlow/research/slim models |
MT_IMAGENET_PATH | path to ImageNet file containing the validation image set |
RESULTCACHING_HOME | directory to cache results (benchmark ceilings) in, ~/.result_caching by default (see https://github.com/mschrimpf/result_caching) |
Installing the TF-slim image models library
TensorFlow-slim does unfortunately not provide an actual pip-installable library here, instead we have to download the code and make it available.
git clone https://github.com/qbilius/models/ tf-models
export PYTHONPATH="$PYTHONPATH:$(pwd)/tf-models/research/slim"
# verify
python -c "from nets import cifarnet; mynet = cifarnet.cifarnet"
Alternatively, you can also move/symlink these packages to your site-packages.
Troubleshooting
Could not find a version that satisfies the requirement brain-score
pip has trouble when dependency links are private repositories (as is the case now for brain-score).
To circumvent, install brain-score by hand before installing candidate_models:
pip install --process-dependency-links git+https://github.com/dicarlolab/brain-score
.
Could not find a version that satisfies the requirement tensorflow
TensorFlow doesn't always catch up with newer Python versions.
For instance, if you have Python 3.7 (check with python -V
), TensorFlow might only work up to Python 3.6.
If you're using conda, it usually installs the very newest version of Python.
To fix, downgrade python: conda install python=3.6
.
Failed to build pytorch
Some versions of PyTorch cannot be installed via pip (e.g. CPU).
Instead, you need to build pytorch from their provided wheel.
Check the website for installation instructions, right now they are (e.g. for Linux, Python 3.6, no CUDA):
pip install http://download.pytorch.org/whl/cpu/torch-0.4.1-cp36-cp36m-linux_x86_64.whl && pip install torchvision
.
Or just use conda, e.g., for CPU: conda install pytorch-cpu torchvision-cpu -c pytorch
No module named `nets` / `preprocessing`
You probably haven't installed TensorFlow/research/slim. Follow the instructions [here](https://github.com/tensorflow/models/tree/master/research/slim#Install).ImportError: cannot import name '_obtain_input_shape'
keras_squeezenet unfortunately does not run with keras > 2.2.0.
To fix, pip install keras==2.2.0
.
tensorflow.python.framework.errors_impl.FailedPreconditionError: Error while reading resource variable
If this happened when running a keras model, your tensorflow and keras versions are probably incompatible. See the setup.py for which versions are supported.
Restoring from checkpoint failed. (...) Assign requires shapes of both tensors to match.
Most likely your passed image_size does not match up with the image size the model expects (e.g. inception_v{3,4} expect 299 insead of 224).
Either let the framework infer what image_size the model needs (run without --image_size
) or set the correct image_size yourself.
MobileNet weight loading failed.
Error message e.g. Assign requires shapes of both tensors to match. lhs shape= [1,1,240,960] rhs shape= [1,1,240,1280]
.
There is an error in the MobileNet implementation which causes the multiplier to not be applied properly: the number of channels sometimes go beyond what they ought to be (e.g. for the last layer). The line in question needs to be prefixed with a conditional:
if i != len(conv_defs['spec']) - 1 or multiplier >= 1:
opdef.multiplier_func(params, multiplier)
This is already done in @qbilius' fork of tensorflow/models.