Using pretrained weights from caffe with THINGSvision

[Alireza Karami]

Hi all

I have a pretrained AlexNet in caffemodel. Do you have any suggestion about how I can use it with THINGSvision toolbox?

Thanks,
Ali

[Johannes Roth]

Hi Ali,

we currently don't support Caffe in thingsvision. You can try to convert your Caffe model into a PyTorch model (e.g. with https://github.com/vadimkantorov/caffemodel2pytorch), which you should then be able to use with get_extractor_for_model (see https://vicco-group.github.io/thingsvision/CustomModels.html).

@LukasMut , what do you think about adding support for Caffe?

[Alireza Karami]

Hi Johannes,

Thanks. Yes, I know about caffemodel2pytorch. There another tool, too (https://github.com/penguinnnnn/Caffe2Pytorch). I tried to convert my file with both of them but I got some unknown error that I could not solve it so far. However, I convert the caffemodel file to npy with the following lines of code:

import numpy as np
import caffe

def net_to_py_readable(prototxt_filename, caffemodel_filename):
net = caffe.Net(prototxt_filename, caffemodel_filename, caffe.TEST) # read the net + weights
pynet_ = []
for li in range(len(net.layers)): # for each layer in the net
layer = {} # store layer's information
layer['name'] = net._layer_names[li]
# for each input to the layer (aka "bottom") store its name and shape
layer['bottoms'] = [(net._blob_names[bi], net.blobs[net._blob_names[bi]].data.shape)
for bi in list(net._bottom_ids(li))]
# for each output of the layer (aka "top") store its name and shape
layer['tops'] = [(net._blob_names[bi], net.blobs[net.blob_names[bi]].data.shape)
for bi in list(net.top_ids(li))]
layer['type'] = net.layers[li].type # type of the layer
# the internal parameters of the layer. not all layers has weights.
layer['weights'] = [net.layers[li].blobs[bi].data[...]
for bi in range(len(net.layers[li].blobs))]
pynet.append(layer)
return pynet

AlexNet_SalObjSub = net_to_py_readable('AlexNet_SalObjSub.prototxt','AlexNet_SalObjSub.caffemodel')

np.save('AlexNet_SalObjSub.npy', AlexNet_SalObjSub, allow_pickle=True)

Is it possible to use npy file as an input to THINGSvision?

[Johannes Roth]

Right now, we only support PyTorch and Tensorflow models. I think your best option would be to implement AlexNet in PyTorch and directly assign the weights that you pulled from the caffemodel, then using that with get_extractor_from_model. Or use extractor=get_extractor(model_name='alexnet', source='torchvision') and alter the extractor.model weights.

[Alireza Karami]

Thank you. I could convert the file using caffemodel2pytorch (https://github.com/vadimkantorov/caffemodel2pytorch). The problem was that I was using Python 3.5, but Python 2.7 was needed.