[feature] Plans to support other pre-trained models?

[David Kossnick]

Are there any plans to support custom pre-trained models, instead of AlexNet/ResNet-50? As background, I am working on a few new models and would like to evaluate them across all the benchmarks you've put together. Thanks!

[Priya Goyal]

Hi @kossnick, thanks for reaching out. Feel free to send PRs :) your contributions are more than welcome. If the models would be valuable to wider audience, we want to definitely support them.

[David Kossnick]

Thanks @prigoyal for the quick reply! As a little more context, we're frequently iterating on model architecture/logic, and have automated tests that run on every PR and assess quality across a range of metrics. I love what you have built here and would love to find a way to incorporate it into our evaluation workflow.

From what I can tell, a number of the fine-tuning tasks are setup specifically based on the pre-trained model. eg: https://github.com/facebookresearch/fair_self_supervision_benchmark/blob/master/self_supervision_benchmark/modeling/jigsaw/resnet_jigsaw_finetune_full.py and https://github.com/facebookresearch/fair_self_supervision_benchmark/blob/master/self_supervision_benchmark/modeling/jigsaw/alexnet_jigsaw_finetune_full.py serve a similar purpose but for different pre-trained models.

If I wanted a way to do this across a range of pre-trained models, what would be best?

1. More naive approach: for each fine-tuning task and for each new model architecture, make a new .py file that pulls out the appropriate features from the pre-trained model. Follow the existing pattern.
2. More complex approach: genericize the fine-tuning logic to take a config, and each new model gets a different config. Move logic about the model out of .py and into a .yml or similar.
3. Other ideas?

Much appreciated!

[Priya Goyal]

thanks for the context @kossnick. The names of the models are bit misleading. These models are no way task specific. They were named like this just so that the benchmark code is clear and easy to read/find.

For your use case, my understanding is that you want to run evaluations of variety of new models on the benchmark tasks? The models could be something new.

If that's the case, all we care about is the model definition used. You can create any model/reuse whatever model and simply define a .py for the model architecture. Concretely, the steps are: 1) create the model.py here and 2) register it here 3) use the model you want by specific the key in the config file example this

This allows you to define any new model and use it for evaluation. Let me know if this helps. Feel free to reach me at [e-mail] for any follow-up.

FYI, we are in the process of building the benchmark in PyTorch which will make it easy to register the new models. Hoping to release it in December or January :)

[Priya Goyal]

closing. Please use VISSL repo now :)