Model Reproducibility

Question

Model Reproducibility

yonigottesman opened this issue 2 years ago · comments

Hi, I think its important to provide scripts to reproduce paper results for the models the package implements. To be some kind of proof everything is working correctly.
What do you think?
Im not sure who can fund these experiments ¯_(ツ)_/¯

Owen Vallis · Answer 1 · Sat Jul 16 2022 04:59:12 GMT+0800 (China Standard Time)

Hi @yonigottesman, we're actually working on this right now and plan to provide benchmarks for all the losses this quarter. Additionally, I've personally reproduced (or as close as I could) the experiments for Multi-Sim, Circle Loss, SimSiam, and Barlow Twins. I've also compared our loss outputs for Multi-Sim and SimSiam against the implementations from the authors.

I have a notebook in the cars196 branch that contains my attempt to repro the multi-sim results on the caltech bird dataset. I even tried to emulate the pytorch version of the random crop and resize. We have an optimized variant of this in KerasCV now, and we plan to switch to use that instead as part of the new benchmarks.

Yoni Gottesman · Answer 2 · Sat Jul 16 2022 11:55:02 GMT+0800 (China Standard Time)

sounds amazing

Elie Bursztein · Answer 3 · Sat Jul 16 2022 22:30:06 GMT+0800 (China Standard Time)

@yonigottesman if you are interested in helping reproduce the experiments (e.g adding losses or more datasets, visualization or other idea you might have - we would love your help. We (Google) are going to train the models and make them available but we could use help to have the best reproduction pipeline possible :)