Cannot reproduce ViG imagenet result using provided pretrained weights. Am I doing something wrong?

[Jirasak Buranathawornsom]

I tried to reproduce the result of ViG using provided pretrained weights. I tried to follow the instruction as close as possible With some slight modification of the code to avoid versioning conflict (I tried to use the given torch and timm version, but it always result in some import error)

• import convert_splitbn_model from timm.layers instead of timm.models
• use ImageDataset instead of Dataset
  The commit different can be seen in here: master...wanburana:Efficient-AI-Backbones:master

I tried 3 models: ViG-S, ViG-B, Pyramid ViG-S
here is the script I use to run the evaluation

python train.py /content/imagenet --model vig_s_224_gelu -b 128 --pretrain_path /content/Efficient-AI-Backbones/vig_pytorch/vig_s_80.6.pth --evaluate

python train.py /content/imagenet --model vig_b_224_gelu -b 128 --pretrain_path /content/Efficient-AI-Backbones/vig_pytorch/vig_b_82.6.pth --evaluate

python train.py /content/imagenet --model pvig_s_224_gelu -b 128 --pretrain_path /content/Efficient-AI-Backbones/vig_pytorch/pvig_s_82.1.pth.tar --evaluate

• ViG-S

  • reproduced ViG-S Top-1: 68.756%
    
  • reported ViG-S Top-1: 80.4 %
    

• ViG-B

  • reproduced ViG-B Top-1: 75.564 %
    

  • reported ViG-B Top-1: 82.3 %-
    

• Pyramid ViG-S

  • reproduced Pyramid ViG-S Top-1: 80.13 %
    
  • reported Pyramid ViG-S Top-1: 82.1 %
    

This google colab can be used to reproduce my result:
https://colab.research.google.com/drive/1lwWvx9jR3CKYX_qLPP0VysRMKCZMX6Qw?usp=sharing

[Kai Han]

I run the code again. The package version: torchvision==0.8.2, torch==1.7.1, timm==0.3.2.

The results are OK (ViG-S):

[Jirasak Buranathawornsom]

Thank you. After I installed the correct dependency, I was able to reproduce the result.