Validate using public checkpoint, Accuracy extremely low

[Li Hao]

hello, i validate the GCViT using your public checkpoint and the accuracy for tiny, xtiny, xxtiny is very low(top1_error 99.9, top5_error 99.75)
I wonder either you public the wrong checkpoints or my validation bash has something wrong?

 python validate.py \
 --model gc_vit_xxtiny \
 --checkpoint ckpts/gcvit_xxtiny_best_1k.pth.tar \
 --data_dir data/imagenet/ \
 --batch-size 32

[Ali Hatamizadeh]

Hi @HowardLi1984

I just checked the accuracy by cloning the repository from scratch and running validation for GC Vity xxtiny and was able to get the following which is the reported accuracy in the paper:

{
    "model": "gc_vit_xxtiny",
    "top1": 79.8,
    "top1_err": 20.2,
    "top5": 95.098,
    "top5_err": 4.902,
    "param_count": 12.0,
    "img_size": 224,
    "crop_pct": 0.875,
    "interpolation": "bicubic"
}

I simply downloaded the model from the ReadMe and ran the folowing:

#!/bin/bash
NUM_PROC=1
DATA_PATH="./ImageNet/ImageNet2012"
checkpoint=./tmp/gcvit_xxtiny_best_1k.pth.tar
BS=32

python validate.py --model gc_vit_xxtiny --checkpoint=$checkpoint --data_dir=$DATA_PATH --batch-size $BS

I don't believe you should get lower accuracy for any of the models, but please do let me know if the issue persists.

Thanks

[Ali Hatamizadeh]

@HowardLi1984 My other suspicion regarding the root cause of this issue is the environment. I'd appreciate it if you may please confirm the version of timm and PyTorch used in your environment.

[Li Hao]

Thanks for your replay. I have found the bug in dataset format.
GCViT's recommand imagenet format:
{
train:
class1: ...
class2:....
valid:
class1:...
class2:...
}
while my imagenet format remove the subclass dictionarys and use an extra file to save all images' labels
{
train: A.jpeg, B.jepg, etc
valid: A.jpeg, B.jepg, etc
}
after changing my imagenet format, i reproduce the model's accurary.

[Ali Hatamizadeh]

@HowardLi1984 , amazing. Great to hear that the model's accuracy can be reproduced.

Best,