Issue of FID score

[Priyanka Upadhyay]

@huiyegit @shihaoji
I have calculated the FID score but I am getting very high value : 35.190974412567414

Am I doing anything wrong? Please let me know if there is any hyper parameter which I need to change to calculate the FID

[huiyegit]

There are the following things that you may have a try:

1. Use the correct source file to calculate FID score(link);
2. Check the fake images and real images for evaluation;
3. For dataset Birds, the best FID score may be not from the last epoch 600. Our pretrained model should reproduce the results in the paper.

[Priyanka Upadhyay]

There are the following things that you may have a try:

1. Use the correct source file to calculate FID score(link);
2. Check the fake images and real images for evaluation;
3. For dataset Birds, the best FID score may be not from the last epoch 600. Our pretrained model should reproduce the results in the paper.

Dear Authors.

1. I have used the correct source file from the given link. --> fid_score.py

2. this is the command I gave to run fid_score.py while following DM-GAN repo (https://github.com/MinfengZhu/DM-GAN) :

cd eval/FID && python fid_score.py --gpu 0 --batch-size 50 --path1 bird_val.npz --path2 ../../models/netG_epoch_600 (fake image folder)

where:
models/netG_epoch_600 ----- > fake image folder
bird_val.npz --> a pre-trained model train on bird dataset to calcualte FID score

3. I am using the pretrained models from your repo to generate fake images. But it is netG_epoch_600.pth. Is there any other model which is trained on less epoch which I should use?

[Priyanka Upadhyay]

There are the following things that you may have a try:

1. Use the correct source file to calculate FID score(link);
2. Check the fake images and real images for evaluation;
3. For dataset Birds, the best FID score may be not from the last epoch 600. Our pretrained model should reproduce the results in the paper.

Dear Authors.

1. I have used the correct source file from the given link. --> fid_score.py
2. this is the command I gave to run fid_score.py while following DM-GAN repo (https://github.com/MinfengZhu/DM-GAN) :

cd eval/FID && python fid_score.py --gpu 0 --batch-size 50 --path1 bird_val.npz --path2 ../../models/netG_epoch_600 (fake image folder)

where: models/netG_epoch_600 ----- > fake image folder bird_val.npz --> a pre-trained model train on bird dataset to calcualte FID score

3. I am using the pretrained models from your repo to generate fake images. But it is netG_epoch_600.pth. Is there any other model which is trained on less epoch which I should use?

I tried both commands

cd eval/FID && python fid_score.py --gpu 0 --batch-size 50 --path1 bird.val.npz --path2 ../../models/netG_epoch_600
got score FID: 35.190974412567414

cd eval/FID && python fid_score.py --gpu 0 --batch-size 50 --path1 ../../data/birds --path2 ../../models/netG_epoch_600
score FID: 28.640264869357054

[huiyegit]

To calculate the FID score, the command should be like this:
python fid_score.py --gpu 0 --batch-size 50 --path1 real_images_bird --path2 fake_images_bird
path1 is the directory containing the real images for evaluation, while path2 is the directory containing the fake images.

The trained models of other epochs have been saved during the training process. You may find them in your machine.

[Priyanka Upadhyay]

To calculate the FID score, the command should be like this: python fid_score.py --gpu 0 --batch-size 50 --path1 real_images_bird --path2 fake_images_bird path1 is the directory containing the real images for evaluation, while path2 is the directory containing the fake images.

The trained models of other epochs have been saved during the training process. You may find them in your machine.

--path1 real_images_bird -- > this path would be test_images path or all real images path ???

all_real_image_path = data/birds/CUB_200_2011/images (11788 image)
test_images_path = data/birds/test_images (2933 image)

The given bird dataset/preprocessed bird dataset does not have explicitly train images and test images folder (we just have pickle files for them). I created my own split according to given class for train (8855 images from 150 classes) and test (2933 images from 50 classes).

and then I used test_images(real images) and generated_images path to get the FID score.

It would be great to know which real_images path you used to generate FID.