Giters

Maluuba / GeNeVA

Code to train and evaluate the GeNeVA-GAN model for the GeNeVA task proposed in our ICCV 2019 paper "Tell, Draw, and Repeat: Generating and modifying images based on continual linguistic instruction"

Home Page:https://www.microsoft.com/en-us/research/project/generative-neural-visual-artist-geneva/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

How to reproduce the best result?

Victarry opened this issue · comments

commented

I follow the instructions in README.md, but I can't get the best results as the paper said.
Here is what I haved done:

  1. Follow the instructions from https://github.com/Maluuba/GeNeVA_datasets/, and generate the data file.
  2. Download the pretrained object detector model from https://www.microsoft.com/en-us/research/project/generative-neural-visual-artist-geneva/
  3. Use the default parameters and run python geneva/inference/train.py @example_args/iclevr-d-subtract.args

Here is my results in iclevr dataset, I trained this model with 2 NVIDIA TITAN Xp about 4 days:

precision
recall
f1
relsim
It seems it's far from the performance in the origin paper.
image

commented

Hi Victarry,

have you solved the problem?
The result of my experiment on i-CLEVR is also about 54% f1-score.

commented

Hi Victarry,

have you solved the problem?
The result of my experiment on i-CLEVR is also about 54% f1-score.

Not yet.
But when I remove the -use_fd flag and keeps -use_fg, I got much better results, about 72% f1-score

commented

Hi friends, I'm also trying to reproduce the scores given in the paper, however current performance of the model is far from those mentioned in the paper. Have you been able to find the reason why the performance is not as good as that of authors?

commented

I'm also trying to reproduce the scores of i-CLEVR as following the instructions noted in README (I mean I used example_args/iclevr-d-subtract.args), but my result is far from the authors, too.

AP AR F1 RSIM
Paper 92.39 84.72 88.39 74.02
Mine 65.84 40.82 49.45 28.33