About the evaluation results
SeaBird-Go opened this issue · comments
Hi, thanks for your work.
I wonder what is the test_list.txt in eval.py file this line. Is it the same with the test_list2.txt?
Besides, I want to know the what the data split you used to obtain the evaluation results in Table 1 and Table 2 in your paper?
Hi! The test_list.txt is the same astest_list2.txtthat I have uploaded in dataset folder. Those corresponds to the official splits we used to get the results reported in the paper. I have modified eval.py. Thanks for your comment!
@Emanuele97x Sorry to comment on this closed issue.
Because I still have some questions about the evaluation results. I download the pretrained models you provided, and I found they are category-specific models, I evaluate these model on each category, I could obtain similar results in your paper.
So I wonder whether the XMFNet should be trained for each category to get the results you presented? I'm also training XMFNet in all categories, but it's too slow so I cannot get any conclusions right now.
Hi @SeaBird-1995, the models are category specific, as those presented in ViPC, the ShapeNet-ViPC dataset is huge so training a single model with all categories could still work, however it takes a lot of time.
@Emanuele97x ， thanks for your clarification. So it means the evaluation results in your papers are all obtained from category-specific models?
Because I want to compare with your results, I need to make sure this one.
@Emanuele97x , thanks for your work.
I want to ask a question since the viewpoint of the input image has many effects on the evaluation metrics, however, when you compute the metrics, the selection of the view image in the ViPCDataLoader is not fixed. So I wonder, in this case, are the quantitative results not comparable?
Dear @zhanghm1995,
Thanks for your interest in this work, during the evaluation the views are selected at random, in this way your evaluation is averaged over all the possible views (good and bad ones), results in the paper are produced in this way to be consistent with previous works (e.g ViPC).
If you use different view selection procedure results will be not comparable anymore.
@Emanuele97x , thanks for your reply. So in the current setting, it would contain randomness to some extent.
What I mean is that the testing set could be randomly selected in advance and then fixing them, then the evaluation results maybe seem more comparable.
Anyway, thanks for your clarification, it helps a lot.
Yes, it could be a possibility, by the way, the error bars on those value are very small, even with multiple random view selection the results are very similar for the same model!
I got it, thanks a lot.