YoutubeVIS Evaluation

[Meryl Fang]

Thank you for sharing the repo!

We are trying to reproduce the YoutubeVIS 2019 results based on your code experiments/youtube_vis.sh.

Without training first, we used coco_seg.pth to get an idea on how to infer, but the results seemed unsatisfactory: just random dots/regions. We'd like to know 1) if this is expected, and it'd be much improved after finetuning; 2) whether this code will reproduce the published YoutubeVIS result (aka AP 32.6).

Thank you very much for your help.

[Youyuan Zhang]

Same question! QAQ

[Jialian Wu]

1. If you use coco.pth for testing, you may want to make sure use coco.py in dataset, because coco and youtubevis have different classes, where coco has 80 classes and youtubevis has 40 classes. I remember I tested it before by directly using coco pretrained model, and the predictions are not good but look reasonable.

2. In our experiments, we observed that the results fluctuated a lot on youtubevis. The possible reason is that the AP metric which is built upon tracklet IOU, which can vary a lot if identity switches or one mask missed.