Giters

LingyiHongfd / LVOS

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Annotations issue

z-jiaming opened this issue · comments

commented

Hi Lingyi,

Congratulations ICCV~
This dataset is an excellent and important work for long-term VOS !

I want some help for the valid split:
There are some new objects (appear in subsequent frames) in your videos.
Unlike the test split, the evaluation of valid is done locally, then the annotations will contain both old and new objects.
It's not very friendly for discovering new targets.

Thus, could you please to provide another version of valid that, like test, contains only the annotation of the first existence of each object?

It seems that only 7K7WVzGG exists as a new object.
image

This is convenient for the dominant frameworks (like XMem and AOT) and easy to use your dataset.

Thanks a lot!

commented

Also, since the single object only appears from frame 00001156 in d83wYdy0, could you please fix it: 00001156 -> 00000001?

image
commented

Hello, Jailing. Thanks for your interest on our work.
(1) It was an oversight on our part not to take the new object problem into account. Indeed, when we construct LVOS, we followed the setting of YouTubeVOS, where new objects will appear in the subsequent frames in many videos. We think that the ability to segment new objects while maintaining tracking of old objects is a very important to VOS model, especially in practical environments. We hope our LVOS can serve as a benchmark to evaluate the performance of VOS models in real application scenarios, so the new object is significant.
(2) The dominant frameworks, such as AOT and XMem, support the emergence of new object. In fact, we just modify the path and initiation code in AOT and XMem to conduct experiments in our paper. We can provide you with the dataloader codes if you need them.
(3) We have checked the validation and test videos to see which ones have new objects. There are some videos containing new objects in not only validation but also test sets. We will follow your advice to provide some codes to enable users to separate the new objects in each video in validation sets by themselves. But, if you use the separated videos, the final performance maybe a little different from origin score.
(4) I have also noticed your new work JointFormer. It is a nice work. If you don't mind, I think maybe we can communicate more via email or wechat afterward. I would also want to get more suggestions and comments from other researchers about LVOS to help me improve it. Of course, I'm eager to cooperate with more researchers too!
Finally, thanks for your suggestion. I will provide some codes to separate the new objects in validation sets and the dataloaders we implemented to help other researchers to develop their own codebase easier.