aiwithshekhar / CFBI

The official implementation of Collaborative Video Object Segmentation by Foreground-Background Integration.

CFBI: Collaborative Video Object Segmentation by Foreground-Background Integration

The official implementation of Collaborative Video Object Segmentation by Foreground-Background Integration (ECCV 2020, Spotlight). [paper] [demo (YouTube)][demo (Bilibili)]

CFBI works fine on both PaddlePaddle and PyTorch. Based on some necessary considerations, we only release the inference code of CFBI here.

If you want to get the training code of CFBI, please contact us by email: [e-mail]. And please inform us of your institution and the purpose of using CFBI in the email. Thank you for your understanding!

Framework:

Some video segmentation results:

Requirements

1. Python3
2. pytorch >= 1.3.0 and torchvision
3. opencv-python and Pillow

Getting Started

1. Prepare datasets:
   • Download the validation split of YouTube-VOS 2018, and decompress the file to datasets/YTB/valid. If you want to evaluate CFBI on YouTube-VOS 2019, please download this split instead.
   • Download 480p TrainVal split of DAVIS 2017, and decompress the file to datasets/DAVIS.
2. Evaluating:
   • YouTube-VOS: Download pretrained CFBI, ResNet101-CFBI, to pretrain_models, and then run bash ytb_eval.sh. After the evaluation, the result will be packed into a Zip file, which you need to send to official evaluation server to calculate a score. For 2019 version, use this server instead. The pretrained CFBI has been trained on YouTube-VOS using a large batch size (20), which boosts the performance (J&F) to 81.8% on the validation split of YouTube-VOS 2018.
   • DAVIS: Download pretrained CFBI, ResNet101-CFBI-DAVIS, to pretrain_models, and then run bash davis_eval.sh. After the evaluation, please use official code to calculate a score, which should be 81.9% (J&F).
   • Fast CFBI: To reduce memory usage, we also provide a fast setting in ytb_eval_fast.sh. The fast setting enables using float16 in the matching process of CFBI. Besides, we apply an atrous strategy in the global matching of CFBI for further efficiency (The discussion of atrous matching will be submitted to our Arxiv paper soon). The fast setting will save a large amount of memory and significantly improve the inference speed of CFBI. However, this will only lose very little performance.
   • Another way for saving memory is to increase the number of --global_chunks. This will not affect performance but will make the network speed slightly slower.

Model Zoo

We recorded the inference speed of CFBI by using one NVIDIA Tesla V100 GPU. Besides, we used a multi-object speed instead of a single-object. Almost every sequence in VOS datasets contains multiple objects, and CFBI is good at processing all of them simultaneously.

F16 denotes using float16 in the matching process. Fast means using both float16 and atrous strategy in the inference stage.

YouTube-VOS (Eval on Val 2018):

In the inference stage, we restricted the long edge of each frame to be no more than 1040 (800 * 1.3) pixels, which is the biggest random-scale size in the training and is smaller than the original size of YouTube-VOS (720p).

Name	Backbone	J Seen	F Seen	J Unseen	F Unseen	Mean	Multi-Obj FPS	Link
ResNet101-CFBI	ResNet101-DeepLabV3+	81.9	86.3	75.6	83.4	81.8	3.48	Click
ResNet101-F16-CFBI	ResNet101-DeepLabV3+	81.9	86.4	75.6	83.3	81.8	4.62 (32.8%↑)	The same as above
ResNet101-Fast-CFBI	ResNet101-DeepLabV3+	81.9	86.4	75.6	83.1	81.8	7.61 (118.7%↑)	The same as above
MobileNetV2-CFBI	MobileNetV2-DeepLabV3+	80.4	84.7	74.9	82.6	80.6	3.88	Click
MobileNetV2-Fast-CFBI	MobileNetV2-DeepLabV3+	80.2	84.6	74.7	82.7	80.6	9.69 (150.0↑%)	The same as above

DAVIS (Eval on Val 2017):

In the inference stage, we ran using the default size of DAVIS (480p).

Name	Backbone	J score	F score	Mean	Multi-Obj FPS	Link
ResNet101-CFBI-DAVIS	ResNet101-DeepLabV3+	79.3	84.5	81.9	5.88	Click
ResNet101-F16-CFBI-DAVIS	ResNet101-DeepLabV3+	79.2	84.4	81.8	7.38 (25.5%↑)	The same as above
ResNet101-Fast-CFBI-DAVIS	ResNet101-DeepLabV3+	77.0	82.7	79.9	10.18 (73.1%↑)	The same as above
MobileNetV2-CFBI-DAVIS	MobileNetV2-DeepLabV3+	76.5	80.3	78.4	6.94	Click
MobileNetV2-Fast-CFBI-DAVIS	MobileNetV2-DeepLabV3+	75.2	78.9	77.1	13.22 (90.5%↑)	The same as above

Using atrous strategy on DAVIS leads to a obvious performance drop. The reason is that DAVIS is so small (only 30 videos in Train split) and easy to be overfitted. Training CFBI with Fast mode should significantly relief the performance drop.

Citing

@inproceedings{yang2020collaborative,
  title={Collaborative video object segmentation by foreground-background integration},
  author={Yang, Zongxin and Wei, Yunchao and Yang, Yi},
  booktitle={Proceedings of the European Conference on Computer Vision},
  year={2020}
}