RCIL

[CVPR2022] Representation Compensation Networks for Continual Semantic Segmentation
Chang-Bin Zhang¹, Jia-Wen Xiao¹, Xialei Liu¹, Ying-Cong Chen², Ming-Ming Cheng¹
¹ _{College of Computer Science, Nankai University}
² _{The Hong Kong University of Science and Technology}

Method

Update

Coming Soon add data folder
Coming Soon init code for Classification
Coming Soon add training scripts for ADE20K and cityscapes
~~09/04/2022 init code for segmentation~~
~~09/04/2022 init readme~~

Benchmark and Setting

There are two commonly used settings, disjoint and overlapped. In the disjoint setting, assuming we know all classes in the future, the images in the current training step do not contain any classes in the future. The overlapped setting allows potential classes in the future to appear in the current training images. We call each training on the newly added dataset as a step. Formally, X-Y denotes the continual setting in our experiments, where X denotes the number of classes that we need to train in the first step. In each subsequent learning step, the newly added dataset contains Y classes.

There are some settings reported in our paper. You can also try it on other any custom settings.

Continual Class Segmentation:
1. PASCAL VOC 2012 dataset:
  - 15-5 overlapped
  - 15-5 disjoint
  - 15-1 overlapped
  - 15-1 disjoint
  - 10-1 overlapped
  - 10-1 disjoint
2. ADE20K dataset:
  - 100-50 overlapped
  - 100-10 overlapped
  - 50-50 overlapped
  - 100-5 overlapped
Continual Domain Segmentation:
1. Cityscapes:
  - 11-5
  - 11-1
  - 1-1
Extension Experiments on Continual Classification
1. ImageNet-100
  - 50-10

Performance

Continual Class Segmentation on PASCAL VOC 2012

Method	Pub.	15-5 disjoint	15-5 overlapped	15-1 disjoint	15-1 overlapped	10-1 disjoint	10-1 overlapped
LWF	TPAMI 2017	54.9	55.0	5.3	5.5	4.3	4.8
ILT	ICCVW 2019	58.9	61.3	7.9	9.2	5.4	5.5
MiB	CVPR 2020	65.9	70.0	39.9	32.2	6.9	20.1
SDR	CVPR 2021	67.3	70.1	48.7	39.5	14.3	25.1
PLOP	CVPR 2021	64.3	70.1	46.5	54.6	8.4	30.5
Ours	CVPR 2022	67.3	72.4	54.7	59.4	18.2	34.3

Continual Class Segmentation on ADE20K

Method	Pub.	100-50 overlapped	100-10 overlapped	50-50 overlapped	100-5 overlapped
ILT	ICCVW 2019	17.0	1.1	9.7	0.5
MiB	CVPR 2020	32.8	29.2	29.3	25.9
PLOP	CVPR 2021	32.9	31.6	30.4	28.7
Ours	CVPR 2022	34.5	32.1	32.5	29.6

Continual Domain Segmentation on Cityscapes

Method	Pub.	11-5	11-1	1-1
LWF	TPAMI 2017	59.7	57.3	33.0
LWF-MC	CVPR 2017	58.7	57.0	31.4
ILT	ICCVW 2019	59.1	57.8	30.1
MiB	CVPR 2020	61.5	60.0	42.2
PLOP	CVPR 2021	63.5	62.1	45.2
Ours	CVPR 2022	64.3	63.0	48.9

Dataset Prepare

PASCVAL VOC 2012
sh data/download_voc.sh
ADE20K
sh data/download_ade.sh
Cityscapes
sh data/download_cityscapes.sh

Environment

conda install --yes --file requirements.txt
Install inplace-abn

Training

Dowload pretrained model from ResNet-101_iabn to pretrained/
We have prepared some training scripts in scripts/. You can train the model by

sh scripts/voc/rcil_10-1-overlap.sh

Inference

You can simply modify the bash file by adding --test, like

CUDA_VISIBLE_DEVICES=${GPU} python3 -m torch.distributed.launch --master_port ${PORT} --nproc_per_node=${NB_GPU} run.py --data xxx ... --test

Reference

If this work is useful for you, please cite us by:

@inproceedings{zhang2022representation,
  title={Representation Compensation Networks for Continual Semantic Segmentation},
  author={Zhang, Chang-Bin and Xiao, Jia-Wen and Liu, Xialei and Chen, Ying-Cong and Cheng, Ming-Ming},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={7053--7064},
  year={2022}
}

Connect

If you have any questions about this work, please feel easy to connect with us (zhangchbin ^ mail.nankai.edu.cn or zhangchbin ^ gmail.com).

Thanks

This code is heavily borrowed from [MiB] and [PLOP].

Awesome Continual Segmentation

There is a collection of AWESOME things about continual semantic segmentation, including papers, code, demos, etc. Feel free to pull request and star.

2022

Representation Compensation Networks for Continual Semantic Segmentation [CVPR 2022] [PyTorch]
Self-training for Class-incremental Semantic Segmentation [TNNLS 2022] [PyTorch]
Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation [TPAMI 2022] [[PyTorch]]

2021

PLOP: Learning without Forgetting for Continual Semantic Segmentation [CVPR 2021] [PyTorch]
Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations [CVPR2021] [PyTorch]
An EM Framework for Online Incremental Learning of Semantic Segmentation [ACM MM 2021] [PyTorch]
SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning [NeurIPS 2021] [PyTorch]

2020

Modeling the Background for Incremental Learning in Semantic Segmentation [CVPR 2020] [PyTorch]

2019

Incremental Learning Techniques for Semantic Segmentation [ICCV Workshop 2019] [PyTorch]

EDGSCOUT / RCIL