This codebase contains a strong re-implementation of FixMatch in the scenario of semi-supervised semantic segmentation, as well as the official PyTorch implementation of our paper:
Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation
Lihe Yang, Lei Qi, Litong Feng, Wayne Zhang, Yinghuan Shi
Preprint 2022
Abstract. In this work, we revisit the weak-to-strong consistency framework, popularized by FixMatch from semi-supervised classification, where the prediction of a weakly perturbed image serves as supervision for its strongly perturbed version. Intriguingly, we observe that such a simple pipeline already achieves competitive results against recent advanced works, when transferred to our segmentation scenario. Its success heavily relies on the manual design of strong data augmentations, however, which may be limited and inadequate to explore a broader perturbation space. Motivated by this, we propose an auxiliary feature perturbation stream as a supplement, leading to an expanded perturbation space. On the other, to sufficiently probe original image-level augmentations, we present a dual-stream perturbation technique, enabling two strong views to be simultaneously guided by a common weak view. Consequently, our overall Unified Dual-Stream Perturbations approach (UniMatch) surpasses all existing methods significantly across all evaluation protocols on the Pascal, Cityscapes, and COCO benchmarks. We also demonstrate the superiority of our method in remote sensing interpretation and medical image analysis.
Labeled images are sampled from the original high-quality training set. Results are obtained by DeepLabv3+ based on ResNet-101 with training size 321.
| Method | 1/115 (92) | 1/57 (183) | 1/28 (366) | 1/14 (732) | 1/7 (1464) |
|---|---|---|---|---|---|
| SupOnly | 45.1 | 55.3 | 64.8 | 69.7 | 73.5 |
| U2PL | 68.0 | 69.2 | 73.7 | 76.2 | 79.5 |
| ST++ | 65.2 | 71.0 | 74.6 | 77.3 | 79.1 |
| PS-MT | 65.8 | 69.6 | 76.6 | 78.4 | 80.0 |
| UniMatch (Ours) | 75.2 | 77.2 | 78.8 | 79.9 | 81.2 |
Results are obtained by DeepLabv3+ based on ResNet-50/101. We reproduce U2PL results on ResNet-50.
| ResNet-50 | 1/30 | 1/8 | 1/4 | ResNet-101 | 1/16 | 1/8 | 1/4 |
|---|---|---|---|---|---|---|---|
| SupOnly | 56.8 | 70.8 | 73.7 | SupOnly | 67.9 | 73.5 | 75.4 |
| U2PL | 59.8 | 73.0 | 76.3 | U2PL | 74.9 | 76.5 | 78.5 |
| UniMatch (Ours) | 64.5 | 75.6 | 77.4 | UniMatch (Ours) | 75.7 | 77.3 | 78.7 |
Results are obtained by DeepLabv3+ based on Xception-65.
| Method | 1/512 (232) | 1/256 (463) | 1/128 (925) | 1/64 (1849) | 1/32 (3697) |
|---|---|---|---|---|---|
| SupOnly | 22.9 | 28.0 | 33.6 | 37.8 | 42.2 |
| PseudoSeg | 29.8 | 37.1 | 39.1 | 41.8 | 43.6 |
| PC2Seg | 29.9 | 37.5 | 40.1 | 43.7 | 46.1 |
| UniMatch (Ours) | 32.2 | 40.4 | 46.2 | 48.7 | 51.2 |
cd UniMatch
conda create -n unimatch python=3.6.9
conda activate unimatch
pip install -r requirements.txt
pip install pip install torch==1.8.1+cu102 torchvision==0.9.1+cu102 -f https://download.pytorch.org/whl/torch_stable.htmlResNet-50 | ResNet-101 | Xception-65
├── ./pretrained
├── resnet50.pth
├── resnet101.pth
└── xception.pth
- Pascal: JPEGImages | SegmentationClass
- Cityscapes: leftImg8bit | gtFine
- COCO: train2017 | val2017 | masks
Please modify the dataset path in configuration files.
The groundtruth mask ids have already been pre-processed. You may use them directly.
├── [Your Pascal Path]
├── JPEGImages
└── SegmentationClass
├── [Your Cityscapes Path]
├── leftImg8bit
└── gtFine
├── [Your COCO Path]
├── train2017
├── val2017
└── masks
# use torch.distributed.launch
sh tools/train.sh <num_gpu> <port>
# or use slurm
# sh tools/slurm_train.sh <num_gpu> <port> <partition>To run on different labeled data partitions or different datasets, please modify:
config, labeled_id_path, unlabeled_id_path, and save_path in train.sh.
Modify the unimatch.py to fixmatch.py in train.sh.
Modify the unimatch.py to supervised.py in train.sh, and double the batch_size in configuration file if you use the same number of GPUs as semi-supervised setting (no need to change lr).
We have another relevant semi-supervised semantic segmentation project: ST++
If you find these projects useful, please consider citing:
@article{unimatch,
title={Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation},
author={Yang, Lihe and Qi, Lei and Feng, Litong and Zhang, Wayne and Shi, Yinghuan},
journal={arXiv:2208.09910},
year={2022}
}
@inproceedings{st++,
title={ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation},
author={Yang, Lihe and Zhuo, Wei and Qi, Lei and Shi, Yinghuan and Gao, Yang},
booktitle={CVPR},
year={2022}
}We thank AEL, CPS, CutMix-Seg, DeepLabv3Plus, PseudoSeg, PS-MT, SimpleBaseline, U2PL and other relevant works (see below) for their amazing open-sourced projects!
- [2017 ICCV] Semi Supervised Semantic Segmentation Using Generative Adversarial Network
- [2019 TPAMI] Semi-Supervised Semantic Segmentation with High- and Low-level Consistency [Code]
- [2020 BMVC] Semi-supervised semantic segmentation needs strong, varied perturbations [Code]
- [2020 CVPR] Semi-Supervised Semantic Segmentation with Cross-Consistency Training [Code]
- [2020 CVPR] Semi-Supervised Semantic Image Segmentation with Self-correcting Networks
- [2020 ECCV] Guided Collaborative Training for Pixel-wise Semi-Supervised Learning [Code]
- [2020 ECCV] Semi-Supervised Segmentation based on Error-Correcting Supervision
- [2021 ICLR] PseudoSeg: Designing Pseudo Labels for Semantic Segmentation [Code]
- [2021 WACV] ClassMix: Segmentation-Based Data Augmentation for Semi-Supervised Learning [Code]
- [2021 CVPR] Semi-supervised Semantic Segmentation with Directional Context-aware Consistency [Code]
- [2021 CVPR] Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision [Code]
- [2021 ICCV] Re-distributing Biased Pseudo Labels for Semi-supervised Semantic Segmentation: A Baseline Investigation [Code]
- [2021 ICCV] Semi-Supervised Semantic Segmentation with Pixel-Level Contrastive Learning from a Class-wise Memory Bank [Code]
- [2021 ICCV] A Simple Baseline for Semi-supervised Semantic Segmentation with Strong Data Augmentation [Code]
- [2021 ICCV] C3-SemiSeg: Contrastive Semi-supervised Segmentation via Cross-set Learning and Dynamic Class-balancing [Code]
- [2021 ICCV] Pixel Contrastive-Consistent Semi-Supervised Semantic Segmentation
- [2021 NeurIPS] Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning [Code]
- [2022 PR] DMT: Dynamic Mutual Training for Semi-Supervised Learning [Code]
- [2022 AAAI] GuidedMix-Net: Semi-supervised Semantic Segmentation by Using Labeled Images as Reference
- [2022 ICLR] Bootstrapping Semantic Segmentation with Regional Contrast [Code]
- [2022 CVPR] ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation [Code]
- [2022 CVPR] Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels [Code]
- [2022 CVPR] Perturbed and Strict Mean Teachers for Semi-supervised Semantic Segmentation [Code]
- [2022 CVPR] Unbiased Subclass Regularization for Semi-Supervised Semantic Segmentation [Code]
- [2022 CVPR] Semi-supervised Semantic Segmentation with Error Localization Network [Code]
- [2022 CVPR] UCC: Uncertainty guided Cross-head Co-training for Semi-Supervised Semantic Segmentation