Giters

Qinying-Liu / Awesome-Open-Vocabulary-Semantic-Segmentation

A curated publication list on open vocabulary semantic segmentation and related area (e.g. zero-shot semantic segmentation) resources..

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Awesome-Open-Vocabulary-Semantic-Segmentation

If you find this project helpful, please consider giving it a star ⭐.

Contents

Open-Vocabulary Semantic Segmentation

Fully-Supervised Open-Vocabulary Semantic Segmentation

The model is trained on fully-supervised semantic segmentation datasets with pixel-level annotations (e.g., COCO Stuff dataset).

  1. [LSeg] | ICLR'22 | Language-driven Semantic Segmentation | [pdf] | [code]
  2. [OpenSeg] | ECCV'22 | Scaling Open-vocabulary Image Segmentation with Image-level Labels | [pdf] | [code]
  3. [Xu et al.] | ECCV'22 | A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model | [pdf] | [code]
  4. [SegCLIP] | ICML'23 | SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation | [pdf] | [code]
  5. [MaskCLIP] | ICML'23 | Open-Vocabulary Universal Image Segmentation with MaskCLIP | [pdf] | [code]
  6. [OVSeg] | CVPR'23 | Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP | [pdf] | [code]
  7. [X-Decoder] | CVPR'23 | Generalized Decoding for Pixel, Image, and Language | [pdf] | [code]
  8. [SAN] | CVPR'23(Highlight) | Side Adapter Network for Open-Vocabulary Semantic Segmentation | [pdf] | [code]
  9. [SAN] | TAPMI'23 | SAN: Side Adapter Network for Open-vocabulary Semantic Segmentation | [pdf] | [code]
  10. [ODISE] | CVPR'23 | Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models | [pdf] | [code]
  11. [FreeSeg] | CVPR'23 | FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation | [pdf] | [code]
  12. [CAT-Seg] | Arxiv'23.03 | CAT-Seg : Cost Aggregation for Open-Vocabulary Semantic Segmentation | [pdf] | [code]
  13. [OpenSeeD] | ICCV'23 | A Simple Framework for Open-Vocabulary Segmentation and Detection | [pdf] | [code]
  14. [GKC] | ICCV'23 | Global Knowledge Calibration for Fast Open-Vocabulary Segmentation | [pdf]
  15. [OPSNet] | ICCV'23 | Open-vocabulary Panoptic Segmentation with Embedding Modulation | [pdf] | [code]
  16. [MasQCLIP] | ICCV'23 | MasQCLIP for Open-Vocabulary Universal Image Segmentation | [pdf]
  17. [DeOP] | ICCV'23 | Open Vocabulary Semantic Segmentation with Decoupled One-Pass Network | [pdf] | [code]
  18. [Li et al.] | ICCV'23 | Open-vocabulary Object Segmentation with Diffusion Models | [pdf] | [code]
  19. [HIPIE] | NeurIPS'23 | Hierarchical Open-vocabulary Universal Image Segmentation | [pdf] | [code]
  20. [FC-CLIP] | NeurIPS'23 | Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP | [pdf] | [code]
  21. [MAFT] | NeurIPS'23 | Learning Mask-aware CLIP Representations for Zero-Shot Segmentation | [pdf] | [code]
  22. [Dao et al] | TMM | Class Enhancement Losses with Pseudo Labels for Open-Vocabulary Semantic Segmentation | [pdf]
  23. [ADA] | Arxiv'23.09 | Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation | [pdf]
  24. [SED] | Arxiv'23.11 | SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation | [pdf]
  25. [SELF-SEG] | Arixv'23.12 | Self-Guided Open-Vocabulary Semantic Segmentation | [pdf]
  26. [SCAN] | Arixv'23.12 | Open-Vocabulary Segmentation with Semantic-Assisted Calibration | [pdf] | [code]
  27. [OpenSD] | Arixv'23.12 | OpenSD: Unified Open-Vocabulary Segmentation and Detection | [pdf] | [code]

Weakly-Supervised Open-Vocabulary Semantic Segmentation

[text-supervised/language-supervised] The model is trained on weakly supervised datasets with only image-level annotations/captions (e.g., CC12M dataset).

  1. [GroupViT] | CVPR'22 | GroupViT: Semantic Segmentation Emerges from Text Supervision | [pdf] | [code]
  2. [ViL-Seg] | ECCV'22 | Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embedding | [pdf]
  3. [MaskCLIP+] | ECCV'22(Oral) | Extract Free Dense Labels from CLIP | [pdf] | [code]
  4. [ViewCo] | ICLR'23 | Viewco: Discovering Text-supervised Segmentation Masks via Multi-view Semantic Consistency | [pdf]
  5. [SegCLIP] | ICML'23 | SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation | [pdf] | [code]
  6. [CLIP-S4] | CVPR'23 | CLIP-S4: Language-Guided Self-Supervised Semantic Segmentation | [pdf]
  7. [PACL] | CVPR'23 | Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning | [pdf]
  8. [OVSegmentor] | CVPR'23 | Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervision | [pdf] | [code]
  9. [SimSeg] | CVPR'23 | A Simple Framework for Text-Supervised Semantic Segmentation | [pdf] | [code]
  10. [TCL] | CVPR'23 | Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairs | [pdf] | [code]
  11. [SimCon] | Arxiv'23.02 | SimCon Loss with Multiple Views for Text Supervised Semantic Segmentation | [pdf]
  12. [Zhang et al.] | Arxiv'23.04 | Associating Spatially-Consistent Grouping with Text-supervised Semantic Segmentation | [pdf]
  13. [ZeroSeg] | ICCV'23 | Exploring Open-Vocabulary Semantic Segmentation from CLIP Vision Encoder Distillation Only | [pdf]
  14. [CLIPpy] | ICCV'23 | Perceptual Grouping in Contrastive Vision-Language Models | [pdf]
  15. [MixReorg] | ICCV'23 | MixReorg: Cross-Modal Mixed Patch Reorganization is a Good Mask Learner for Open-World Semantic Segmentation | [pdf]
  16. [CoCu] | NeurIPS'23 | Bridging Semantic Gaps for Language-Supervised Semantic Segmentation | [pdf] | [code]
  17. [PGSeg] | NeurIPS'23 | Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentation | [pdf] | [code]
  18. [SAM-CLIP] | Arixv'23.10 | SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding | [pdf]
  19. [CLIP-DINOiser] | Arixv'23.12 | CLIP-DINOiser: Teaching CLIP a few DINO tricks | [pdf] | [code]
  20. [TagAlign] | Arixv'23.12 | TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification | [pdf] | [code]
  21. [S-Seg] | Arixv'24.01 | Exploring Simple Open-Vocabulary Semantic Segmentation | [pdf] | [code]
  22. [CLIPSelf] | ICLR'24(Spotlight) | CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction | [pdf] | [code]
  23. [Uni-OVSeg] | Arixv'24.02 | Open-Vocabulary Segmentation with Unpaired Mask-Text Supervision | [pdf]

Training-Free Open-Vocabulary Semantic Segmentation

The model is modified from the off-the-shelf large models (e.g., CLIP, Diffusion models) without an additional training phase.

  1. [MaskCLIP] | ECCV'22(Oral) | Extract Free Dense Labels from CLIP | [pdf] | [code]
  2. [ReCo] | NeurIPS'22 | ReCo: Retrieve and Co-segment for Zero-shot Transfer | [pdf] | [code]
  3. [CLIP Surgery] | Arxiv'23.04 | CLIP Surgery for Better Explainability with Enhancement in Open-Vocabulary Tasks | [pdf] | [code]
  4. [OVDiff] | Arxiv'23.06 | Diffusion Models for Zero-Shot Open-Vocabulary Segmentation | [pdf]
  5. [DiffSegmenter] | Arxiv'23.09 | Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter | [pdf]
  6. [CLIP-DIY] | WACV'24 | CLIP-DIY: CLIP Dense Inference Yields Open-Vocabulary Semantic Segmentation For-Free | [pdf]
  7. [IPSeg] | Arxiv'23.10 | Towards Training-free Open-world Segmentation via Image Prompting Foundation Models | [pdf]
  8. [PnP-OVSS] | Arxiv'23.11 | Plug-and-Play, Dense-Label-Free Extraction of Open-Vocabulary Semantic Segmentation from Vision-Language Models | [pdf]
  9. [SCLIP] | Arxiv'23.12 | SCLIP: Rethinking Self-Attention for Dense Vision-Language Inference | [pdf]
  10. [GEM] | Arxiv'23.12 | Grounding Everything: Emerging Localization Properties in Vision-Language Transformers | [pdf] | [code]
  11. [CaR] | Arxiv'23.12 | CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor | [pdf] | [code]
  12. [FOSSIL] | WACV'24 | FOSSIL: Free Open-Vocabulary Semantic Segmentation through Synthetic References Retrieval | [pdf]
  13. [TagCLIP] | AAAI'24 | TagCLIP: A Local-to-Global Framework to Enhance Open-VocabularyMulti-Label Classification of CLIP Without Training | [pdf] | [code]

Others

  1. [Zhou et al.] | Arxiv'23.11 | Rethinking Evaluation Metrics of Open-Vocabulary Segmentation | [pdf] | [code]

Zero-Shot Semantic Segmentation

Different from open-vocabulary segmentation (cross-dataset), zero-shot methods split each dataset to seen classes and unseen classes.

  1. [ZegFormer] | CVPR'22 | ZegFormer: Decoupling Zero-Shot Semantic Segmentation | [pdf] | [code]
  2. [Xu et al.] | ECCV'22 | A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model | [pdf] | [code]
  3. [ZegCLIP] | CVPR'23 | ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation | [pdf] | [code]
  4. [PADing] | CVPR'23 | Primitive Generation and Semantic-related Alignment for Universal Zero-Shot Segmentation | [pdf] | [code]
  5. [DeOP] | ICCV'23 | Open Vocabulary Semantic Segmentation with Decoupled One-Pass Network | [pdf] | [code]
  6. [SPT] | AAAI'24 | Spectral Prompt Tuning: Unveiling Unseen Classes for Zero-Shot Semantic Segmentation | [pdf] | [code]

Referring Image Segmentation

Fully-Supervised Referring Image Segmentation

  1. [CARIS] | ACM MM'23 | CARIS: Context-Aware Referring Image Segmentation | [pdf] | [code]
  2. [BKINet] | TMM'23 | Bilateral Knowledge Interaction Network for Referring Image Segmentation | [pdf] | [code]
  3. [Group-RES] | ICCV'23 | Advancing Referring Expression Segmentation Beyond Single Image | [pdf] | [code]
  4. [RIS-DMMI] | ICCV'23 | Beyond One-to-One: Rethinking the Referring Image Segmentation | [pdf] | [code]
  5. [ETRIS] | ICCV'23 | Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentation | [pdf] | [code]
  6. [SEEM] | ArXiv'23.04 | Segment Everything Everywhere All at Once | [pdf] | [code]

Weakly-Supervised Referring Image Segmentation

  1. [Kim et al.] | ICCV'23 | Shatter and Gather: Learning Referring Image Segmentation with Text Supervision | [pdf] | [code]
  2. [Liu et al.] | ICCV'23 | Referring Image Segmentation Using Text Supervision | [pdf] | [code]

Open-Vocabulary Object Detection

  1. [RO-ViT] | CVPR'23(Highlight) | Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers | [pdf] | [code]
  2. [CAT] | CVPR'23 | CAT: LoCalization and IdentificAtion Cascade Detection Transformer for Open-World Object Detection | [pdf] | [code]
  3. [DetCLIPv2] | CVPR'23 | DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignment | [pdf]
  4. [CondHead] | CVPR'23 | Learning to Detect and Segment for Open Vocabulary Object Detection | [pdf]
  5. [CORA] | CVPR'23 | CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching | [pdf] | [code]
  6. [ovdet] | CVPR'23 | Aligning Bag of Regions for Open-Vocabulary Object Detection | [pdf] | [code]
  7. [OADP] | CVPR'23 | Object-Aware Distillation Pyramid for Open-Vocabulary Object Detection | [pdf] | [code]
  8. [F-VLM] | ICLR'23 | F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models | [pdf] | [code]
  9. [mm-ovod] | ICML 2023 | Multi-Modal Classifiers for Open-Vocabulary Object Detection | [pdf] | [code]
  10. [SGDN] | Arxiv'23.07 | Open-Vocabulary Object Detection via Scene Graph Discovery | [pdf]
  11. [MMC-Det] | Arxiv'23.08 | Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detection | [pdf]
  12. [IPL] | Arxiv'23.08 | Improving Pseudo Labels for Open-Vocabulary Object Detection | [pdf]
  13. [DITO] | Arxiv'23.09 | Detection-Oriented Image-Text Pretraining for Open-Vocabulary Detection | [pdf] | [code]
  14. [EdaDet] | ICCV'23 | EdaDet: Open-Vocabulary Object Detection Using Early Dense Alignment | [pdf] | [code]
  15. [LP-OVOD] | WACV'24 | LP-OVOD: Open-Vocabulary Object Detection by Linear Probing | [pdf] | [code]
  16. [DST-Det] | Arxiv'23.10 | DST-Det: Simple Dynamic Self-Training for Open-Vocabulary Object Detection | [pdf]
  17. [CoDet] | NeurIPS'23 | CoDet: Co-Occurrence Guided Region-Word Alignment for Open-Vocabulary Object Detection | [pdf] | [code]
  18. [PLAC] | Arxiv'23.12 | Learning Pseudo-Labeler beyond Noun Concepts for Open-Vocabulary Object Detection | [pdf]
  19. [Sambor] | Arxiv'23.12 | Boosting Segment Anything Model Towards Open-Vocabulary Learning | [pdf] | [code]
  20. [DVDet] | ICLR'24 | LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors | [pdf]

Universal Semantic Segmentation

  1. [Semantic-SAM] | Arxiv'23.10 | Semantic-SAM: Segment and Recognize Anything at Any Granularity | [pdf] | [code]
  2. [Open-Vocabulary SAM] | Arxiv'24.01 | Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively | [pdf] | [code]
  3. [OMG-Seg] | Arxiv'24.01 | OMG-Seg: Is One Model Good Enough For All Segmentation? | [pdf] | [code]

Related Survey

  1. Towards Open Vocabulary Learning: A Survey | [pdf]
  2. A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future | [pdf]

Feedback

If you have any suggestions or find missing papers, please don't hesitate to contact me via lydyc@mail.ustc.edu.cn.

About

A curated publication list on open vocabulary semantic segmentation and related area (e.g. zero-shot semantic segmentation) resources..