zhang-tao-whu / HIPIE

Code release for "Hierarchical Open-vocabulary Universal Image Segmentation"

HIPIE: Hierarchical Open-vocabulary Universal Image Segmentation

We present HIPIE, a novel HIerarchical, oPen-vocabulary and unIvErsal image segmentation and detection model that is capable of performing segmentation tasks at various levels of granularities (whole, part and subpart) and tasks, including semantic segmentation, instance segmentation, panoptic segmentation, referring segmentation, and part segmentation, all within a unified framework of language-guided segmentation.

Hierarchical Open-vocabulary Universal Image Segmentation
Xudong Wang*, Shufan Li*, Konstantinos Kallidromitis*, Yusuke Kato, Kazuki Kozuka, Trevor Darrell
Berkeley AI Research, UC Berkeley; Panasonic AI Research

[project page] [arxiv] [paper] [bibtex]

Currently, this repo contains only codes for demos. Stay tuned.

Installation

Please refer to INSTALL.md for more details.

Demos

• See Demo-Main for Panoptic, Part, Instance and Referring Segmentation
• See Demo-SD for Combining our model with Stable Diffusion
• See Demo-SAM for Combining our model with Segment Anything

HIPIE is also capable of labeling segmentation masks from SAM and can even identify additional masks that may have been overlooked by SAM.

Please check our project page for more demos!

Model Zoo

We release two checkpoints at the moment.

• ResNet-50 Pretrained with O365,COCO,RefCOCO,Pascal Panoptic Parts
• ViT-H Pretrained with O365,COCO,RefCOCO

ToDos

We will release training and evlautation code with more checkpoints soon.

License

The majority of HIPIE is licensed under the MIT license. If you later add other third party code, please keep this license info updated, and please let us know if that component is licensed under something other than CC-BY-NC, MIT, or CC0.

How to get support from us?

If you have any general questions, feel free to email us at Xudong Wang, Shufan Li and Konstantinos Kallidromitis. If you have code or implementation-related questions, please feel free to send emails to us or open an issue in this codebase (We recommend that you open an issue in this codebase, because your questions may help others).

Citation

If you find our work inspiring or use our codebase in your research, please consider giving a star ⭐ and a citation.

@misc{wang2023hierarchical,
  title={Hierarchical Open-vocabulary Universal Image Segmentation}, 
  author={Xudong Wang and Shufan Li and Konstantinos Kallidromitis and Yusuke Kato and Kazuki Kozuka and Trevor Darrell},
  year={2023},
  eprint={2307.00764},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}