Yang Liu1*, Muzhi Zhu1*, Hengtao Li1*, Hao Chen1, Xinlong Wang2, Chunhua Shen1
1Zhejiang University, 2Beijing Academy of Artificial Intelligence
Powered by large-scale pre-training, vision foundation models exhibit significant potential in open-world image understanding. Even though individual models have limited capabilities, combining multiple such models properly can lead to positive synergies and unleash their full potential. In this work, we present Matcher, which segments anything with one shot by integrating an all-purpose feature extraction model and a class-agnostic segmentation model. Naively connecting the models results in unsatisfying performance, e.g., the models tend to generate matching outliers and false-positive mask fragments. To address these issues, we design a bidirectional matching strategy for accurate cross-image semantic dense matching and a robust prompt sampler for mask proposal generation. In addition, we propose a novel instance-level matching strategy for controllable mask merging. The proposed Matcher method delivers impressive generalization performance across various segmentation tasks, all without training. For example, it achieves 52.7% mIoU on COCO-20i for one-shot semantic segmentation, surpassing the state-of-the-art specialist model by 1.6%. In addition, our visualization results show open-world generality and flexibility on images in the wild.
- Online Demo
- Release code and models
vos_demo.mp4
The content of this project itself is licensed under LICENSE.
If you find this project useful in your research, please consider cite:
@article{liu2023matcher,
title={Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching},
author={Liu, Yang and Zhu, Muzhi and Li, Hengtao and Chen, Hao and Wang, Xinlong and Shen, Chunhua},
journal={arXiv preprint arXiv:2305.13310},
year={2023}
}