hustvl / QueryInst

Instances as Queries (ICCV 2021)

Home Page:https://arxiv.org/abs/2105.01928

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Instances as Queries

PWC PWC PWC PWC

QueryInst-VIS Demo
  • [News]

    • Apr, 2022: If you like QueryInst for instance segmentation, you might also like TeViT (CVPR 2022, oral, paper / code & models) for high-performance video instance segmentation!.
    • Oct, 2021: QueryInst (ICCV 2021) is now officially included by mmdetection library, with new checkpoints, corresponding logs, and augmented training settings. We suggest you to use the newest QueryInst implementation in mmdetection, meanwhile this repo will be maintained too. Issues are welcomed if you have problems using QueryInst to reproduce the COCO AP reported in our paper.
    • Sep, 2021: We are now busy adding QueryInst to mmdetection library (open-mmlab/mmdetection#6050, open-mmlab/mmdetection#6000). The model will be augmented, please stay tuned. We will also extend QueryInst based on mmdetection library to other tasks, e.g., panoptic segmentation.
  • TL;DR: QueryInst (Instances as Queries) is a simple and effective query based instance segmentation method driven by parallel supervision on dynamic mask heads, which outperforms previous arts in terms of both accuracy and speed.

  • Our QueryTrack (i.e., Tracking Instances as Queries, tech report) based on QueryInst won the 2nd place (AP = 52.3 @ test set, AP = 54.3 @ val set) in video instance segmentation (VIS) track with single online end-to-end model, single scale testing & without using extra video training data in the 3rd Large-scale Video Object Segmentation Challenge, CVPR 2021.

  • For the first time, we demonstrate that an end-to-end query based framework driven by parallel supervision is competitive with well-established and highly-optimized methods in a wide range of instance-level recognition tasks (object detection, instance segmentation and video instance segmentation).

Instances as Queries

by Yuxin Fang*, Shusheng Yang*, Xinggang Wang†, Yu Li, Chen Fang, Ying Shan, Bin Feng, Wenyu Liu.

(*) equal contribution, (†) corresponding author.

ICCV2021 Paper

QueryInst

  • This repo serves as the official implementation for QueryInst, based on mmdetection and built upon Sparse R-CNN & DETR. Implantations based on Detectron2 will be released in the near future.

  • This project is under active development, we will extend QueryInst to a wide range of instance-level recognition tasks.

Main Results on COCO test-dev

Configs Aug. Weights Box AP Mask AP
QueryInst_Swin_L_300_queries (single scale testing) 400 ~ 1200, w/ Crop baidu / google 56.1 49.1

Main Results on COCO val

Configs Aug. Weights Box AP Mask AP
QueryInst_R50_3x_300_queries 480 ~ 800, w/ Crop baidu / google 46.9 41.4
QueryInst_R101_3x_300_queries 480 ~ 800, w/ Crop baidu / google 48.0 42.4
QueryInst_X101-DCN_3x_300_queries 480 ~ 800, w/ Crop - 50.3 44.2
QueryInst_Swin_L_300_queries (single scale testing) 400 ~ 1200, w/ Crop baidu / google 56.1 48.9

Notes:

  • Accesscode for baidu is QIst.

Getting Started

python setup.py develop
  • Prepare datasets:
mkdir data && cd data
ln -s /path/to/coco coco
  • Training QueryInst with single GPU:
python tools/train.py configs/queryinst/queryinst_r50_fpn_1x_coco.py
  • Training QueryInst with multi GPUs:
./tools/dist_train.sh configs/queryinst/queryinst_r50_fpn_1x_coco.py 8
  • Test QueryInst on COCO val set with single GPU:
python tools/test.py configs/queryinst/queryinst_r50_fpn_1x_coco.py PATH/TO/CKPT.pth --eval bbox segm
  • Test QueryInst on COCO val set with multi GPUs:
./tools/dist_test.sh configs/queryinst/queryinst_r50_fpn_1x_coco.py PATH/TO/CKPT.pth 8 --eval bbox segm

Citation

If you find our paper and code useful in your research, please consider giving a star and citation 📝 :

@InProceedings{Fang_2021_ICCV,
    author    = {Fang, Yuxin and Yang, Shusheng and Wang, Xinggang and Li, Yu and Fang, Chen and Shan, Ying and Feng, Bin and Liu, Wenyu},
    title     = {Instances As Queries},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2021},
    pages     = {6910-6919}
}
@article{QueryTrack,
  title={Tracking Instances as Queries},
  author={Yang, Shusheng and Fang, Yuxin and Wang, Xinggang and Li, Yu and Shan, Ying and Feng, Bin and Liu, Wenyu},
  journal={arXiv preprint arXiv:2106.11963},
  year={2021}
}

TODO

  • QueryInst training and inference code.
  • QueryInst with Swin-Transformer and Test-Time-Augmentation.
  • QueryInst configurations for Cityscapes and YouTube-VIS.
  • QueryInst pretrain weights.
ezoic increase your site revenue

About

Instances as Queries (ICCV 2021)

https://arxiv.org/abs/2105.01928

License:MIT License


Languages

Language:Python 99.9%Language:Dockerfile 0.1%Language:Shell 0.1%