mRobotit / UnSniffer

[CVPR 2023] Official Pytorch code for Unknown Sniffer for Object Detection: Don’t Turn a Blind Eye to Unknown Objects

Unknown Sniffer for Object Detection: Don’t Turn a Blind Eye to Unknown Objects (CVPR 2023)

Wenteng Liang^*, Feng Xue^*, Yihao Liu, Guofeng Zhong, Anlong Ming

(:star2: denotes equal contribution)

Paper Bilibili Youtube Slides Project

Introduction

The recently proposed open-world object and open-set detection have achieved a breakthrough in finding never-seen-before objects and distinguishing them from known ones. However, their studies on knowledge transfer from known classes to unknown ones are not deep enough, resulting in the scanty capability for detecting unknowns hidden in the background. In this paper, we propose the unknown sniffer (UnSniffer) to find both unknown and known objects. Firstly, the generalized object confidence (GOC) score is introduced, which only uses known samples for supervision and avoids improper suppression of unknowns in the background. Significantly, such confidence score learned from known objects can be generalized to unknown ones. Additionally, we propose a negative energy suppression loss to further suppress the non-object samples in the background. Next, the best box of each unknown is hard to obtain during inference due to lacking their semantic information in training. To solve this issue, we introduce a graph-based determination scheme to replace hand-designed non-maximum suppression (NMS) post-processing. Finally, we present the Unknown Object Detection Benchmark, the first publicly benchmark that encompasses precision evaluation for unknown detection to our knowledge. Experiments show that our method is far better than the existing state-of-the-art methods.

Requirements

pip install -r requirements.txt

In addition, install detectron2 following here.

Dataset Preparation

The datasets can be downloaded using this link.

PASCAL VOC

Please put the corresponding json files in Google Cloud Disk into ./anntoations

Please download the JPEGImages data from the link provided by VOS.

The VOC dataset folder should have the following structure:

 └── VOC_DATASET_ROOT
     |
     ├── JPEGImages
     ├── voc0712_train_all.json
     ├── voc0712_train_completely_annotation200.json
     └── val_coco_format.json

COCO

Please put the corresponding json files in Google Cloud Disk into ./anntoations

The COCO dataset folder should have the following structure:

 └── COCO_DATASET_ROOT
     |
     ├── annotations
        ├── xxx (the original json files)
        ├── instances_val2017_coco_ood.json
        ├── instances_val2017_mixed_ID.json
        └── instances_val2017_mixed_OOD.json
     ├── train2017
     └── val2017

Training

python train_net.py --dataset-dir VOC_DATASET_ROOT --num-gpus 2 --config-file VOC-Detection/faster-rcnn/UnSniffer.yaml --random-seed 0 --resume

The pretrained models for Pascal-VOC can be downloaded from UnSniffer. Please put the model in UnSniffer/detection/data/VOC-Detection/faster-rcnn/UnSniffer/random_seed_0/.

Pretesting

The function of this process is to obtain the threshold, which only uses part of the training data.

sh pretest.sh

Evaluation on the VOC

python apply_net.py --dataset-dir VOC_DATASET_ROOT --test-dataset voc_custom_val  --config-file VOC-Detection/faster-rcnn/UnSniffer.yaml --inference-config Inference/standard_nms.yaml --random-seed 0 --image-corruption-level 0 --visualize 0

Evaluation on the COCO-OOD

sh test_ood.sh

Evaluation on the COCO-Mix

sh test_mixed.sh

Visualize prediction results

sh vis.sh

License

This repository is released under the Apache 2.0 license as found in the LICENSE file.

Citation

If you use UnSniffer, please consider citing:

@inproceedings{liang2023unknown,
title={Unknown Sniffer for Object Detection: Don't Turn a Blind Eye to Unknown Objects},
author={Liang, Wenteng and Xue, Feng and Liu, Yihao and Zhong, Guofeng and Ming, Anlong},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2023}
}

Acknowledgments:

UnSniffer builds on previous works code base such as VOS and OWOD. If you found UnSniffer useful please consider citing these works as well.