TFOD Benchmark for Few-Shot Object Detection

TFOD is the first benchmark dataset for Task-Focused Few-Shot Object Detection. Why? We find that detection is not reliable outside of its initial training setting for many robot tasks. However, many researchers do not have a robot or even access to data to evaluate few-shot detection algorithms in a robotics setting. Notably, few-shot is exactly as it sounds, having to perform detection with very few annotated examples. Thus, we created the TFOD Benchmark in a challenging robot manipulation setting, which provides highly variable image characteristics for a consistent set of objects. This evaluation will help guide innovation toward increasingly reliable few-shot detection for robotics.

Contact: Brent Griffin (griffb at umich dot edu)

Benchmark Example.

Using TFOD

Run ./demo/tfod_manual_data_demonstration.py to manually load TFOD data.
[native Python]

Run ./demo/tfod_detectron2_data_demonstration.py to automatically load data to detectron2.
[native Python, has detectron2 dependency]

Here are the commands we used to set up a virtual environment for detectron2 and TFOD:

python3 -m venv ~/tfod
source ~/tfod/bin/activate
pip install --upgrade pip
pip install torch torchvision IPython 
pip install git+https://github.com/facebookresearch/detectron2.git

Benchmark

The TFOD Benchmark uses MS-COCO AP metrics and k few-shot examples across 12 object classes.

Method	k	AP	AP50	AP75	APs	APm	APl
ClickBot	1	14.1	19.9	17.2	0.0	32.9	22.8
ClickBot	2	18.3	24.3	22.5	0.0	32.1	27.7
ClickBot	4	35.0	46.0	42.0	1.7	57.4	39.0

Is your technique missing although the paper and code are public? Let us know and we'll add it. We average our baseline TFOD results across ten consecutive trials. Use this approach to report results if your method is nondeterministic.

Using ClickBot Baseline on TFOD Benchmark

Run ./demo/tfod_clickbot_baseline_demonstration.py to replicate our ClickBot baseline results.
[native Python, has detectron2 dependency]

ClickBot Per-Object Benchmark Results.

Publication

Please cite our paper if you find it useful for your research.

@inproceedings{Gr23,
  author = {Griffin, Brent},
  title = {Mobile Robot Manipulation using Pure Object Detection},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year = {2023}
}

TFOD Experiment Videos

https://youtu.be/77oVdhyY4vU

https://youtu.be/giiSYDwZM4c

https://youtu.be/OhQfMPuZQlg

Use

This code is available for non-commercial research purposes only.

griffbr / TFOD