dbolya / tide

A General Toolbox for Identifying Object Detection Errors

Home Page:https://dbolya.github.io/tide

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

OpenImages support would be great

rodrigob opened this issue · comments

Would be great to also support the OpenImages dataset.
(15M boxes over 600 categories; 2.7M instance segmentations over 350 categories)

This dataset was part of the RVC 2020 challenge and its own Kaggle competitions in 2019.

Thanks for the suggestion!

I don't have much experience with OpenImages, so would it be possible for you to implement a dataset driver for it as a pull request? You can use the COCO one as reference:

tide/tidecv/datasets.py

Lines 60 to 107 in 49a5d2a

def COCO(path:str=None, name:str=None, year:int=2017, ann_set:str='val', force_download:bool=False) -> Data:
"""
Loads ground truth from a COCO-style annotation file.
If path is not specified, this will download the COCO annotations for the year and ann_set specified.
Valid years are 2014, 2017 and valid ann_sets are 'val' and 'train'.
"""
if path is None:
path = download_annotations(
'COCO{}'.format(year),
'http://images.cocodataset.org/annotations/annotations_trainval{}.zip'.format(year),
force_download)
path = os.path.join(path, 'annotations', 'instances_{}{}.json'.format(ann_set, year))
if name is None: name = default_name(path)
with open(path, 'r') as json_file:
cocojson = json.load(json_file)
images = cocojson['images']
anns = cocojson['annotations']
cats = cocojson['categories'] if 'categories' in cocojson else None
# Add everything from the coco json into our data structure
data = Data(name, max_dets=100)
image_lookup = {}
for idx, image in enumerate(images):
image_lookup[image['id']] = image
data.add_image(image['id'], image['file_name'])
if cats is not None:
for cat in cats:
data.add_class(cat['id'], cat['name'])
for ann in anns:
image = ann['image_id']
_class = ann['category_id']
box = ann['bbox']
mask = f.toRLE(ann['segmentation'], image_lookup[image]['width'], image_lookup[image]['height'])
if ann['iscrowd']: data.add_ignore_region(image, _class, box, mask)
else: data.add_ground_truth (image, _class, box, mask)
return data

The official toolkit from the RVC competition might be helpful: https://github.com/ozendelait/rvc_devkit/tree/master/objdet

Specifically, it downloads the original CSV annotations from OID (V5|V6) and resorts to https://github.com/bethgelab/openimages2coco to convert them into COCO instance/segmentation format, which can then be a drop-in replacement for TIDE.

The steps to produce a working solutions seem to be as follow:

  1. remove unnecessary lines used to download OID images https://github.com/ozendelait/rvc_devkit/blob/c986717abc24eba99a259e203a9ce4e182b2124e/objdet/download_oid_boxable.sh#L21 and run download_oid_boxable.sh
  2. Modify the line https://github.com/ozendelait/rvc_devkit/blob/c986717abc24eba99a259e203a9ce4e182b2124e/objdet/convert_oid_coco.sh#L32 to conform to the change in https://github.com/bethgelab/openimages2coco, where convert.py has been renamed to convert_annotations.py and run the conversion script
  3. For bounding box annotations, it seems a dummy 'segmentation' field is needed, or one can fork TIDE and make necessary adjustments. For mask annotations, somehow openimages2coco decides to use 'segments_info' as the field name https://github.com/bethgelab/openimages2coco/blob/8991d9bccbd3d91f32b87f04dab60b2a61cb608e/utils.py#L238 , so that needs to be converted as well.

Then simply substitute the OID path:

tide = TIDE()
tide.evaluate(
    datasets.COCO(path_to_oid_converted), 
    datasets.COCOResult(path_to_preds), 
    mode=TIDE.BOX
    ) # Use TIDE.MASK for masks
tide.summarize()  # Summarize the results as tables in the console
tide.plot()       # Show a summary figure. Specify a folder and it'll output a png to that folder.