Panoptic Segmentation
Internship @ AI Without Borders
Week - 1
Literature Study - Panoptic Segmentation
| Paper | bottom line | Constraints | Code | Dataset used |
|---|---|---|---|---|
| Panoptic Segmentation by FAIR | PQ - Considers segmentation quality and a recognition quality. PQ is insensitive to class imbalance by calculating for each class independently and average over classes. | Just an introduction to the task of panoptic segmentation | No code | Vistas, CityScapes, ADE20k |
| Panoptic Feature Pyramid Networks | Detailed insights into the single-network architecture | Requires generating a coherent scene segmentation that is rich and complete. | No Code | |
| UPSNet: A Unified Panoptic Segmentation Network | 2 head network: Semantic Segmentation using Deformable Convolutions & Instance Segmentation using Mask R-CNN | Complex and mutli-headed architecture, could be tricky to get it to work | Uber Research | COCO, CityScapes |
| Panoptic Segmentation with an End-to-End Cell R-CNN for Pathology Image Analysis | Panoptic segmentation of various cancer cells | Access to dataset. The paper does not mention details of the dataset nor if it was further modified to all pixel segmentation | No code | MICCAI 2017 digital pathology challenge dataset |
Week - 2&3
Datasets and corresponding literature
Specifically for Panoptic Segementation
| Datasets | Description |
|---|---|
| COCO 2018 | Things, regions (Road, grass, water) |
| ADE20k | Things, regions and parts of things |
| Mapillary Vistas | Street Scenes |
| CityScapes | City/Street Scenes |
Medical Image Datasets
| Datasets | Description |
|---|---|
| 2017 MICCAI Digital Pathology Challenge dataset | Segmentation of Nuclei in Images. Dataset used in a paper for panoptic segmentation. Could not gain access to the dataset |
| Leukemia ALL-IDB | (White) Blood cell classification. Looks suitable but have to download to know how the data is. |
| Multi-class artefact detection in video endoscopy | Perfect dataset! But annotations are bounding box and not segementation. Have time to annotate? |
| Blood Cells Detection | Again perfect dataset! But annotations are bounding box and not segementation. Have time to annotate? |
| BACH | Breast Cancer Histology Images. 1 huge whole slides with multiple classes and instances but cropped into 400 which mostly 1 instance and 1 class |
| Melanoma, Skin Cancer | (Object segmentation) Multi class segmentaion with 1 instance. |
| Segmentation of neuronal structures in EM stacks Home | Cell boundaries hence good for normal segmentation only |
| Nuclear Segmentation | Good segmentation dataset but only nuclei, boundaries and background. Bascially 2 classes |
| Medical Segmentation Decathlon | Huge number of datasets with no proper description |
| Broad Bioimage Benchmark Collection | Database - with excellent description |
| The cell Image Library | Database |
| ImageJ | Database - Small |
| List of Cancer Cell Datasets for DL | Good datasets with multiclass classification but mostly for object detection (1 instance) |
Week - 4
Annotation tools and label format conversions
Coco Annotator
Quick annotaion of objects using 'Magic wand' to annotate disconnected objects or with the help of an API that fetches annotations from a semi-trained network. But may be too challenging not to miss any pixels while using magic wand or brush tool.
PixelAnnotationTool
Pesudo Semi automated annotation tool which uses opencv watershed segmentation for annotaing all pixels. May be ideal for our needs
voc2coco
Convert annotation format from voc to coco. Also contains direct instructions for the BCCD dataset Blood Cells Detection mentioned above.
Downlaoding and reading annotation and trying the tools to make custom annotations
Setting up coco annotator docker. Downlaoding and coverting BCCD from VOC to coco. Takes quite some time to fully annotate each image. Very imbalanced instances. Reading artifact detection EAD dataset. Too many instances of blur and contrast which is immpossible to annotate for panoptic.
Scraping the web one last time for a good dataset
no luck
Week - 5&6
UPSNet
Going through the code of UPSNet and trying to get some predictions