Instance Segmentation

[abhigoku10]

@rstrudel thanks for opensourcing the code base can we perform instance segmentation also from the segmenter ? if so how to do it ? what all changes have to be made in the code base
THnaks in advance

[rstrudel]

Hi @abhigoku10 ,
Yes Segmenter can be extended to instance segmentation, however the paper is about semantic segmentation and we do not plan to support instance segmentation in this repository.
To extend it to instance segmentation, one could replace the class tokens with object tokens and use a hungarian macthing algorithm, DETR style. For the matching, you need a similarity loss between predicted and ground truth masks and Max-DeepLab proposed a great framework for both instance and panoptic segmentation. This is definitely doable with Segmenter, the model modifications are minor, but the training pipeline modifications are not trivial.

[abhigoku10]

@rstrudel thanskf or the response will try to do it , but great work