VSainteuf / utae-paps

I run some tests with your implementation for the panoptic metric and got a curious result.

I made a simple test case with two instance who overlap the void class -> they should get filtered out.

if I have only one of those instances it works properly and I don’t get FP.
If I now have both instances they don’t get filtered out and I get two FP

It might have to do with

utae-paps/src/panoptic/metrics.py

Line 55 in 9e54d3c

*torch.unique(void_mask, return_counts=True)

since you here do the unique over basicly a binary mask if I understand correctly. I supect you wanted to do here instead of void_mask -> instance_ture[batch_idx]

Hi @marckatzenmaier,
Thanks very much for pointing this out.
This is indeed a bug. Instead of iterating over all the individual binary instance masks of void segments, this loop only runs once on the total binary mask of all void objects. So instead of matching predicted segments with void segments, this part of the code tries to match predicted segments with the union of all void segments, which probably never happens.

So at the end of the day, segments predicted for void objects are not ignored (though they should be), resulting in "false FP" (false positives that shouldn't be false positives).
This artificially reduces the RQ value and thus the PQ score.

I think this line should be
*torch.unique(instance_true[batch_idx]*void_mask, return_counts=True).

This is fixed now.
I re-evaluated the models on PASTIS with the new code and the bug fix results in a ~2-3pt PQ increase thanks to the reduced number of False Positives.

Problem with filtering out multiple void instances