Davis dataset
bhack opened this issue · comments
Why have you excluded an evaluation on very high quality annotations like Davis dataset and related F measure on boundaries?
Following PolygonRNN [L. Castrejon, K. Kundu, R. Urtasun, and S. Fidler. Annotating object instances with a polygon-rnn. In CVPR, 2017], our model is trained and compared on Cityscapes dataset. All other datasets (i.e ADE) are for cross domain experiments.
And thank you for your suggestion, Davis dataset and F measure are definitely worth ours attention and further investigation.
With a single polygon you cannot have enclosed holes in the segmentation target. The error related to miou is relative only to the distance of the polygons points from the true boundaries and to the number of used polygon points when you need to approximate curved shapes. So I think that F over boundaries could better test how your approach is valid expecially if you challenge high resolution and high quality segmentations like in Davis.
Looking at the code in the Davis dataset, it seems like the seg2bmap function also adds boundaries in holes in the object before calculating the F-score.
(https://github.com/fperazzi/davis/blob/master/python/lib/davis/measures/f_boundary.py#L72)
For PolygonRNN++, we follow the style of actual human annotation. A human annotates the outer boundary polygon and hole polygons separately, and in the presence of a hole, they would mark a subtraction (either through depth ordering or through explicit mention of subtraction). Therefore, the F score might not be exactly fair for us as we aim to only approximate the outer boundaries of the object (or the hole) (as one would do while interactively annotating). That said, our IoU calculations are still with our boundary polygon prediction and the actual segmentation mask with holes, i.e. we do not remove the holes to show unfair improvements in IoU w.r.t pixel wise methods in our experiments.
On the whole, we agree that the F score is a good metric and from the experiments in the DAVIS paper, seems like it is also well correlated to the well accepted Jaccard metric.
Thanks for the suggestion, we are now looking into working on the Davis dataset as well.
Yes this is correct. But i think that also the second factor is important on how your implicit polygon curve approximation (as a sequence of lines) will impact on F score. Will you need to introduce a curve to achieve high level of precision over the boundaries?
Agreed. For a similar reason, we added a Graph Neural Net, that adds a point between each edge and moves them to better approximate the contour. But maybe a Bezier curve for each edge would be a better choice and would be an interesting direction to explore and measure with the F score.
Yes probably a Bezier/B-splines approach could be interesting to explore with F score also if probably the interaction will be not as easy as with polygons for average users.
That is true and maybe it should be a post-processing step. It would be very interesting if Davis had tracks where human supervision was not only scribble based but also polygon based (as that allows for explicit control rather than an implicit level of control). Thanks for the interesting discussion!
Check also the new "interactive teaser" in Davis 2018 probably it will inspire you with this post-processing approach.
@amlankar I share your view but I think that to recover the last % points on F it needs a form of explicit control. Cause the implicit one can require too much interactive rounds without achieving a full coverage of the boundaries.
So I think that your approach and the postprocessing extension could be mixed with some scribbles bootstrap rounds. If you have any interest on this we can discuss somewhere.
Closing because of unrelatedness with the code. We would be interested in discussing this further offline, thanks for the suggestions!
Well done https://github.com/fidler-lab/curve-gcn