Inconsistent SPICE scores
brandbir opened this issue · comments
It was found that in pycocoevalcap/spice/spice.py, line 29 is sorting the img_ids prior the generation of SPICE metrics. The result metrics are then extracted and mapped to the corresponding image ids in pycocoevalcap/eval.py by the setImgToEvalImgs() function. However, the latter function is using an unsorted img_ids list. This incompatibility between the sorted and unsorted lists makes the SPICE metrics being mapped to incorrect image ids. This does not have any effect on the overall SPICE metric as it is only a problem of misalignment between the computed SPICE metrics and their corresponding image ids. For consistency, no sorting is needed for imgIds in pycocoevalcap/spice/spice.py.
