The some "Evaluation Metric" for seen classes and novel classes.
jingzhengli opened this issue · comments
Hi, thanks for sharing the great work!
I found that the same Evaluation Metric, i.e., the Hungarian algorithm, is used for seen classes and novel classes. In fact, I think that the standard "classification accuracy" should be used for seen classes, to avoid the mis-matching issue on seen classes.
I think the use of the Hungarian algorithm here is because the method does not have a parametric classifier for both seen and novel classes, so the standard 'classification accuracy' is not applicable for both seen and novel classes.
I think the use of the Hungarian algorithm here is because the method does not have a parametric classifier for both seen and novel classes, so the standard 'classification accuracy' is not applicable for both seen and novel classes.
Thanks for your reply. I agree with you in this case.
I provide a way to measure the performance of a seen category using standard classification accuracy, similar to a prototype network: maybe we can directly calculate the classification accuracy of unlabelled instances within the cluster to which the labeled seen category belongs.
I think the use of the Hungarian algorithm here is because the method does not have a parametric classifier for both seen and novel classes, so the standard 'classification accuracy' is not applicable for both seen and novel classes.
If seen class uses Hungarian algorithm as metric, we cannot compare it with other baselines based on standard 'classification accuracy'.(not fair)
Hi both, apologies for the delay in pitching in on this!
It is indeed an interesting point. 'Classification Accuracy' could be seen as 'Old ACC' if we do not perform the Hungarian Assignment over these classes and instead force the ground truth 'cluster_index --> class_index' mapping for the classes for which we have labels. In this way, Classification Accuracy is an upper bound to Old ACC (though note that assuming the GT mappings for the Old classes necessarily reduces ACC on the New categories).
We considered this discrepancy, but for now decided to do the Hungarian algorithm over ALL categories, as it allows for a 1-1 comparison with the fully unsupervised methods such as k-means clustering. In future works we may explore other options!
Hi both, apologies for the delay in pitching in on this!
It is indeed an interesting point. 'Classification Accuracy' could be seen as 'Old ACC' if we do not perform the Hungarian Assignment over these classes and instead force the ground truth 'cluster_index --> class_index' mapping for the classes for which we have labels. In this way, Classification Accuracy is an upper bound to Old ACC (though note that assuming the GT mappings for the Old classes necessarily reduces ACC on the New categories).
We considered this discrepancy, but for now decided to do the Hungarian algorithm over ALL categories, as it allows for a 1-1 comparison with the fully unsupervised methods such as k-means clustering. In future works we may explore other options!
Thank you very much for your reply
Hi both, apologies for the delay in pitching in on this!
It is indeed an interesting point. 'Classification Accuracy' could be seen as 'Old ACC' if we do not perform the Hungarian Assignment over these classes and instead force the ground truth 'cluster_index --> class_index' mapping for the classes for which we have labels. In this way, Classification Accuracy is an upper bound to Old ACC (though note that assuming the GT mappings for the Old classes necessarily reduces ACC on the New categories).
We considered this discrepancy, but for now decided to do the Hungarian algorithm over ALL categories, as it allows for a 1-1 comparison with the fully unsupervised methods such as k-means clustering. In future works we may explore other options!
Hi, Using clustering to obtain pseudo-labels in "Contrastive adaptation network for unsupervised domain adaptation'' solved this problem.