Giters

open-mmlab / mmeval

A unified evaluation library for multiple machine learning libraries

Home Page:https://mmeval.readthedocs.io/en/latest/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Calling for volunteers for adding new metrics!

ice-tong opened this issue · comments

commented

Dear community,

We are excited to introduce our new evaluation library: MMEval, a unified evaluation library for multiple machine learning libraries.

With the release of MMEval, we have some metrics from the OpenMMLab algorithm library that have not yet been added. We list the metrics below and call for community help.

If you are interested, you can claim a metric by replying to this issue in the following format:

Metric No. : <The Metric No. you claim>
Status :  Apply | Submit
Links : The repo links you fork (Apply status) / The PR links you open (Submit status)
No. Metric Algorithm library Level of Difficulty Assigned to Status / PR
1 CityScapesMetric mmdet / mmseg ⭐️⭐️⭐️ @Muyun99 Apply
2 CocoPanopticMetric mmdet ⭐️⭐️⭐️ @Even-ok Apply
3 LVISMetric mmdet ⭐️⭐️ @dmucby Apply
4 CrowdHumanMetric mmdet ⭐️⭐️ @tianleiSHI Apply
5 DOTAMetric mmrotate ⭐️⭐️ @YanxingLiu #65
6 RotatedCocoMetric mmrotate ⭐️⭐️ @YanxingLiu
7 IndoorMetric mmdet3d ⭐️⭐️⭐️
8 InstanceSegMetric mmdet3d ⭐️⭐️⭐️ @Pzzzzz5142 Apply
9 KittiMetric mmdet3d ⭐️⭐️⭐️
10 LyftMetric mmdet3d ⭐️⭐️⭐️
11 NuScenesMetric mmdet3d ⭐️⭐️⭐️
12 WaymoMetric mmdet3d ⭐️⭐️⭐️
13 OneMinusNEDMetric mmocr ⭐️⭐️
14 WordMetric mmocr ⭐️⭐️
15 CharMetric mmocr ⭐️⭐️
16 CocoWholeBodyMetric mmpose ⭐️⭐️
17 NME mmpose ⭐️⭐️
18 AUC mmpose ⭐️⭐️
19 EPE mmpose ⭐️⭐️
20 PoseTrack18Metric mmpose ⭐️⭐️
21 ANetMetric mmaction2 ⭐️⭐️
22 AccMetric mmaction2 ⭐️⭐️
23 FlowOutliers mmflow ⭐️⭐️ @fengsxy Apply
24 MOTChallengeMetrics mmtracking ⭐️⭐️
25 ReIDMetrics mmtracking ⭐️⭐️
26 SOTMetric mmtracking ⭐️⭐️
27 TAOMetric mmtracking ⭐️⭐️
28 YouTubeVISMetric mmtracking ⭐️⭐️ @qianlian-mozi Apply
29 MultiScaleStructureSimilarity mmediting ⭐️⭐️
30 FrechetInceptionDistance & TransFID mmediting ⭐️⭐️
31 InceptionScore & TransIS mmediting ⭐️⭐️
32 SAD mmediting ⭐️⭐️ @xuan07472 #76
33 MattingMSE mmediting ⭐️⭐️ @xuan07472 #71
34 ConnectivityError mmediting ⭐️⭐️ @xuan07472 #79
35 GradientError mmediting ⭐️⭐️ @xuan07472 #78
36 PerceptualPathLength mmediting ⭐️⭐️
37 PrecisionAndRecall mmediting ⭐️⭐️
38 SlicedWassersteinDistance mmediting ⭐️⭐️
39 NIQE mmediting ⭐️⭐️
40 Equivariance mmediting ⭐️⭐️

NOTES:

  1. Documents of MMEval can be found at: https://mmeval.readthedocs.io/en/latest/
  2. The contributing guides can be found at: https://github.com/open-mmlab/mmeval/blob/main/CONTRIBUTING.md

Examples:

We provide some examples showing how to add Metric from the OpenMMLab algorithm library to MMEval

  1. mmeval.Accuracy from mmcsl
  1. mmeval.MeanIoU from mmseg
  1. mmeval.VOCMeanAP from mmdet
  1. mmeval.COCODetectionMetric from mmdet

Submitted Content:

  1. The PR in MMEval that adds the new metric (with complete documentation and testing)
  2. The PR in original algorithm library that uses the new metric (with evaluation comparison between using MMEval and before)

No1. CityScapesMetric

  • Add the CityScapesMetric from mmdet / mmseg to MMEval, and use the MMEval's CityScapesMetric in mmdet / mmseg.
  • Technical Tags: Python; Object detection; Semantic segmentation;
  • The original CityScapesMetric in mmdet
  • The original CityScapesMetric in mmseg

No2. CocoPanopticMetric

  • Add the CocoPanopticMetric from mmdet to MMEval, and use the MMEval's CocoPanopticMetric in mmdet.
  • Technical Tags: Python; Object detection; Panoptic segmentation;
  • The original CocoPanopticMetric in mmdet

No3. LVISMetric

  • Add the LVISMetric from mmdet to MMEval, and use the MMEval's LVISMetric in mmdet.
  • Technical Tags: Python; Object detection;
  • The original LVISMetric in mmdet

No4. CrowdHumanMetric

  • Add the CrowdHumanMetric from mmdet to MMEval, and use the MMEval's CrowdHumanMetric in mmdet.
  • Technical Tags: Python; Object detection;
  • The original CrowdHumanMetric in mmdet

No5. DOTAMetric

  • Add the DOTAMetric from mmrotate to MMEval, and use the MMEval's DOTAMetric in mmrotate.
  • Technical Tags: Python; Rotated object detection;
  • The original DOTAMetric in mmrotate

No6. RotatedCocoMetric

  • Add the RotatedCocoMetric from mmrotate to MMEval, and use the MMEval's RotatedCocoMetric in mmrotate.
  • Technical Tags: Python; Rotated object detection;
  • The original RotatedCocoMetric in mmrotate

No7. IndoorMetric

  • Add the IndoorMetric from mmdet3d to MMEval, and use the MMEval's IndoorMetric in mmdet3d.
  • Technical Tags: Python; 3D object detection;
  • The original IndoorMetric in mmdet3d

No8. InstanceSegMetric

  • Add the InstanceSegMetric from mmdet3d to MMEval, and use the MMEval's InstanceSegMetric in mmdet3d.
  • Technical Tags: Python; 3D instance segmentation;
  • The original InstanceSegMetric in mmdet3d

No9. KittiMetric

  • Add the KittiMetric from mmdet3d to MMEval, and use the MMEval's KittiMetric in mmdet3d.
  • Technical Tags: Python; 3D object detection;
  • The original KittiMetric in mmdet3d

No10. LyftMetric

  • Add the LyftMetric from mmdet3d to MMEval, and use the MMEval's LyftMetric in mmdet3d.
  • Technical Tags: Python; 3D object detection;
  • The original LyftMetric in mmdet3d

No11. NuScenesMetric

  • Add the NuScenesMetric from mmdet3d to MMEval, and use the MMEval's NuScenesMetric in mmdet3d.
  • Technical Tags: Python; 3D object detection;
  • The original NuScenesMetric in mmdet3d

No12. WaymoMetric

  • Add the WaymoMetric from mmdet3d to MMEval, and use the MMEval's WaymoMetric in mmdet3d.
  • Technical Tags: Python; 3D object detection;
  • The original WaymoMetric in mmdet3d

No13. OneMinusNEDMetric

  • Add the OneMinusNEDMetric from mmocr to MMEval, and use the MMEval's OneMinusNEDMetric in mmocr.
  • Technical Tags: Python; OCR;
  • The original OneMinusNEDMetric in mmocr

No14. WordMetric

  • Add the WordMetric from mmocr to MMEval, and use the MMEval's WordMetric in mmocr.
  • Technical Tags: Python; OCR;
  • The original WordMetric in mmocr

No15. CharMetric

  • Add the CharMetric from mmocr to MMEval, and use the MMEval's CharMetric in mmocr.
  • Technical Tags: Python; OCR;
  • The original CharMetric in mmocr

No16. CocoWholeBodyMetric

  • Add the CocoWholeBodyMetric from mmpose to MMEval, and use the MMEval's CocoWholeBodyMetric in mmpose.
  • Technical Tags: Python; Pose estimation;
  • The original CocoWholeBodyMetric in mmpose

No17. NME

  • Add the NME from mmpose to MMEval, and use the MMEval's NME in mmpose.
  • Technical Tags: Python; Pose estimation;
  • The original NME in mmpose

No18. AUC

  • Add the AUC from mmpose to MMEval, and use the MMEval's AUC in mmpose.
  • Technical Tags: Python; Pose estimation;
  • The original AUC in mmpose

No19. EPE

  • Add the EPE from mmpose to MMEval, and use the MMEval's EPE in mmpose.
  • Technical Tags: Python; Pose estimation;
  • The original EPE in mmpose

No20. PoseTrack18Metric

  • Add the PoseTrack18Metric from mmpose to MMEval, and use the MMEval's PoseTrack18Metric in mmpose.
  • Technical Tags: Python; Pose estimation;
  • The original PoseTrack18Metric in mmpose

No21. ANetMetric

  • Add the ANetMetric from mmaction2 to MMEval, and use the MMEval's ANetMetric in mmaction2.
  • Technical Tags: Python; Video understanding;
  • The original ANetMetric in mmaction2

No22. AccMetric

  • Add the AccMetric from mmaction2 to MMEval, and use the MMEval's AccMetric in mmaction2.
  • Technical Tags: Python; Video understanding;
  • The original AccMetric in mmaction2

No23. FlowOutliers

  • Add the FlowOutliers from mmflow to MMEval, and use the MMEval's FlowOutliers in mmflow.
  • Technical Tags: Python; Optical flow;
  • The original FlowOutliers in mmflow

No24. MOTChallengeMetrics

  • Add the MOTChallengeMetrics from mmtracking to MMEval, and use the MMEval's MOTChallengeMetrics in mmtracking.
  • Technical Tags: Python; Multiple object tracking;
  • The original MOTChallengeMetrics in mmtracking

No25. ReIDMetrics

  • Add the ReIDMetrics from mmtracking to MMEval, and use the MMEval's ReIDMetrics in mmtracking.
  • Technical Tags: Python; Re-identification;
  • The original ReIDMetrics in mmtracking

No26. SOTMetric

  • Add the SOTMetric from mmtracking to MMEval, and use the MMEval's SOTMetric in mmtracking.
  • Technical Tags: Python; Single object tracking;
  • The original SOTMetric in mmtracking

No27. TAOMetric

  • Add the TAOMetric from mmtracking to MMEval, and use the MMEval's TAOMetric in mmtracking.
  • Technical Tags: Python; Object tracking;
  • The original TAOMetric in mmtracking

No28. YouTubeVISMetric

  • Add the YouTubeVISMetric from mmtracking to MMEval, and use the MMEval's YouTubeVISMetric in mmtracking.
  • Technical Tags: Python; Video instance segmentation;
  • The original YouTubeVISMetric in mmtracking

No29. MultiScaleStructureSimilarity

  • Add the MultiScaleStructureSimilarity from mmediting to MMEval, and use the MMEval's MultiScaleStructureSimilarity in mmediting.
  • Technical Tags: Python; GAN;
  • The original MultiScaleStructureSimilarity in mmediting

No30. FrechetInceptionDistance & TransFID

  • Add the FrechetInceptionDistance and TransFID from mmediting to MMEval, and use the MMEval's FrechetInceptionDistance and TransFID in mmediting.
  • Technical Tags: Python; GAN;
  • The original FrechetInceptionDistance in mmediting
  • The original TransFID in mmediting

No31. InceptionScore & TransIS

  • Add the InceptionScore and TransIS from mmediting to MMEval, and use the MMEval's InceptionScore and TransIS in mmediting.
  • Technical Tags: Python; GAN;
  • The original InceptionScore in mmediting
  • The original TransIS in mmediting

No32. SAD

  • Add the SAD from mmediting to MMEval, and use the MMEval's SAD in mmediting.
  • Technical Tags: Python; GAN;
  • The original SAD in mmediting

No33. MattingMSE

  • Add the MattingMSE from mmediting to MMEval, and use the MMEval's MattingMSE in mmediting.
  • Technical Tags: Python; Matting;
  • The original MattingMSE in mmediting

No34. ConnectivityError

  • Add the ConnectivityError from mmediting to MMEval, and use the MMEval's ConnectivityError in mmediting.
  • Technical Tags: Python; Matting;
  • The original ConnectivityError in mmediting

No35. GradientError

  • Add the GradientError from mmediting to MMEval, and use the MMEval's GradientError in mmediting.
  • Technical Tags: Python; Matting;
  • The original GradientError in mmediting

No36. PerceptualPathLength

  • Add the PerceptualPathLength from mmediting to MMEval, and use the MMEval's PerceptualPathLength in mmediting.
  • Technical Tags: Python; GAN;
  • The original PerceptualPathLength in mmediting

No37. PrecisionAndRecall

  • Add the PrecisionAndRecall from mmediting to MMEval, and use the MMEval's PrecisionAndRecall in mmediting.
  • Technical Tags: Python; GAN;
  • The original PrecisionAndRecall in mmediting

No38. SlicedWassersteinDistance

  • Add the SlicedWassersteinDistance from mmediting to MMEval, and use the MMEval's SlicedWassersteinDistance in mmediting.
  • Technical Tags: Python; GAN;
  • The original SlicedWassersteinDistance in mmediting

No39. NIQE

  • Add the NIQE from mmediting to MMEval, and use the MMEval's NIQE in mmediting.
  • Technical Tags: Python; GAN;
  • The original NIQE in mmediting

No40. Equivariance

  • Add the Equivariance from mmediting to MMEval, and use the MMEval's Equivariance in mmediting.
  • Technical Tags: Python; GAN;
  • The original Equivariance in mmediting
commented

Metric No.1 CityScapesMetric
Status : Apply
Links : https://github.com/Muyun99/mmeval

commented

Metric No.5 DOTAMetric
Status : Apply
Links : https://github.com/YanxingLiu/mmeval.git

commented

Metric No.24 FlowOutliers
Status : Apply
Links : https://github.com/fengsxy/mmeval

commented

Metric No.3 LVISMetric
Status : Apply
Links : https://github.com/dmucby/mmeval

commented

Metric No.33 MattingMSE
Status : Apply
Links : https://github.com/xuan07472/mmeval

commented

Metric No.32 SAD
Status : Apply
Links : https://github.com/xuan07472/mmeval

commented

Metric No.33 MattingMSE
Status : Submit
Links : #71

commented

Metric No.32 SAD
Status : Submit
Links : #76

commented

Metric No.5 DOTAMetric
Status : Submit
Links : #65

commented

Metric No.35 GradientError
Status : Apply
Links : https://github.com/xuan07472/mmeval

commented

Metric No.35 GradientError
Status : Submit
Links : #78

commented

Metric No.34 ConnectivityError
Status : Apply
Links : https://github.com/xuan07472/mmeval

commented

Metric No.34 ConnectivityError
Status : Submit
Links : #79