Consolidate metric definitions
romanlutz opened this issue · comments
Through various iterations of dashboards we have created metric definitions across the code base. However, there is no single source of truth. Worse yet, they are not necessarily all covered by unit tests.
In some cases we've defined metrics for binary classification and regression, but left out multiclass classification. In other cases the code is specific to certain visual object classes (e.g., heatmap from error analysis). A more general definition might be better.
As we move towards supporting larger datasets and thereby the need for calculating metrics at a larger scale arises, we may need to consolidate these definitions. This may end up being outside of UI / TS code which is perfectly fine, of course.
FYI @imatiach-msft I created this based off of your comment in #1314
FYI @gaugup