Giters

microsoft / responsible-ai-toolbox

Responsible AI Toolbox is a suite of tools providing model and data exploration and assessment user interfaces and libraries that enable a better understanding of AI systems. These interfaces and libraries empower developers and stakeholders of AI systems to develop and monitor AI more responsibly, and take better data-driven actions.

Home Page:https://responsibleaitoolbox.ai/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Consolidate metric definitions

romanlutz opened this issue · comments

commented

Through various iterations of dashboards we have created metric definitions across the code base. However, there is no single source of truth. Worse yet, they are not necessarily all covered by unit tests.

In some cases we've defined metrics for binary classification and regression, but left out multiclass classification. In other cases the code is specific to certain visual object classes (e.g., heatmap from error analysis). A more general definition might be better.

As we move towards supporting larger datasets and thereby the need for calculating metrics at a larger scale arises, we may need to consolidate these definitions. This may end up being outside of UI / TS code which is perfectly fine, of course.

FYI @imatiach-msft I created this based off of your comment in #1314
FYI @gaugup