magnusja / metriculous

Measure and visualize machine learning model performance without the usual boilerplate.

metriculous

Measure, visualize, and compare machine learning model performance without the usual boilerplate. Breaking API improvements to be expected.

Installation

$ pip install metriculous

Or, for the latest unreleased version:

$ pip install git+https://github.com/metriculous-ml/metriculous.git

Comparing Regression Models

Click to see more code

import numpy as np

# Mock the ground truth, a one-dimensional array of floats
ground_truth = np.random.random(300)

# Mock the output of a few models
perfect_model = ground_truth
noisy_model = ground_truth + 0.1 * np.random.randn(*ground_truth.shape)
random_model = np.random.randn(*ground_truth.shape)
zero_model = np.zeros_like(ground_truth)

import metriculous

metriculous.compare_regressors(
    ground_truth=ground_truth,
    model_predictions=[perfect_model, noisy_model, random_model, zero_model],
    model_names=["Perfect Model", "Noisy Model", "Random Model", "Zero Model"],
).save_html("comparison.html").display()

This will save an HTML file with common regression metrics and charts, and if you are working in a Jupyter notebook will display the output right in front of you:

Comparing Classification Models

Click to see more code

import numpy as np


def normalize(array2d: np.ndarray) -> np.ndarray:
    return array2d / array2d.sum(axis=1, keepdims=True)


class_names = ["Cat", "Dog", "Pig"]
num_classes = len(class_names)
num_samples = 500

# Mock ground truth
ground_truth = np.random.choice(range(num_classes), size=num_samples, p=[0.5, 0.4, 0.1])

# Mock model predictions
perfect_model = np.eye(num_classes)[ground_truth]
noisy_model = normalize(
    perfect_model + 2 * np.random.random((num_samples, num_classes))
)
random_model = normalize(np.random.random((num_samples, num_classes)))

import metriculous

metriculous.compare_classifiers(
    ground_truth=ground_truth,
    model_predictions=[perfect_model, noisy_model, random_model],
    model_names=["Perfect Model", "Noisy Model", "Random Model"],
    class_names=class_names,
    one_vs_all_figures=True,
).display()

Development

Poetry

This project uses poetry to manage dependencies. Please make sure it is installed for the required python version. Then install the dependencies with poetry install.

Makefile

A Makefile is used to automate common development workflows. Type make or make help to see a list of available commands. Before commiting changes it is recommended to run make format check test.