ucalyptus2 / error-parity

Achieve error-rate fairness between societal groups for any score-based classifier.

error-parity

Fast postprocessing of any score-based predictor to meet fairness criteria.

The error-parity package can achieve strict or relaxed fairness constraint fulfillment, which can be useful to compare ML models at equal fairness levels.

Installing

Install package from PyPI:

pip install error-parity

Or, for development, you can clone the repo and install from local sources:

git clone https://github.com/socialfoundations/error-parity.git
pip install ./error-parity

Getting started

See detailed example notebooks under the examples folder.

from error_parity import RelaxedThresholdOptimizer

# Given any trained model that outputs real-valued scores
fair_clf = RelaxedThresholdOptimizer(
    predictor=lambda X: model.predict_proba(X)[:, -1],   # for sklearn API
    # predictor=model,  # use this for a callable model
    constraint="equalized_odds",
    tolerance=0.05,     # fairness constraint tolerance
)

# Fit the fairness adjustment on some data
# This will find the optimal _fair classifier_
fair_clf.fit(X=X, y=y, group=group)

# Now you can use `fair_clf` as any other classifier
# You have to provide group information to compute fair predictions
y_pred_test = fair_clf(X=X_test, group=group_test)

How it works

Given a callable score-based predictor (i.e., y_pred = predictor(X)), and some (X, Y, S) data to fit, RelaxedThresholdOptimizer will:

1. Compute group-specific ROC curves and their convex hulls;
2. Compute the r-relaxed optimal solution for the chosen fairness criterion (using cvxpy);
3. Find the set of group-specific binary classifiers that match the optimal solution found.
   • each group-specific classifier is made up of (possibly randomized) group-specific thresholds over the given predictor;
   • if a group's ROC point is in the interior of its ROC curve, partial randomization of its predictions may be necessary.

Features and implementation road-map

We welcome community contributions for cvxpy implementations of other fairness constraints.

Currently implemented fairness constraints:

• equality of odds (Hardt et al., 2016);
  • i.e., equal group-specific TPR and FPR;
  • use constraint="equalized_odds";
• equal opportunity;
  • i.e., equal group-specific TPR;
  • use constraint="true_positive_rate_parity";
• predictive equality;
  • i.e., equal group-specific FPR;
  • use constraint="false_positive_rate_parity";

Road-map:

• demographic parity;
  • i.e., equal group-specific predicted prevalence;

Citing

This repository contains code and supplementary materials for the following preprint:

André F. Cruz and Moritz Hardt. "Unprocessing Seven Years of Algorithmic Fairness." arXiv preprint, 2023.