FalaahArifKhan / fairness-variance

Studying model uncertainty and instability for different demographic groups

Fairness-Variance

Studying model uncertainty and instability for different demographic groups.

Structure of the repo

• configs folder includes all general configurations used for experiments.
• data folder includes datasets used in experiments. Rest of the datasets are present in the Virny library.
• results folder contains tuned hyper-parameters for all models used in experiments.
• source folder includes Python modules for experimental pipelines and visualizations.
• notebooks folder includes Jupyter notebooks used for all experiments. It has the following subfolders:
  • notebooks/diff_fairness_interventions_exp contains notebooks for the experiment with different fairness interventions.
  • notebooks/diff_train_set_sizes_exp contains notebooks for the experiment with different train set sizes and in-domain / out-of-domain settings.
  • notebooks/mult_repair_levels_exp contains notebooks for the experiment with different repair levels for Disparate Impact Remover.
  • notebooks/visualizations_for_all_datasets contains notebooks with visualizations for all experiments aggregated over all datasets and model types.

Table with used parameters for each dataset and fairness intervention pair

	ACS Income	ACS PublicCoverage	Law School	Student Performance
Disparate Impact Remover (DIR)	repair_level = 0.7	repair_level = 0.6	repair_level = 0.6	repair_level = 0.7
Learning Fair Representations (LFR)	{'k': 5, 'Ax': 0.01, 'Ay': 1.0, 'Az': 50.0}	{'k': 10, ‘Ax’: 0.1, ‘Ay’: 1.0, 'Az': 2.0}	{'k': 5, 'Ax': 0.01, 'Ay': 1.0, ‘Az’: 50.0}	{'k': 10, 'Ax': 0.1, ‘Ay’: 1.0, 'Az': 2.0}
Equalized Odds Postprocessor (EOP)	Apply (no parameters)	Apply (no parameters)	Apply (no parameters)	Apply (no parameters)
Reject Option Classification (ROC)	Apply with default settings	Apply with default settings	Apply with default settings	Apply with default settings
Exponentiated Gradient Reduction (EGR)	constraints = 'DemographicParity'	constraints = 'DemographicParity'	constraints = 'DemographicParity'	constraints = 'DemographicParity'
Adversarial Debiasing (ADB)	num_epochs = 200, debias = True	num_epochs = 200, debias = True	num_epochs = 200, debias = True	num_epochs = 200, debias = True