VBT-X is a diversity-aware fairness testing approach for black-box machine learning models. VBT-X improves verification-based testing (VBT), another fairness testing approach, with a sampling technique. We first reimplement VBT, referring to the source code of VBT, and then implement the sampling technique.
Refer to our paper IST'23 for more technical details.
The experimental results of the evaluation of VBT-X are available in
.
VBT-X was proposed by Zhenjiang Zhao, Takahisa Toda and Takashi Kitamura. The implementation was performed by Zhenjiang Zhao.
The experiments were based on Python 3.8.10
.
We recommend using a virtual environment to run and test VBT-X,
to avoid dependency conflicts between packages.
We show two ways to create a virtual environment.
- If you already have
Python 3.8.10
installed, you can usevenv
to create a virtual environment. - If not, you can
Make sure Python 3.8.10
and venv
are installed.
Create a virtual environment called env_vbtx
:
python3 -m venv env_vbtx
Activate the environment:
source env_vbtx/bin/activate
Install required packages:
pip install -r requirements.txt
Make sure conda
is installed.
Create a virtual environment called env_vbtx
, and install Python 3.8.10
and pip
in it:
conda create --name env_vbtx python=3.8.10 pip
Activate the environment:
conda activate env_vbtx
Install required packages:
pip install -r requirements.txt
Use the following command to run VBT-X:
python exp.py <dataset> <protected_attr> <model> <vbtx_version> [runtime] [loop_times]
The possible values for each parameter are listed below:
- dataset: Adult, Credit, Bank
- protected_attr: sex, race, age
- model: LogReg, NB, RanForest, DecTree
- vbtx_version: naive, improveds10, improved
- runtime: the running time in seconds (default=1200)
- loop_times: the number of repeated runs (default=31)
The parameter 'vbtx_version' specifies the version of VBT-X to be used:
naive refers to Basic VBT-X(s=10)
,
improveds10 refers to VBT-X(s=10)
and improved refers to VBT-X
.
The folder FairnessTestCases
contains 12 machine learning models prepared for fairness testing.
The folder Datasets
contains 3 training datasets, on which the 12 models were trained.
As an example, the following command means to use VBT-X
to perform a fairness testing on the configuration (Adult, sex, NB) within 60 seconds:
python exp.py Adult sex NB improved 60 1
Each run outputs two .csv
files, located in two folders DiscData
and TestData
, respectively.
The results contained in each folder are as follows:
DiscData
: the results of detected discriminatory instancesTestData
: the results of generated test cases
For the running example, suppose we obtain a result file DiscData/improved-NB-Adult-sex-60-0.csv
, and the first six rows of this file are shown below:
$ head -6 DiscData/improved-NB-Adult-sex-60-0.csv
1,35,54,1,6,1,0,2,1,50,1,9,74,1
1,35,54,1,6,1,0,2,0,50,1,9,74,0
3,0,28,1,4,55,0,0,1,74,7,81,58,1
3,0,28,1,4,55,0,0,0,74,7,81,58,0
5,26,11,3,4,0,4,0,1,3,3,7,57,1
5,26,11,3,4,0,4,0,0,3,3,7,57,0
For interpreting this file, a row represents an individual, and a pair of two consecutive rows represents a discriminatory instance. For example, the pair of row 1 and row 2 is a discriminatory instance, the pair of row 3 and row 4 is another discriminatory instance, and so on.
An individual (i.e., a row) is represented as a sequence of attribute values. The order of these attributes is the same as the order of the attributes of the training data. For example, you can find the list of attributes for Adult with the following command:
$ head -1 Datasets/Adult.csv
age,workclass,fnlwgt,education,martial_status,occupation,relationship,race,sex,capital_gain,capital_loss,hours_per_week,native_country,Class
The results in DiscData
are interpreted in the same as above,
except that a pair of two consecutive rows represents a test case.
If you use the VBT-X, please cite our paper IST'23.
Bibtex:
@article{zhao2023ist,
title = {Diversity-aware fairness testing of machine learning classifiers through hashing-based sampling},
journal = {Information and Software Technology},
volume = {167},
pages = {107390},
year = {2024},
issn = {0950-5849},
doi = {https://doi.org/10.1016/j.infsof.2023.107390},
url = {https://www.sciencedirect.com/science/article/pii/S0950584923002458},
author = {Zhenjiang Zhao and Takahisa Toda and Takashi Kitamura},
keywords = {Algorithm fairness, Fairness testing, SAT/SMT solving, Constraint sampling, Hashing-based technique}
}
VBT-X is licensed under The MIT License.