Code for Quantifying Ignorance in Individual-Level Causal-Effect Estimates under Hidden Confounding
$ git clone git@github.com:anndvision/quince.git
$ cd quince
$ conda env create -f environment.yml
$ conda activate quince
Find the best hyperparameters using the tune
function, on a dataset like ihdp
for an ensemble
model.
$ quince \
tune \
--job-dir ~/experiments/quince/tuning/ \
--max-samples 500 \
--gpu-per-trial 0.2 \
ihdp \
ensemble
Here, we use the train
function to fit an ensemble
of mixture density networks on 10 realizations of the ihdp
with hidden confounding dataset. For the full results change --num-trials 1000
$ quince \
train \
--job-dir ~/experiments/quince/ \
--num-trials 10 \
--gpu-per-trial 0.2 \
ihdp \
ensemble \
--dim-hidden 200 \
--num-components 5 \
--depth 4 \
--negative-slope 0.3 \
--dropout-rate 0.5 \
--spectral-norm 6.0 \
--learning-rate 5e-4 \
--batch-size 200 \
--epochs 500 \
--ensemble-size 10
Plots will be written to the experiment-dir
$ quince \
evaluate \
--experiment-dir ~/experiments/quince/ihdp/hc-True_beta-None/ensemble/dh-200_nc-5_dp-4_ns-0.3_dr-0.5_sn-6.0_lr-0.0005_bs-200_ep-500/ \
compute-intervals \
--gpu-per-trial 0.2 \
compute-intervals-kernel \
--gpu-per-trial 0.2 \
plot-deferral \
plot-errorbars \
--trial 0
$ quince \
train \
--job-dir ~/experiments/quince/ \
--num-trials 50 \
--gpu-per-trial 0.2 \
synthetic \
--gamma-star 1.65 \
ensemble \
--dim-hidden 200 \
--num-components 5 \
--depth 4 \
--negative-slope 0.0 \
--dropout-rate 0.1 \
--spectral-norm 6.0 \
--learning-rate 1e-3 \
--batch-size 32 \
--epochs 500 \
--ensemble-size 10
$ quince \
evaluate \
--experiment-dir ~/experiments/quince/synthetic/ne-1000_gs-1.65_th-4.00_be-0.75_si-1.00_dl-2.00/ensemble/dh-200_nc-5_dp-4_ns-0.0_dr-0.1_sn-6.0_lr-0.001_bs-32_ep-500/ \
compute-intervals \
--gpu-per-trial 0.2 \
compute-intervals-kernel \
--gpu-per-trial 0.2 \
plot-ignorance \
print-summary \
print-summary-kernel \
paired-t-test
Repeat the above for --gamma-star 2.72
and --gamma-star 4.48
.
$ quince \
train \
--job-dir ~/experiments/quince/ \
--num-trials 20 \
--gpu-per-trial 0.5 \
hcmnist \
--gamma-star 1.65 \
ensemble \
--dim-hidden 200 \
--num-components 5 \
--depth 2 \
--negative-slope 0.0 \
--dropout-rate 0.15 \
--spectral-norm 3.0 \
--learning-rate 5e-4 \
--batch-size 200 \
--epochs 500 \
--ensemble-size 5
$ quince \
evaluate \
--experiment-dir ~/experiments/quince/hcmnist/gs-1.65_th-4.00_be-0.75_si-1.00_dl-2.00/ensemble/dh-200_nc-5_dp-2_ns-0.0_dr-0.15_sn-3.0_lr-0.0005_bs-200_ep-500/ \
compute-intervals \
--gpu-per-trial 1.0 \
print-summary
Repeat the above for --gamma-star 2.72
and --gamma-star 4.48
.