This is an implementation of Score-based Hypothesis Testing for Unnormalized Models
See hst/requirements.txt
and ood/requirements.txt
- Perform Goodness of Fit Test through
hst/test_gof.py
.- Global hyperparameters are configured in config.yml
- Experimental setup are listed in make.py
- Hyperparameters can be found at process_control() in utils.py
- Perform the heuristic HST for Out-of-Distribution (OOD) detection on Network Intrusion Data through
hst/test_ood.py
. - Perform the heuristic HST for Out-of-Distribution (OOD) detection on Image Data through
ood/test_ood.py
. Our implementation is based on the official code for "Score-Based Generative Modeling through Stochastic Differential Equations: (https://github.com/yang-song/score_sde).
- Prepare datasets for Goodness of Fit Test
python hst/make_datasets.py
- Generate scripts to perform Goodness of Fit Test with multivariate Normal distribution by varying the perturbation level of mean and logvar
python hst/make.py --mode ptb --data MVN --run test
- Generate scripts to perform Goodness of Fit Test with Gauss-RBM by varing the sample size n
python hst/make.py --mode ds --data RBM --run test
- Perform KSD-U test with multivariate Normal distribution at perburbation level of 0.1 on mean and 0.0 on logvar and sample size n=100
python hst/test_gof.py --control_name MVN_ksd-u_0.1-0.0_100_0
- Perform HST (Simple) test with Exponential family at perburbation level 0.1 on tau and sample size n=100
python hst/test_gof.py --control_name EXP_hst-b-g_0.1_100_0
- Perform HST (Composite) test with Gauss-RBM at perburbation level 1.0 on W and sample size n=30
python hst/test_gof.py --control_name RBM_hst-b-e_1.0_30_0
- Perform computation comparison between LRT (Simple) and HST (Simple)
python hst/run_exp_time.py
- Perform OOD Detection on Network Intrusion Data (n=1)
python hst/train_ood.py --control_name KDDCUP99_rbm python hst/test_ood.py --control_name KDDCUP99_hst_none_1
- Perform OOD Detection on CIFAR-10(in-distribution) versus Tiny-ImageNet (out-of-distribution)
python ood/test_ood.py
- Multivariate Normal Distribution, power comparison by changing the perturbation level of
$\log(\Sigma)$ $(n=100)$
- Gauss-Bernoulli RBM, Power comparison of various tests varying sample size and the perturbation level
$\sigma_{ptb} = 0.03$
- (a) ROC curves and (b, c) histograms of test statistics of HST for OOD Detection on CIFAR10 (in-distribution) and Tiny ImageNet datasets (out-distribution)
Suya Wu
Enmao Diao
Khalil Elkhalil
Jie Ding
Vahid Tarokh