Uncertainty Quantification of the Black Hole Mass Estimation
Predicting virial black hole masses using neural network model and quantifying their uncertainties.
This repository is to accompany the paper Uncertainty Quantification of the Virial Black Hole Mass with Conformal Prediction.
Setup and Requirement
git clone https://github.com/yongsukyee/uncertain_blackholemass.git && cd uncertain_blackholemass
pip install -r requirement.txt
Pipeline
-
Get data
- Setup configs:
config/config.yaml
: Main config fileconfig/config_getdata.yaml
: Additional configs to specify the data directory
- To run:
$ python get_data.py
- See Dataset on the catalogue and data used.
- Setup configs:
-
Feature extractor using fully-connected neural network model
- Setup configs:
config/config.yaml
- To run:
$ python run_featureextractor.py
- Generated output files:
datasplitidx.pkl
: dict of data split indices {'train' train_idx, 'test': test_idx}yscaler.pkl
: sklearn.preprocessing scalermodel.pth
: Trained PyTorch modelloss.pkl
: DataFrame of 'train' and 'test' lossesfeatures.pkl
: DataFrame of features, 'objid', 'label', 'scaled_label', 'output'
- Setup configs:
-
Uncertainty quantification for regression using MAPIE
- Setup configs:
config/config.yaml
: Main config fileconfig/config_runuqregressor.yaml
: Additional configs to select saved log directory fromrun_featureextractor.py
- To run:
$ python run_uqregressor.py
- Generated output files:
estimator_optim.pkl
: Optimized GradientBoostingRegressor model for normal regressionestimatorq_optim.pkl
: Optimized GradientBoostingRegressor model for quantile regressionmapieuq_<STRATEGY>.pkl
: Fitted MapieRegressor for specified strategymapieuq_pred.pkl
: y_pred, y_pis, sorted {'target', 'pred', 'lower', 'upper', 'pierr_metric'}mapieuq_picp_alpha.pkl
: DataFrame of prediction interval coverage probability (PICP) for different uncertainty quantification methods with alpha as indexmapieuq_mpiw_alpha.pkl
: DataFrame of mean prediction interval width (MPIW) for different uncertainty quantification methods with alpha as index
- Setup configs:
Dataset
Dataset from Sloan Digital Sky Survey (SDSS) Data Release 16 quasar properties are publicly available for:
- Catalogue: http://quasar.astro.illinois.edu/paper_data/DR16Q/
- Spectra: http://quasar.astro.illinois.edu/paper_data/DR16Q/fits/
Example Exploratory Notebooks
notebooks/inspectdata.ipynb
: Inspect data from SDSSnotebooks/featureextraction.ipynb
: Examine outputs of feature extractionnotebooks/uncertaintyquantification.ipynb
: Analyse various uncertainty quantification methods for the black hole mass estimation
Reproducible Results
The reproducible_output/
directory contains the generated logs and output files that are used for the analysis in our paper. For a list of output files, see Pipeline.
Citation
If you find this repository useful, please cite the paper:
TODO: insert citation
References and Resources
- Taquet, Vianney, V. Blot, Thomas Morzadec, Louis Lacombe and Nicolas J.-B. Brunel. “MAPIE: an open-source library for distribution-free uncertainty quantification.” ArXiv abs/2207.12274 (2022). [Paper | Code]
- Romano, Yaniv, Evan Patterson and Emmanuel J. Candès. “Conformalized Quantile Regression.” Neural Information Processing Systems (2019). [Paper | Code]
- Angelopoulos, Anastasios Nikolas and Stephen Bates. “A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification.” ArXiv abs/2107.07511 (2021). [Paper | Code]