roc_curve_with_confidence_intervals
Python version
import platform
print(platform.python_version())
3.7.5
Notes
Run you jupyter notebook
positioned on the stackoverflow
project folder.
The the following notebook cell will append to your path the current folder where the jupyter notebook is runnig, in order to be able to import auc_delong_xu.py
script for this example.
import os
import sys
import pandas as pd
import numpy as np
from sklearn import datasets
notebook_folder_path = !pwd
prj_path = os.path.abspath(os.path.join(notebook_folder_path[0], '', ''))
sys.path.append(prj_path)
print('Append to path: %s' % prj_path)
Append to path: /Users/lsanchez/roc_curve_with_confidence_intervals
The data
I used the iris dataset to create a binary classification task where the possitive class corresponds to the setosa
class.
The y_score
is simply the sepal length
feature rescaled between [0, 1]
.
data = pd.DataFrame(
datasets.load_iris().data,
columns=datasets.load_iris().feature_names)
target = pd.Series([
datasets.load_iris().target_names[x]
for x in datasets.load_iris().target])
data.head()
sepal length (cm) | sepal width (cm) | petal length (cm) | petal width (cm) | |
---|---|---|---|---|
0 | 5.1 | 3.5 | 1.4 | 0.2 |
1 | 4.9 | 3.0 | 1.4 | 0.2 |
2 | 4.7 | 3.2 | 1.3 | 0.2 |
3 | 4.6 | 3.1 | 1.5 | 0.2 |
4 | 5.0 | 3.6 | 1.4 | 0.2 |
The AUC and Delong Confidence Interval is calculated via the Yantex's implementation of Delong (see script: auc_delong_xu.py
for further details)
from auc_delong_xu import auc_ci_Delong
y_true = (target != 'setosa').astype(int)
y_score = data['sepal length (cm)']
y_score = (y_score - y_score.min()) / (y_score.max() - y_score.min())
y_pred = (y_score > .5).astype(int)
auc, auc_var, ci = auc_ci_Delong(
y_true=y_true,
y_scores=y_score)
print('ROC AUC: %s, Conf.' % auc)
print('Confidence Interval: %s (95%% confidence)' % str(ci))
ROC AUC: 0.9586000000000001, Conf.
Confidence Interval: [0.93020851 0.98699149] (95% confidence)
R Version
version
_
platform x86_64-apple-darwin13.4.0
arch x86_64
os darwin13.4.0
system x86_64, darwin13.4.0
status
major 3
minor 6.1
year 2019
month 07
day 05
svn rev 76782
language R
version.string R version 3.6.1 (2019-07-05)
nickname Action of the Toes
The data
I used the iris dataset to create a binary classification task where the possitive class corresponds to the setosa
class.
The y_score
is simply the sepal length
feature rescaled between [0, 1]
.
library(pROC)
library(datasets)
data(iris)
y_true = as.integer(iris$Species == 'setosa')
y_score = iris$Sepal.Length
y_score = (y_score - min(y_score)) / (max(y_score) - min(y_score))
y_pred = as.integer(y_score > .5)
roc = roc(y_true, y_score)
roc
Type 'citation("pROC")' for a citation.
Attaching package: ‘pROC’
The following objects are masked from ‘package:stats’:
cov, smooth, var
Setting levels: control = 0, case = 1
Setting direction: controls > cases
Call:
roc.default(response = y_true, predictor = y_score)
Data: y_score in 100 controls (y_true 0) > 50 cases (y_true 1).
Area under the curve: 0.9586
print(ci(roc))
95% CI: 0.9302-0.987 (DeLong)