AUPRC as a loss function

Question

AUPRC as a loss function

pedro-ribeirosantiago opened this issue 3 years ago · comments

Pedro Henrique Ribeiro Santiago commented 3 years ago

Hi all,

Thanks for the excellent package! Is there any way to use the Area Under the Precision-Recall Curve (AUPRC) as the measure to evaluate cross-validation? This would be useful in cases of an imbalanced dataset.

Saito, T., & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PloS one, 10(3), e0118432.

Cheers,
Pedro

Jared Huling · Answer 1 · Mon May 17 2021 22:08:27 GMT+0800 (China Standard Time)

I would be happy to look at a pull request if you want to code it up

Pedro Henrique Ribeiro Santiago · Answer 2 · Wed Jun 16 2021 14:37:13 GMT+0800 (China Standard Time)

Hello Jared,

I believe I figured out the code to use the AUPRC as the measure to evaluate cross-validation. However, it will not be possible to account for observation weights (as in the case of the AUC using the function glmnet:::auc.mat) since there is no R function to compute the Weighted AUPRC. Should I go ahead and open a pull request?

Cheers,
Pedro

Jason E. Aten, Ph.D. · Answer 3 · Wed Jul 12 2023 20:11:38 GMT+0800 (China Standard Time)

I would be interested in AUPRC too. @pedro-ribeirosantiago could you describe how far you got, or post a pull request here? Thanks!

Jason E. Aten, Ph.D. · Answer 4 · Wed Jul 12 2023 20:26:41 GMT+0800 (China Standard Time)

Ah! I see you did submit a PR.

#24

@jaredhuling Could you comment? Would this be okay to merge in? Thanks guys.

Pedro Henrique Ribeiro Santiago · Answer 5 · Fri Jul 14 2023 07:59:32 GMT+0800 (China Standard Time)

Hey @glycerine and @jaredhuling,

I did the pull request last year. The build had a failure only in the 'R_VERSION=devel, R_ARCH=x64'. I'm experienced with R programming but, unfortunately, I was unable to identify the problem since the modifications were built correctly in the "stable" and "patched" versions. I'm happy to provide further support and go ahead. Please let me know how I can help!

Cheers,
Pedro

Jason E. Aten, Ph.D. · Answer 6 · Fri Jul 14 2023 18:16:37 GMT+0800 (China Standard Time)

Thanks @pedro-ribeirosantiago ! I was able to merge and load the patch just fine on my machine. I'm using linux, amd64, and R 4.1.0.

I tried to look at why the resulting oem package would install fine on stable and patched but not on devel. However the artifact log gets deleted after a month, and so AppVeyor does not have it any more.

@pedro-ribeirosantiago Could you make a small tiny change (to a comment perhaps) in the pull request and push it that so that AppVeyor will re-run the check and we can inspect the C:/projects/oem/oem.Rcheck/00install.out file which should contain the error.

Pedro Henrique Ribeiro Santiago · Answer 7 · Mon Jul 17 2023 12:11:52 GMT+0800 (China Standard Time)

Dear @glycerine and @jaredhuling,

I just made a new pull request and now it passed all tests, so it would be ready for merging. Please let me know if you need anything from my side and I'm happy to keep making changes if needed!

Cheers,
Pedro