Confidence band for the ROC curve

jlevy13 · February 9, 2020, 2:16pm

How would you construct a confidence band for the ROC curve?

I do not mean a confidence interval for the AUC or for other parameters that can be derived from the ROC curve, but a confidence band for the curve itself.

I came across this paper by Macskassy, Provost & Rosset, but it is relatively old paper (from 2005), and also this paper by Kerekes from 2008 (but it does not provide a methodology which was not considered by Macskassy et. al.

I was thinking about using bootstrap, but then the question is how to order the bootstraped curves to get the 2.5% and 97.5% percentiles. One possibility is to order the curves by their respective AUCs, or another one-dimentional parameter (e.g. Youden index).

Any idea will be welcomed.

f2harrell · February 9, 2020, 5:38pm

You really need a simultaneous confidence region, but every point on the curve is a simple proportion so you can just use pointwise Wilson confidence intervals and try to convert them to rectangles for the two dimensions. But stop to question why you want an ROC curve in the first place since it is completely at odds with decision making and does not provide the needed insights. See this for example.

jlevy13 · February 10, 2020, 9:34am

Hi Thank you for the answer.
I am aware that ROC curves has many limitations, and I also think that a confidence band for the curve is not useful, to say the least. My excuse is that a co-worker asked me about it and I said that I will look at this question.