The current practice generally is to use the point with the Youden index as the best threshold. Being based on sensitivity and specificity, this is good for discrimination or classification of already known outcomes. But it is not good for predicting an unknown outcome. For this, we need the threshold where the PPV+NPV is the highest. We call it the P-index ( Use of ROC curve analysis for prediction gives fallacious results: Use predictivity-based indices - PMC ). Do you agree that the best threshold for prediction is not the Youden index but the P-index? In fact, the use of the AUROC curve is an inappropriate measure for assessing the prediction performance of a model.
1 Like
No. Maximum likelihood estimation (or the Bayesian extension of it) is what gives us optimum prediction. This uses a logarithmic probability scoring rule. Bigger picture: It is a mistake to consider thresholds when building a model. Thresholds can only be determined when a utility function is available for an individual patient, and every patient may have a different utility function. If any dichotomization is to be done it must be done on outputs (never covariates) and done in such a way as to approximate maximizing expected utilities.
The concepts of PPV and NPV generally have an extremely narrow scope: when there is no pre-test probability and when the test is binary. This is quite far from risk estimation, which applies to extremely general situations.
3 Likes
Thanks for the response. Could you please cite a reference for this maximum likelihood estimation for optimal prediction?
Threshold based an individual-based utility could be an ideal solution, but it seems impractical as of now.
PPV and NPV may have a narrow scope but they are the right indicators to measure predictivity in a binary setup. Pre-test probability is required for retrospective studies, and not for prospective studies. PPV and NPV too are probabilities and should have the same applicability as the risk estimates. I will greatly appreciate your elaboration. Thanks.
A nice discussion can be found in:
Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477), 359-378.
The argument @f2harrell is making is based upon proper scoring rules. From the abstract of the paper referenced above:
A scoring rule is proper if the forecaster maximizes the expected score for an observation drawn from the distribution F if he or she issues the probabilistic forecast F, rather than G \ne F. It is strictly proper if the maximum is unique. In prediction problems, proper scoring rules encourage the forecaster to make careful assessments and to be honest.
1 Like
That’s a great reference. On a related note, maximum likelihood estimates tend to require smaller sample sizes, or get your more performance out of higher sample sizes. They can also accommodate censoring, clustering, and other complexities. We should use predictive performance measures that are aligned with maximum likelihood which is why I like pseudo adjusted R^2 measures as in R2
Agreed. My colleague and I published some work a while back that looks at thresholds explicitly from the perspective of the institution’s utility function: https://doi.org/10.1093/jamia/ocad042
It’s not very patient-specific, but we figured an average utility would be better than ignoring it entirely.
1 Like