The current practice generally is to use the point with the Youden index as the best threshold. Being based on sensitivity and specificity, this is good for discrimination or classification of already known outcomes. But it is not good for predicting an unknown outcome. For this, we need the threshold where the PPV+NPV is the highest. We call it the P-index ( Use of ROC curve analysis for prediction gives fallacious results: Use predictivity-based indices - PMC ). Do you agree that the best threshold for prediction is not the Youden index but the P-index? In fact, the use of the AUROC curve is an inappropriate measure for assessing the prediction performance of a model.
1 Like
No. Maximum likelihood estimation (or the Bayesian extension of it) is what gives us optimum prediction. This uses a logarithmic probability scoring rule. Bigger picture: It is a mistake to consider thresholds when building a model. Thresholds can only be determined when a utility function is available for an individual patient, and every patient may have a different utility function. If any dichotomization is to be done it must be done on outputs (never covariates) and done in such a way as to approximate maximizing expected utilities.
The concepts of PPV and NPV generally have an extremely narrow scope: when there is no pre-test probability and when the test is binary. This is quite far from risk estimation, which applies to extremely general situations.
1 Like