Dr. Frank Harrell encouraged me via Twitter to post my question here. Thank you in advance for your feedback.
Background:
I developed a predictive model which can be used to predict a (continuous) score. (The model is nothing fancy - a linear regression model with multiple predictors.)
The model will be applied on a per subject basis to predict their score from input predictor values. Given a subject’s predicted score, the overall goal is to recommend a particular course of action for that subject based on whether (i) their predicted score is greater than or equal to a pre-specified threshold or (ii) their predicted score is strictly less than that pre-specified threshold. For example, the course of action for (i) could be “do nothing” and for (ii) could be “take more training to improve your score”.
Question:
This overall goal makes me wonder whether one should check whether or not the entire prediction interval for the given subject contains or not that threshold?
Comments:
On Twitter, Dr. Harrell mentioned the following:
“That’s a super question, worth a new topic on http://datamethods.org if you want. We tend to use point estimates for decisions but the optimal Bayes decision takes uncertainties also into account.”
I should clarify that I operate in a Frequentist setting.
4 Likes
I don’t think that this important topic has been explored enough in the literature, so I look forward to others’ responses. To add a bit of background, it is very difficult to make an optimum decision without using a Bayesian approach. That’s because the Bayes optimum decision, e.g., one that maximizes expected utility / minimizes expected loss, takes consequences of the decisions into account in a formal quantitative fashion. Expected utility is a function of the posterior distribution of the unknown parameter(s) and the utility function. The utility function can be asymmetric, e.g., the cost of missing cancer may be greater than the cost (monetary and otherwise) of using a radical treatment on a non-existent cancer. The posterior distribution can also be asymmetric just as a confidence interval can go farther out on the “harm” side than on the “benefit” side. These asymmetries can result in an optimum decision that is not obvious. This is to say that the question is difficult.
We tend to use point estimates too quickly. At least looking at the prediction interval will give pause.
5 Likes
Hi @Isabella_Ghement, nice question. I don’t have an answer, but would like to ask for a clarification: let’s say for the sake of argument we achieve consensus that one should check the prediction interval. In that case the recommended action is clear/unambiguous if the prediction interval does not contain the decision threshold. But what if it does? It seems we’d go from the binary recommended action to ternary, which is clinically quite different situation (not saying that is worse; but surely different)