Dr. Frank Harrell encouraged me via Twitter to post my question here. Thank you in advance for your feedback.
I developed a predictive model which can be used to predict a (continuous) score. (The model is nothing fancy - a linear regression model with multiple predictors.)
The model will be applied on a per subject basis to predict their score from input predictor values. Given a subject’s predicted score, the overall goal is to recommend a particular course of action for that subject based on whether (i) their predicted score is greater than or equal to a pre-specified threshold or (ii) their predicted score is strictly less than that pre-specified threshold. For example, the course of action for (i) could be “do nothing” and for (ii) could be “take more training to improve your score”.
This overall goal makes me wonder whether one should check whether or not the entire prediction interval for the given subject contains or not that threshold?
On Twitter, Dr. Harrell mentioned the following:
“That’s a super question, worth a new topic on http://datamethods.org if you want. We tend to use point estimates for decisions but the optimal Bayes decision takes uncertainties also into account.”
I should clarify that I operate in a Frequentist setting.