I wrote to Dr. Harrell directly and he suggested I post this question here. Please be gentle.
I’m starting to get a grasp on why ROC curve cutoffs (and just about any cutoff) are inadequate and I’m sold on predictive regression modeling.
So, where I’m curious is . . . what I’m calling the intersection of prediction and causality. After you build a model, and make a prediction, what guidance can the model give a clinician in terms of actionable information.
Say a patient comes in, and you’ve trained a valid and robust predictive model with the help of rms (both the package and book!). You, or the EMR system, enters the patient’s information into the model and they have a bad prognosis of disease X. You look at their information and you notice that their lab count Z is well outside of “normal range” (let’s say, the ones provided on a common lab sheet). For simplicity let’s assume lab count Z can be directly manipulated by drugs and not dependent, say, on a failing organ.
Okay, so here we are.
Use the model to see how much this lab value is affecting the prediction for the individual patient?
Use the model to look at alternative scenarios for that individual patient (given the assumptions of the model, would getting their lab value back to a certain range, or increasing/decreasing it by an interval, affect their model-driven prognosis)?
Use the model to define a new data-driven “normal range” for lab value Z for the “small world” of this sample? I could see the problems in this as perhaps no one in the sample (depending on the sample) has a “good” lab value for Z. Also, it’s a problematic overall average, but perhaps the clinician is nuanced and just considers it as a piece of evidence among many.
So, I love this quote from Dr. Harrell on his blog about his journey to Bayes. “Null hypothesis testing is simple because it kicks down the road the gymnastics needed to subjectively convert observations about data to evidence about parameters.”
So, here we are again, we know the patient has a bad prognosis and we now need to go from data observations to actionable information and we know we can’t rely on cutoffs/p-values for easy answers. In other words, we need to gamble (thank you Nate Silver) on a course of action with the patient. Can any of the numbered steps above provide valid intel/evidence to help the clinician make a good bet? Or should it all be instinct, training, gestalt from this point forward?
What would you do? Thanks!
Update - I see Dr. Harrell has written about this here a bit under “What is a good global strategy for making optimum decisions for individual patients?” I would love to see more extensive explanations, opinions, sources, and optimally a well-written book on this! Do such things exist?