What should MDs-in-training know about medical prediction?

prediction

#21

Just remember that Bayes’ rule is not needed for the diagnostic problem. And using it there doesn’t get students excited about the many great applications of Bayes where Bayes thinking is truly needed. In a prospective cohort study you can directly get the probabilities of real interest.


#22

I am not sure that this the right way to put it- and I’ve been thinking about it for a while now and wished I had a good answer (I don’t) but these are the lines along which I’m thinking-

Group probabilities versus individual binary fate-
it is the tension between probabilities and the fact that as an individual, you are one point in there but have no control over where you fall into that probability and the outcome for yourself is binary, not a probability. So 90% survival sounds good- but is useless for you and your family if you are in the 10% who don’t survive.
As a medic, you can home in peace and sleep well that with a survival chance of 90%, your patient most likely will be fine. As a patient, you wonder ‘what if I am in those 10%’- and I tell you, you don’t sleep fine with that. We personally have always lived by ‘hope for the best, prepare for the worst’- but this is usually not what your physician will tell you.
Slightly off topic but I am currently reading ‘Skin in the game’ - very cynical but absolutely worth reading- and it feels very much like the same situation.

Impact of outcomes
Linked to the issue of probability is the fact that we aren’t taught at medical school of the psychological impact of a severe diagnosis. My husband was diagnosed with Stage 4 Melanoma out of the blue- and I was shell-shocked to realise that what I had thought of a medical problem in fact comes with a mental burden of at least the same size. Probabilities are often about worsening of disease, progression or downright survival- and unless we get a better understanding of what this means to a person our communication will always be off.

The combination of ‘90% survival good for the group’ plus a severe underestimation of the psychological burden to the patient leads then to really poor communication results. Patient pushing physicians to give them an answer ‘how long do I have to live’. Physician evasive (patient ‘he/she got no idea’), physician giving probability (things go worse- patient ‘they haven’t done their best’, things go better ‘my doctors were wrong’ ) - our patient forums are FULL of them, none of these scenarios are good and they ultimately undermine trust.

Then, not directly to the question of how to communicate probability but building on the comments earlier on this thread- there is a lot of sloppy thinking out there which even doesn’t require advanced statistics to solve, e.g.

  • physicians giving Melanoma (sorry, that’s the disease of my interest, so therefore the examples are very loop-sided toward there ;-)) patients long-term survival data. We had over 10 approvals of new and effective Melanoma drugs since 2011- so we simply CANNOT have valid long-term OS data. 10 years OS data takes…more than 10 years to acquire.

  • just yesterday, someone was quoted the efficacy of chemotherapy in Melanoma- it doesn’t work, just like we learned in Med School ages ago- but given RECENT data- with cross-over from highly effective therapies that now make the rubbish comparator look good as they are able to salvage some.

So, in addition in how to communicate…the data has to be correct.


#23

My knee-jerk reaction here is that this is the bread and butter of health economics, although there the focus is usually on the health system choosing between a sequence of decisions based on population averages. Individual patient simulation, using appropriately modeled predictive distributions for parameter uncertainty is probably the answer you want clinically. This is not realistic for 99% of practicing clinicians though.

That said, some resources I would recommend are:

Textbooks

  1. The “blue book” by Michael Drummond Methods for the Economic Evaluation of Health Care Programmes
  2. The “green book” by Briggs and Claxton https://www.amazon.ca/Decision-Modelling-Health-Economic-Evaluation/dp/0198526628/ref=sr_1_1?s=books&ie=UTF8&qid=1534334947&sr=1-1&keywords=decision+modelling+health+economics

Open source references (general):

  1. ISPOR good modeling practices State-Transition Modeling: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force-3
  2. Markov Decision Processes: A Tool for Sequential Decision Making under Uncertainty
  3. Guidelines for the Economic Evaluation of Health Technologies: Canada

Open source resource (individual participant simulation)

  1. NICE TSD 15 - Patient-level simulation

Bonus that I think @f2harrell might like:

  1. RFP Topic: Use of Network Meta-analysis to Inform Clinical Parameters in Economic Evaluations

Note that as per @f2harrell all the health economic methods really do is built a loss function which you then minimize. The tricky bit then becomes (as @f2harrell states) whose loss function do you use, and what happens if you specify it incorrectly.


#24

Please discuss this particular topic on the other topic page we have set up for teaching probability. I think you are falling into the trap of assuming that probabilities don’t apply to individuals. If this were the case there would be no gambling on individual football games or horses, and you would think differently (or not at all) when deciding whether to carry a large metal stick out on a golf course in an electrical storm. It’s best not to use the term group probabilities. The probabilities we calculate in medicine are learned from a group but are intended to apply to individuals, at least if such individuals were adequately represented in the heterogeneous group. Let’s pick this up on the other topic. It’s all about playing the odds, and “in the 10% who don’t survive” is a phrase that does not help because it is a retrospective way of looking at things and is not useful at the early decision point IMHO.


#25

Thank you @bryll. One point you bring up that I had not considered: some patients will be very upset to learn that their risk of stroke is 0.01, others will be unaffected. Guiding individual patients through their unique experiences of risk, maybe we could call it “risk management,” is important and should be addressed in a lecture on medical prediction.

So, we have, in temporal order, (1) risk estimation, (2) risk interpretation, (3) risk communication, and (4) risk management. I have focused on (1) and (2). There is a decent literature on (3) including Spiegelhalter’s work (https://understandinguncertainty.org) as mentioned by @f2harrell. Can (4) be taught as well? Maybe stories on the subjective experience of risk would help.


#26

Great points, and this makes me wonder of the usefulness to patients of simultaneously communicating reference risks. Perhaps Spiegelhalter has discussed this. For example, compare the 0.01 to the risk of getting killed in a car accident with 5 years of 10,000 miles per year of driving.


#27

Yes something to anchor the risk to risks people generally accept and to risks they generally don’t accept


#28

Throughout my, albeit short, medical career I have sought out a “reference set” of risks to compare to when discussing risks with my patients. Patients usually only hear about probability when discussing distressing medical information. They have no frame of reference, nothing to compare to. So as they are being told about diseases and procedures they don’t understand, they are also being communicated to in probability and risk with no “patient-world” tie-ins to guide them.

I think MDs underestimate this lack of a frame of reference. Most MDs are married to MDs, their friends are MDs - they are constantly marinated in medical terminology. I teach my students to cherish and ruminate on their meetings with banks and mortgage loan officers, to remember how stupid they felt, how they agreed and nodded to questions and statements they did not understand, how they signed papers in blind trust of this person they had never met before, partially because they didn’t feel like they could admit how little they understood of what was just said.

I wish there were a intuitive reference list of risk from the day-to-day lives of the population. Fender-benders and other car-related statistics seem like they would fulfil this criteria, as you mentioned. Local weather patterns maybe, “the same probability as snow in June”.