Misunderstandings about the types of causal inferences we can make from RCTs abound. They make their way into journals, propagating like viruses and lodging in the brains of other researchers and clinicians. Many have tried in vain to dispel them, attempting, futilely, to undo damage that has already been done.
These misunderstandings can impact clinical care. Every day, clinicians address causal questions posed by their patients. The most ubiquitous inferential error made by patients in assessing their own symptoms is “post hoc, ergo propter hoc.” This is not an easy error to explain to patients and some physicians make it too. And when a clinician makes it, he ends up misinforming his patients.
Maybe it could help to provide examples of the types of cause/effect questions that clinicians might face from patients and to reason through the responses that the clinician should provide, in layman’s terms.
The purpose of this post is to promote understanding by eliciting accurate, yet simple, narrative explanations (without statistical jargon) for commonly-misunderstood causal RCT concepts. All discussion is appreciated.
“Toy” examples
Example 1- A pharmaceutical company conducted a randomized, placebo-controlled clinical trial, in which patients with moderate to severe asthma were randomized to add either a new drug inhaler or a placebo inhaler to their usual treatment. Patients were followed for 3 years and all of their asthma exacerbations were recorded by their family physicians. At the end of 3 years, the exacerbation rate in each arm of the trial was compared. The rate was 30% higher in the placebo arm than in the new treatment arm.
Susan was enrolled in this clinical trial and her family physician had a record of all of her asthma exacerbations in the years leading up to the trial. After her trial participation ended and the study’s results had been published, Susan found out that she had been receiving the new drug (i.e. non-placebo) inhaler during the trial. Her family physician noted that the frequency of her asthma exacerbations had been lower during the time she was enrolled in the trial. After the conclusion of the trial, he tells Susan: “That new inhaler worked really well for you!”
Question- Is Susan’s physician’s inference/statement defensible? Why or why not?
Example 2- A pharmaceutical company studied the ability of a new diabetes drug to lower blood sugar, as compared with approved “standard of care” drugs. In order to ensure that the new drug was not having unintended adverse cardiovascular effects, the drug regulator mandated that the trial follow a large number of patients for several years. At the end of the trial, while analyzing adverse events recorded during the trial, the regulator noticed a higher rate of fractures among patients in the new treatment arm. In spite of this safety signal, the drug was ultimately approved, with the product monograph flagging the observed increased fracture rate among patients enrolled in the trial.
Cathy was enrolled in this clinical trial. During the course of the trial, she suffered a painful vertebral compression fracture. After seeing the product monograph labelling for the approved drug, Cathy asks her family physician if the new drug had “caused” her fracture.
Question- How should Cathy’s family physician respond?
Example 3- A global public health crisis caused by a novel infection prompted some physicians to begin prescribing an already-approved medication for “off-label” purposes, on the basis of mechanistic reasoning, hoping that it might work in treating the novel disease. A large RCT studying this question was ongoing but wouldn’t be completed for several months. Arguing that there was “nothing to lose,” Dr.Jones prescribed the drug off-label to any of his patients who asked for it. Other doctors were more circumspect than Dr.Jones, preferring to wait to see the results of the RCT. At the end of the trial, 3% of patients in each arm of the RCT had died. In spite of the RCT results, Dr.Jones continued to prescribe the drug off-label, arguing that:
“All of the patients to whom I have prescribed the drug have survived,” which is a pretty strong signal (to me, at least) that the drug MUST be doing something, in at least SOME people. MAYBE, instead of concluding that the drug is a “dud,” we should consider the possibility that the RCT result appeared neutral because the drug is actually capable of helping some people but harming others, AND that the proportions of those helped and harmed IN EACH ARM of the trial (as a result of being offered or not offered the drug) BALANCED OUT PERFECTLY, to yield the same mortality rate in each arm (?) AND MAYBE, if we were to just leave patients to their own devices and stop standing in their way (like we did during the RCT), those who were “destined” to benefit would somehow INSTINCTIVELY seek out the drug (patients know their bodies best, after all), while those “destined” to be harmed would not seek it. This is a free country after all…who am I to stand in the way of what a patient wants?”
Question- What’s wrong with Dr.Jones’ reasoning?