Practically it is often challenging to rely only on RCT data when making inferences about patients we see in clinic. This is because RCTs, being experiments, prioritize patient selection over representativeness to reduce variance and maximize the information gained on relative treatment efficacy. These inferences need to be fused with large-scale observational data to facilitate patient-specific inferences.
Thus, I think Pearl and @scott have their hearts in the right place even if I do certainly share the concerns stated in this thread and on twitter. Here is a more mature attempt for such fusion between RCT and observational datasets.
Granted, there are ill-defined RCTs and there can be biased/confounded observation. The case presented in the blog displays a surprisingly large gap between RCT and observation data yet claiming that both are based on similar population. Neither can RCT answer counterfactual questions down to individual cases, nor can observational data as long as only statistical information is available. Pearl’s rung 3 requires deterministic modeling / knowledge.
Observational data may help refine RCT, assessing confounding factors and thus e.g. trying to zoom in as much as possible towards a sub-population of interest. But if observation contradicts well-defined RCT results then it probably misses a lot of important features or is plagued by selection biases. RCT is also useful to find or prove causal arrow direction.
Data proposed in the blog can be modeled with following causal Bayesian network.
Treatment:
Drug No drug Gender Setup
0.5 0.5 F RCT
0.6998 0.3002 F Choice
0.5 0.5 M RCT
0.6998 0.3002 M Choice
Gender:
F M
0.5 0.5
Outcome:
Survived Died Treatment Gender Setup
0.489022 0.510978 Drug F RCT
0.270328 0.729672 Drug F Choice
0.49002 0.50998 Drug M RCT
0.699715 0.300285 Drug M Choice
0.210579 0.789421 No drug F RCT
0.699336 0.300665 No drug F Choice
0.210579 0.789421 No drug M RCT
0.699336 0.300665 No drug M Choice
Setup:
RCT Choice
0.5 0.5
As @stephen and others have written before, RCTs do not require representativeness in the usual sense, and do not profit from it aside from specific issues surrounding interactions.
Which is exactly why RCTs alone usually cannot be used for patient-specific inferences, i.e., at rung 3 of Pearl’s ladder of causation. For that we need additional knowledge and assumptions.
The same applies for all well-designed experiments, including those we do in the lab. Alone they are not enough to guide any major decisions. Knowledge from multiple domains needs to be integrated.
I totally accept the principle as pointed out by you @f2harrell and @stephen that RCTs can only be expected to estimate risk ratios or odds ratios from ‘non-representative’ populations and that this information is applied to an individual by using her or his personal baseline risk of the trial outcome. However this baseline risk should not be estimated in any old way. (For example some cardiovascular risk calculators will predict that a lipid lowering drug will reduce CV risk in a person with an extremely high blood pressure when the lipid levels in that person are impeccably low risk!)
The baseline risk should be based on a data examined during the trial (e.g. as disease severity or staging) by the investigators in planned way. If this was not done at the time (as by @Scott and JP), I can’t see a subsequent ad hoc observational study as suggested by @Pavlos_Msaouel, @Scott and JP providing the required baseline risks. It will need another RCT or planned study by randomising to different test thresholds (or different tests) as described in my recent pre-print [1] and the chosen model based on collapsible ORs or RRs calibrated as described in Section 5.7 and Figure 5 of that pre-print. The principle behind this design is to assess the RR or OR in a sub-population where there is equipoise. The population is therefore highly non-representative. Section 2.1 of the preprint sets out the assumptions on which this is based (the subsequent proof for this is in section 5.1 and thus why ‘representativeness’ is not required in general).
@Stephen has been very supportive of my ideas. Do you or anyone know of other groups who are trying to do something similar to me?
Why not? When making individualized inferences we have to assume that the patient we see in clinic shares certain stable causal properties with the RCT population. One could indeed assume that this causal structure is only present in the RCT cohort and apply the RCT findings only to patients from the exact same population, including the center they were treated and all other eligibility criteria. But no clinician ever does that, certainly not in oncology. Instead, we assume that the patient we see in clinic shares the same key causal structures that allow the transportability of inferences from an RCT that may, for example, have been done in a different country. This is a reasonable approach, in part because the RCT eligibility criteria are often very poor discriminators of patient baseline risk, as opposed to large-scale observational studies to develop prognostic risk scores.
But now note that the assumptions needed to transport RCT inferences to the patient we see in clinic are generally stronger than the ones needed to integrate knowledge on the relative treatment effect from RCTs with knowledge on baseline risks from observational datasets.
I am writing a paper on this with the hope that it will be accessible to statisticians and methodology-minded clinicians. The biggest challenge is notation. Each community focuses on different aspects of a problem and uses their own terminology and mathematical notation, and this can be confusing. I was raised within the Bayesian statistical school and I am much more familiar with that notation. But for this and other related projects, in the past few months I dived much more into the notation used by computer scientists to also help our statistical collaborators digest and incorporate it in our work. It is not an easy task but it is well worth it.
As a very simple but practical example of how valuable this integration can be, see this post discussing how problematic it can be to judge a surrogate endpoint based on how strongly it correlates with overall survival. Instead, we need first to consider the causal networks that generated these endpoints.
I challenge that statement. If the RCT’s patient entry criteria are reasonably broad, the RCT provides the best data for estimating outcome tendencies of the patients in the control arm. Among other things, RCTs have a well-defined time zero and do not lose patients to follow-up. Observational data, besides having higher measurement error and often tragically higher amounts of missing data, tend to have trouble deciding exactly which patients to include in the analysis and trouble keeping track of patients in follow-up.
“However this baseline risk should not be estimated in any old way. (For example some cardiovascular risk calculators will predict that a lipid lowering drug will reduce CV risk in a person with an extremely high blood pressure when the lipid levels in that person are impeccably low risk!)”
Are you questioning the validity of currently-used CV risk calculators (e.g., Framingham- derived from massive, decades-long observational studies) or are you highlighting the fact that CV risk is multifactorial?
If you are suggesting that our estimates of “baseline” CV risk for individual patients should be based on observations derived from RCTs rather than large, longterm observational cohorts, then I’m confused…how exactly could we derive valid risk estimates from RCTs (which might be orders of magnitude smaller and short-term) ?…
As an aside, I have patients with LDL values close to 2.0 (not on a statin), who were subsequently diagnosed with advanced coronary and/or carotid atherosclerosis. I (and their cardiologists) still treat them with statins to reduce their future MI risk.
Beautiful arguments related to the open research question of how best to debias the different types of such datasets. To prevent these considerations from distracting us from the main theme, we use in the aforementioned upcoming paper observational data derived from RCTs. This is consistent with the use of externally developed risk models described in the PATH statement.
Having said that, prospective rigorously conducted observational studies focusing on data collection across departments/institutions/countries can also address these considerations as well as, or better than, external RCTs used to develop risk models. This is because the purpose of RCTs is to estimate relative treatment efficacy whereas such protocols are specifically designed for risk modeling. See here for an example of a companion prospective technology-enabled data collection protocol that has now enrolled hundreds of patients and is allowing us to redefine disease states related to the novel toxicities observed with immune checkpoint therapies.
The Framingham data is used to estimate total CV risk. I have no quarrel with that. My concern is that a RR (e.g. 0.5) based on the treatment of one risk factor (e.g. a statin for hyperlipidaemia) should only be applied to the risk component due to that risk factor alone (e.g. of 2%). This individual risk reduction of 2% x 0.5 = 1% should then be subtracted from the total risk due to hyperlipidaemia, smoking, high BP, diabetes, gender etc. (e.g. 10%-1% to give a reduced total risk of 9% - if one assumes additive risk factors). Instead, some risk tools appear to apply the RR of 0.5 to the total risk (e.g. 0.5 x 10% = 5%, which exaggerates the risk reduction. Something along these lines would be my response also to @Pavlos_Msaouel 's “Why not?” Ideally the estimated risks (e.g. of MI) should be calibrated against the frequency of outcomes (e.g. of MI) in an observational study if the risks were constant enough over time.
Yes, I think we are pretty much aligned. What I do also want to highlight is that the computer scientists have developed syntaxes (graphical and mathematical) that can be used to explicitly represent these considerations. These tools are powerful and upcoming generations of biostatisticians (or data analysts or whatever we want to call them) will very likely be taking advantage of them a lot.
That approach to gauging the benefits of statins doesn’t seem consistent with the inclusion criteria for the large pivotal statin RCTs- these criteria didn’t hinge on patients having some specified pretreatment LDL level, but rather on the presence of clinical evidence of cardiovascular disease. In other words, the thing being randomized was not an LDL level, but rather a high risk patient..
Since the patients enrolled in the pivotal statin trials were “high-risk” for cardiovascular events not simply due to their LDL levels, but rather as a result of their clinical cardiovascular disease (which in turn would have reflected their global risk profiles), it seems kosher (to me) to apply the relative risk reduction seen in these trials to patients’ global CV risk (as estimated from observational studies), and not just to the LDL-attributable CV risk. I’m not an expert on this topic, but the implication that statins would only confer meaningful CV risk reduction for patients with elevated LDL seems at odds with everything I’ve ever been taught…
You are right. The rough estimate of the risk ratio (RR) for statins from high CV risk trials irrespective of lipid levels is about 0.8. Therefore if the untreated risk is 10% over 10 years, then on average, taking a statin will reduce it to 8%. This hides a variation in risk reduction (AKA heterogeneity of treatment effect) which is estimated as 20% reduction per 1mmol/l reduction in LDL cholesterol.
This implies when statins are given without knowing the LDL cholesterol, the average reduction in LDL cholesterol is 1mmol/l giving a reduced risk of 20% and the above RR of 0.8. However, if the LDL cholesterol is reduced by 3mmol/l from 4 to 1mmol/l then there will be a roughly 3 x 20 = 60% reduction in CV risk so the RR is 0.4. However, if the LDL cholesterol is already 1mmol/l then it is unlikely that taking a statin will reduce it beyond measurement error and achieve any significant risk reduction.
You are right that public health policy from the results of overall CV risk trials is to treat irrespective of the individual person’s LDL cholesterol. This avoids time consuming testing, interpretation etc and is estimated to be the most cost effective approach from an economic point of view based on doctor’s time, cost of tests, etc. However, the personal cost of this approach is that many with low LDL cholesterol are treated when they have little prospect of benefit from a statin and may be subjected to its adverse effects.
Many object to this blunderbuss approach especially those who belong to the Over-diagnosis / over-treatment movement. Perhaps if a patient suffers adverse effects (e.g. muscle pain) then it might be sensible to practice ‘personalised’ (as opposed to public health) medicine and to stop the statin, measure the LDL and HDL cholesterol and if they show very low risk for that particular patient, then advise not starting again. If such a patient then suffers a CV event, then the ‘cause’ is probably due to some other risk factor, not lipids. Perhaps that @Scott and JP might like to comment on this.
Hmm- it would be interesting to hear from others on this topic. Although this thread didn’t start out with a focus on statins, the topic provides a perfect clinical example of the challenges of applying RCT findings to individual decision-making…
There are clearly differences of opinion around the potential benefits of statins even at lower LDL. I’d have trouble withholding a statin from a patient with an LDL of 2.0 and documented CAD (and, in my experience, so would cardiologists).
The section of UptoDate entitled “Management of low density lipoprotein cholesterol (LDL-C) in the secondary prevention of cardiovascular disease” includes the following statement in the Summary section:
“For patients with CVD, independent of the baseline low density lipoprotein cholesterol (LDL-C) level, we recommend lifelong high-intensity statin therapy ([atorvastatin] 40 to 80 mg or [rosuvastatin] 20 to 40 mg) rather than moderate intensity statin or no LDL-C lowering therapy ([Grade 1A]. For patients who do not tolerate these doses, the maximally tolerated dose of a statin should be used.”…
And while my own approach as a family physician has been to tailor statin treatment (including dose) primarily to the patient’s CV risk rather than a number (LDL), it’s certainly always important to consider patient preference. Fortunately, statins are extremely well-tolerated for the vast majority of patients.
I accept what you say but would point out that in the case of people with existing CVD we are talking about secondary prevention where it is widely accepted that statins should be prescribed irrespective of lipid profiles, perhaps because of a high prior probability of lipid abnormalities. This is supported by a widely quoted theory that statins also have an anti-inflammatory effect that reduces or reverses atheroma formation (perhaps via LDL cholesterol reduction as there is also a theory that latter causes endovascular inflammation). I am unaware of RCTs to assess the efficacy of statins on the frequency of CV events in people with low LDL cholesterol but below is an observational study where about 40% of people already had CVD so that it was partly a case of secondary prevention [1].
The risk reductions that I was referring to in my previous post concerned primary prevention when no CV events have yet taken place. I agree with you that this discussion illustrates the complexities physicians face when trying to interpret RCTs and observational studies and practising personalised medicine.
What’s the net N for the sum total of all statin RCTs? How are statins not a solved problem by now?
A) The trials weren’t designed/conducted right
B) The trials weren’t analyzed right
C) We’re making progress, but need to stay the course and do more trials
D) Something else is wrong
I think that the answer is D because ‘diagnostic’ tests are not assessed properly. In my opinion, they are not assessed properly for use to provide conditional probabilities of outcomes on intervention and control and the absolute risk (or probability) differences. Instead, risk ratios or odds ratios are applied incorrectly to total risk as explained in post 46. The assessment of medical tests focuses on estimating sensitivities and specificities that only give a rough idea of a test’s value in screening. Tests are also used to give lists of differential diagnoses, to differentiate between them and to form sufficient and necessary diagnostic criteria. The latter are closely related to their use in providing conditional probabilities of outcomes on treatment and control. So in my view the problem is not with RCTs but the backwardness of implementing their results and in the use of diagnosis tests. I explain how much of this might be put right in the Oxford Handbook of Clinical Diagnosis (e.g. http://oxfordmedicine.com/view/10.1093/med/9780199679867.001.0001/med-9780199679867-chapter-13#med-9780199679867-chapter-13-div1-20) and in a recent preprint (see post 38).
What is the problem you’re referring to David? Is it the problem of accurately estimating 10 year MI risk in a primary prevention setting (i.e, are you highlighting the suboptimal nature of existing CV risk estimators/calculators in common use)? Or are you questioning whether it’s valid to extrapolate the relative benefits of statins from secondary prevention trials to a primary prevention context? Or are you questioning whether the relative efficacy of statins might decrease as a patient’s baseline LDL decreases?
From a purely practical standpoint, I don’t really perceive a major problem, provided that we consider that the primary role of statins is to reduce CV risk rather than just LDL.
I know none of the MDs on this site need to hear this, but non MDs might be interested in how primary care physicians and cardiologists usually approach discussions around statin treatment:
Everyone agrees that patients with clinical evidence of CAD (e.g., angina or a history of MI) need statins- these patients have already proven that they have vascular lipid plaque. This treatment context is called “secondary” prevention;
For those without symptoms (e.g., no angina or past history of MI), we use risk calculators derived from large, longterm observational cohort studies (as imperfect as they may be) to estimate 10 year MI risk. Examples of risk factors included in these calculators include: age, smoking status, lipid levels, presence/absence of diabetes and hypertension, etc…We share the 10 year absolute risk estimate with patients and we then usually quote the relative statin effect, as derived from large historical statin RCTs (which were conducted in a secondary prevention context). We apply this relative statin effect to each patient’s estimated absolute baseline risk, thereby providing them with an estimate of the absolute risk reduction they might expect over a given period of time if they were to start a statin;
A third group of patients has arisen in the past few years and the size of this group is expanding rapidly as our imaging techniques improve. These are patients who have imaging done for reasons unrelated to their vascular health, whose CT scans and ultrasounds (chest, abdomen) incidentally show evidence of atherosclerosis. There are no guidelines for how to manage these asymptomatic patients who nonetheless have anatomical evidence of atherosclerosis. My informal observation is that cardiologists tend to recommend statins (usually high intensity) for them (and this has been my own practice as well). These patients fall somewhere “between” the primary and secondary prevention context, not really fitting into either category.
Was there a better way to design the original large statin trials? If so, what would they have looked like? Is there a better way for us to be managing statin discussions than the method we’re using now?
Erin, part of what instigated my MCQ was the tentative-sounding heuristic you had mentioned in an earlier post:
Shouldn’t we be doing studies in such a way that we could by now approach these questions in a less purely heuristic/phenomenalistic manner? Or is this type of therapy too marginal to have such expectations of it? Is precision medicine practicable only in cases where the risks and risk reductions are large in absolute terms?
I’m not sure I’d describe this approach as a “heuristic.” I haven’t really read anything that convinces me (at least in terms I can understand, not coming from a math background), that we’re all committing an egregious error by practising the way we have been now for many years.
I don’t like prescribing pills and most patients don’t like taking them. But it’s important to remember that risk stratification can fail in both directions. Since our predictive ability is suboptimal, one person might be overtreated, but another undertreated. From a public health standpoint and also the standpoint of a doctor who likes her patients, I’m much more concerned that our current risk calculators will still misclassify an occasional young man as “low risk” for coronary disease, only to have him suffer a fatal MI at age 50, than I am that I might be prescribing a statin for a couple of decades to a person who was never “destined” to develop significant atherosclerosis.
I think there’s a risk in overthinking things, at least where this particular drug class is concerned. Considerations will certainly be different for medications with more significant toxicity and unestablished efficacy. But as far as statins go, we’re talking about a dirt cheap drug class (all now generic) with decades of safety data and a generally benign side effect profile.
You seem to be wondering why, after all these years, we still have not developed a more “personalized” approach to statin therapy in the primary prevention context. I would like that too, but easier said than done. As you know, this is basically the idea underlying coronary calcium scoring (the utility of which has proven to be fairly controversial for several reasons). And maybe let’s just all agree not to go down the polygenic risk score rabbit hole
There are undoubtedly many very smart people who have been thinking long and hard about how to achieve what you’re suggesting using a method that is accurate, widely accessible, and easily affordable by healthcare systems. I hope they succeed some day.
