Just to notify that our correspondence on the recent meta-analysis of statin-related adverse effects has now been published in The Lancet.
The authors’ reply leaves several key methodological issues unresolved:
First, their reply too readily dismisses the issue of the relevant loss function in safety assessment; specifically, it does not adequately address the consequences of failing to detect real effects. This is particularly problematic when wide interval estimates that include the null remain compatible with clinically important increases in risk, yet are treated as reassuring because they are not “statistically significant” - especially under a strategy that favors false negatives.
Second, the reply also dismisses the problem of dilution in intention-to-treat estimates by invoking the protection afforded by randomization. This does not really address the critical point: in safety analyses, what matters is not only the effect of treatment assignment but also the effect of actual exposure to the treatment. Under non-adherence, intention-to-treat estimates can dilute risk contrasts relative to per-protocol effects.
Third, the authors formally acknowledge that absence of evidence is not evidence of absence, yet their final recommendation appears to move in the opposite direction. They conclude that “labelling and other official sources of health information should be revised” in light of their findings.
This concern is not hypothetical. The public communication surrounding the study has already moved well beyond careful wording. The University of Oxford itself published the headline: “Statins do not cause the majority of side effects listed in package leaflets.”
I apologize for the two consecutive posts, but I think it is relevant to share the open letter (attached) - currently collecting signatures - that we wrote to the University of Oxford. Indeed, since both sides agree that “absence of evidence” is not equivalent to “evidence of absence”, we believe that Oxford should publish a correction.
I think this point is central to DataMethods, as it concerns the communication of the outcomes of applying specific methodological criteria, and therefore of the possibilities attributed to those criteria.
Asking the Oxford authors to word their findings as follows might be a reasonable “middle ground” that would not risk aggravating a serious public health problem (statin denialism):
“Many of the AEs currently listed in statin product monographs were added to labelling many years ago because of a reported temporal relationship to statin use in individual patients in the postmarket setting. Rigorous evidence for a causal role of the statin in individual cases was not required to include these AEs in the label. Rather, many AEs were added as a precautionary measure, because a causal role for the drug “could not be excluded.” But we have now accrued high numbers of reports of many of these AEs in the context of decades of statin clinical trials. Trial-level evidence allows us to see that many of the AEs reported over the years in the postmarket setting (outside the context of trials) have occurred with similar frequency in statin-treated and placebo-treated trial subjects. We can draw the following conclusion from this observation: IF statins are capable of causing these AEs, they would only do so at rates so low that the signal has been undetectable after decades of clinical trials. This fact suggests that that the risk for the labelled AEs in question would have to be so low that it would not be expected to influence clinical decision-making. AEs that occur at very low rates are often those that occur unpredictably in idiosyncratically-vulnerable patients, for reasons we don’t understand- as such, these AEs can usually not be avoided through pre-treatment patient screening. If no safety signal for an AE has been observed after many accrued AEs of this type in statin clinical trials, we can conclude that IF statins are capable of causing this AE (perhaps in idiosyncratically-vulnerable patients), the risk is likely to be lower than 1 in …”
First, the Oxford headline does not merely refer to statins in general: it presents the findings of a study conducted by Oxford researchers in a way that appears to turn trial-level non-detection of excess adverse events into causal evidence of absence of such effects. That is the core problem, especially considering the strong bias towards false negatives.
This is not a minor wording issue, as it legitimizes a pseudo-methodological move (in pharmacovigilance) that has no sound basis and is inconsistent with decades of warnings in epidemiology and statistics: absence of evidence is notevidence of absence. Importantly, we are not imposing an external standard: the authors themselves have acknowledged this distinction.
Such a methodological and infodemiological problem has consequences as serious as those associated with statin denialism. It may even be more insidious, because it often goes unnoticed given that the error is technically subtle, accessible to relatively few people, and at the same time it is amplified and given credibility by a major academic institution such as Oxford (of course, Oxford is not the only institution where this type of problem has occurred). Again, the literature I have already mentioned in this thread reports striking examples of these phenomena.
The targeted failure is independent of the specific topic - statins, vaccines, cancer drugs, or any other medical intervention. Indeed, we are talking about a general principle of evidence communication. Institutions should not disseminate this kind of severe misinformation, especially when it concerns causal interpretation of safety evidence.
A second issue is why such miscommunication occurred. There are several competing or alternative explanations, and these should be considered analytically rather than dismissed as speculation. One possibility is simple incompetence: that Oxford researchers and/or communication officers failed to recognize the difference between “absence of evidence” and “evidence of absence.” Another possibility is that ordinary academic dynamics contributed to the overstatement of the findings: the incentive to present results in stronger, more visible, and more prestigious terms. A further possibility is that the long list of declared conflicts of interest that can have influenced, directly or indirectly, the framing and communication of the results. These are not “conspiracy theories”, but explanatory hypotheses for the observed “data”.
As for the specific discussion about statins, the “middle ground” solution seems to ignore what has already been repeatedly discussed in previous comments: again, the proposed narrative is only one of several possible narratives compatible with the observed data. Its persuasive force is substantially reduced once we consider well-known sources of bias, including the influence of pharmaceutical companies, selective framing of trial evidence, under-ascertainment or under-reporting of adverse events, and the persistent myth that randomized trials are the only legitimate source of causal evidence.
None of this means that statins lack important cardiovascular benefits when they are properly evaluated and appropriately prescribed - and, in some respects “paradoxically”, I reach this conclusion only after integrating information derived from the direct (observational) experience of the clinicians who prescribe them. The crucial question is that, in light of a highly controversial landscape that continues to rely on ritualistic practices and to perpetuate striking errors such as the one shown here, the scientific community should make a collective effort both to estimate and to limit the impact of these biases on the production of medical evidence.
If methodologists themselves fail to do this because they are misled by formal plausibility - not unlike the plausibility generated by a large language model - or because they are influenced by ideological, political, social, or economic pressures, then I do not see who else can realistically do it. And that would cause serious, unavoidable, reputational damage to medical research. What I (have to) expect, as a researcher, is not to be reassured with a convenient narrative; I expect serious countermeasures to prevent similar failures from continuing to occur.
In this respect, when explanations are produced for the public, we should stop attributing responsibility for claims to abstract formal entities such as “the evidence” or “the studies.” Evidence and studies acquire meaning only through human interpretation, validation, and judgment. Therefore, if Oxford wishes to communicate health recommendations to the public, it should state more explicitly that its analysts judge the underlying uncertainty to be sufficiently low to support those recommendations. At the same time, it should engage openly with those raising substantive methodological concerns, rather than leaving those concerns unanswered.
This is a lot of words. I was with you for the first two paragraphs. But then I just started to get angry. Your writing betrays a strong ideological overtone that goes way beyond the statistical point you’re trying to make.
“Such a methodological and infodemiological problem has consequences as serious as those associated with statin denialism.”
No, in this particular case, it doesn’t- not even remotely close. Are you a clinician who has to grapple, day in and day out, with the consequences of statin denialism for real patients with lives on the line?
“…or because they are influenced by ideological, political, social, or economic pressures…”
You are implicitly accusing clinicians of somehow profiting from the promotion/sale of generic medications that have been on the market for decades and cost patients pennies per day. You’re implying that being forced to “admit” that these drugs might have risks would somehow cut into those profits or force us to acknowledge some embarrassing “truth” about statins that could have wide-reaching clinical consequences. Hogwash. How, exactly, in this day and age, do you propose that clinicians benefit from prescribing statins to their patients?..The answer is that we don’t.
There’s a trope - perpetuated for many years by drug safety researchers whose egos have been bruised by run-ins with defensive pharmaceutical companies, either in print or in court- that paints physicians as either stupid or corrupt or both. The (extremely insulting) insinuation is that we are zombie-like prescribing machines who are incapable of independently assessing the strength of the evidence behind our practices. As though medicine isn’t actually a profession, like any other. As though we don’t care enough to ensure that what we do every day at work makes sense. As though statins are some type of sacred cow that we “ritualistically” defend, with glazed eyes, from any criticism, no matter how well-founded. Newsflash- none of these things are true. Believe it or not, we actually just like keeping our patients alive as long as possible and out of congestive heart failure. There’s no other medication class on the market (aside, perhaps, from contraceptive pills) that has been studied as widely as this one, with benefits demonstrated over and over…and over again. These pills save lives- full stop. Are they perfect? No. Can they have side effects, like every other medication? Yes. Should doctors talk about them? Yes.
Do I think the Oxford team should have said that statins “don’t cause” certain AEs? The answer is “no”- statistically speaking, it’s impossible to “prove” a complete absence of risk. I happen to agree with you that this was a poor choice of wording. And I don’t fault you for calling them out on it. In my opinion, the wording I suggested above would have been more accurate and appropriate and would have put reasonable bounds on the residual uncertainty around statin-related AEs, after all these years. But if you are suggesting (as you seem to be) that there are multiple clinically important risks for this very old class of drugs and that the Oxford researchers are actively trying to bury them with their analysis (in order to “save face” after promoting them heavily or because they are somehow in bed with pharmaceutical companies), any credibility you might have been afforded for your “absence of evidence” point is going to go out the window.
Could your group’s crusade to make your statistical point cost lives given our current clinical climate? Maybe. So I’d suggest that you think very hard about the tone and wording of your rebuttals. In spite of your protests to the contrary, I detect a very strong “anti-pharma”/conspiratorial overtone in your response. I’m not sure how you acquired it, but it’s something you should probably curb - at least when discussing an issue of such massive public health importance.
You have completely misunderstood and misread what I wrote. Paradoxically, I argued precisely the opposite of what you suggest, as I was referring to methodologists who can be overly influenced by various pressures, while explicitly stating my greater confidence in the judgments made by clinicians in their day-to-day practice.
See my comments
If methodologists themselves fail to do this because they are misled by formal plausibility - not unlike the plausibility generated by a large language model - or because they are influenced by ideological, political, social, or economic pressures, then I do not see who else can realistically do it.
and
None of this means that statins lack important cardiovascular benefits when they are properly evaluated and appropriately prescribed - and, in some respects “paradoxically”, I reach this conclusion only after integrating information derived from the direct (observational) experience of the clinicians who prescribe them.
I can only note that you continue to disregard the literature I have provided, which also highlights the cognitive limitations that affect all of us. These limitations can only be overcome through collaboration, since none of us comes even remotely close to possessing all the expertise required to fully evaluate the evidence in its entirety.
In light of this, I reject all the “insinuations” you suggest I have made. At the same time, I cannot meaningfully respond to what are mere personal opinions, especially when they extend beyond the substance of the discussion and into judgments about my character or motives.
Unfortunately, this is not the first time - nor will it be the last - that a call for methodological rigor is dismissed as “conspiracy thinking” whenever the topic is sensitive. Yet methodological rigor necessarily requires consideration of the full range of hypotheses that may explain the observed data, including those that may be uncomfortable or inconvenient.
Contrary to what you suggest, I also consider the possibility that such attitudes arise from a sincere and genuine concern for patient health. Nevertheless, my assessment that this approach is mistaken remains unchanged whenever that concern translates into rigidity rather than openness to dialogue with individuals who bring different expertise and perspectives to the discussion.
This view is informed not only by the literature I study but also by the information and insights I have gained through my daily collaborations with clinicians and physicians. I do not feel compelled to document those experiences, however, because I do not perceive an impartial interest in evaluating them on their merits.
Apologies, but I’m going to disengage at this point, Alessandro. You’re telling me that I’m completely misunderstanding what you’re saying, but then the next thing you say reinforces my initial impression. My reading comprehension must be terrible. It appears that we’re just not communicating effectively with each other- probably best to stop now. That’s okay. Thanks for the dialogue.
I think at this point it would be most helpful for Alessandro to re-draft a statement that is minimalistic, sticking to just the most directly related facts in the case. A possible statistical addition that would be very worthwhile would be simple Bayesian calculations of the probability of clinically relevant safety signals, e.g., P(absolute risk increase > 0.01).
I don’t have a dog in this fight given that I have found the evidence for primary prevention compelling but remain concerned about paternalistic minimization of risk promulgation.
However my advice for you as a zealous young research warrior for transparency is that you should learn now to expect that you WILL be “reassured with a convenient narrative”. Further you should expect to be attacked if you deviate from it and you should let that hot water roll off your ice cold back.
Not even definitive mathematical proof is effective against a convenient narrative, methodological or otherwise. Furthermore; and you can quote me on this;
“The science of trial methodology provides no refuge from unyielding dogmatism. Indeed, there is no refuge in science from that.”
@Alessandro_Rovetta I have to agree with @llynn that while you may have a point here that is important, you have to let go of the hot water and stick to the methodology otherwise it will get drowned out
Thank you for these words, which I will take to heart as a person before I do as a researcher. I also thank @s_doi for the advice that followed.
Precisely on the basis of what both of you have suggested, I would like to emphasize that the COIs section - just like the limitations section - should not be treated as an academic formality, since it was designed to inform readers about competing hypotheses that can contribute to explaining the observed results.
If this part is simply excluded from a “methodological discussion”, then such a discussion becomes idealized - as if it referred to an ideal world free from human influences.
Bayesian calculations of the probability of clinically relevant safety signals, e.g., P(absolute risk increase > 0.01)
That’s exactly what’s needed, IMO.eOf course it’s hard to define a"clinically relevant safety signals", or a risk-benefit ratio, when both risks and benefits are rather rare.
There’s no doubt that the need to specify a prior is the main reason why some people continue to oppose Bayesian approaches. Bayes is clearly right, and the problem of priors seems to me to show us that doing good science is harder problem than most people, including many statisticians, thought until recently.
Luckily, the problem of priors has several possible solutions. For example: 1,use reverse Bayes, 2, do the calculations with a skeptical prior (e,g, half of the prior probability on the null) and an ‘optimistic prior". 3. Use likelihoods or Bayes’ factors.
It may be possible to estimate the distribution of a “universe” of safety effects from analyzing hundreds of safety outcomes on >100,000 patients if we can assume some kind of comparability to apply to the statin population. This universe could help in defining a prior.
All of us are concerned that “statin fear” will cause increase in CV mortality, but statin myopathy should not be discounted for paternalistic reasons. The statin wars are long over and statins won.
It’s time to take a hard look at the myopathy risk. A patient can suffer for years due to insidious statin myopathy, with thigh weakness and pain on rising from a chair, knelling or squatting that they thought was just due to aging.