I am attaching here the letter that I just sent to Prof. Juan Franco, editor in chief of BMJ Evidence Based Medicine, requesting the retraction of “Likelihood ratio interpretation of the relative risk”
Dear Prof. Franco,
A few days ago, you received my resignation as an editorial board member at BMJ Evidence Based Medicine. This resignation was in response to the publication of the manuscript “Likelihood ratio interpretation of the relative risk” by Suhail Doi et al. I am writing this letter to follow up on my resignation, and to request that you appoint an external expert to evaluate whether the paper must be retracted. The following is a summary of the reasons behind this request.
Is the conventional interpretation of the relative risk in conflict with Bayes Theorem?
The “key messages” section of this paper states that the study adds the following to the literature:
⇒ It is demonstrated that the conventional interpretation of the relative risk is in conflict with Bayes’ theorem.
⇒ The interpretation of the relative risk as a likelihood ratio connecting prior (unconditional) intervention risk to outcome conditional intervention risk is required to avoid conflict with
I will refer to the first bullet point as “Doi’s Conjecture”. Doi’s Conjecture is stated in the second section of the main text, where it is claimed that “the usual interpretation (33% increase in the +ve outcome under treatment) contravenes Bayes Theorem”.
No attempt is made within the text to prove Doi’s Conjecture. But perhaps more worryingly, no attempt is made to define the term “interpretation”, a term which is also not defined in standard probability theory. The meaning of Doi’s Conjecture is therefore ambiguous. Without guidance from the text about the intended meaning of the central claim in the manuscript, I can only speculate about some possible intended meanings.
My first reading of Doi’s Conjecture is as a claim that the relative risk, which is defined as the ratio of two probabilities, cannot be interpreted as the ratio of those two probabilities. This claim would appear postmodernist in nature. It is worth noting that when considering the data in the hypothetical randomized trial shown in table 1, the risk of the outcome is indeed 33% higher among the treated than the non-treated. I am puzzled by the apparent claim that the observed results in a trial can be a violation of Bayes Theorem; it seems obvious (at least to to me) that the logical validity of Bayes’ theorem is invariant to how the investigator interprets a mathematical object. If this was commonly held to be an inconsistency, probability theory would be in deep conceptual trouble.
In the section “problem statement”, it is written ”If we accept Bayes’ theorem as the correct way to generate a predictive value or posterior probability, then it becomes clear that the RR should best be interpreted in the same way as a LR”. A second attempt to interpret Doi’s Conjecture in light of this claim, then, is as a logical implication not of Bayes Theorem, but of the use of Bayes’ Theorem to generate posterior probabilities. Again, in order to make sense of the claim, it is necessary determine (among other things) how the term “interpretation” is formalized, which is not answered within the manuscript. In this context, it is worth noting that there are multiple correct (and mathematically equivalent) ways to determine the posterior probability, using different inputs and different mathematical objects to represent the reasoning. If these methods result in different posterior probabilities given the same data, then at least one of them is incorrect. Fortunately for scientists, every valid approach results in the same conclusions, and there would appear to exist no reason for one valid approach to take precedence over another valid approach.
A final attempt to make sense of Doi’s Conjecture, this time in light of the “example” section on the manuscript, is that approaches based on the relative risk can result in invalid predictions if used to transfer effects to different populations. For example, if the treatment is associated with a 33% increase in risk in all groups, then if investigators make predictions based on an assumption that the relative risk is the same in another population, this might result in predictions outside the range of valid probabilities. This is not a new observation, and has been known for as long as statisticians have used relative risks. I refer to this phenomenon as “closure” of the effect measure The relative risk is not closed, this is uncontested, but this can only be understood as a violation of Bayes Theorem if the term “interpretation” is taken to imply stability of the effect measure across populations. In my view, if stability is the relevant consideration, then this should be tackled heads on instead of oblique references to “interpretations”.
To summarize: Doi’s Conjecture, which is the main contribution of this paper, is a vaguely stated and unproven claim. As far as I can tell, every reasonable attempt to make the conjecture more precise results in a false statement. In my view, Doi’s Conjecture is not so much “wrong” as “not even wrong”
On the Switch Relative Risk
While the larger context of the manuscript is not directly relevant to whether it should be retracted, it is worth noting that the arguments in the manuscript were previously made by Prof. Doi within a 367-post long discussion thread on the Datamethods discussion forum at http://discourse.datamethods.org, in response to my work on a mathematical object called The switch relative risk or The switch risk ratio (interchangeable terms for the same object). The switch relative risk is defined as the relative risk if treatment reduces risk, and as the survival ratio (also known as the complementary relative risk) if treatment increases risk. The switch relative risk was first described by van der Laan et al in Journal of the Royal Statistical Society: Series B (2007), and is motivated by a large body of work spanning several decades and several academic fields, including medicine, psychology, toxicology and philosophy.
The motivation behind the Doi et al paper, is in part to argue against usage of the switch relative risk in clinical medicine. This is for example illustrated by the fact that immediately after the paper was published by BMJ Evidence Based Medicine, Prof. Doi posted the following on the Datamethods forum:
In this thread there was a discussion by @AndersHuitfeldt about the utility of the switch risk ratio and I argued that this does not solve the problem. This has now been elucidated in detail in this paper and brings some closure to this issue.
Remarkably, the switch relative risk is mentioned only once in the paper:
It also explains why the RR divided by the complementary or switch RR equals the OR.
This sentence is not correct: If the RR is divided by the switch relative risk and the risk among the exposed is lower than the baseline risk, you will get 1 and not the OR. In this sentence, Doi et al is using the term “the switch RR” to refer to the complementary RR, a usage which is completely new to this paper, and which can only serve to confuse readers: He claims to have brought closure to the discussion about the switch relative risk by redefining it as the complementary relative risk! When forum participants suggested that he might correct the article by removing the mention of the switch relative risk, he defended his redefinition by noting there is no copyright on the English word “switch” and continued arguing that he had conclusively “put closure on anything that emanates from the RR or its complement”
Even in a hypothetical world where Doi is correct in his claim that he has put closure to both the RR and its complement, he makes no argument as to why this should generalize to “anything that emanates” from them. Here, it is important to note that lack of closure is the most charitable interpretation of why Prof. Doi thinks he has conclusively shown the relative risk to be inconsistent with Bayes Theorem, and that in contrast to the relative risk and complementary relative risk, the switch relative risk is closed on [0,1].
Some suggestions for BMJ Evidence Based Medicine going forward
As discussed, I have stepped down from my position on the Editorial Board of BMJ Evidence Based Medicine. But I do want to give one final piece of advice for the future of this journal: If you intend to continue publishing methodological work of a mathematical nature, this will require heightened scrutiny during the review process, particularly for manuscripts written by clinicians whose primary training was not in the mathematical sciences.
Yours sincerely,
Anders Huitfeldt MB BCh BAO (NUI), LRCS&PI, ScM, ScD
Department of Mathematics
École Polytechnique Fédérale de Lausanne
