Should one derive risk difference from the odds ratio?

Sincere question for the statistical establishment: Do you all see what is happening here? Can someone please go on record to clarify for the public that Suhail makes absolutely no sense?

In particular, it would be helpful if anyone could corroborate the following claims, which are not easily disputed:

(1) Suhail claimed to have conclusively won the discussion against me, based on a paper in which he defined the “switch risk ratio” to be equal to the complementary risk ratio
(2) This illustrates that Suhail has never even understood what we mean when we use the term “switch risk ratio”, and that he clearly has made no attempt to understand our argument
(3) Suhail has never tried to explain what an “interpretation” is, how an interpretation can be “correct” or “incorrect”, and why this matters to choice of effect measure
(4) Suhail has made absolutely no attempt to explain why the connection between RCTs and diagnostic tests matters to choice of effect measure in RCTs

It would also be helpful if someone could clarify to the public that my arguments, while simple, are structured as proper mathematics (i.e. it is clear what my assumptions are, and how the conclusions follow from the assumptions) whereas Suhail’s paper is an unstructured mess.

I worry a lot that, due to Suhail being a professor and me being a nobody, that he is able to use his academic authority to win by default in the court of public opinion. Now, I would be the first person to recognize that in a hypothetical world where I am wrong, I am acting like a crazy person. So if he wins in the court of public opinion, most people will conclude I am crazy. It is kind of important that people who see what is happening here come to my defense.

Suhail, would you be interested in borrowing a method of conflict resolution from the judicial system, ie. some sort of mock trial, a live streamed event where we could both give presentations to a jury of statisticians, and call witnesses to corroborate our claims?

1 Like

@s_doi’s paper is a valuable contribution to the literature focusing on something that he is interested in. But it is true that the paper does not discuss @AndersHuitfeldt’s switch RR or his related papers, which is a distinct line of investigation that should be continued independently. A practical solution to clear the air may be to issue an addendum removing the term “switch RR” (used only once in the manuscript) and use only the term “complementary RR” throughout.

5 Likes

I am also a “nobody”, and I am not a member of the “statistical establishment” (I’m not even a statistician). But I recommend that you tone down the language. I don’t know @s_doi personally (or his work), but I do know that it is inappropriate and unnecessary to state that he “does not have the required intelligence to take part in this discussion”. Even if that were true, it is not the way to get people to “come to your defense”. I’m exposing a personal bias of mine, but I tend to discount people who interact that rudely. Even if, say, @s_doi was to argue that the earth is flat, I would hesitate to side with someone being so antagonistic.

3 Likes

I think its best to retain the word switch as a synonym for complementary as in any case proponents of the so called switch RR also base this on the probability of the complement of the outcome event anyway and there should be no copyright on the English word switch - which implies that outcomes are being switched aka the complementary outcome

Thank you, but I am actually in favor of the earth being round :grinning:

In that case it would have been more appropriate to cite @AndersHuitfeldt’s work and directly engage with his arguments in your manuscript.

7 Likes

This was discussed by the author group after it was raised by the reviewer and editors but then we realized that since we are suggesting that the RR (or complementary RR) should be interpreted as ratios of odds then there is nothing to engage anymore with this argument as they become invalid under the constraints we set for the interpretation of the ratio.

I am attaching here the letter that I just sent to Prof. Juan Franco, editor in chief of BMJ Evidence Based Medicine, requesting the retraction of “Likelihood ratio interpretation of the relative risk”

Dear Prof. Franco,

A few days ago, you received my resignation as an editorial board member at BMJ Evidence Based Medicine. This resignation was in response to the publication of the manuscript “Likelihood ratio interpretation of the relative risk” by Suhail Doi et al. I am writing this letter to follow up on my resignation, and to request that you appoint an external expert to evaluate whether the paper must be retracted. The following is a summary of the reasons behind this request.

Is the conventional interpretation of the relative risk in conflict with Bayes Theorem?

The “key messages” section of this paper states that the study adds the following to the literature:

⇒ It is demonstrated that the conventional interpretation of the relative risk is in conflict with Bayes’ theorem.

⇒ The interpretation of the relative risk as a likelihood ratio connecting prior (unconditional) intervention risk to outcome conditional intervention risk is required to avoid conflict with

I will refer to the first bullet point as “Doi’s Conjecture”. Doi’s Conjecture is stated in the second section of the main text, where it is claimed that “the usual interpretation (33% increase in the +ve outcome under treatment) contravenes Bayes Theorem”.

No attempt is made within the text to prove Doi’s Conjecture. But perhaps more worryingly, no attempt is made to define the term “interpretation”, a term which is also not defined in standard probability theory. The meaning of Doi’s Conjecture is therefore ambiguous. Without guidance from the text about the intended meaning of the central claim in the manuscript, I can only speculate about some possible intended meanings.

My first reading of Doi’s Conjecture is as a claim that the relative risk, which is defined as the ratio of two probabilities, cannot be interpreted as the ratio of those two probabilities. This claim would appear postmodernist in nature. It is worth noting that when considering the data in the hypothetical randomized trial shown in table 1, the risk of the outcome is indeed 33% higher among the treated than the non-treated. I am puzzled by the apparent claim that the observed results in a trial can be a violation of Bayes Theorem; it seems obvious (at least to to me) that the logical validity of Bayes’ theorem is invariant to how the investigator interprets a mathematical object. If this was commonly held to be an inconsistency, probability theory would be in deep conceptual trouble.

In the section “problem statement”, it is written ”If we accept Bayes’ theorem as the correct way to generate a predictive value or posterior probability, then it becomes clear that the RR should best be interpreted in the same way as a LR”. A second attempt to interpret Doi’s Conjecture in light of this claim, then, is as a logical implication not of Bayes Theorem, but of the use of Bayes’ Theorem to generate posterior probabilities. Again, in order to make sense of the claim, it is necessary determine (among other things) how the term “interpretation” is formalized, which is not answered within the manuscript. In this context, it is worth noting that there are multiple correct (and mathematically equivalent) ways to determine the posterior probability, using different inputs and different mathematical objects to represent the reasoning. If these methods result in different posterior probabilities given the same data, then at least one of them is incorrect. Fortunately for scientists, every valid approach results in the same conclusions, and there would appear to exist no reason for one valid approach to take precedence over another valid approach.

A final attempt to make sense of Doi’s Conjecture, this time in light of the “example” section on the manuscript, is that approaches based on the relative risk can result in invalid predictions if used to transfer effects to different populations. For example, if the treatment is associated with a 33% increase in risk in all groups, then if investigators make predictions based on an assumption that the relative risk is the same in another population, this might result in predictions outside the range of valid probabilities. This is not a new observation, and has been known for as long as statisticians have used relative risks. I refer to this phenomenon as “closure” of the effect measure The relative risk is not closed, this is uncontested, but this can only be understood as a violation of Bayes Theorem if the term “interpretation” is taken to imply stability of the effect measure across populations. In my view, if stability is the relevant consideration, then this should be tackled heads on instead of oblique references to “interpretations”.

To summarize: Doi’s Conjecture, which is the main contribution of this paper, is a vaguely stated and unproven claim. As far as I can tell, every reasonable attempt to make the conjecture more precise results in a false statement. In my view, Doi’s Conjecture is not so much “wrong” as “not even wrong”

On the Switch Relative Risk

While the larger context of the manuscript is not directly relevant to whether it should be retracted, it is worth noting that the arguments in the manuscript were previously made by Prof. Doi within a 367-post long discussion thread on the Datamethods discussion forum at http://discourse.datamethods.org, in response to my work on a mathematical object called The switch relative risk or The switch risk ratio (interchangeable terms for the same object). The switch relative risk is defined as the relative risk if treatment reduces risk, and as the survival ratio (also known as the complementary relative risk) if treatment increases risk. The switch relative risk was first described by van der Laan et al in Journal of the Royal Statistical Society: Series B (2007), and is motivated by a large body of work spanning several decades and several academic fields, including medicine, psychology, toxicology and philosophy.

The motivation behind the Doi et al paper, is in part to argue against usage of the switch relative risk in clinical medicine. This is for example illustrated by the fact that immediately after the paper was published by BMJ Evidence Based Medicine, Prof. Doi posted the following on the Datamethods forum:

In this thread there was a discussion by @AndersHuitfeldt about the utility of the switch risk ratio and I argued that this does not solve the problem. This has now been elucidated in detail in this paper and brings some closure to this issue.

Remarkably, the switch relative risk is mentioned only once in the paper:

It also explains why the RR divided by the complementary or switch RR equals the OR.

This sentence is not correct: If the RR is divided by the switch relative risk and the risk among the exposed is lower than the baseline risk, you will get 1 and not the OR. In this sentence, Doi et al is using the term “the switch RR” to refer to the complementary RR, a usage which is completely new to this paper, and which can only serve to confuse readers: He claims to have brought closure to the discussion about the switch relative risk by redefining it as the complementary relative risk! When forum participants suggested that he might correct the article by removing the mention of the switch relative risk, he defended his redefinition by noting there is no copyright on the English word “switch” and continued arguing that he had conclusively “put closure on anything that emanates from the RR or its complement”

Even in a hypothetical world where Doi is correct in his claim that he has put closure to both the RR and its complement, he makes no argument as to why this should generalize to “anything that emanates” from them. Here, it is important to note that lack of closure is the most charitable interpretation of why Prof. Doi thinks he has conclusively shown the relative risk to be inconsistent with Bayes Theorem, and that in contrast to the relative risk and complementary relative risk, the switch relative risk is closed on [0,1].

Some suggestions for BMJ Evidence Based Medicine going forward

As discussed, I have stepped down from my position on the Editorial Board of BMJ Evidence Based Medicine. But I do want to give one final piece of advice for the future of this journal: If you intend to continue publishing methodological work of a mathematical nature, this will require heightened scrutiny during the review process, particularly for manuscripts written by clinicians whose primary training was not in the mathematical sciences.

Yours sincerely,

Anders Huitfeldt MB BCh BAO (NUI), LRCS&PI, ScM, ScD

Department of Mathematics

École Polytechnique Fédérale de Lausanne

We will respond if the editor asks us to do so but I was also aiming to respond on the blog. However, after reading through several times, I am unable to find anything precise enough to respond to. Perhaps anyone on the blog can rephrase some specific questions from the letter above, if of interest, and will attempt to respond.

The only thing clear in the letter thus far is that we used switch RR as an alternate for the complementary RR and that “switch” must not be used this way and must always refer to the van der Laan usage of the term. Just to be clear, what van der Laan suggested is that the conditional RR given by
P(Y=1)|X,Z,C) ÷ P{Y(0)=1|X,Z,C}
be used for subjects with values (X,Z,C) for
which P(Y = 1|X,Z,C) ≤ P{Y(0) = 1|X,Z,C}
and be replaced by the below for all remaining subjects:
P(Y=0)|X,Z,C) ÷ P{Y(0)=0|X,Z,C}
This is just to clarify what the argument is about and perhaps then we should also request van der Laan to also add an addendum because Deeks used the word switch several times in his 2002 paper to refer to the complementary effect measure switching outcomes and therefore van der Laan had no right to use it.

However, after reading through several times, I am unable to find anything precise enough to respond to

How about responding to the claim that the conjecture presented as the main contribution of your paper, is not even wrong? If I am wrong about that, you should easily be able to tell us precisely what you think you have proved and how you think you have proved it.

perhaps then we should also request van der Laan to also add an addendum because Deeks used the word switch several times in his 2002 paper to refer to the complementary effect measure switching outcomes and therefore van der Laan had no right to use it.

Deeks never referred to the complementary risk ratio as the switch risk ratio, but as RR(H). As demonstrated by your activity in this thread, you used the term in order to “bring closure to” a discussion about the switch relative risk, in a setting where that term was unambiguously used to refer to the van der Laan effect measure. You had no need to add an extra term to refer to the complementary relative risk, which you had already discussed earlier in the paper. In my view, this is intellectual dishonesty, and I won’t engage further with your justifications.

Our paper brings some closure to the use of the complementary RR as a solution to the problems of the classic RR NOT because we use the term switch but because we demonstrate it is actually a LR. I am not sure why there is so much emphasis on a term that is not even yours - Robins discussed inference on a conditional RR in the 90’s and van der Laan expanded on this and coined this term - its not even in your purview to protest.

This is not about “ownership” of terms, I am not accusing you of “appropriating” a term that belongs to someone else. I am accusing you of trying to win a debate by stealthily and unilaterally (and with no other purpose) redefining a term that had a preexisting and unambiguous definition shared by all participants in this discussion

.A summary of this recent discussion is in order for those interested in this thread

a) Any formulation of the RR that uses the complementary RR independent of the standard RR e.g. survival ratios or switch ratios combined with standard ratios for different subjects are not solutions to the problem of the RR because both the RR and its complement with the switched outcome are likelihood ratios.
b) For a likelihood ratio to be meaningful as an effect measure, both likelihood ratios need to be used on the same subjects and then combined e.g. the ratio of likelihood ratios
c) to disprove a) and b) requires that a mathematical flaw be identified in the proof in the appendix of our paper that demonstrates the likelihood ratio interpretation of the RR
d) This discussion has been derailed by the extreme focus on the word switch that has no bearing on the results of our study - with or without this term the handful of people that suggested use of the RR with a switched outcome or with van der Laan’s suggestion are not supported. I would hope that van der Laan stumbles on this blog and could make a comment about his SRR proposal in the light of our paper.

Addendum: If anyone would like to engage in a proper discussion of this paper please PM me offline and will be happy to engage with your thoughts and move the ideas further as this will avoid this incessant tirade and unprofessionalism from one poster in this thread

Your appendix consists in its entirety of the following parts:
(1) A proof that RR(positive) equals LR(positive)
(2) A proof that RR(negative) equals LR(negative)
(3) A proof that RR(positive)/RR(negative)=OR
(4) A seemingly confused statement that if we “perceive” the RR as a ratio of risks, then we can rewrite the risk difference as R0*RR-R0. In fact, this trivial rewriting of the risk ratio is always true, regardless of how we “perceive” anything. Of note, it should be a warning signal when you think you can prove that something follows logically from your “perception” of a mathematical object, “perception” is a term which has no meaning in probability theory.
(5) A similar statement for consequences of “perceiving” the OR

It is a complete mystery to me why you think your statements a) and b) above follow from these trivial and irrelevant claims.

I would also like to note:

  • You have made no attempt to clarify what you mean by a “meaningful” effect measure, this makes it impossible to evaluate any proof of whether certain conditions are required for “meaingfulness”
  • In the switch relative risk, the same effect measure is used on all subjects. You clearly have not understood at all how this works.

I want to be clear that when I ask for retraction of your paper, this has very little to do with the switch relative risk, and everything to do with the fact that the entire paper (which barely even discusses the switch relative risk) is complete nonsense.

There is a separate discussion about your behavior on this forum; this discussion is not so much about the minor error you made in the paper by referring to the switch relative risk, it is about how you tried to lean on your irrelevant (and irredeemably flawed) paper to “win” a serious academic discussion by redefining the terms under discussion.

2 Likes

Suhail, do you agree that the key message of your paper is “The conventional interpretation of the relative risk is in conflict with Bayes Theorem”, and therefore, if this sentence is not true, the paper should be retracted?

Having read the paper, I think my main concern would be its effect on the young approaching this material without guidance — since apparently the editors of BMJ EBM have relinquished all responsibility in that regard. This quotation from Schopenhauer comes to mind:

How could minds strained and ruined in the freshness of youth by the nonsense of Hegelianism still be capable of following Kant’s profound investigations? They are early accustomed to regard the hollowest of verbiage as philosophical thoughts, the most miserable sophisms as sagacity, and silly craziness as dialectic; and by accepting frantic word-combinations in which the mind torments and exhausts itself in vain to conceive something, their heads are disorganized.

— Arthur Schopenhauer, Preface to Second Edition of ‘The World as Will and Representation’ (E.F.J. Payne, tr.)

2 Likes

I don’t believe the editors have much responsibility for the type of “young” that this refers to and the lead-in to what you quoted is more illustrative:

> It has never done more than peruse him hastily and impatiently, or listen to an account at second hand; and this again is due to its having in consequence of bad guidance , wasted its time on the philosophemes of ordinary, and hence officious and intrusive, heads, or even of bombastic sophists, which have been irresponsibly commended to it. Hence the confusion in the first conceptions, and generally the unspeakable crudity and clumsiness that appear from under the cloak of affectation and pretentiousness in the philosophical attempts of the generation thus brought up. But the man who imagines he can become acquainted with Kant’s philosophy from the descriptions of others, labors under a terrible mistake. On the contrary, I must utter a serious warning against accounts of this kind, expecially those of recent times. In fact in the most recent years in the writings of the Kantian philosophy which really reach the incredible.

I would have had @Sander_Greenland more in mind as the fitting analogue to Kant. :wink:

I would agree with you

But I also need to write a disclaimer - Schopenhauer bitterly denounced G. W. F. Hegel as a “charlatan” (hence Hegelianism above). I have no interest in agreeing with Schopenhauer about any such thoughts about anyone - everyone brings their thoughts of any type and eventually what is true emerges…

I am less interested in philosophical analogies and more interested in having a real discussion about the merits of the methodological disagreement. In order to try to get more statisticians on record with their views, I am going to state my position once more, hopefully in a clearer and more concise language than my previous post:

Doi et al have shown that it is possible to interpret the relative risk as a likelihood ratio. This is not contested, but it also a very trivial and unsurprising observation. Additionally, they have claimed to show that every other interpretation of the relative risk is incorrect (“in violation of Bayes’ theorem”). This claim is unsubstantiated and incoherent, and must be expunged from the scientific record, either via a correction or a retraction.

Is it possible that a correction might be sufficient in place of a retraction? In my view, if they changed the text to argue only that “the relative risk can be interpreted as a likelihood ratio” (and not that other interpretations are invalid), it would not be incorrect, but this would be much too trivial to warrant publication in a medical journal. Such a correction would be a pure face saving move, and leave future readers puzzled over why a medical journal thought this observation was of sufficient novelty or interest to merit a scientific paper.

I guess it is theoretically possible that one can show a clinical use for the likelihood interpretation of the relative risk. However, I want to state emphatically that Doi et al did not do that. Nobody has suggested a clinically useful decision making procedure where we use the outcome as a diagnostic test for whether the patient was treated. If Doi is able to propose a clinical situation where this is the appropriate decision making procedure, I would not object to them correcting the paper to make this argument rather than retracting it (assuming they also removed any claim that other interpretations are incorrect or in violation of basic probability theory). I do however have significant doubts over whether it will be possible to do this in a non-contrived way.

In my view, the whole situation is analogous to a hypothetical scenario where a medical journal somehow published a paper claiming, as its key message, that “π can only be interpreted as 180° converted to a radian scale, interpreting π as the ratio of the circumference to the diameter is in violation of the parallel postulate”.