Analogy between clinical trials and diagnostic tests: using likelihood ratios to interpret negative trials

[R_cubed]

@Robert_Matthews has already published a better method of doing this from the Bayesian perspective using the reported confidence intervals. Matthew’s Analysis of credibility procedure can give a wide range of plausible values for an effect that could justify continued research into a topic, in spite of a failure to achieve “significance” in a set of studies.

Frequentist vs Bayesian debate applied to real-world data analysis: any philosophical insights?

Robert Matthews did all of the creative work in deriving the algebraic expression. He has described them in this paper, as well as the one I linked to above. Both are worth reading. Further work in this area has been done by Leonhard Held. Conventional reports dichotomize results into “significant” and “not significant”. Given only the CI, plug in the limits into this formula to derive the “advocacy bound” when given a “not significant” report. For ratios, the advocacy limit (AL) is: AL…

From the original link:

Among 169 statistically nonsignificant primary outcome results of randomized trials published in 2021, the hypotheses of lack of effect (null hypothesis) and of clinically meaningful effectiveness (alternate hypothesis) were compared using a likelihood ratio to quantify the strength of support the observed trial findings provide for one hypothesis vs the other; about half (52.1%) yielded a likelihood ratio of more than 100 for the null hypothesis of lack of effect vs the alternate.

The premise of the article is attempting to defend a logical fallacy with empirical data.
They just picked 2 arbitrary points on a likelihood function (each that have precisely 0 probability of being true from a strict axiomatic perspective) to defend the idea that nonsignificant studies provide evidence in favor of the null. I can’t access the paper because of the paywall, but the use of the term “likelihood ratio” vs. “likelihood function” makes me suspicious this is what was done.

It is flawed because of:

1. Coming to conclusive results based on single studies directly contradicts the idea that meta-analysis can magnify power and precision of any test or estimate. Any experiment is a simultaneous test of the experimental procedure in addition to the scientific question. Neyman denied the possibility of drawing firm conclusions in individual cases, and interpreted outputs as decisions for behavior only. AFAIK, even Fisher insisted on replication of experiments that yielded positive tests of significance.

2. Their meta-analysis is flawed for all of the reasons @Sander alluded to in that giga-thread on odds ratios. Using empirical data to justify a statistical procedure is at best, exploratory.

Should one derive risk difference from the odds ratio?

Doi: Why do you keep repeating that the RR is not portable? We agree so that was never an issue, so why bring it up? We also agree that, unlike the RR and RD, the OR is not mathematically constrained by baseline risks, which is a trivial fact known many decades ago, so why bring that up? We also agree that, because of that fact, when you looked over a vast range of studies crudely with no attempt to stratify them by topic, you saw little association of the OR with baseline risks, especially comp…

Key quote:

Without some extraordinary considerations, we do not transport estimates for the effect of antihistamines on hives to project effects of chemotherapies on cancers, nor do we combine these disparate effects in meta-analyses.

LIkewise, we don’t transport likelihoods from heterogeneous studies either.

Related Reading
https://www.sciencedirect.com/science/article/abs/pii/S1047279712000221

The discussion and some of the links are worth follow up.

Most reasonably hypothesised effects cannot be exactly zero?

It is easy for those of us who have never had to grapple with the logical issues discussed in a mathematical analysis class, to get totally confused on this point. I am fortunate enough to have mentors with Ph.D level mathematics education to help me get over the hurdle. These 2 papers also helped me understand the issues much better. Abstractly, the cardinality of the reals is so vast, that the probability of picking any particular real number is zero. That also makes it difficult to…

Here is an interesting discussion on Andrew Gelman’s blog on 4 interpretations of p-values that is related to this paper. @Sander_Greenland has a number of valuable comments in this thread as well.