# Analogy between clinical trials and diagnostic tests: using likelihood ratios to interpret negative trials

**URL:** <https://discourse.datamethods.org/t/analogy-between-clinical-trials-and-diagnostic-tests-using-likelihood-ratios-to-interpret-negative-trials/6888>\
**Category:** general\
**Created:** [June 21, 2023, 10:23am UTC](https://discourse.datamethods.org/t/analogy-between-clinical-trials-and-diagnostic-tests-using-likelihood-ratios-to-interpret-negative-trials/6888 "2023-06-21T10:23:41Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Elias\_Eythorsson](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/elias_eythorsson/32/2524_2.png) [@Elias\_Eythorsson](https://discourse.datamethods.org/u/Elias_Eythorsson)\
**Post date:** [June 21, 2023, 10:23am UTC](https://discourse.datamethods.org/t/analogy-between-clinical-trials-and-diagnostic-tests-using-likelihood-ratios-to-interpret-negative-trials/6888/1 "2023-06-21T10:23:42Z")

</div>

I am intrigued by a recent publication in JAMA titled [Evidence of Lack of Treatment Efficacy Derived From Statistically Nonsignificant Results of Randomized Clinical Trials](https://jamanetwork.com/journals/jama/fullarticle/2806151) that proposes interpreting negative trials using likelihood ratios, finding that many negative trials provide substantial evidence for the null hypothesis of no treatment effect. Some of the nice properties of using this approach are summarized in an editorial by Roger J. Lewis titled [Revisiting the Analogy Between Clinical Trials and Diagnostic Tests by Interpreting a Negative Trial as a Negative Test for Efficacy](https://jamanetwork.com/journals/jama/fullarticle/2806175?utm_source=twitter) . As (primarily) a clinician, this approach seems very intuitive and relatively easy to understand. I would love to participate in a discussion regarding the merits of this approach and specifically, its relationship to P-value functions and S-values.

---

<div class="post-metadata">

**Author:** ![R\_cubed](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/r_cubed/32/1518_2.png) [@R\_cubed](https://discourse.datamethods.org/u/R_cubed)\
**Post date:** [June 21, 2023, 11:33am UTC](https://discourse.datamethods.org/t/analogy-between-clinical-trials-and-diagnostic-tests-using-likelihood-ratios-to-interpret-negative-trials/6888/2 "2023-06-21T11:33:11Z")

</div>

@Robert_Matthews has already published a better method of doing this from the Bayesian perspective using the reported confidence intervals. Matthew’s Analysis of credibility procedure can give a wide range of plausible values for an effect that could justify continued research into a topic, in spite of a failure to achieve “significance” in a set of studies.

> [@Frequentist vs Bayesian debate applied to real-world data analysis: any philosophical insights?](https://discourse.datamethods.org/t/frequentist-vs-bayesian-debate-applied-to-real-world-data-analysis-any-philosophical-insights/1695/9):
>
> Robert Matthews did all of the creative work in deriving the algebraic expression. He has described them in this paper, as well as the one I linked to above. Both are worth reading. Further work in this area has been done by Leonhard Held. Conventional reports dichotomize results into “significant” and “not significant”. Given only the CI, plug in the limits into this formula to derive the “advocacy bound” when given a “not significant” report. For ratios, the advocacy limit (AL) is: AL…

From the original link:

> Among 169 statistically nonsignificant primary outcome results of randomized trials published in 2021, **the hypotheses of lack of effect (null hypothesis) and of clinically meaningful effectiveness (alternate hypothesis) were compared using a likelihood ratio to quantify the strength of support the observed trial findings provide for one hypothesis vs the other**; about half (52.1%) yielded a likelihood ratio of more than 100 for the null hypothesis of lack of effect vs the alternate.

The premise of the article is attempting to defend a logical fallacy with empirical data.  
They just picked 2 arbitrary points on a likelihood **function** (each that have precisely 0 probability of being true from a strict axiomatic perspective) to defend the idea that nonsignificant studies provide evidence **in favor** of the null. I can’t access the paper because of the paywall, but the use of the term “likelihood ratio” vs. “likelihood function” makes me suspicious this is what was done.

It is flawed because of:

1. Coming to conclusive results based on single studies directly contradicts the idea that meta-analysis can magnify power and precision of any test or estimate. Any experiment is a simultaneous test of the experimental procedure in addition to the scientific question. Neyman denied the possibility of drawing firm conclusions in individual cases, and interpreted outputs as decisions for behavior only. AFAIK, even Fisher insisted on replication of experiments that yielded positive tests of significance.

2. Their meta-analysis is flawed for all of the reasons @Sander alluded to in that giga-thread on odds ratios. Using empirical data to justify a statistical procedure is at best, exploratory.

> [@Should one derive risk difference from the odds ratio?](https://discourse.datamethods.org/t/should-one-derive-risk-difference-from-the-odds-ratio/4403/25):
>
> Doi: Why do you keep repeating that the RR is not portable? We agree so that was never an issue, so why bring it up? We also agree that, unlike the RR and RD, the OR is not mathematically constrained by baseline risks, which is a trivial fact known many decades ago, so why bring that up? We also agree that, because of that fact, when you looked over a vast range of studies crudely with no attempt to stratify them by topic, you saw little association of the OR with baseline risks, especially comp…

Key quote:

> Without some extraordinary considerations, we do not transport estimates for the effect of antihistamines on hives to project effects of chemotherapies on cancers, nor do we combine these disparate effects in meta-analyses.

LIkewise, we don’t transport likelihoods from heterogeneous studies either.

Related Reading  
[https://www.sciencedirect.com/science/article/abs/pii/S1047279712000221](https://www.sciencedirect.com/science/article/abs/pii/S1047279712000221)

The discussion and some of the links are worth follow up.

> [@Most reasonably hypothesised effects cannot be exactly zero?](https://discourse.datamethods.org/t/most-reasonably-hypothesised-effects-cannot-be-exactly-zero/1041/9):
>
> It is easy for those of us who have never had to grapple with the logical issues discussed in a mathematical analysis class, to get totally confused on this point. I am fortunate enough to have mentors with Ph.D level mathematics education to help me get over the hurdle. These 2 papers also helped me understand the issues much better. Abstractly, the cardinality of the reals is so vast, that the probability of picking any particular real number is zero. That also makes it difficult to…

🆕 Here is an interesting discussion on Andrew Gelman’s blog [on 4 interpretations of p-values](https://statmodeling.stat.columbia.edu/2023/04/14/4-different-meanings-of-p-value-and-how-my-thinking-has-changed/) that is related to this paper. @Sander_Greenland has a number of valuable comments in this thread as well.

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [June 21, 2023, 12:29pm UTC](https://discourse.datamethods.org/t/analogy-between-clinical-trials-and-diagnostic-tests-using-likelihood-ratios-to-interpret-negative-trials/6888/3 "2023-06-21T12:29:51Z")

</div>

The JAMA article has a flaw. To bring evidence for similarity we need a similarity zone for the treatment effect from each study. The authors took the effect size used in the power calculation as the similarity boundary. This is not correct in general; the similarity boundary needs to be smaller than the powered effect size. And the Bayesian approximation to all this works so much better than likelihood ratios.

There is also a key problem in using the two-discrete-point likelihood ratio. For a choice of two effect sizes, such likelihood ratios quantify the relative support for one effect size vs. the other. One effect size may appear to be much more likely, but it could be way off; it’s just better than the other point which is even farther off. In other words one can have false confidence in the more likely point (actually this is the point that makes the **data** more likely) when using a relative method such as likelihood ratios. The Bayesian approach does not have this problem and provides much more intuitive results. The paper should have IMHO used a simple Bayesian approximation throughout. Note that I was a reviewer on the paper and I detailed all these issues in my review.

---

<div class="post-metadata">

**Author:** ![Elias\_Eythorsson](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/elias_eythorsson/32/2524_2.png) [@Elias\_Eythorsson](https://discourse.datamethods.org/u/Elias_Eythorsson)\
**Post date:** [June 21, 2023, 1:04pm UTC](https://discourse.datamethods.org/t/analogy-between-clinical-trials-and-diagnostic-tests-using-likelihood-ratios-to-interpret-negative-trials/6888/4 "2023-06-21T13:04:22Z")

</div>

Presumably, the likelihood function for any alternative hypothesis can be computed and statements made such as “The likelihood ratio supports the null hypothesis of no treatment effect compared to a range of plausible smallest meaningful clinical effect sizes”?

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [June 21, 2023, 1:07pm UTC](https://discourse.datamethods.org/t/analogy-between-clinical-trials-and-diagnostic-tests-using-likelihood-ratios-to-interpret-negative-trials/6888/5 "2023-06-21T13:07:49Z")

</div>

Yes, the key idea being good in selecting a threshold for non-trivial clinical effects. Buy why use a method that is less intuitive and actionable than Bayes?

---

<div class="post-metadata">

**Author:** ![R\_cubed](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/r_cubed/32/1518_2.png) [@R\_cubed](https://discourse.datamethods.org/u/R_cubed)\
**Post date:** [June 21, 2023, 2:00pm UTC](https://discourse.datamethods.org/t/analogy-between-clinical-trials-and-diagnostic-tests-using-likelihood-ratios-to-interpret-negative-trials/6888/6 "2023-06-21T14:00:02Z")

</div>

> [@f2harrell](#):
>
> The JAMA article has a flaw. To bring evidence for similarity we need a similarity zone for the treatment effect from each study. The authors took the effect size used in the power calculation as the similarity boundary. This is not correct in general; the similarity boundary needs to be smaller than the powered effect size. And the Bayesian approximation to all this works so much better than likelihood ratios.

This subtle point you bring up reminded me of Jeffrey Blume’s interval testing procedure he calls “second generation p-values”. While I think the procedure deserves a more descriptive name, with “p-\*” being too overloaded in stats and math, I think it is another useful way to extract value from frequentist results.

> [@What are credible priors and what are skeptical priors?](https://discourse.datamethods.org/t/what-are-credible-priors-and-what-are-skeptical-priors/580/22):
>
> I will have to look at Magnitude Based Inference (aka Magnitude Based Decisions) a bit more carefully, but Jeffrey Blume has developed an approach that is superficially similar called Second Generation P values. His approach is derived from likelihood theory, and has attractive properties I don’t think the MBI/MBD people are cognizant of yet. I have not yet proven to myself Blume’s Second Generation P values and MBI are equivalent. It may be the case that these scholars stumbled upon Blume’s …

> **[Second-generation p-values: Improved rigor, reproducibility, &...](https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0188299)**
>
> Verifying that a statistically significant result is scientifically meaningful is not only good scientific practice, it is a natural way to control the Type I error rate. Here we introduce a novel extension of the p-value—a second-generation p-value...

**Jeffrey D. Blume, Robert A. Greevy, Valerie F. Welty, Jeffrey R. Smith & William D. Dupont (2019)** An Introduction to Second-Generation p-Values, The American Statistician, 73:sup1, 157-167 ([link](https://amstat.tandfonline.com/doi/full/10.1080/00031305.2018.1537893))

I’d strongly prefer a nice clean separation of the actual data, whether it be interval estimates, likelihood functions, or “confidence” distributions, from the storytelling in the discussion and methods.

Related thread

> [@Non inferiority Margin without Placebo controlled Trial](https://discourse.datamethods.org/t/non-inferiority-margin-without-placebo-controlled-trial/4250):
>
> How to estimate the Non inferiority Margin/Equivalence Margin if we do not have Placebo controlled Trial

---

<div class="post-metadata">

**Author:** ![simongates](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/simongates/32/367_2.png) [@simongates](https://discourse.datamethods.org/u/simongates)\
**Post date:** [June 21, 2023, 2:34pm UTC](https://discourse.datamethods.org/t/analogy-between-clinical-trials-and-diagnostic-tests-using-likelihood-ratios-to-interpret-negative-trials/6888/7 "2023-06-21T14:34:56Z")

</div>

Great points, very much agree (haven’t read the paper in detail yet though).

What’s annoying/frustrating is that the habit of referrring to trials with a “non-significant” primary outcome is impossible to kill. Here it is again in JAMA.

---

<div class="post-metadata">

**Author:** ![Pavlos\_Msaouel](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/pavlos_msaouel/32/739_2.png) [@Pavlos\_Msaouel](https://discourse.datamethods.org/u/Pavlos_Msaouel)\
**Post date:** [June 21, 2023, 9:48pm UTC](https://discourse.datamethods.org/t/analogy-between-clinical-trials-and-diagnostic-tests-using-likelihood-ratios-to-interpret-negative-trials/6888/8 "2023-06-21T21:48:43Z")

</div>

> [@Elias\_Eythorsson](#):
>
> As (primarily) a clinician, this approach seems very intuitive and relatively easy to understand. I would love to participate in a discussion regarding the merits of this approach and specifically, its relationship to P-value functions and S-values.

We once explored a similar likelihood-based approach ([second generation p-values](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0188299)) to [interpret the pivotal RCTs in kidney cancer](https://www.sciencedirect.com/science/article/pii/S2468294220300034). It really did not catch on for various reasons, one being the need to specify a null hypothesis interval. We will still sometimes toy around with similar concepts as needed (see “indifference zone” [here](https://journals.sagepub.com/doi/10.1177/17588359221103199)).

Concurrently [in 2019](https://oncologytube.com/video/38160/changing-paradigms-in-statistical-interpretation) we began exploring the use of S-values. This worked far better. Much more intuitive, actionable, and clinicians, biostatisticians, and other researchers love it.

---

<div class="post-metadata">

**Author:** ![Elias\_Eythorsson](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/elias_eythorsson/32/2524_2.png) [@Elias\_Eythorsson](https://discourse.datamethods.org/u/Elias_Eythorsson)\
**Post date:** [June 23, 2023, 11:44am UTC](https://discourse.datamethods.org/t/analogy-between-clinical-trials-and-diagnostic-tests-using-likelihood-ratios-to-interpret-negative-trials/6888/9 "2023-06-23T11:44:34Z")

</div>

Devil’s advocate: if in the future I publish a pragmatic RCT and a priori define and justify a minimally clinically important treatment effect given cost and resource constraints, and the trial turns out negative. Why should I not present the trial using this framework?

---

<div class="post-metadata">

**Author:** ![R\_cubed](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/r_cubed/32/1518_2.png) [@R\_cubed](https://discourse.datamethods.org/u/R_cubed)\
**Post date:** [June 23, 2023, 12:11pm UTC](https://discourse.datamethods.org/t/analogy-between-clinical-trials-and-diagnostic-tests-using-likelihood-ratios-to-interpret-negative-trials/6888/10 "2023-06-23T12:11:33Z")

</div>

> [@Elias\_Eythorsson](#):
>
> Devil’s advocate: if in the future I publish a pragmatic RCT and a priori define and justify a minimally clinically important treatment effect given cost and resource constraints, and the trial turns out negative. Why should I not present the trial using this framework?

I think this comment addresses your question:

> [@f2harrell](#):
>
> **There is also a key problem in using the two-discrete-point likelihood ratio.** For a choice of two effect sizes, such likelihood ratios quantify the relative support for one effect size vs. the other. One effect size may appear to be much more likely, but it could be way off; it’s just better than the other point which is even farther off. **In other words one can have false confidence in the more likely point (actually this is the point that makes the data more likely) when using a relative method such as likelihood ratios.**

Why can’t you just present the likelihood function or p value curve (which can be used to extract the likelihood function) for all \alpha levels itself? What is the attraction of dichotomizing the likelihood function? State your interpretation, but present all of the data so the reader can judge for him/herself.

---

<div class="post-metadata">

**Author:** ![Elias\_Eythorsson](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/elias_eythorsson/32/2524_2.png) [@Elias\_Eythorsson](https://discourse.datamethods.org/u/Elias_Eythorsson)\
**Post date:** [June 23, 2023, 12:24pm UTC](https://discourse.datamethods.org/t/analogy-between-clinical-trials-and-diagnostic-tests-using-likelihood-ratios-to-interpret-negative-trials/6888/11 "2023-06-23T12:24:22Z")

</div>

Thank you. I suppose my answer was just poorly worded, I personally would always just present the likelihood function and describe my interpretation in the main text based on an already defined minimally important effect size of interest. Having said this, I am trying to understand why the likelihood function in particular is better or worse than a p-value function or the Bayesian approach that you recommended in the first comment (that I am slowly working through, thank you for the suggestion).

---

<div class="post-metadata">

**Author:** ![R\_cubed](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/r_cubed/32/1518_2.png) [@R\_cubed](https://discourse.datamethods.org/u/R_cubed)\
**Post date:** [June 23, 2023, 12:52pm UTC](https://discourse.datamethods.org/t/analogy-between-clinical-trials-and-diagnostic-tests-using-likelihood-ratios-to-interpret-negative-trials/6888/12 "2023-06-23T12:52:18Z")

</div>

There are 3 different issues involved here:

1. In terms of the JAMA likelihood ratio approach, my assessment from the abstract is that the authors were attempting to justify the fallacy p(\hat\theta | \theta\_0) \> 0.05 \implies p(\theta) = 0 . The reverse Bayes approach described by Matthews illustrates that there is a distribution of outcomes, and there usually isn’t enough information in a single study to come to a definitive conclusion that the fallacious use of hypothesis testing implies.

2. In terms of communicating the information from a single experiment, the likelihood function may be preferable in the sense that it is easy to use in a meta-analysis. A complete p-value curve can be used to compute a likelihood function, and in some cases, it can be preferable to combine entire confidence curves directly. I think likelihood functions and p-value curves ( or the equivalent “confidence distributions”) are complementary. **Importantly, If readers disagree with your choice of MCID, they can re-do the analysis using your summary stats if presented as likelihood functions and confidence distributions.**

3. There are different perspectives on data analysis in statistics. While there are strong normative arguments in favor of a Bayesian approach, the point of science is to have data collection procedures that lead honest and principled scholars to converge on the the true state of nature, at least temporarily ending dispute on a point of fact. Likelihood ratios and confidence distributions are a compromise that allows Bayesians and Frequentists to communicate productively.

**Schweder, T., & Hjort, N. L. (2013, August).** Integrating confidence intervals, likelihoods and confidence distributions. In Proceedings of the 59th World Statistics Congress, 25-30 August 2013, Hong Kong, Amsterdam International Statistical Institute (Vol. 1, pp. 277-282).

Take al look at the R package for meta-analysis based on confidence distributions:

> **[gmeta: Meta-Analysis via a Unified Framework of Confidence Distribution](https://cran.r-project.org/web/packages/gmeta/index.html)**
>
> An implementation of an all-in-one function for a wide range of meta-analysis problems. It contains three functions. The gmeta() function unifies all standard meta-analysis methods and also several newly developed ones under a framework of combining...

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [June 23, 2023, 1:17pm UTC](https://discourse.datamethods.org/t/analogy-between-clinical-trials-and-diagnostic-tests-using-likelihood-ratios-to-interpret-negative-trials/6888/13 "2023-06-23T13:17:50Z")

</div>

To reiterate some of the thread, presentation of the full likelihood function (as a function of the unknown treatment difference) is a great idea and shows the **relative** compatibility of possible effects and the data. The likelihood function is equivalent to a Bayesian posterior probability distribution under a non-informative (flat) prior distribution on a specific parameter scale. Since all values of the effect are not equally likely, this is not so reasonable. For example we know that therapies in chronic disease are not curative (except maybe in hepatitus C) so it is not reasonable to allow the odds ratio to be zero, but would be reasonable to put a normal prior on the odds ratio that makes it unlikely for the OR to be \< 0.25. So going full Bayes is still recommended, and is easier to interpret.

Besides the JAMA article misusing likelihood ratios, the bigger problem is the failure of clinical trial designers to use MCIDs in power calculations, thereby frequently launching futile clinical trials. The JAMA article assumed that effects used i power calculations are MCIDs, which is probably true about a third of the time.