From the paper:
We consider trials with sequential enrollment and interim analyses at pre-specified sample sizes
𝑘_1 , … , 𝑘_𝑚, with a final analysis performed after 𝑛 = 500 efficacy measures have been > observed. We consider both a single final analysis and group-sequential designs, with
𝑚 ∈ {1, 2, 5, 10, 20, 50, 100, 250, 500} the number of analyses.
This is an interesting coincidence, as I had just watched a video Scott Berry gave on this topic. I strongly suggest people interested in this watch it.
My initial take is that this is an entirely unrealistic simulation of a Bayesian trial, Perhaps @f2harrell could give an opinion. Even likelihoodists know that the likelihood can be misleading at small sample sizes. You can see this “Type I inflation” in their Figure I. Royall prefers the term “misleading evidence”, and there is a bound k, such that the likelihood ratio will not exceed it, thereby providing evidence that H > 0, when H \leq 0. That bound k he discusses is closely related to the Tippett minimal p value test sometimes used in meta-analysis.
As Scott discussed in the video, sequential trials described as having a \alpha = 0.05 are almost always tested using 1 tail at 0.025. I take the authors at their word, and when they simulated a trial at 0.05, they used a 1 tail test (ie. Z = 1.64), which doesn’t represent practice as those who design studies describe them.
One thing that concerns me, is the use of the p-value to compute the posterior, under certain assumptions. In table 2, the authors state the following:
e_i : The frequentist concludes efficacy if p-value < e_i.
The Bayesian concludes efficacy if P(\mu > 0 | X_1 ... X_k) > 1 - e_i
Having thought about this in the context of meta-analysis, I am familiar with their assumptions and the calculations. I discussed them in a number of threads (see below).
Interim p-values are almost invariably much lower than the 0.05 level used in their simulations. These adjustments to the p-value are entirely consistent with Bayes theorem.
Their assumptions essentially translate to a prior on the probit scale, instead of doing formal likelihood calculations. On the probit scale, Kulinskaya, Morgenthaler, and Statute acknowledge that the observed Z score has a random component with a standard error of \pm 1.
I’d like to know what the actual likelihood looked like in those simulations.
Contrast this to a likelihood curve in this post. I’d bet a fairly large number of false discoveries in their simulations would have relatively flat likelihood functions that would not persuade anyone to stop the trial. Their “early stopping” is a function of how they compute the posterior, which is related to all of the flaws of p-values.
The following paper describes my intuition regarding their computation of the Bayesian posterior via a transformation of the p-value.
Wouter Kager, Ronald Meester (2026) On the relation between likelihood ratios and p-values for testing success probabilities of Bernoulli trials
It is well known that there is no direct one-to-one relation between p-values and likelihood ratios or Bayes factors, since their relation crucially involves the sample size n. We investigate their (asymptotic) relation in a coin-tossing context where the hypotheses of interest address the success probability of the coin, and where detailed computations are possible. This leads to useful insights in the nature of p-values and likelihood ratios. Our results imply, for instance, that under mild conditions, a p-value of 0.05 cannot correspond to a likelihood ratio larger than 7.5, for any hypothesis versus a null hypothesis that the success probability has a specific value. We also show it is unlikely one can obtain a large likelihood ratio by tossing a fair coin until the number of heads deviates from the mean by several standard deviations.
The Bayes factor bound 1/-e \times p \times ln(p) is the best case Bayes factor. For p = 0.05, the best case Bayes factor is no more that about 2.5 to 1 in favor of the alternative over the null.
I re-read more closely a section of the paper. They used different variances to give weight to the prior, so my initial criticism (that their analysis was based on a naive uninformative prior) was mistaken.
I still maintain their “Bayesian computation” suffers from confusing the hypothesis
H > 0 vs H \leq 0 with H = 0 vs H > 0. That is also discussed in the Wouter and Meesler paper.
They note:
Here, for u > 0, we recognize Φ(−u) as the 1-sided p-value for testing the null hypothesis H0. So we have found a one-to-one relation, which does not involve n, between the p-value and a likelihood ratio, after all. But this particular likelihood ratio does not consider the null hypothesis at all, and provides a rather unexpected Bayesian interpretation of the p-value.
It is odd to compute the equivalent of a LR for the sign, then argue that this is a Type 1 error by conditioning on an exact value of 0, when strictly speaking, the probability the true parameter is exactly 0, is itself 0.
The relevant alternative hypothesis for a one-sided p value is the parameter on the opposite side of the null, that has the same absolute value as the observed estimate – ie. what is the probability of seeing the observed sign when our estimate comes from a model where the true value has the opposite sign?
It is very easy to confuse these closely related hypotheses. I’ve done it countless times when working through calculations.
In a Bayesian interpretation of p-values, it merely provides a lower bound on the possibility that the observed sign is wrong.
Jeff Blume’s intro on likelihood functions is also valuable:
Recall that the coverage probability of a CI can change even when the data, and the likelihood function, do not. The natural interpretation of an unadjusted CI as the ‘collection of hypotheses that are best supported by the data’ is justified by the Law of Likelihood.
I suspect something similar is happening here, except that rejection probabilities are occurring under likelihood ratios are close to the bound where they might still give misleading evidence.
He continues:
The universal bound applies to both fixed sample size and sequential study designs. In a sequential study, the probability of observing misleading evidence does increase with the number of looks at the data. However, the amount by which it increases shrinks to zero as the sample size grows. As a result, the overall probability can remain bounded (Blume 2007). The probability of observing misleading evidence in a fixed sample size study is less than the corresponding probability in a sequential study, and both are less than 1/k. This is shown below, where the subscript on P denotes the “true” hypothesis.
Some interesting links on likelihood theory
Here is a nice primer on likelihood methods for epidemiologists with @Sander as one of the listed authors:
Some relevant threads:
The Kass paper on data translated likelihoods would be helpful to those working through this paper.