# Is Repeated Bayesian Interim Analysis Consequence-Free? **URL:** \ **Category:** journal club\ **Created:** [August 20, 2025, 10:56am UTC](https://discourse.datamethods.org/t/is-repeated-bayesian-interim-analysis-consequence-free/28342 "2025-08-20T10:56:16Z")\ **Posts on this page:** 1\ **Showing post:** 12

**Author:** ![R\_cubed](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/r_cubed/32/1518_2.png) [@R\_cubed](https://discourse.datamethods.org/u/R_cubed)\ **Post date:** [July 1, 2026, 9:14pm UTC](https://discourse.datamethods.org/t/is-repeated-bayesian-interim-analysis-consequence-free/28342/12 "2026-07-01T21:14:41Z")

From the paper: > We consider trials with sequential enrollment and interim analyses at pre-specified sample sizes > 𝑘\_1 , … , 𝑘\_𝑚, with a final analysis performed after 𝑛 = 500 efficacy measures have been \> observed. We consider both a single final analysis and group-sequential designs, with > 𝑚 ∈ {1, 2, 5, 10, 20, 50, 100, 250, 500} the number of analyses. This is an interesting coincidence, as I had just watched a video Scott Berry gave on this topic. I strongly suggest people interested in this watch it. [![](https://discourse.datamethods.org/uploads/default/original/2X/6/645320f8fef31168cfdbac24360e7441476d9d95.jpeg "Spending Alpha") ](https://www.youtube.com/watch?v=aHuL41OpDB8) My initial take is that this is an entirely unrealistic simulation of a Bayesian trial, Perhaps @f2harrell could give an opinion. Even likelihoodists know that the likelihood can be misleading at small sample sizes. You can see this “Type I inflation” in their Figure I. Royall prefers the term “misleading evidence”, and there is a bound k, such that the likelihood ratio will not exceed it, thereby providing evidence that H \> 0, when H \leq 0. That bound k he discusses is closely related to the Tippett minimal p value test sometimes used in meta-analysis. As Scott discussed in the video, sequential trials described as having a \alpha = 0.05 are almost always tested using 1 tail at 0.025. I take the authors at their word, and when they simulated a trial at 0.05, they used a 1 tail test (ie. Z = 1.64), which doesn’t represent practice as those who design studies describe them. One thing that concerns me, is the use of the p-value to compute the posterior, under certain assumptions. In table 2, the authors state the following: > e\_i : The frequentist concludes efficacy if p-value \< e\_i. > The Bayesian concludes efficacy if P(\mu \> 0 | X\_1 ... X\_k) \> 1 - e\_i Having thought about this in the context of meta-analysis, I am familiar with their assumptions and the calculations. I discussed them in a number of threads (see below). Interim p-values are almost invariably much lower than the 0.05 level used in their simulations. These adjustments to the p-value are entirely consistent with Bayes theorem. Their assumptions essentially translate to a prior on the probit scale, instead of doing formal likelihood calculations. On the probit scale, Kulinskaya, Morgenthaler, and Statute acknowledge that the observed Z score **has a random component** with a standard error of \pm 1. I’d like to know what the actual likelihood looked like in those simulations. Contrast this to a likelihood curve in this post. I’d bet a fairly large number of false discoveries in their simulations would have relatively flat likelihood functions that would not persuade anyone to stop the trial. Their “early stopping” is a function of how they compute the posterior, which is related to all of the flaws of p-values. > **[Data Analysis in the Geosciences](http://stratigrafia.org/8370/lecturenotes/likelihood.html)** 🆕 The following paper describes my intuition regarding their computation of the Bayesian posterior via a transformation of the p-value. **Wouter Kager, Ronald Meester (2026)** On the relation between likelihood ratios and p-values for testing success probabilities of Bernoulli trials > **[On the relation between likelihood ratios and p-values for testing success...](https://arxiv.org/abs/2408.12905)** > > It is well known that there is no direct one-to-one relation between $p$-values and likelihood ratios or Bayes factors, since their relation crucially involves the sample size $n$. We investigate their (asymptotic) relation in a coin-tossing context... > **It is well known that there is no direct one-to-one relation between p-values and likelihood ratios or Bayes factors, since their relation crucially involves the sample size n.** We investigate their (asymptotic) relation in a coin-tossing context where the hypotheses of interest address the success probability of the coin, and where detailed computations are possible. This leads to useful insights in the nature of p-values and likelihood ratios. **Our results imply, for instance, that under mild conditions, a p-value of 0.05 cannot correspond to a likelihood ratio larger than 7.5, for any hypothesis versus a null hypothesis that the success probability has a specific value.** We also show it is unlikely one can obtain a large likelihood ratio by tossing a fair coin until the number of heads deviates from the mean by several standard deviations. The Bayes factor bound 1/-e \times p \times ln(p) is the **best case Bayes factor.** For p = 0.05, the best case Bayes factor is no more that about 2.5 to 1 in favor of the alternative over the null. I re-read more closely a section of the paper. They used different variances to give weight to the prior, so my initial criticism (that their analysis was based on a naive uninformative prior) was mistaken. I still maintain their “Bayesian computation” suffers from confusing the hypothesis H \> 0 vs H \leq 0 with H = 0 vs H \> 0. That is also discussed in the Wouter and Meesler paper. They note: > Here, for u \> 0, we recognize Φ(−u) as the 1-sided p-value for testing the null hypothesis H0. So we have found a one-to-one relation, which does not involve n, between the p-value and a likelihood ratio, after all. **But this particular likelihood ratio does not consider the null hypothesis at all, and provides a rather unexpected Bayesian interpretation of the p-value.** It is odd to compute the equivalent of a LR for the sign, then argue that this is a Type 1 error by conditioning on an exact value of 0, when strictly speaking, the probability the true parameter is exactly 0, is itself 0. The relevant alternative hypothesis for a one-sided p value is the parameter **on the opposite side of the null, that has the same absolute value as the observed estimate** – ie. what is the probability of seeing the observed sign when our estimate comes from a model where the true value has the opposite sign? It is very easy to confuse these closely related hypotheses. I’ve done it countless times when working through calculations. In a Bayesian interpretation of p-values, it merely provides a **lower bound** on the possibility that the observed sign is wrong. > **[Living with P Values: Resurrecting a Bayesian Perspective... : Epidemiology](https://journals.lww.com/epidem/abstract/2013/01000/living_with_p_values__resurrecting_a_bayesian.9.aspx)** > > ull, when they are no such thing. This misuse may be lessened by recognizing correct Bayesian interpretations. For example, under weak priors, 95% confidence intervals approximate 95% posterior probability intervals, one-sided P values approximate... Jeff Blume’s intro on likelihood functions is also valuable: > **[Statistical Evidence | Likelihod](https://www.statisticalevidence.com/likelihood)** > > Describes likelihood methods for measuring statistical evidence and discussed the law of likelihood. > **Recall that the coverage probability of a CI can change even when the data, and the likelihood function, do not.** The natural interpretation of an unadjusted CI as the ‘collection of hypotheses that are best supported by the data’ is justified by the Law of Likelihood. I suspect something similar is happening here, except that rejection probabilities are occurring under likelihood ratios are close to the bound where they might still give misleading evidence. He continues: > The universal bound applies to both fixed sample size and sequential study designs. In a sequential study, the probability of observing misleading evidence does increase with the number of looks at the data. However, the amount by which it increases shrinks to zero as the sample size grows. As a result, the overall probability can remain bounded (Blume 2007). **The probability of observing misleading evidence in a fixed sample size study is less than the corresponding probability in a sequential study, and both are less than 1/k. This is shown below, where the subscript on P denotes the “true” hypothesis.** **Some interesting links on likelihood theory** Here is a nice primer on likelihood methods for epidemiologists with @Sander as one of the listed authors: > **[Maximum Likelihood, Profile Likelihood, and Penalized Likelihood: A Primer](https://pmc.ncbi.nlm.nih.gov/articles/PMC3873110/)** > > The method of maximum likelihood is widely used in epidemiology, yet many epidemiologists receive little or no education in the conceptual underpinnings of the approach. Here we provide a primer on maximum likelihood and some important extensions ... Some relevant threads: The Kass paper on data translated likelihoods would be helpful to those working through this paper. > [@Some thoughts on uniform prior probabilities when estimating P values and confidence intervals](https://discourse.datamethods.org/t/some-thoughts-on-uniform-prior-probabilities-when-estimating-p-values-and-confidence-intervals/7508/160): > > Why do you assume a flat prior after your first estimate? That amounts to ignoring the likelihood function from the first study. Please study the following: Kass, R. E. (1990). Data-translated likelihood and Jeffreys’s rules. Biometrika, 77(1), 107-114. ([PDF](https://www.stat.cmu.edu/~kass/papers/jeffreys.pdf)) > [@Using p-values, S-values, and the Bayesian Rejection Ratio to re-examine a meta-analysis on knee abduction kinematics and ACL injury risk](https://discourse.datamethods.org/t/using-p-values-s-values-and-the-bayesian-rejection-ratio-to-re-examine-a-meta-analysis-on-knee-abduction-kinematics-and-acl-injury-risk/3887): > > A colleague brought to my attention the following peer-reviewed meta-analysis of 12 prognostic studies that attempted to predict future ACL injury risk from knee kinematics. Title: Do knee abduction kinematics and kinetics predict future anterior cruciate ligament injury risk? A systematic review and meta-analysis of prospective studies The authors found 9 studies, using 3 different assessments, that attempt to detect excess angular motion as a predictor of future ACL injury. They reported … > [@Relating S-values to other measures of statistical information](https://discourse.datamethods.org/t/relating-s-values-to-other-measures-of-statistical-information/4293): > > After reading this paper again: I wanted to share some R code I wrote to improve my intuition on how information measures relate to other common statistical quantities (ie Z-scores). # Converting among various measures of information and evidence # Shannons, p values, Z\_scores, Bayes Factor Bounds. bayes\_factor \<- function(p) { e \<- 2.71828 if (p \>= 1/e) { bayes\_factor \<- 1.00 } else { bayes\_factor \<- 1/(-e\*p\*log(p)) } } shannon\_table \<- function(lower, upper) { bits … > [@Stability of results from small RCTs](https://discourse.datamethods.org/t/stability-of-results-from-small-rcts/2287/12): > > Blockquote The ANDROMEDA-SHOCK trial discussion on this platform discussed how a Bayesian approach might allow a more nuanced interpretation of smaller RCTs whose results don’t reach “statistical significance,” but where early data seems to be “leaning” in one direction. While I agree with our host that Bayesian methods are generally preferable, there are simple transforms of p values that can minimize this error. I took a recommendation of Sander Greenland seriously, and started studying p… --- _[View the full topic](https://discourse.datamethods.org/t/is-repeated-bayesian-interim-analysis-consequence-free/28342)._