Interim Analysis & "Alpha Spending" in RCT's using Bayesian Analyses

This is an excellent question and an area where we need to create more educational material. The best way to explain the irrelevance of multiplicity adjustments for sequential Bayesian analysis is simple simulations as done here. And the R code provided there exposes what is going on. If the “population of unknown treatment effects”, a.k.a. the prior distribution, is the same one used for the analyses, then Bayesian posterior probabilities computed at the moment of stopping the trial are perfectly calibrated. In other words, if you simulate unknown efficacies under a certain prior and use that prior also for analysis all is well. If the analysis prior is more conservative than that, all is also well. All a Bayesian has to show is that posterior probabilities are well calibrated to the decision maker. The stopping rule and data look schedule are irrelevant. You’ll see a nice byproduct of Bayes in that simulation: if you stop early for efficacy, the posterior mean is perfectly calibrated. Contrast that with the frequentist point estimate, which is biased away from the null when you stop early.

The code exposes another very important point. Frequentist null hypothesis testing is concerned with controlling how often you assert an effect when none exists. In other words, if there is no effect how often do we get a signal inconsistent with that one value. Bayesian analysis is concerned with attempting to recover the unknown efficacy generating the data no matter what amount of efficacy exists.

Perhaps the following point should be communicated to collaborators before the points above. Though often worried about, type I error probabilities \alpha, like power, are really pre-study concepts and not “how do we interpret results at the end” concepts. And less controversially, \alpha is not a false positive probability in the least. This confusion has prevented many investigators (and statisticians) from seeing things clearly. \alpha is a conditional probability (conditional on the effect being exactly zero). What event or condition is it a probability of? It is the probability of making an assertion of efficacy. It is not the prob. that the treatment doesn’t work. Another way of saying this is that \alpha is a probability about data, not a prob. about a parameter.

So in the task of trying to preserve \alpha researchers and statisticians have lost sight of the fact that type I errors are not really errors in the sense we need them to be, but are just probabilities of assertions. Because of that, Bayesian analysis does not need to be concerned with them.

3 Likes