I’m planning a few behavior intervention trials for the coming year (smoking cessation, alcohol reduction, etc.). I’m thinking of adopting a Bayesian group sequential design for these trials, as the analysis plans are fully Bayesian, and it doesn’t match well with thinking in terms of traditional power calculations and significance test.
One thing that I haven’t really seen a discussion about is the re-starting of a trial due to criteria that previously were met no longer being so. Here is an example:
- Let us assume that I have a success criteria P1 and a futility criteria P2.
- I have a follow-up interval of 3 months from randomization.
- I recruit for 6 months, and analyzing those who have responded to the 3 month follow-up at this time reveals that P2 is fulfilled. It seems that it is futile to continue recruitment, so I stop.
- However, there are still participants in the trial that are yet to report 3 month outcomes, and as they start reporting the P2 criteria is no longer fulfilled. Now it seems that it is not futile to continue, but P1 is not fulfilled.
My question is, should I then re-start recruitment after having stopped? What are your concerns with doing so versus not doing so?
From my perspective a Bayesian group sequential design should be the norm in behavior intervention trials, so guidance is very much welcome to help set a standard.
One of the beautifies of Bayesian design is that you don’t need to group the analyses; you can analyze the data continuously and make decisions any time you want as discussed here. This is especially true if you use a skeptical prior for the effect of interest, because it will tilt estimates downward if you stop early because the effect seems to be large.
To answer your question, consider the way that you would cheat with a Bayesian sequential design. You decide to restart a study and get more data. You don’t like the new data because it’s less impressive, so you decide to stick with the earlier data. As long as you don’t cheat, your’re OK. New data merely supersede old data and you base your analysis on all available data at any moment. The Bayesian sequential approach is fair, because you might extend the study and get less impressive results. Nothing is guaranteed.
Thank you @f2harrell , thinking about this from the cheating perspective is useful.
Following the reasoning that new data supersedes old data, it seems to me that the right thing to do is to restart the trial. Not restarting the trial would be cheating, as I would be ignoring new data. Skeptical priors will obviously help from this happening (as it will pull estimates downward in early looks), but there will be pathological cases where early data looks impressive and will override even a very skeptical prior.
Ignoring data not yet collected is not exactly cheating, but will result in suboptimal decisions.
Sorry, I was referring to the data collected after making the initial decision to stop (ie. data from participants already enrolled but for whom follow-up data was not available at the time of decision).
1 Like