I have become increasingly interested in the role of simulation in developing plans for analysis. I first came across the idea when reading about estimating frequentist properties of Bayesian procedures, and now more recently see it being advocated as a routine part of model building in the Bayesian context (e.g. over at the Stan forums specifically in the context of making sure your code works as you intend it to). This procedure makes a lot of sense to me:
- Specify your data generating model (common to frequentist and Bayesian)
- Simulate parameters and data from that model that reflect your planned data collection strategy
- Assess whether your proposed model can recover the simulated parameters with some predetermined measure of accuracy
- (optional) evaluate how potential problems might undermine your goals and evaluate the robustness of alternative models
It seems to me that if you did this at the outset of your trial then you would:
- Know exactly how you need to set up your data collection forms
- Be confident that your code correctly returns the true value given your assumptions
- Be able to assess plausibility that violation of those assumptions would change your findings
- Directly investigate issues related to what-ifs (e.g. if a client is adament about using propensity score matching, or stepwise regression, or AIC for model selection etc… etc…)
I suppose the easy answer is to trust published simulations, or that this is simply too much work. On its face it appeals to me though in terms of addressing some of the clinical arguments that have come up here in the past (e.g. dichotomania, importance of regularizing coefficients in highly dimensional settings, hazards of stepwise procedures). Maybe all we need for some of these is a collection of citations to simulations related of major issues?
I’m almost certain the statisticians and trialists here will think this just seems like an obvious thing, and yet I can’t recall working with anyone at our (admittedly small) centre that takes this approach.
An example I am working on now is a protocol for a review where we will be specifying a stochastic loss function across outcomes for use in decision making. Due to time constraints this isn’t something we can do while meeting deadlines, but it seems like it would be entirely worthwhile to look ahead at how a p-value based decision rule would conflict with what we are proposing.