How to pre-specify the statistical analysis strategy in RCTs?

Hi everyone,

We would like to get your feedback on a preprint we’ve written on how the statistical analysis approach in a randomised trial should be pre-specified: https://arxiv.org/ftp/arxiv/papers/1907/1907.04078.pdf

We wrote this paper as a response to what we perceived as a major problem in the way stats analysis sections in trial protocols were being written: for example, reviews of protocols have found that around 11-20% do not say what statistical model they plan to use to analyse the primary outcome, 42% do give the model, but not enough detail to implement it, and 19% let the investigators choose their analysis model after seeing the trial data.

Our concern is that these protocols are written in a way that essentially allows the investigators to wait until they get their data, run a number of different analyses, and then choose the one that gives the answer they want.

Our preprint is gives a set of rules for how we think analysis strategies should be pre-specified; these rules are designed to ensure that statistical methods cannot be chosen after seeing trial data in order to get a more favourable result (i.e. to limit the possibility of p-hacking, or at least to help people identify when p-hacking occurred). These rules are mainly based on what is in the SPIRIT and ICH-E9 guidelines. Our hope is that this checklist could be used to help people design an analysis strategy for their own RCT, or used to check whether published protocols/RCTs may be at risk of bias from lack of appropriate pre-specification.

The five rules are:

  1. Pre-specify the analysis methods before recruitment to the trial begins;

  2. Specify a single primary analysis strategy (if multiple analyses are planned, one should be identified as the primary);

  3. Plan all aspects of the analysis, including the analysis population, statistical model, use of covariates, handling of missing data, and any other relevant aspects (eg prior);

  4. Enough detail should be provided so that a third party could independently perform the analysis (ideally this could be achieved by providing the planned statistical code); and

  5. Adaptive analysis strategies which use the trial data to inform some aspect of the analysis should use deterministic decision rules

The rationale and further description of these rules is available in the pre-print (https://arxiv.org/ftp/arxiv/papers/1907/1907.04078.pdf); there’s also an example of a trial with a problematic protocol, and how these rules could be used to fix the issues. This is still a work in progress, so we would really like your feedback on it; does it make sense, what needs further clarification/explanation, what needs fixing, etc. Thanks!

5 Likes

a very worthwhile idea. i wonder to what extent the 11-20% of authors would claim that the analysis strategy is obvious because there is some convention in place. I personally wouldn’t buy the argument but it is a little true that reviewers at journals encourage some consistency in the analysis and thus would request the ‘usual analysis’ if it is missing. A different analysis can always be justified retrospectively though

re rule #1. I don’t think i would make this a requirement. Eg, before the blind is broken is sufficient? Someone on datamethods recently queried their primary analysis after recruitement and before trial completion: here That seems reasonable to me, ie they will learn from the data and will do a better, tenable analysis (without some simple definition of outcome imposed for the sake of pre-specification)

there was a well-known example of switching the primary outcome due to low events (ie “after seeing the trial data”) - but in this case it was described in the analysis plan (this study was likely discussed on this message board somewhere, the journal was circ outcomes i think).

but then the analysis plan grows as you need to cover all the possible scenarious! They often end up saying eg “we will use unstructured cov, and if that doesn’t converge we will then try …” You can feel yourself wasting time when preparing analyses in such detail

i will read your paper in a moment, but i wonder if you cover this ie you’re suggesting the analysis plan is a section of the protocol or a separate document? Putting it in the protocol means the analysis won’t be over-specified - that’s good

But then you can also discuss the value of output shells etc etc and the sap grows and grows and gets a life of its own. I left industry because i wanted to enjoy the room-to-move that you have in academia, which usually means more intelligent analyses, and i wouldn’t like to see that constrained by expanding SAPs and SOPs.

cheers

edit: i have read the paper and have small additional thoughts:
-criterion (1) restricts to prospective studies, but retrospective studies and meta-analysis is where this thinking is most needed (ie where bias is most likely and analyses are most complex)? researchers have to write a proposal to get access to eg registry databases and the proposal includes intended analyses, thus it is likely happening to some extent already, it is just not monitored?
-good idea re give code in analysis plan (if possible) and the rule: “enough detail should be provided so that a 3rd party can reproduce the results”; it rules out data-dependent methods of covariate selection, all good
-i wonder how easy it is to pre-specify the analysis population. It is sometimes hard to define ‘spurious data’ or gauge data quality that would warrant exclusion. Maybe it requires clinical expertise or consideration of multiple variables as a collection
-it’s quite possible i missed something in my reading but you seem to cover everthying. Thanks for writing it and promoting good research practice

1 Like

Being a big fan of generating simulated (aka fake) data and analyzing it using the methods proposed for the actual analysis, I would suggest exactly that, in addition to what you’ve already proposed.

  1. Set a seed value
  2. Generate an appropriately simulated dataset (set up some reasonable equations for mean treatment response, between-patient variability, within-patient variability, dealing with missing data, etc, etc)
  3. Analyze the simulated data using the proposed methods, generating all appropriate estimates/graphs/etc

Basically, get as close as you can to running the code with a single call, and at the appropriate time replace the code that generates the fake data (items 1 and 2 above) with reading in realdata.csv, with no other changes to the code.

In addition to heading off some of the problems your paper addresses, it’s amazing how helpful setting up and running the process of analyzing fake data is to getting clarity from stakeholders about what questions they really want answered, and if the proposed experiment will actually answer those questions.

2 Likes

Thanks PaulBrownPhD, this is really helpful! Lots for us to think about.

I think this is a really nice idea - we like the idea of pre-specifying the statistical code, and this is a good way of making sure the code actually works/does what you want.

How about adding something about the need to pre-specify an equivalence margin to help with study interpretation in the event that one obtains a null result.