a very worthwhile idea. i wonder to what extent the 11-20% of authors would claim that the analysis strategy is obvious because there is some convention in place. I personally wouldn’t buy the argument but it is a little true that reviewers at journals encourage some consistency in the analysis and thus would request the ‘usual analysis’ if it is missing. A different analysis can always be justified retrospectively though
re rule #1. I don’t think i would make this a requirement. Eg, before the blind is broken is sufficient? Someone on datamethods recently queried their primary analysis after recruitement and before trial completion: here That seems reasonable to me, ie they will learn from the data and will do a better, tenable analysis (without some simple definition of outcome imposed for the sake of pre-specification)
there was a well-known example of switching the primary outcome due to low events (ie “after seeing the trial data”) - but in this case it was described in the analysis plan (this study was likely discussed on this message board somewhere, the journal was circ outcomes i think).
but then the analysis plan grows as you need to cover all the possible scenarious! They often end up saying eg “we will use unstructured cov, and if that doesn’t converge we will then try …” You can feel yourself wasting time when preparing analyses in such detail
i will read your paper in a moment, but i wonder if you cover this ie you’re suggesting the analysis plan is a section of the protocol or a separate document? Putting it in the protocol means the analysis won’t be over-specified - that’s good
But then you can also discuss the value of output shells etc etc and the sap grows and grows and gets a life of its own. I left industry because i wanted to enjoy the room-to-move that you have in academia, which usually means more intelligent analyses, and i wouldn’t like to see that constrained by expanding SAPs and SOPs.
cheers
edit: i have read the paper and have small additional thoughts:
-criterion (1) restricts to prospective studies, but retrospective studies and meta-analysis is where this thinking is most needed (ie where bias is most likely and analyses are most complex)? researchers have to write a proposal to get access to eg registry databases and the proposal includes intended analyses, thus it is likely happening to some extent already, it is just not monitored?
-good idea re give code in analysis plan (if possible) and the rule: “enough detail should be provided so that a 3rd party can reproduce the results”; it rules out data-dependent methods of covariate selection, all good
-i wonder how easy it is to pre-specify the analysis population. It is sometimes hard to define ‘spurious data’ or gauge data quality that would warrant exclusion. Maybe it requires clinical expertise or consideration of multiple variables as a collection
-it’s quite possible i missed something in my reading but you seem to cover everthying. Thanks for writing it and promoting good research practice