Raj, I found this forward looking post . I see @ADAlthousePhD point also. l debated some of the points below with him on twitter and I still do not know if we are in agreement.
I think there has to be a checklist for basic components for all; clinician scientist , statistician and trialists. One that all can engage and discuss which, for example, assures the validity and reproducibility of the entry measurement (e.g criteria) as well as any tools used for adjustment in relation to both the population under test as well as the population to which the results will be generalized have been checked.
This is particularly true in critical care. I think few on twitter understood why I was so concerned. My admonitions were not views which were widely held before the Pandemic and during the heady days of massive multicenter RCT application to consensus critical care syndromes like sepsis.
Steps for massive RCT
- Create a criteria set for a consensus syndrome (eg by Delphi)
- Apply an RCT to the syndrome using the set.
- Write protocol based on results of RCT generalized to all that meet the criteria.
- Promulgate the protocol as the standard of care for all meeting the criteria
…
This paper shows the weakness of oversimplifying.
…
Protocol Failure Detection: The Conflation of Acute Respiratory Distress Syndrome, SARS-CoV-2 Pneumonia and Respiratory Dysfunction - PubMed
.In this paper we show that the pandemic exposed a profound flaw in standard critical care RCT design and more importantly the Trialists interpretation and generalization of the implications of their results in these large high quality Randomized Controlled Trials. The consequence of the cognitive error identified in this paper likely contributed to dangerous early intubation as a standard of care of COVID 19 early in the pandemic.
The cognitive error was to perceive, believe, and vigorously promulgate that the results of an RCT applied to a syndrome were applicable to a new disease because it “met the criteria” of that syndrome. This error is easy to see in retrospect but… why did such an obvious mistake happen?.
Trialists were overconfident that the basics of RCT design and in the value of consensus “Science”. They .thought RCT could be applied to consensus syndromes as if a consensus syndrome was a disease like diabetic ketoacidosis. . There was a narrow perception of the meaning of the term “heterogeneity of treatment effects” . They did not understand that the choice of criteria for the syndrome under test may cause amplification of treatment effect heterogeneity. Trialists were overconfident in the power of randomization and adjustments to overcome these issues.
These have been the warnings I have been providing for the past decade. Trialists should read that paper at least twice and then figure out how to design a checklist which prevents this type of cognitive catastrophe from occurring again. The potential adverse consequences of this for the public are beyond discussion but the point is it is time to abandon political expediency and raise this issue front and center in the elite medical literature…