I have linked an article below published this month. This provides more evidence that the measurement tools used for critical care RCT esp. for sepsis and mortality are often fake. Those who have studied this issue did not need more evidence but perhaps if enough of these studies are published the perceived bright star of SOFA will fade (as SIRS has faded). This study is no surprise given there has not been a single reproducibly positive sepsis RCT for 30 yrs.
Now fake is a harsh word and I dont use it lightly. We woud not hesitate to use that word if a homeopathic study was quoting some measurement standard that was guessed in the remote past.
So what makes a measurement used in RCT “fake”. One answer is the measurement is fake if it based on guessed thresholds. A guess is bad enough but it might be right. Guessed thresholds…I will let the statisticians tell us the probabilty that those are valid measurements.
Guessed thresholds cannot be used as a tool to measure other things or as a gold standard surrogate for a dependent variable in science. It does not matter how old the guess is or how many experts believe and promulgate the guess.
A guess does not become a measurement in science because it has “survived the test of time”. The history of science has shown that time is not a test.
I have heard that statisticians must use the tools they are given because its mandated by the PI or the reviewer. This argument suggests the statistician is a powerless worker who needs to use a potentially fake measurement, if mandated by the powerful, to do her work. Anyone reading this datamethods forum, and seeing the intellectual power here, knows that is nonsense. Statisticians are at the top of the RCT decision making hierarchy not the bottom, and for good reason.
Another argument has been that its not the statistician’s job to determine the origin or validity of the clinical measurement tool used. This “head in sand argument” ignores the fact that, in 2020, it is very easy to check the origin, history, and validity and its within the capability of statisticians to do so. It futher ignores the role of the statistician as the gate keeper of the RCT math, which I have pointed out before, is not comprised of siloed measurements disconnected from statistics but rather all the RCT math is an interdependent continuum.
So I have “enjoyed” (with some head wagging) all of the writings here discussing the meaning and the statistical math of RCT using SOFA and SIRS (guessed in 1989 and 1996 respectively). I’m reminded of the “deep” math of the Claudius Ptolemy. Were those fake measurements because the model was wrong? No. Good science is often wrong! Rather, they were fake because the model was guessed.
Now the paper I cite here below won’t be the end. SOFA will live on as a RCT measurement or endpoint because it has social momentum. The measurement of SIRS (guessed in 1989) is still used in research (as sepsis 2) but it is fading because it has lost social momentum. SOFA (guessed in 1996) is still going strong and used as a surrogate for sepsis as Sepsis 3. It has social momentum. Social momentum is the armor which deflects counterinstances.
Yet, next time someone asks you to do complex statistical analysis of data using SIRS or SOFA you will have to decide if you have the courage to ask for evidence that this is a valid, reproducible surrogate. The answer, and the studies they give you (if any) to support the measure will be more enlightening than the math, if you still choose to do it.