Information, Evidence, and Statistics for critical research appraisal

[R_cubed]

@ESMD

I think most practising physicians would agree that the traditional pyramid (RCT at the top) applies in assessments of treatment efficacy but not necessarily in assessment of treatment safety. I’ve noticed that those who tend to argue against an efficacy pyramid are often those in non-clinical professions.

I’m glad you expressed this, because this common attitude encouraged by evidence hierarchies is neither ethically optimal, nor statistically correct.

Richard Royall (who I mentioned above) wrote an excellent paper discussing the ethics and scientific issues around the ECMO trial for infants. I cannot express the logic better than by offering some quotes and recommending reading it.

https://projecteuclid.org/journals/statistical-science/volume-6/issue-1/Ethics-and-Statistics-in-Randomized-Clinical-Trials/10.1214/ss/1177011934.full

Some key quotes:

1. From the abstract:

Blockquote
We urge that the view that randomized clinical trials are the only scientifically valid means of resolving controversies about therapies is mistaken, and we suggest that a faulty statistical principle is partly to blame for this misconception.

2. In discussing the 1985 ECMO trial where the investigators felt “compelled to conduct a prospective, randomized study” on infants with severe respiratory distress without equipoise, Royall has this to say

Blockquote
This is particularly disturbing to me as a statistician, because it is we statisticians who are largely responsible for creating attitudes and assumptions that compelled this study… In the next section, I propose that the above doctrine springs at least in part, adherence to a faulty statistical principle.

3. In presenting alternatives to the randomized “play the winner” design, he says:

Blockquote
Science desires randomized clinical trials, it does not demand them. [my emphasis] Moreover, the importance of randomization is exaggerated when historical controls are the only alternative discussed…many of the weaknesses of historical controls can be avoided by using concurrent, nonrandomized controls. And of course, comparing treatment to historical and concurrent controls can provide even greater evidence of efficacy.

4 In discussing the randomization principle (ie. only randomized experiments provides a basis for inference) he writes:

Blockquote
Statistical theory explains why the randomization principle is unacceptable. It does this in terms of the concepts of conditionality and likelihood…the conditional randomization distribution is degenerate, assigning one to the actual allocation used … the only “inference” on the observed data is “I saw what I saw”