Blockquote
If you are going to claim that my position is “logically false”, then please show me that it relies on a logically invalid syllogism. Alternatively, please stop appropriating terms that have precise definitions in order to borrow authority for your position.
While I think the entire history of human beings learning things long before randomization was discovered is proof enough, Royall already rebutted this demand for randomization in the paper I cited in his discussion of the Likelihood Principle:
Blockquote
Statistical theory explains why the randomization principle is unacceptable. It does this in terms of the concepts of conditionality (ancillarity) and likelihood… The conditionality principle asserts that when there is an ancillary statistic C present, inferences should be based upon the observed value of C. The problem this creates for the randomization principle is that the statistic representing the result of the randomization is ancillary; thus the conditional randomization distribution is degenerate, assigning one to the actual allocation used … the only “inference” on the observed data is “I saw what I saw”.
My in thread post of a quote from Dennis Lindley RE: randomization being useful but not necessary is also relevant.
Design considerations are very important before the data is collected (for the experimenter), but the Likelihood Principle does not use that to create “hierarchies of evidence” after the data is collected. There should be no a priori reason for a reader to grant randomized studies greater weight than observational ones simply by a pre-data design criterion.
Goutis, C., & Casella, G. (1995). Frequentist Post-Data Inference. International Statistical Review / Revue Internationale de Statistique, 63(3), 325–344. Frequentist Post-Data Inference on JSTOR
Blockquote
The end result of an experiment is an inference, which is typically made after the data have been seen (a post-data inference). Classical frequency theory has evolved around pre-data inferences, those that can be made in the planning stages of an experiment, before data are collected. Such pre-data inferences are often not reasonable as post-data inferences, leaving a frequentist with no inference conditional on the observed data.
None of this is important if powerful interests can manipulate what studies are published, and which ones are not, via subversion of the peer review process, which Ioannidis discusses in those papers.
Much to the disappointment of those who want easy answers, there are no rules based upon design features alone, that can decide this a priori, especially when there is an appreciable probability of fraud or deception. My position is that needs to be done on a case by case basis, according to formal decision theoretic principles. Michael Rawlins presents many historical examples of medical learning from what we know are imperfect data, that many EBM proponents would find problematic.
The world we have to deal with are:
- large economically and politically powerful, and coordinated actors who can afford to produce randomized designs (or the illusion of them) vs
- disorganized and politically weak agents who might be able to rebut biased presentations of RCTs with observational evidence or approximate the relevant RCT when powerful interests have no incentive to conduct a credible RCT.
That is the problem I am worried about, and am working out how to rigorously formalize the notion.
Related Reading
Rubin, D. B. (1992). Meta-Analysis: Literature Synthesis or Effect-Size Surface Estimation? Journal of Educational Statistics, 17(4), 363–374. https://doi.org/10.3102/10769986017004363
Blockquote
In contrast to these average effect sizes of literature synthesis, I believe that the proper estimand is an effect-size surface, which is a function only of scientifically relevant factors, and which can only be estimated by extrapolating a response surface of observed effect sizes to a region of ideal studies. This effect-size surface perspective is presented and contrasted with the literature synthesis perspective.
In a Robust Bayesian Meta-Analytic approach, design considerations could be treated as a nuisance factor, and integrated out.
There are some errors in statistical reasoning here, but the main points are important.
Blockquote
Seemingly well designed, executed, and reported, RCTs with exciting results can also be misleading due to the hijacked research agenda. These trials are designed to deceive and the methods of deception are alarmingly simple, but effective. The main tactics used relate to the choice of comparators, the choice of outcomes, and the manipulation of statistics to produce desired outcomes, and selectively report them.
Nothing in EBM textbooks or literature prepares a scholar for the adversarial context of the “real world.”