Blockquote
[information about why treatments are selected in the field] prior information is all important and seldom obtained.
It is true that in the total absence of information on a prognostic variable, randomization is the minimax optimal experimental design. My concern is the use of reported randomization as a post hoc criterion of study quality that automatically trumps observational research.
Statistician Paul Rosenbaum describes the considerations that go into the evaluation of observational studies:
Blockquote
Dismissive criticism rests on the authority of the critic and is so broad and vague that its claims cannot be studied empirically
The retrospective consideration of design factors as measures of quality (commonly seen in meta-analytic guidelines) strike me as meeting Rosenbaum’s definition of “dismissive criticism.” The relative credibility to place on a reported estimate is context sensitive. It isn’t the lack of randomization that needs to be noticed, but the assertion of a prognostic variate that should (and could) be controlled, but was not. Dismissal of data based on design features alone, strikes me as not scientifically constructive, if not completely irrational.
Your point about confounding by indication is important, and I concede there are likely many scenarios where I would want a randomized study.
My problem is that rehabilitation research is not going to get the funds to run RCTs that a drug or medical device would get, so the large sample sizes for credible RCTs are outside of the typical budget. The decision theoretic approach to experiments advises optimization over randomization when samples are small. Most interesting are mixed strategies, that inject a bit of randomness into algorithms that create a priori balanced groups. This would satisfy the frequentists who demand randomization, while Bayesian decision theorists are happy that expected information is being maximized by prior information.
Absent well done small experiments, I’d like to see well done observational studies, where there is a lot of room for improvement.
Nathan Kallus (Cornell) has a number of recent papers that goes into the technical details and decision theoretic analysis regarding the utility of the algorithmic creation of a priori balanced groups for controlled experiments and modifications of them for observational causal inference.
Some recent papers on the a priori use of nonrandom construction of balanced groups, as well as a recent paper with Art Owen (who developed empirical likelihood) about using observational studies to inform RCTs:
Blockquote
Relying on modern optimization methods, kernel allocation, which ensures nearly perfect covariate balance without biasing estimates under model mis-specification, offers sizable advantages in precision and power as demonstrated in a range of real and synthetic examples. We provide strong theoretical guarantees on variance, consistency and rates of convergence and develop special
algorithms for design and hypothesis testing
Blockquote
Motivated by an observational study in spine surgery, in which positivity is violated and the true treatment assignment model is unknown, we present the use of optimal balancing by kernel optimal matching (KOM) to estimate ATE. By uniformly controlling the conditional mean squared error of a weighted estimator over a class of models, KOM simultaneously mitigates issues of possible
mis-specification of the treatment assignment
Blockquote
We propose a novel framework for matching estimators for causal effect from observational data that is based on minimizing the dual norm of estimation error when expressed as an operator. We show that many popular matching estimators can be expressed as optimal in this framework, including nearest-neighbor matching, coarsened exact matching, and mean-matched sampling. This reveals their motivation and aptness as structural priors formulated by embedding the effect in a particular functional space. This also gives rise to a range of new, kernel-based matching estimators that arise when one embeds the effect in a reproducing kernel Hilbert space. Depending on the case, these estimators can be found using either quadratic optimization or integer optimization. We show that estimators based on universal kernels are universally consistent without model specification. In empirical results using both synthetic and real data, the new, kernel-based estimators outperform all standard causal estimators in estimation error.
Branson, Z. (2021). Randomization Tests to Assess Covariate Balance When Designing and Analyzing Matched Datasets. Observational Studies 7(2), 1-36. doi:10.1353/obs.2021.0031. (link)
Blockquote
In this work, we develop a randomization test for the hypothesis that a matched dataset approximates a particular experimental design, such as complete randomization, block randomization, or rerandomization. Our test can incorporate any experimental design, and it allows for a graphical display that puts several designs on the same univariate scale, thereby allowing researchers to pinpoint which design—if any—is most appropriate for a matched dataset. After researchers determine a plausible design, we recommend a randomization based approach for analyzing the matched data, which can incorporate any design and treatment effect estimator. Through simulation, we find that our test can frequently detect violations of randomized assignment that harm inferential results
Blockquote
The increasing availability of passively observed data has yielded a growing interest in “data fusion” methods, which involve merging data from observational and experimental sources to draw causal conclusions. Such methods often require a precarious tradeoff between the unknown bias in the observational dataset and the often-large variance in the experimental dataset. We propose an alternative approach, which avoids this tradeoff: rather than using observational data for inference, we use it to design a more efficient experiment.