Design of Experiments in Economics vs Medicine: a Decision Theory POV

[R_cubed]

I think there is room for broad agreement among decision theorists and randomistas with the use adaptive randomization procedures that:

1. minimize sample size requirements.
2. permit model-free inference via permutation tests or bootstrap based on design.

I also find Rosenbaum’s perspective on observational studies as approximations to RCTs a valuable way to look at things.

The following 2014 paper discuses both asymptotic and small sample results of biased coin as well as deterministic allocation procedures, and makes connections to the theory of optimal experiments (both Bayesian and Frequentist). There have been other recent papers (no earlier than 2020) on adaptive allocation that extend these results.

It builds upon the series of Lachin’s 1988 papers by also including an interesting Bayesian allocation rule that is balanced at small sample sizes, and becomes increasingly close to random allocation at larger sample sizes, which is what I would expect from a decision theory perspective. It all comes down to the bias/variance trade one wishes to make.

Atkinson AC. (2014). Selecting a Biased-Coin Design, Statistical Science, Statist. Sci. 29(1), 144-163

https://projecteuclid.org/journals/statistical-science/volume-29/issue-1/Selecting-a-Biased-Coin-Design/10.1214/13-STS449.full

Random allocation can be considered a clear dividing line between controlled research designs (being the least efficient from a strict expected information perspective) vs observational ones, with the latter needing larger effective sample sizes and rigorous data collection to control for factors undermining credibiliy – ie. confounding by indication.

Appropriate statistical tests for my study?

I would recommend study of Paul Rosenbaum’s work on using an observational study to approximate an RCT. It would complement the recommendations of @f2harrell noted below. Since you do not control the treatment allocation mechanism, you bear a burden of proof to demonstrate that no explanations other than the hypothesized treatment, are plausible. You will need to think very hard about the potential biases and confounding variables in order to create plausibly exchangeable groups for compa…

An example that used covariate-adaptive allocation (minimization in this case):

Stinear, C. M., Petoe, M. A., Anwar, S., Barber, P. A., & Byblow, W. D. (2014). Bilateral priming accelerates recovery of upper limb function after stroke: a randomized controlled trial. Stroke, 45(1), 205-210. (link)

Intervention allocation was concealed and randomized using customized software (www.rando.la) that minimized between-group differences in age, baseline ARAT score, PREP stratification,2 and brain-derived neurotrophic factor genotype derived from a single baseline blood sample because this may influence plasticity and learning

Given the sample size (28 control, 29 treated) minimization on predictive covariates was correct, but there were a number of other flaws that detracts from analysis (ie. mean on stroke score impact scale). It is also strange to report p values in Table 1 when the study was explicit in using a minimization procedure.