Random sampling versus random allocation/randomization- implications for p-value interpretation

[R_cubed]

@ESMD

1. What exactly does a p value mean in the context of an RCT, given that “random sampling” from an underlying population of interest has not occurred?

I remember posting a link that discusses this in a very old thread:

Parametric Analyses In Randomized Clinical Trials

The textbooks discuss the distinction between model based inference (ie. 2 sample t-test), vs design based inference (again either 2 sample t test or permutation test). The design gives justification to the calculation of a p value in an RCT.

Updates to NEJM Statistical Guidelines for Authors

P values in clinical trials can always be justified (and calculated) by permutation tests based upon the design. Wasn’t that Fisher’s key justification for the randomization procedure? The same cannot be said for observational studies. I’m not sure how big of an error this is, however. At least in simulations, the t test results generally matches the permutation test results. Addendum: I re-read the first paper I posted by Berger, Lunneborg, Ernst, and Levene who argue that RCTs should …