I am some what new to randomized trials and have been reading [1], [2], [3] unsystematically over the last few days.
It seems to me that many online platforms that run randomized experiments use Bernoulli trials, in which treatment assignment is independent across all units. These online platforms have very large samples in their experiments. Fisher’s tea-tasting experiment seems to be a completely randomized trial, in which a fixed number of units were randomly chosen to receive the treatment.
What are the similarities and differences between Bernoulli trials and completely randomized trials?
Three points:
- In both designs, the assignment is unconfounded so the treatment effect can be estimated without bias. So, no difference on this point.
- The obvious difference is that in a Bernoulli trial, there is a small but non-zero probability that all units may end up in the treatment or control state (completely randomized trials, of course, guarantee that there would be a fixed number of units in each treatment state).
- For test of significance, Fisher provides an example to demonstrate that Bernoulli trials are more sensitive (Chap 2 of [2]). However, my understanding is that Fisher’s example works if we are thinking about the uncertainty due to the treatment assignments (exact test) so that we observe only one potential outcome for each treatment state but assume the unobserved under the exact/strict null hypothesis. I also understand that in many if not most empirical analyses, the presented measures of uncertainty are those arising from sampling (§2.4 of [3] has a discussion).
Is the above list accurate and complete? If not, what are the inaccuracies and what is missing?
Also, how would conventional sampling-based standard errors [3] differ between Bernoulli trials and completely randomized trials.
[1] Fisher, Ronald Aylmer. “The design of experiments.” 2nd Ed (1937).
[2] Imbens, Guido W., and Donald B. Rubin. “Causal inference in statistics, social, and biomedical sciences.” Cambridge University Press, 2015.
[3] Athey, Susan, and Guido W. Imbens. “The econometrics of randomized experiments.” In Handbook of economic field experiments, vol. 1, pp. 73-140. North-Holland, 2017.