A useful lens I have used in thinking about this topic is the distinction between internal and external validity (Campbell, 1957 and later publications with coauthors). Internal validity refers to the ability to attribute a causal link to the intervention, by managing sources of confounding (random allocation, blinding, and concurrent control being some of the tools to accomplish this). External validity refers to the ability to generalize from a sample to the population. Random sampling is a mechanism to achieve this. In the clinical trials literature, similar distinctions have been made (eg, Schwartz & Lellouch 1967, who refer to explanatory vs pragmatic, respectively). See also
Turning to the question at hand:
Mead, et al. (2012) write that “In designed experiments, the very careful control which is exercised often makes it difficult to identify a population for which the sample is relevant.” They raise the question “of whether the conclusions from such rigidly controlled experiments have any validity for future practical situations. This is a difficult question, which extends beyond statistics” (p. 233).
Similarly, Simon (2006) writes that “Researchers, trying to minimize variation, will use exclusion criteria to create more homogeneous groups…If it is difficult to extrapolate results from a very tightly controlled and homogeneous clinical trial to the variation of patients seen in your practice, then the research has limited value to you” (p. 39).
Senn (2007) seems to echo @Pavlos_Msaouel . He writes, “In experiments, provided that we obtain suitable material, we rarely worry about its being representative. Inference is comparative and…this is generally the appropriate attitude for clinical trials” (p. 40).
I credit the randomization test community for thinking carefully about this issue, as @R_cubed alluded to already. They seem to have provided the most uncompromising answer I’ve seen so far. Edgington and Onghena (2007) state that “statistical inferences about treatment effects must be restricted to the subjects (or other experimental units) used in an experiment” (p. 8) due to the inability to randomly sample subjects from the population. “Inferences about treatment effects for other subjects must be nonstatistical inferences-inferences without a basis in probability. Nonstatistical generalization is a standard scientific procedure. We generalize from our experimental subjects to individuals who are quite similar in those characteristics that we consider relevant.” Finally, “the main burden of generalizing from experiments always has been, and must continue to be, carried by nonstatistical rather than statistical logic.”
Finally, one could push the questionable logic of statistical inference all the way off the cliff. Berk & Freedman (2005) wrote that “inferences to imaginary populations are also imaginary.”
Historically, Fisher’s century old classic (Fisher 1922) founded statistical inference on the random sampling from a population model, but he also was an advocate of random assignment in designed experiments (see his 1935 book Design of Experiments and other writings). I think Austin Bradford Hill was the first to take these ideas into the clinical trials arena.
References:
R. Berk and D. A. Freedman, 2005: Statistical assumptions as empirical commitments. In
Law, Punishment, and Social Control: Essays in Honor of Sheldon Messinger, 2d ed., edited
by T. G. Blomberg and S. Cohen (Aldine, 235-254).
D. T. Campbell, 1957: Factors relevant to the validity of experiments in social settings.
Psychological Bulletin, 54: 297-312.
E. S. Edgington and P. Onghena, 2007: Randomization Tests. Fourth edition. Chapman & Hall/CRC.
R. A. Fisher, 1922. On the Mathematical Foundations of Theoretical Statistics, Philosophical Transactions of the Royal Society of London A, 222, 309–368.
R. Mead, S. G. Gilmour, and A. Mead, 2012: Statistical Principles for the Design of Experiments: Applications to Real Experiments. Cambridge University Press.
D. Schwartz and J. Lellouch, 1967: Explanatory and pragmatic attitudes in therapeutic trials. Journal of Chronic Diseases, 20: 637-648. Reprinted in Journal of Clinical Epidemiology, 62: 499-505.
S. Senn, 2007: Statistical Issues in Drug Development. Second edition. Wiley.
S. D. Simon, 2006: Statistical Evidence in Medical Trials: What do the Data Really Tell Us? Oxford University Press.