How to interpret “confidence intervals” in observational studies

[R_cubed]

I think this old post by @Sander is worth study:

@Sander

Next, I think this statement is commonly believed and simply wrong from an information perspective: “we cannot interpret the interval except for its properties in the long run.” No: As noted just below that statement, the 95% interval can be interpreted for this single data analysis as the interval of all parameter values for which p>0.05. Thus the interval shows the parameter values for which the data supply less than log2(1/0.05) = 4.3 bits of information against them, given the model used to compute the interval. This interpretation does rely on the repeated-sampling property that, given the model, the random P across studies is uniform at the correct value; this property ensures that the S-value captures the refutational information in the test statistic (note that posterior predictive P-values are not uniform and thus do not sustain this interpretation).

The frequentist domination of statistics has caused the emphasis on randomization as a methodology to obscure what it was supposed to do in the first place: to create exchangeable observations for comparison.

Bernardo, J. M. (1996). The concept of exchangeability and its applications. Far East Journal of Mathematical Sciences, 4, 111-122. (PDF)

This confusion stems from the language in basic stats textbooks not being formally powerful enough to represent uncertainty about the entire experimental model, which can be done using hierarchical models, as Bernardo notes in his abstract (quoted below).

The general concept of exchangeability allows the more flexible modelling of most experimental setups. The representation theorems for exchangeable sequences of random variables establish that any coherent analysis of the information thus modelled requires the specification of a joint probability distribution on all the parameters involved, hence forcing a Bayesian approach. The concept of partial exchangeability provides a further refinement, by permitting appropriate modelling of related experimental setups, leading to coherent information integration by means of so-called hierarchical models. Recent applications of hierarchical models for combining information from similar experiments in education, medicine and psychology have been produced under the name of meta-analysis.

The concept of exchangeability is fundamental to the frequentist analysis of data using permutation tests.

Good, P. I. (2002). Extensions of the concept of exchangeability and their applications. Journal of Modern Applied Statistical Methods, 1(2), 34.

Extensions Of The Concept Of Exchangeability And Their Applications

I think Bernardo might be a bit extreme when he writes about coherence demanding a Bayesian approach, when he has done extensive study of reference Bayesian methods, that attempt to maximize the weight of information in the observed data, while still obtaining a proper probability distribution.

If we view any data collection effort as attempting to minimize all sources of uncertainty (ie aleatory and epistemic), we are faced with a multi-objective decision problem, which is widely known to have multiple solutions.

Multiple-criteria decision analysis

Related Threads

Random sampling versus random allocation/randomization- implications for p-value interpretation

The questions below are very basic and have probably been asked thousands of times by statistically-naive MDs- apologies in advance. Unfortunately, I keep getting hung up on them and can’t find clear answers in my reading. I’ll understand if there’s no way to explain these concepts in layman’s terms. 1. What exactly does a p value mean in the context of an RCT, given that “random sampling” from an underlying population of interest has not occurred? This is the definition of a p-value provide…

Using Bayesian statistics in observational studies

My understanding is that @f2harrell argues that the utilization of Bayesian statistics for the analysis of observational data would “protect us” against false results. What are the main mechanisms for this? Utilization of skeptical priors? What other advantages does the Bayesian approach bring to the table in the world of observational studies? Are there any papers or books (with convincing examples ) specifically dedicated to this topic?