How to interpret “confidence intervals” in observational studies

222

Chris- thanks for the link to the 1990 Greenland paper. I had read it some time ago but forgotten about it. It’s just great. Arguably, it would be hard to find another paper that speaks so directly to the question at the heart of this thread. I’ll have to read it several more times to understand it properly, but I think that the effort will be worthwhile.

I see what Suhail is saying here; this ostensible inconsistency has caught my attention as well.

At first glance, it appears that Dr.Greenland’s views might have changed over the years. Several of his earlier papers highlight that confidence intervals presented in observational studies are difficult (if not impossible) to interpret given the absence of random sampling or randomization. Those papers propose that confidence intervals, if presented in the observational context, should be viewed as reflecting the “minimum” degree of uncertainty in the point estimate- a scenario that would only exist if some very unrealistic conditions were in play (e.g., random sampling, which doesn’t occur in clinical observational research). In recent years, however, it appears (to me at least) that his emphasis has shifted. Now, instead of emphasizing the importance of being adequately restrained when interpreting observational study results, he seems to emphasize the importance of not being overly restrained. Specifically, he’s talked a lot in recent years about the perils of “nullism” (i.e., the tendency to disregard potential therapeutic harm signals simply because their confidence intervals cross the null).

It’s challenging to reconcile this apparent shift in focus over time; but, at my own peril, I’ll take a shot at trying to explain it. I expect that top epidemiology experts are acutely aware that many decades of recklessly over-interpreted observational research, churned out by under-qualified researchers, has resulted in few people taking observational evidence seriously these days. And so, in an effort to move the pendulum back in the other direction, he is now cautioning research consumers about the risk of adopting an overly nihilistic view of observational evidence (i.e., disregarding point estimates that could reflect potentially important public health effects, but which lie within confidence intervals that cross the null). Given his apparent agreement, back in 1995, regarding the challenges inherent in interpreting small relative effects (https://www.science.org/doi/pdf/10.1126/science.7618077), I find his more recent exhortations to avoid under-interpreting small effects a bit hard to understand…

A couple of quotes from the 1995 article:

“As a result, most epidemiologists interviewed by Science said they would not take seriously a single study reporting a new potential cause of cancer unless it reported that exposure to the agent in question increased a person’s risk by at least a factor of 3-which is to say it carries a risk ratio of 3. Even then, they say, skepticism is in order unless the study was very large and extremely well done and biological data support the hypothesized link. Sander Greenland, a University of California, Los Angeles, epidemiologist, says a study reporting a twofold increased risk might then be worth taking seriously-“but not that seriously.”

But later:

“There’s nothing sinful about checking for confounding variables. The sin comes in believing a causal hypothesis is true because your study came up with a positive result, or believing the opposite because your study was negative."

While his earlier papers seem to focus on the first listed “sin” (i.e., believing a causal hypothesis is true…), his primary concern in more recent years seems to be reflected in the bolded phrase.

While maybe agreeing in theory with the bolded phrase, many clinicians will disagree with it in practice. Any suggestion, in clinical journals or the media, that clinicians should “act” on weak, uncertain potential harm signals acknowledges only the numerator in the the therapeutic risk/benefit ratio. Epidemiologists rush to trumpet the numerator in spite of the fact that they are not trained to understand the denominator. But it’s the ratio that clinicians need to consider when advising their patients. And those who don’t deeply understand the ratio are in no position to provide clinical advice.

2 Likes
223

Fisher’s 1922 paper was shockingly lackadaisical when it came to random sampling:

It should be noted that there is no falsehood in interpreting any set of independent
measurements as a random sample from an infinite population ; foi any such set of
numbers are a random sample from the totality of numbers produced by the same
matrix of causal conditions : the hypothetical population which we are studying is an
aspect of the totality of the effects of these conditions, of whatever nature they may be.
The postulate of randomness thus resolves itself into the question, “ Of what population
is this a random sample ? ” which must frequently be asked by every practical statistician.

Of course, Fisher was not a frequentist. It was Neyman’s 1934 paper that emphasized the importance of random sampling: “Random sampling means the method of including in the sample single elements of the population with equal chances for each element.” He then generalized this concept to cases such as stratified sampling where the probabilities of selection are equal within the strata, etc.

When it came to randomized trials, Fisher was more rigorous in 1935 (Design of Experiments; I quote from the 8th edition of 1966). In explaining the Lady Tasting Tea, he wrote:

The phrase “random order” itself, however, must be regarded as an incomplete instruction, standing as a kind of shorthand symbol for the full procedure of randomisation, by which the validity of the test of significance may be guaranteed against corruption by the causes of disturbance which have not been eliminated.

In explaining an agricultural experiment in randomized blocks, he wrote about “assignment made at random”:

This does not mean that the experimenter writes down the names of the varieties, or letters standing for them, in any order that may occur to him, but that he carries out a physical experimental process of randomisation, using means which shall ensure that each variety has an equal chance of being tested on any particular plot of ground. A satisfactory method is to use a pack of cards numbered from 1 to 100, and to arrange them in random order by repeated shuffling.

See p. 567 of Neyman’s paper, and Ch. 2, Secs. 9-10, and Ch. 4, Secs. 22 & 26 of Fisher’s Design of Experiments, 8th ed., for further discussion.

3 Likes
224

At the moment I also don’t quite understand everything in the 1990 paper either. Specifically, one thing I don’t quite get yet is the discussion of “nonprobabilistic” interpretations of statistical inferences.

3 Likes
225

I guess I would read his various positions as a common critique of statistical inference methods from single observational studies. A statistically significant result doesn’t make it “true” any more than a nonsignificant result makes it “false”. He seems to be saying publish the results on the basis of quality of study design, execution, and scientific relevance, with whatever uncertainties are found (regardless of significance or nonsignificance), allowing the work to be added to the “synthesis of all relevant evidence”. https://sites.stat.columbia.edu/gelman/research/unpublished/Amrhein_Gelman_Greenland_McShane2019.pdf

3 Likes
226

I was surprised by Frank’s position rather than that of Sander who has clearly changed position since that paper 35 years ago and the fact that this is all that can be cited is telling.

Gelman in his book on regression says that even if data have not been collected by any random process, for statistical inference it is helpful to assume some probability model for the data. For example, in Section 7.1 we fit and interpret a regression predicting presidential election outcomes from the national economy. The 16 elections in our dataset are not a random sample from any larger set of elections, nor are elections the result of some random process. Nonetheless, we assume the model yi = a + bxi + ei and work out the sampling distribution implied from that. The sampling distribution is said to be a generative model in that it represents a random process which, if known, could generate a new dataset. Next we discuss how to use the sampling distribution to define the statistical properties of estimates

It was not at all lackadaisical - it was spot on. Fisher’s point is that every study is just one set of observations from a much larger universe of possible outcomes, and statistics works by asking: “What bigger group or process do these patients stand for?” This is exactly in keeping with Gelman’s framing of a generative model that lets us simulate or imagine new datasets under the same assumptions. Any dataset can be thought of as part of something bigger. It means the hypothetical set of all possible outcomes that could be produced by the same causes. Randomness is just a way of saying: outcomes could have been different. If we ran the same study again with a new group of patients, the results wouldn’t be identical — not because of error or mistakes, but because biological and clinical responses naturally vary. The real question is always: “What larger group or process am I assuming my patients represent?” That’s what a researcher must make clear before interpreting results.

227

To conceive of generating a new dataset (required only in the frequentist world) you have to have some idea of how persons got into your dataset and which biases are present, understood, and are replicable. Many OS do not qualify in these ways. The “inference” from these OS is ill-defined in the first place.

1 Like
228

I agree that we have to have some idea of how persons got into our dataset and which biases are present, understood, and are replicable etc. However I will ask you to share your thoughts on what Fisher said: “it should be noted that there is no falsehood in interpreting any set of independent measurements as a random sample from an infinite population ; for any such set of numbers are a random sample from the totality of numbers produced by the same matrix of causal conditions : the hypothetical population which we are studying is an aspect of the totality of the effects of these conditions, of whatever nature they may be”. Clearly, this matrix of causal conditions applies to the statistical model and the method of estimation derived from it (estimator) and its target of estimation need not be the population we have in mind if the matrix of causal conditions do not fit this population. However, there must be some other population (in the words of Fisher “Of what population is this a random sample?”) of which data collected without any clear probability sampling are a random sample of and the UI is then an interval of probability models (or test hypotheses) compatible with the data - one of which may align with this other population. Do you disagree with this? If so, why?
NB: last sentence edited for clarity

229

It sounds too much like a rationalization and is not convincing. He clearly envisioned studies in which you could apply the results outside the study, and the word ‘infinite’ is also problematic. Most OS’s do not qualify in these ways. Many OS’s are so problematic that even if they contained the entire world population we would not know what to make of the results.

Coffee drinking caused pancreatic cancer only for a few days and the epidemiologist authors (McManus et al JAMA 1980) were not even rational enough to realize there weren’t enough pancreatic cancer cases worldwide for it to be caused by something as common as coffee drinking.

I am doubtful that I will benefit from further interchanges on this issue. This strikes me as going to extraordinary lengths not to be Bayesian.

2 Likes
230

I too, am going to “bow out” after this reply. I don’t think I have anything more to add to the thread at this point. Also, I have too much other work to do.

You seem to be highlighting my lack of epidemiology training and therefore scholarship. I acknowledge both freely. Yes, a lot can change in 30 years (including experts’ opinions), and I’m going to be ignorant of 99.9% of these changes because this isn’t my field. The last thing I want to do is go off half-cocked about topics I don’t understand. Maybe this has already happened in this thread and I’ve made a fool out of myself by continuing this dialogue- others can be the judge. But, so far, the answers I’ve received to the question in the original post are anything but reassuring. I wasn’t playing “gotcha” here, Suhail- I was trying to understand. And I don’t understand why any change in the field of epidemiology over the past 30 years (including DAG-based methods and the “causal revolution”) would negate the fact that the interpretability of confidence intervals hinges on randomness being present- and it’s not present in observational research (though you clearly feel that it is present in some very contorted sense of the word).

Maybe the question in the original post sounds naïve and simplistic to you. But I guarantee you that I’m not the only student of epi/stats who has sought an answer to it.

2 Likes
231

Okay fair enough - we have all said what we wanted to - there is a lot of information on this thread and readers can decide where they go from here…

2 Likes
232

Perhaps you are right … I might make the leap one day like you did when I come across a need …

2 Likes
233

I fear opening a can of worms and I don’t want to derail this thread, but the Bayesian world requires the likelihood. When I try to think about the “probability of the data [given some value of the parameters]”, to me this implies some unpredictable process that takes those parameters and is capable of generating a dataset according to the given probability model. Whether that actually occurred only once or an infinite number of times doesn’t matter: I can’t think of what a likelihood of (e.g.) 0.2 could mean except that this (hypothetical) data generating process would produce that dataset about 20% of the time. It is very possible that I am suffering from a lack of imagination.

I am very receptive to the Bayesian interpretation of probability, and having seen Bayesians advocate for the “generative model” approach towards defining the likelihood (including Gelman’s “…even if the data have not been collected by any random process, for statistical inference it is helpful to assume some probability model for the data…” quoted above), I struggle to imagine how you might define a likelihood without invoking the question of “what datasets could we have seen, given some underlying parameter values?” So I am not sure why this conception is required only in the frequentist world.

Have I completely missed the point? Besides the obvious (that one would typically be a hypothetical simplified model for the other), is there some fundamental difference between an unknown data generating process that did produce the observed data and the probability model that is encoded in the likelihood that implicitly could produce a new dataset? If, as has seemingly been suggested in this thread, we contend that there is no randomness at all in the former in an observational study, then I can’t see how a Bayesian analysis would help either.

3 Likes
234

The Fisher quote seems to be suggesting we treat a nonrandom sample as if it were random, and in fact this is often done in laboratory settings, such as (designed) assay validation studies, where other sources of variation (systematic and random) are explicitly built into the study design. When the technology is well understood, there is probably little harm. However for survey sampling, the level of control achievable in the lab is not present. However when it comes to comparative studies, even Fisher demanded an explicit random assignment procedure (shuffling a deck of cards) and he relentlessly attacked the observational studies used to link smoking and lung cancer because (among other things) of their lack of randomization.

2 Likes
235

Your questions are excellent ones and I hope others will respond more cogently than my attempt here. I return to a neuroscience example to illustrate that Bayes is about uncovering hidden truths that are in operation to produce the one dataset of interest. We observe a specific brain stimulation when a specific stimulus its given, no inference needed when this can be replicated within a person. When several stimuli are active and so are several brain activations, Bayes’ rules is used to attempt to trace back the stimulus that caused a specific activation.

The likelihood function just allows us to vary the unknown truths and to compute the relative likelihood that the observed data would have been observed under each of an infinite array of unknowns truths. The simplest application of the likelihood is what Richard Royall championed, e.g., compute the ratio of likelihoods of \theta = 0 vs. \theta = 3 to gauge the relative evidence for the true \theta being 0 or 3 given the one observed dataset, no sampling distributions required.

So the likelihood in considering the chances that the one observed dataset would have been observed under different hidden truths is devoid of any “data process” and concentrates on the “\theta process”. In the likelihoodist school of statistics, the \theta process is unfiltered but provides only relative evidence. With Bayes, the \theta process is filtered/focused by the prior distribution to provide absolute evidence (actual probability that \theta is in some interval specified by the user).

236

And we know today that his explanation for the relationship was wrong. Thanks to Sensitivity Analysis.

237

Because my thoughts on this are mainly from what I’ve read from Paul Rosenbaum’s work, i decided to email him. I summarized what he said here:

Observational studies lack the independence between treatment assignment and potential outcomes that randomization secures. Frequentist approaches interpret them by positing the role randomization would have played, providing a benchmark of balance and independence and then attempt to approximate it via design or analysis-based methods (matching, stratification, propensity scores, etc.). Without this reference to randomization, causal inference risks degenerating into unchecked association. Rubin’s potential outcomes framework circumvents Fisher’s randomization-based mathematics by reworking the model of treatment assignment and outcomes, but this introduces new technical challenges such as the dependence structure of potential outcomes.

1 Like
238

This is very helpful. The term “would have played” is the heart of the problem. It’s as severe an assumption as saying that a PS contained all the confounders.

1 Like
239

The problem being that the Propensity score is only conditioned on observed confounders. Right?

240

That’s only part of the point I was trying to make. The condition that PR said was necessary is equivalent in severity to a researcher stating without evidence that PS was adequate, i.e., that it included all confounders.

1 Like
241

That’s where Sensitivity Analysis comes in right?