Regarding “if assumptions hold” it’s a much better accomplishment by Bayes than what frequentist analyses do: make the same model assumptions and still do not get accurate inferential quantities in not-huge samples.
There is nothing in the data to estimate those parameters unless you have a smaller RCT to include in the analysis. For the pure observational data case, estimation of bias parameters comes solely from the prior, and an observational researcher needs to be able to strongly defend the bounds put on the bias prior. It’s like a formal sensitivity analysis.
I do like the emphasis on practical demonstrations, but those who are trained only in frequentist methods or who have profited from “publishable p-values” are usually unswayed by even impressive practical demonstrations. That leaves another tactic: direct attacks on the very foundations of frequentist inference. For example frequentists embrace the notion of “error rates” but \alpha doesn’t have anything to do with error probabilities (and it’s not a rate anyway). An error probability is an unconditional probability in the sense that it does not condition on an unknown truth but must be based on what was known at the moment that a decision was made. An error probability would be the probability of making a mistake in rejecting H_0, i.e., P(\theta \leq 0) when p < 0.05 and \theta > 0 implies treatment benefit. Frequentist methods can’t provide that probability so it can’t provide error probabilities no matter how often frequentists use the term error. Controllling \alpha is not a noble aim, now or when it was first invented.
Likewise, we have serious problems with frequentist confidence limits as this topic has so well demonstrated.
I’ve tried to tiptoe around criticisms of frequentism for years but no longer. It’s time for subjective Bayesians to quit being apologetic, and it’s also time for the field of epidemiology to grow up.
The real issue (I agree with Sander here) is that many researchers are deeply entrenched in cognitive biases that misunderstanding and poor language (confidence and significance) allow them to maintain, such as ignoring model assumptions in interpretation of uncertainty intervals. These biases won’t disappear with theoretical critiques; they will persist until a better framework demonstrates clearly why they were thinking wrongly. Cognitive biases and assumptions are unavoidable in any inferential approach, but if we can demonstrate to clinicians in applications that Bayesian methods help reform the way we think and thereby make assumptions explicit and decisions clearer, then I agree with you, there is a real opportunity here for those interested in advocacy in this area. We all have priorities, and I must admit that solving cognitive biases through advocacy on inferential methods does not rank high on mine – perhaps because I have been involved in many other issues that I feel rank higher (e.g. the many problems with evidence synthesis or choice of relative effect measures etc)
There are more educational efforts to give, more volunteering to review for medical and epilepsy journals, … One project that might be influential though requiring a lot of work is to take a few debunked episodes findings and re-analyze them using more thoughtful and careful Bayesian approaches.
I think you can. Have you read any of Paul Rosenbaum’s work? That’s where I’m borrowing some of this. But given my understanding of why Randomization is used to justify Significance tests/Hypothesis Tests. If we can make a reasonable justification for the assumption of ignorability using the Design and Sensitivity Analyses. I think you can justify your Inferential Statistics.
Regarding sensitivity analysis, is the reader owed a sensitivity analysis of the priors themselves?
I’ve argued at length elsewhere that most of the time statistical inference is too premature to apply (or at least be take literally) for most situations, and observational studies would seem to be a superb example of this, regardless of which inferential framework is used. For example, doesn’t the likelihood principle (on which Bayesian inference is based) require prespecification of the model? How often can such prespecification be done with observational data?
I agree. That project will certainly make an impact if it produces results that clinicians can relate to. A lot of work is involved but also requires shared data.
Randomization is used to justify not having random samples, while getting high quality unbiased data. I don’t see how that idea spills over to observational data.
Yes there’s always model dependency, in equal doses to Bayesian and frequentist, then Bayes has prior dependency. Along the lines of a formal sensitivity analysis, we simulate the primary Bayesian operation characteristic (probability of making the right decision) under a sampling prior that may disagree with the analysis prior.
Regarding model dependencyt, there are parameters that really matter for robustness of main findings, and parameters/model misspecification that don’t.
Reply to Frank:
Randomization is used to justify Significance/Hypothesis Tests. This goes back to R A Fisher’s Design of Experiments book. It spills over to Observational Studies because the goal is to design it as if it were an Randomized Experiment. Using Observational methods, we can argue how far away that was.
Chris’ point that any method for presenting observational study findings will be fraught with assumptions is well-taken. But observational research will continue to be published and it might not be realistic to expect that it will only be published following an exhaustive triangulation of multiple lines of evidence. Until robust triangulation (an evidentiary “web”) is achieved, a top priority should be to ensure that clinicians and patients don’t allow important medical decisions to be unduly impacted by weak and highly uncertain threads of observational evidence. To this end, it seems like a subjective Bayesian approach to observational research could be a significant improvement over the status quo frequentist approach.
Bayesian presentations of observational study findings could contain a detailed description of how researchers arrived at their preferred prior. Mandatory performance of a sensitivity analysis, including assessment of how a highly skeptical prior would affect the results, would allow research consumers to downgrade their assessment of the incremental evidentiary impact of the study if they suspected that it was a lot more exploratory than the authors are admitting (e.g., if HARK’ing is suspected).
In addition to considering a switch to Bayesian presentation of observational study findings, it also seems important to consider the role that modern DAG-based epidemiologic methods might realistically play in medicine. Pavlos has shown how they can be used in elegant ways to improve the design and interpretation of complex oncology RCTs involving sequential therapeutic decisions. They can also be used to criticize, retroactively, the design of observational studies that were performed without prior construct of a DAG- but this is a destructive function, rather than a constructive one. We need to emphasize, bluntly, that clinicians are unlikely, in most situations, to consider observational studies flagging therapeutic harm signals- even those designed with DAGs- to be sufficient, in isolation, to alter clinical decision-making. Confronted with a symptomatic patient, and in possession of a therapy that trials have shown might offer relief, physicians are unlikely to allow highly uncertain observational harm signals for that same therapy to alter a decision to treat. In situations where therapeutic efficacy is unclear and/or other efficacious therapeutic options exist, the risk/benefit calculus might be different. For asymptomatic patients, observational evidence of possible therapeutic harm will need to be weighed against the potential risk of a more important adverse clinical outcome in the absence of treatment. Whereas a poorly-constructed (i.e., DAG-free) observational study will usually stand no chance of contributing meaningfully to an accumulating evidence base or to clinical risk/benefit assessments in asymptomatic patients, studies that use more rigorous observational methods might at least stand a chance of contributing to eventual evidence triangulation or decision-making in situations where potential therapeutic benefits are more unclear.
Observational evidence can and should play a role in clinical decision-making. But observational researchers need to accept that the clinical impact of their work is going to be much more modest than they have historically expected. And that impact will be determined most responsibly by bodies of experts that generate clinical practice guidelines, not by individual epidemiologists grabbing a bullhorn. Since the results of any given study will not likely be considered compelling enough to alter clinical decision-making (no matter how vigilant the design), individual studies should not be presented to clinicians and patients with this expectation in mind. But, if conducted rigorously, the study could contribute meaningfully to future triangulation of many lines of evidence.
If the goal is to design an observational study as if it was a randomized trial, that has very little to do with the OS being treatable as a randomized study. So I missed that point. On a related note, an OS study cannot be said to mimic a clinical trial if it’s not fully prospective with protocolized data collection as all RCTs do.
Contrast this discussion with the classic paper Strong Inference by Platt in 1964. The amount of work that good experimentalists have to do to triangulate a scientific finding is huge. Why have we always made it so easy on observational researchers?
To Frank: I don’t think I follow what you said. So perhpas you could explain to me what you think of when someone says, design an Observational Study as if it were a randomized experiment. Because no one would say that means that Observational Studies are mimicing the data collection part of an experiment. That wouldn’t make sense.
What domains are you thinking of when you say things are easier for Observational Researchers? Outside of obvious stuff like data collection. And what do you mean by that? I’m sure we can all pull our favorite examples of Observational Studies that have little point in being done in the first place(especially the ones that call themselves Observational Studies with zero interest in a causal research question). But there are also some very clever ones that you’ll find in Paul Rossenbaum’s Design of Observational Studies book.
Given the ongoing discussion I realized that what has been misunderstood is that we are not talking about descriptive studies here where, when the target of estimation is off by a wide margin, the UI becomes uninterpretable. In an analytical study (observational or experimental) there is no problem if the target of estimation is completely off the population value, so long as it is similarly off target in both groups. That is why random sampling of groups is much less important than comparability of groups (aka internal validity). I have now updated the Table again to make this point very clear.
Decision scenarios for interpreting uncertainty intervals in analytical studies (observational and experimental)
Interval type
Practical meaning
Driven by probability model (precision)
Driven by statistical model (validity)
Clinical utility
Note on group targets (external relevance)
Typical study designs
Imprecise (any plausibility)
Wide interval → high uncertainty; many population models could explain the data.
Low information → interval wide.
Validity of assumptions matters little because imprecision dominates.
Rarely useful for clinical decisions; mainly contributes to cumulative evidence.
Even if groups are off-target, this adds little since the estimate is already uninformative.
Small RCTs, underpowered cohorts, rare-event case–control studies.
Precise + Comparison plausible (internally valid)
Narrow interval, and the two groups are sufficiently comparable that the observed difference plausibly reflects the intervention effect.
High information → interval narrow.
Assumptions of the statistical model are credible → groups comparable except for intervention.
High utility: provides a trustworthy estimate of the effect within the study. Useful for decisions once consistent with other evidence.
The groups may still not represent the true target population (external validity issue), so generalization requires caution.
Well-conducted RCTs; well-designed observational studies with strong design and analysis where biases affect both groups similarly.
Narrow interval, but the groups are not comparable, so the observed difference cannot be attributed confidently to the intervention.
High information → interval narrow.
Assumptions of the statistical model fail → groups differ in bias structure, comparison distorted.
Misleading: creates false confidence in the wrong effect estimate. Should not guide care until cumulative evidence resolves the bias.
Groups may also be off-target, further compounding the problem.
Poorly designed and executed observational studies; RCTs with major flaws (e.g., inadequate concealment, differential measurement error, selective loss to follow-up).
I was merely saying that pretending to mimic a setting for which frequentist statistics is supported does not make it so. This strikes me as the emperor’s new clothes …
Comparing with Platt’s approach reveals the answer. Briefly, I see very few OS posit alternative explanations or even both to do very easy things such as simulating the effect of an unmeasured confounder on the key results. OS researchers have gotten off very easy compare to experimentalists, not even doing basic things that don’t require experiments.
I could only read the first page, but it is no surprise that Platt was a physicist. I suspect that a study of the history of science (or science + engineering) would do more to teach people how to reason with data than a course on statistical inference, which too often is used to short circuit the process of triangulation. As George Box noted, the Wright brothers did lots of experiments, triangulated like mad, but never calculated a p-value, UI, Bayes factor, etc.