Frequentist vs Bayesian debate applied to real-world data analysis: any philosophical insights?


I’ve been reading about the benefits of the Bayesian versus frequentist approach in clinical trials. However, I don’t know if there are any specific insights applicable to the real-world data scenario, with observational studies that have an increased risk of bias. What can be said about the Bayesian - frequentist debate that is specifically useful for observational studies? Does anyone have any kind of pretty philosophical ideas about it?



The attributes of Bayes that make the approach attractive for randomized experiments are also available for observational research. Here are a few:

  • Most users of frequentist methods do not realize that except in rare cases such as when normality and constant variance assumptions hold, p-values and confidence intervals are approximate. The most frequently used approximations, based on Wald statistics, become worse the more non-quadratic the log-likelihood function. The binary logistic model is a good case in point, and so are mixed effects models. Bayesian calculations are exact.
  • As detailed in Nate Silver’s The Signal and the Noise, a Bayesian approach provides a way to obtain actionable evidence when randomization is impossible. His beautiful example is the evidence for cigarette smoking causing lung cancer. None other than Ronald Fisher himself argued that one cannot learn from observational data in this context because “lung cancer could cause smoking”, so without randomization no conclusions could be made (he had significant consulting income from the tobacco industry). Silver showed how a Bayesian analysis that is very skeptical about the effect of smoking on lung cancer is still definitive, and would have probably made the US Surgeon General issue his warning 10 years sooner.
  • Bayesian methods allow for incorporation of non-data information, which for observational studies is even more important than for experiments. Think about all the studies claiming to show that food x is associated with health outcome y. Bayesian analysis can place extreme skepticism on such associations.
  • There’s always the property of Bayesian posterior probabilities that they are in predictive, actionable, mode rather than a backwards-information-flow mode.
  • The job of a Bayesian analysis is to reveal the hidden truth in the sense of what data generating mechanism generated the observed data. This mechanism may be a one-time event with no repetitions possible. Frequentists must envision an infinite repetition of the same experiment to get long-run operating characteristics. Bayesians are more interested in “what do we have now?”.
  • As Andrew Gelman has written, type S errors (getting the wrong sign on an effect) are real dangers, especially in observational research. Some of the risk of type S errors is due to effectively using a flat prior for the parameter of interest, either by being frequentist or by using a Bayesian non-informative prior. Informative priors that disfavor extreme values and completely rule out impossible values (e.g., smoking improving lung function; aging making you less susceptible to arthritis) improve the situation.


not sure if this fits ‘observational study’, but i read this paper recently: Handling Multiplicity in Neuroimaging through Bayesian Lenses with Hierarchical Modeling

It was attacked on pubpeer by one individual, but i thought the authors handled it well: pubpeer comments

but i’m interested in the Q and wish i knew more about it… I’ll read @f2harrell 's comments…



Wonderful answer indeed, thank you very much. A condensed lecture in a post. From this answer, I understand that one of the specific advantages of Bayesian analysis, applied specifically to real-world data (RWD), is that using priors offsets some of the biases inherent in observational studies. This could mean a specific advantage of Bayesian analysis in these type of designs. Right?
For example, in situations where there is a lot of evidence from RCTs, one could use the background information from these studies, and try to improve applicability to special subgroups (e.g., the elderly, or patients with special risk factors…) by applying Bayesian regressions to new RWD focusing on specific populations. What do you think?
PS: As an oncologist, I’ve been fascinated by the story of Silver and tobacco. I’m going to buy the book.



I like those thoughts. And I highly recommend Silver’s book. It’s both scholarly and highly readable.

1 Like