Using Bayesian statistics in observational studies

Thanks R_cubed for the comments and citations. Subjective logics have been a big area in AI/CS and philosophy for at least the past half century. I think Ramsey and DeFinetti viewed probability as the normative form for subjective logic, which would make Bayesian bias analysis a subfield of subjective logic.

Unlike bias analysis though, the more general formalisms don’t appear to me to have made serious inroads into applied statistics (as opposed to AI/ML systems, where by the 1980s there were things like Pearl’s Bayes nets), perhaps because they don’t seem to lend themselves as easily to familiar software formats.

Not that bias analysis has made huge inroads. Its prototypical forms appeared in the 1940s-1970s (e.g., Berkson, Cornfield, Leamer) - exactly the era in which instead mindless significance testing came to overwhelm research output. And bias analysis still seems far removed from the mass of teaching and research. That is unsurprising given how even today, few researchers seem to grasp why a P-value is not “the probability that chance alone produced the association”, and few evince an accurate understanding of either frequentist or Bayesian methodology - as displayed by their misinterpreting P-values as if they were justified posterior credibilities and CIs as if they were justified credibility intervals.

You remarked that my criticism of reference (“objective”) Bayes seems to assume that an analysis based upon a reference posterior will be naively interpreted as definitive. That’s not my assumption so much as my observation from my experience. The same problem happens with “classical” statistical methods; it’s just that the Bayesian applications I see are no better.

For hypothetical ideal users (and maybe us here, but not most users) Fisherian-frequentist and reference-Bayesian methods each supply summaries of data information filtered through their assumed models, in the form of interval estimates. In typical med research their summaries are almost identical numerically, and reference-Bayes estimates can be viewed as bias-adjusted frequentist estimates (e.g., see Firth, Biometrika 1993).

The problem I see is that reference-Bayes results will get misinterpreted as supplying justified credibilities when they are no such thing in reality. The very existence of medical trials on a relation demonstrates there must be a lot of background (prior) information - enough to have gotten the trials approved and funded. This background voids the reference-Bayes output as supplying a scientific Bayesian inference, revealing it instead to be a model-based study-information summary of Bayesian instead of frequentist probability form. As long as these distinctions are made clear, I’m fine with reference Bayes.

3 Likes