Since we are discussing papers by @Sander, I’d like to add a commentary he wrote for Statistical Science that discusses the application of both “Frequentist” and “Bayesian” concepts in a reasonable way. A number of sections touch directly upon the question asked in the original post.
Greenland, S. (2010). Comment: The need for syncretism in applied statistics. Statistical Science, 25(2), 158-161. link
He discusses how hierarchical models provide a framework for integrating the considerations of prior information that motivate a particular investigation, with the methodological concerns for study designs that have the characteristics that a frequentist would accept (proper coverage of interval estimates or prediction intervals in repeated use).
The topic of hierarchical models is very important; I’d claim all of Sander’s insights and experience have been formalized by the engineers, computer scientists, and logicians who work in the area known as Subjective logic, which is an adaptation of Dempster-Shafer belief functions so they are compatible with the probability calculus. Applications include diverse areas from electronic sensor fusion to intelligence analysis.
Oren, N., Norman, T. J., & Preece, A. (2007). Subjective logic and arguing with evidence. Artificial Intelligence, 171(10-15), 838-854. link
From the link on Subjective Logic:
Subjective logic is a calculus for probabilities expressed with degrees of epistemic uncertainty where sources of evidence can have varying degrees of trustworthiness. In general, subjective logic is suitable for modeling and analysing situations where information sources are relatively unreliable and the information they provide is expressed with degrees of uncertainty. For example, it can be used for modeling subjective trust networks and subjective Bayesian networks, and for intelligence analysis.
The entire literature on statistics makes it seem as if there are distinct techniques of “Bayesian” statistics and “Frequentist” statistics. As Sander’s papers demonstrate, this is an illusion for any practical problem.
After much reading on the topic, I’d hope the nomenclature of “Frequentist vs Bayesian” disappears, and focuses on the real area of disagreement – how much weight to place on prior (often qualitative, imprecise, but critically relevant) information with the information obtained from a data collection protocol.
Bayesians and “Frequentists” disagree on how prior information should be used to derive a data collection protocol, with Frequentist designs often formally ignoring prior information.
This difference of opinion between proponents of a model, and skeptics, can be given a game theoretic interpretation, which also happens to be an algorithm to derive a decision procedure that will settle the dispute.
Game Theoretic Probability and Statistics
A complementary method of bridging the gap between Bayesian and “Frequentists” is the Objective Bayes perspective, with the most work I am familiar with being done by James Berger and Jose Bernardo. Papers can be found under the term Reference Bayes or Reference Analysis.
A reference posterior distribution is an effort to obtain a posterior distribution, using mathematics alone, that has interpretations that would satisfy a Frequentist (who is concerned about the frequency coverage of the interval generated by data collection procedure) and mostly satisfy a Bayesian (who is concerned about the information contained in the experiment and how it relates to the scientific questions).
A more precise, technical definition of a reference prior (used to produce an Objective Bayes posterior) is:
Reference analysis uses information-theoretical concepts to make precise the
idea of an objective prior which should be maximally dominated by the data, in
the sense of maximizing the missing information (to be precisely defined later)
about the parameter.
Sander has this to say about Reference Bayes:
Accuracy of computation becomes secondary to prior specification, which is too often neglected under the rubric of “objective Bayes” (a.k.a. “please don’t bother me with the science” Bayes).
My question to Sander:
The criticism of Objective Bayes seems to assume that an analysis based upon a reference posterior will naively be interpreted as definitive.
This isn’t necessarily so. Why can’t reference posteriors be used as inputs to the probabilistic bias analysis you described in the numerous papers cited? The cognitive science literature done in the context of training intelligence analysts strongly suggests expressing uncertainty so that it obeys the probability calculus improves reasoning ability. Why can’t there be room for both an “Objective Bayes Posterior” and a “Subjective Bayesian” probabilistic bias analysis? As Senn has noted: “Bayesians are free to disagree with each other.”
Karvetski, C. W., Mandel, D. R., & Irwin, D. (2020). Improving probability judgment in intelligence analysis: From structured analysis to statistical aggregation. Risk Analysis, 40(5), 1040-1057. link
As I had pointed out in other threads, it is often useful to see a consensus predictive distribution to determine the value of your own subjective knowledge. This occurs routinely in financial valuation contexts.