Who should define reasonable priors and how?

[R_cubed]

I thought I’d bump this thread after finding a valuable discussion between Andrew Gelman and Sander Greenland on this precise issue of setting priors.

Specifying a prior distribution for a clinical trial: What would Sander Greenland do?
https://statmodeling.stat.columbia.edu/2008/02/05/specifying_a_pr_1/

Specifying a prior distribution for a clinical trial
https://statmodeling.stat.columbia.edu/2008/01/24/specifying_a_pr/

My posts in this thread were some speculations I had about how 2 honest Bayesian scientists might negotiate in order to derive an experiment that would settle a question of fact, that is in dispute.

In the real world information is not free. So it would be important for parties of the dispute to know how much it would cost to run this (hopefully) definitive experiment (ie expected number of observations), and if that new information is worth paying for. In the decision theory and economic perspectives, this is known as Value of Information.

Value of information

This is the role I see for retrospective meta-analyses – constraining the family of possible informative priors in order to decide on what future experiments (broadly defined, also including observational studies) to perform, to settle disagreements on fact and to aid in selecting a policy.

In Greenland’s terminology, he would call them “covering priors”, a concept I think has substantial merit.

Relevant Discussions

Is it sensible to consider meta-analysis the strongest level of evidence for or against an intervention?

i was reading a bayes re-analysis yesterday that did this, it might not be the best example tho: bayes re-analysis, ECMO RCT. They say: “data-derived prior distributions were developed based on relevant studies18-20 from a meta-analysis of ECMO for ARDS.21 The treatment effects in these previous studies were combined with the observed data from this trial in a Bayesian hierarchical random-effects model (that itself used minimally informative priors)” they make too much of the convention of …

2019 Nobel Prize in Economic Sciences is about limitations of RCT

This article summarizes the 2019 Nobel Price in Economics" What randomisation can and cannot do: The 2019 Nobel Prize Their critique of randomized controlled trials (RCT) is three-fold: They make no specific claim about external validity. In economics it requires a pre-existing system the study is embedded within. In a plumbing metaphor, the studies focus on problems in existing plumbing–not upon how to engineer and install a system. Optimal experimental design is generally not an RCT. From …

A Ph.D dissertation by a recent Vanderbilt graduate who was supervised by both Dr. Harrell and Dr. Blume. It is likely that rational scientists, in fields with small samples (especially my field of rehabilitation and physical disabilities) would implement some sort of adaptive randomization to balance the need for efficiency with the need for robustness (ie credibility).

I don’t remember the dissertation going into actual details, but AFAIK, it shouldn’t be hard to prove this type of design is admissible for both parties.

Chipman, Jonathan Joseph (2019) Sequential Rematched Randomization and Adaptive Monitoring with the Second-Generation p-Value to Increase the Efficiency and Efficacy of Randomized Clinical Trials. Vanderbilt University, ProQuest Dissertations Publishing, 2019. 13900836.

http://etd.library.vanderbilt.edu/available/etd-06142019-180645/unrestricted/_main.pdf