Thanks ADA for the perceptive comments!
One of your comments raised concerns for me:
“One of my perceived advantages of Bayesian reporting would be the natural replacement of NHST p-values with a posterior probability that seems more intuitively useful in context of the clinical question. Isn’t it more appealing to report results as “based on the observed difference between groups, there’s a 93% posterior probability that Drug>Placebo” as opposed to “there was a 7% chance of the observed difference between groups if Drug = Placebo” (numbers made up, of course)”
Yes that’s a perceived advantage of Bayes, but “perceived” does not mean genuine. “Numbers made up” is the matter of course for Bayes since one has to make up a prior. My biggest fear for Bayes is the tendency to take phrases like “there’s a 93% posterior probability …” as if 93% were a singular quantity: There’s an infinite number of posterior probabilities, depending on the prior. So at best you could report “Using our prior and data-generation model (DGM) there’s a 93% posterior probability”; but thanks to the prior that will be contestable (and should be contested) in even more ways than the P-value (which is also sensitive to the DGM).
The prior opens up a whole new avenue for biasing and gaming the results. When told to switch from P-values to confidence intervals (CIs), many researchers mastered the art of hacking the intervals to include the hypotheses they liked and exclude the hypotheses they didn’t like; others simply learned to describe CIs to focus the reader toward desirable and away from undesirable results. With the prior in your control it’s even easier to hack posterior intervals than CIs, especially to include nulls (few question null-centered priors even when they abhor NHST); plus, all the biased-description skills learned for P-values and CIs transfer wholesale to posterior probabilities and intervals.
That said, I have long taught and promoted understanding of Bayesian methods, have published studies that used them, and think Bayesian perspectives on methods are as essential as frequentist perspectives. But while Bayesian methods can be an invaluable supplement to simpler methods, I doubt they are a safe substitute for those methods: As I see it, the problems of human bias that plague analyses and reporting are simply too deep to be addressed by a switch to Bayes or any statistical method.
6 Likes