Clinician and very amateur stats nerd here, trying to take in the BBR course one step at a time. I wanted to ask a question that was inspired by the discussions around the course but doesn’t directly relate to any chapter and therefore decided to open a separate thread.
My question is regarding the definition of reasonable/realistic priors for Bayesian analysis. What triggered me was the discussion on David Spiegelhalter’s blog between him and John Ioannidis (see here) regarding the Bayesian re-analysis of the ANDROMEDA study. Ioannidis argues that the skeptical priors used in the analysis were not skeptical enough. Thus the question arises, who could decide on what skeptical (or optimistic) enough is and more importantly, how?
The one who definitely cannot be expected to decide on this is the reader who in most cases lack both statistical and subject-matter knowledge to answer this. However, I got the impression that probably neither Ioannidis or Spiegelhalter are optimally suited for it either, unless they possess in-depth subject matter knowledge about septic shock, which they might. I guess the onus is on the journal editor to make sure priors, as other statistical assumptions are reasonable, however I feel that this fact might be one big impetus for journals to accept non-frequentist statistical methods. Still, I feel that maybe the opinion of an editor and 2-3 reviewers might still be called into question by other experts.
I am wondering about the possibility of constructing a Delphi-like process which could help subject matter experts reach consensus about reasonable priors. Professional societies would be, in my view optimally suited to deliver such consensus-based recommendations for prior distributions to be used in their field, as they do with other similar recommendations that are based on expert opinion. However I believe that these societies would need a sort of handbook or guideline as to how to conduct such a process. Maybe that could be something that the statistician community could contribute with? What do you think? Maybe this is something that already exists?
Áron Kerényi, MD PhD