In the clinical prediction model framework, many suggest doing internal validation after model development to quantify and adjust for optimism/overfitting. The most recommended procedure is doing non-parametric bootstrapping 200-500 times.
However, doing such a procedure with a Bayesian model and a large dataset is infeasible due to computational burden. In the Bayesian literature, model checking with posterior predictive checks and even leave-one-out cross-validation are commonly recommended. Yet I have not yet seen these procedures in the clinical prediction literature.
How should one internally validate a Bayesian clinical prediction model?
2 Likes
Once the LLM bubble bursts, there will be plenty of spare compute for this.
2 Likes
Bayesian predictive checks and leave-out-one log-likelihood-based methods need to be featured in the new Bayesian workflow, and someone needs to write clinical papers to make this understandable. Bigger picture: Bayes doesn’t have overfitting in one sense; it just has priors that were too wide in the opinion of an outside critic. The real work of Bayes in this setting is specifying the priors. the new work on R2D2 priors provides one path for doing this in a global fashion so as to no have to specify priors on individual predictor effects.
2 Likes
I found this by @avehtari insightful:
2 Likes
I agree that there really is a need for guidance @f2harrell . Your book does a tremendous job for giving applied researchers a blueprint for frequentist model development, but I too am I bit at a loss for a Bayesian prediction model workflow.
1 Like
Aki’s work is definitely the first place to look, but keep in mind the importance of the priors. If you have a flat prior for many regression coefficients you are saying that highly extreme predictions are likely. Someone else might say that’s overfitting but you would have to view it as expected given the model/prior specification. Of course that specification would be silly but it’s the default for many. I’d like to have an option in the rmsb package so that you could easily set priors on interquartile-range covariate effects, which makes things easier whether you spline variables or not.