Statistically Efficient Ways to Quantify Added Predictive Value of New Measurements

[R_cubed]

@Claus_G

I wish to compare the classical with the AI model. I run a training/validating process and then test the performance on an independent holdout test set.

If I understand your process correctly, I think you will need to seriously evaluate if you have enough observations to do the validation by holding out data. Much of the recommendations of @f2harrell are based upon important papers by Julian Faraway:

Faraway, J. J. (1992). On the cost of data analysis. Journal of Computational and Graphical Statistics, 1(3), 213-229. (PDF)

A complete discussion on the use of split data methods must also include:

Faraway, J. J. (2016). Does data splitting improve prediction?. Statistics and computing, 26, 49-60. (PDF)

At least for medical applications, a strong argument can be made that split data approaches lose too much information relative to model specification and bootstrap validation.

Split-Sample Model Validation – Statistical Thinking

Related thread:

Rigor and Reproducibility in a Research Network

I found this interesting paper by Julian Faraway that describes a more nuanced perspective on the issue of split sample vs resampling, that I offer up for discussion. From reading the paper, it seems as if there are more areas of agreement than disagreement with the opinions expressed in this thread, but I’m curious if the preferences for resampling might be modified slightly after reading this. Faraway, J. J. (2016). Does data splitting improve prediction?. Statistics and computing, 26, 49-60…