ABCLOC: bootstrap method for overfitting-corrected model performance metrics

data analysis model validation
1

@f2harrell, I want to apply this method in my project and I want to double-check if I understood your conclusions correctly:

The “sd2rev wtd4” method was the best. This method can be described as:

  • For each bootstrap iteration b=1,\dots,B, we computed a variate V_b:
    • V_b = bootstrap-sample performance (P_{boot}) - 1.25 \times original-sample performance (P_{test})
  • Calculate the lower and upper standard deviations of these variates (SE_{lower}, SE_{upper}) using Hmisc::dualSD

If possible, I am interested in constructing the 95% confidence intervals for the optimism-corrected estimate. How can we derive it from SE_{lower}, SE_{upper}? Maybe using the standard normal approximation -1.96 \times SE_{lower}, + 1.96 \times SE_{upper} ? This doesn’t seem correct though given the assymetrical assumtion of dualSD.

Here is a simulated example applying the method above: ABCLOC "sd2rev wtd4" method · GitHub

Is it correct? Thanks

1 Like
2

Look at the predab.resample R code which has the specifics. Note how it reverses top and bottom limits.

3

Exactly what I was looking for. Thank you very much.

I suggest you to mention in the blog post you apply the method in this function

4

That’s in the help file for calibrate.orm and related validation functions.

1 Like