# ABCLOC: bootstrap method for overfitting-corrected model performance metrics

**URL:** <https://discourse.datamethods.org/t/abcloc-bootstrap-method-for-overfitting-corrected-model-performance-metrics/28632>\
**Category:** model validation\
**Created:** [February 5, 2026, 5:56pm UTC](https://discourse.datamethods.org/t/abcloc-bootstrap-method-for-overfitting-corrected-model-performance-metrics/28632 "2026-02-05T17:56:52Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![arthur\_albuquerque](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/arthur_albuquerque/32/3269_2.png) [@arthur\_albuquerque](https://discourse.datamethods.org/u/arthur_albuquerque)\
**Post date:** [February 5, 2026, 5:56pm UTC](https://discourse.datamethods.org/t/abcloc-bootstrap-method-for-overfitting-corrected-model-performance-metrics/28632/1 "2026-02-05T17:56:52Z")

</div>

@f2harrell, I want to apply this method in my project and I want to double-check if I understood your conclusions correctly:

> **[Bootstrap Confidence Limits for Bootstrap Overfitting-Corrected Model...](https://www.fharrell.com/post/bootcal/)**
>
> The Efron-Gong optimism bootstrap has been used for decades to obtain reliable estimates of likely performance of statistical models on new data. It accomplishes this by estimating the bias (optimism) from overfitting and subtracting that bias from...

The “sd2rev wtd4” method was the best. This method can be described as:

- For each bootstrap iteration b=1,\dots,B, we computed a variate V\_b:
  - V\_b = bootstrap-sample performance (P\_{boot}) - 1.25 \times original-sample performance (P\_{test})

- Calculate the lower and upper standard deviations of these variates (SE\_{lower}, SE\_{upper}) using `Hmisc::dualSD`

If possible, I am interested in constructing the 95% confidence intervals for the optimism-corrected estimate. How can we derive it from SE\_{lower}, SE\_{upper}? Maybe using the standard normal approximation -1.96 \times SE\_{lower}, + 1.96 \times SE\_{upper} ? This doesn’t seem correct though given the assymetrical assumtion of `dualSD`.

Here is a simulated example applying the method above: [ABCLOC "sd2rev wtd4" method · GitHub](https://gist.github.com/arthur-albuquerque/880b0689849bbd018767904405eb8212)

Is it correct? Thanks

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [February 7, 2026, 2:53pm UTC](https://discourse.datamethods.org/t/abcloc-bootstrap-method-for-overfitting-corrected-model-performance-metrics/28632/2 "2026-02-07T14:53:10Z")

</div>

Look at the `predab.resample` R code which has the specifics. Note how it reverses top and bottom limits.

---

<div class="post-metadata">

**Author:** ![arthur\_albuquerque](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/arthur_albuquerque/32/3269_2.png) [@arthur\_albuquerque](https://discourse.datamethods.org/u/arthur_albuquerque)\
**Post date:** [February 7, 2026, 3:00pm UTC](https://discourse.datamethods.org/t/abcloc-bootstrap-method-for-overfitting-corrected-model-performance-metrics/28632/3 "2026-02-07T15:00:05Z")

</div>

Exactly what I was looking for. Thank you very much.

I suggest you to mention in the blog post you apply the method in this function

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [February 7, 2026, 3:31pm UTC](https://discourse.datamethods.org/t/abcloc-bootstrap-method-for-overfitting-corrected-model-performance-metrics/28632/4 "2026-02-07T15:31:38Z")

</div>

That’s in the help file for `calibrate.orm` and related validation functions.
