# Using the double bootstrap for validating prediction models

**URL:** <https://discourse.datamethods.org/t/using-the-double-bootstrap-for-validating-prediction-models/5090>\
**Category:** model validation\
**Created:** [November 15, 2021, 1:05am UTC](https://discourse.datamethods.org/t/using-the-double-bootstrap-for-validating-prediction-models/5090 "2021-11-15T01:05:24Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![bklamer](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/b/c67d28/32.png) [@bklamer](https://discourse.datamethods.org/u/bklamer)\
**Post date:** [November 15, 2021, 1:05am UTC](https://discourse.datamethods.org/t/using-the-double-bootstrap-for-validating-prediction-models/5090/1 "2021-11-15T01:05:24Z")

</div>

I have a few questions about using the double bootstrap for the purposes of validating prediction model performance.

1. When is the double bootstrap more preferable than the basic bootstrap?
2. What is a useful definition of the double bootstrap algorithm?

The double bootstrap seems suitable for two cases: 1) when a bias adjusted estimate of optimism is desired or 2) to both select from competing models and summarize the performance of the selected model (in the manner of nested k-fold cross validation).

Depending on those cases, my best guess at the appropriate double bootstrap algorithms are the following.

Double bootstrap for bias adjusted optimism correction:

1. Fit the model to the original data and calculate the apparent performance estimate S\_{app}.
2. For k = 1, ..., K
  1. Generate a bootstrapped dataset (with replacement) from the original data.
  2. Fit the model to resample k.
  3. Calculate the apparent bootstrapped performance estimate S\_{k\_{b}}.
  4. Calculate an additional performance estimate S\_{k\_{b}:orig} by evaluating the  
bootstrapped model on the original data.
  5. For j = 1, ..., J
    1. Generate a bootstrapped dataset (with replacement) from resample k.
    2. Fit the model to resample j.
    3. Calculate the apparent double bootstrapped performance estimate S\_{jk\_{bb}}.
    4. Calculate an additional performance estimate S\_{jk\_{bb}:k\_b} by evaluating the double bootstrapped model on resample k.

3. Calculate the optimism of the bootstrapped performance estimate.
  - O\_{b} = \frac{1}{JK} \sum\_{1}^{K} \sum\_{1}^{J} (S\_{jk\_{bb}} - S\_{jk\_{bb}:k\_{b}})

4. Calculate the optimism of the original performance estimate.
  - O\_{orig} = \frac{1}{K} \sum\_{1}^{K} (S\_{k\_b} - S\_{k\_{b}:orig})

5. Calculate the bias-corrected optimism.
  - Only worth it if the bias of the optimism is large compared to the standard error of the optimism?
  - O\_{adj} = O\_{orig} - (O\_{orig} - O\_{b})

6. Calculate the optimism adjusted performance estimate using the bias-corrected optimism.
  - S\_{adj} = S\_{app} - O\_{adj}

7. Report the model’s validated performance estimate using S\_{adj}.

This seems more of a two-step procedure than “double” bootstrap, but feels appropriate for model selection and summarization.

1. For each model i = 1, \dots, I
  1. Fit model i to the original data.
  2. Calculate the apparent performance estimate S\_{i\_{app}}.

2. Determine which model has best performance.
  1. For resamples j = 1, ..., J
    1. Generate a bootstrapped dataset (with replacement) from the original data.
    2. For each model i = 1, \dots, I
      1. Fit model i to resample j.
      2. Calculate the performance estimate S\_{ij\_{boot}}.
      3. Calculate an additional performance estimate S\_{ij\_{boot:orig}} by evaluating bootstrapped model i on the original data.

  2. Calculate the optimism of the apparent performance estimate. For each model i,
    - O\_i = \frac{1}{J} \sum\_{1}^{J} (S\_{ij\_{boot}} - S\_{ij\_{boot:orig}})

  3. Calculate the optimism adjusted performance estimate. For each model i,
    - S\_{i\_{adj}} = S\_{i\_{app}} - O\_i

  4. Determine which model has best prediction performance and proceed using selected model method i.

3. Estimate the prediction performance of the selected model.
  1. For resamples k = 1, ..., K
    1. Generate a bootstrapped dataset (with replacement) from the original data.
    2. Fit the selected model to resample k.
    3. Calculate the performance estimate S\_{k\_{boot}}.
    4. Calculate an additional performance estimate S\_{k\_{boot:orig}} by evaluating the bootstrapped model on the original data.

  2. Calculate the optimism of the apparent performance estimate.
    - O\_{top} = \frac{1}{K} \sum\_{1}^{K} (S\_{k\_{boot}} - S\_{k\_{boot:orig}})

  3. Calculate the optimism adjusted performance estimate.
    - S\_{{top}\_{adj}} = S\_{{top}\_{app}} - O\_{top}

  4. Report the model’s validated performance estimate using S\_{{top}\_{adj}}.

Am I on the right track or are there any fundamental errors here?

Thanks!

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [November 15, 2021, 12:21pm UTC](https://discourse.datamethods.org/t/using-the-double-bootstrap-for-validating-prediction-models/5090/2 "2021-11-15T12:21:05Z")

</div>

The double bootstrap was created to fix problems with the regular bootstrap, especially in constructing confidence intervals that have the right tail areas on both sides. I don’t think the regular Efron-Gong optimism bootstrap is broken enough in the settings you have described to warrant going to the trouble of a double bootstrap.

---

<div class="post-metadata">

**Author:** ![bklamer](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/b/c67d28/32.png) [@bklamer](https://discourse.datamethods.org/u/bklamer)\
**Post date:** [November 16, 2021, 12:32am UTC](https://discourse.datamethods.org/t/using-the-double-bootstrap-for-validating-prediction-models/5090/3 "2021-11-16T00:32:22Z")

</div>

Thanks, just to confirm, competing models can be fit within each bootstrap iteration and their resulting optimism adjusted performance estimates can be used to both select the top performing model and report final performance?

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [November 16, 2021, 12:42pm UTC](https://discourse.datamethods.org/t/using-the-double-bootstrap-for-validating-prediction-models/5090/4 "2021-11-16T12:42:56Z")

</div>

Now I see the question. To oversimplify I’d say that if you had two competing modeling procedures you could select the better one with a single bootstrap, but if you were selecting from among many then the double bootstrap is called for.

---

<div class="post-metadata">

**Author:** ![bklamer](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/b/c67d28/32.png) [@bklamer](https://discourse.datamethods.org/u/bklamer)\
**Post date:** [November 16, 2021, 3:20pm UTC](https://discourse.datamethods.org/t/using-the-double-bootstrap-for-validating-prediction-models/5090/5 "2021-11-16T15:20:22Z")

</div>

I’m not familiar with a practical implementation for this case and don’t know of any references. Is it a two-step procedure, like the second algorithm I outlined above, where one bootstrap procedure is used to compare competing models, then a second bootstrap procedure is used to estimate the final prediction performance? Or, would it be like the first algorithm above, which requires an outer loop of bootstrap resampling from the original data and an inner loop of bootstrap resampling from the resampled data?

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [November 16, 2021, 4:48pm UTC](https://discourse.datamethods.org/t/using-the-double-bootstrap-for-validating-prediction-models/5090/6 "2021-11-16T16:48:07Z")

</div>

The second. I’m sure someone has written this up somewhere. Nested cross-validation is even more common.

---

<div class="post-metadata">

**Author:** ![brawthy](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/brawthy/32/4027_2.png) [@brawthy](https://discourse.datamethods.org/u/brawthy)\
**Post date:** [July 19, 2023, 10:58pm UTC](https://discourse.datamethods.org/t/using-the-double-bootstrap-for-validating-prediction-models/5090/7 "2023-07-19T22:58:03Z")

</div>

Good Afternoon Professor,

Sorry to revive this thread, but I wanted to make sure I was following this correctly: You are saying that @bklamer’s second algorithm in the original post is the correct implementation of the double bootstrap from one-among-many models, correct?

I’ve been trying to track down some sources for how the double bootstrap is defined, especially in the context of validation, but online results seem contradictory at times (probably due to the lack of a clear definition of a ‘double bootstrap’?). Thanks!

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [July 20, 2023, 2:25pm UTC](https://discourse.datamethods.org/t/using-the-double-bootstrap-for-validating-prediction-models/5090/8 "2023-07-20T14:25:13Z")

</div>

Not sure. I hope someone knows the answer.