# Internal validation: Imputation nested within bootstrap

**URL:** <https://discourse.datamethods.org/t/internal-validation-imputation-nested-within-bootstrap/5608>\
**Category:** data analysis\
**Tags:** prediction, validation\
**Created:** [May 9, 2022, 7:20am UTC](https://discourse.datamethods.org/t/internal-validation-imputation-nested-within-bootstrap/5608 "2022-05-09T07:20:45Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![pravleen](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/p/9f8e36/32.png) [@pravleen](https://discourse.datamethods.org/u/pravleen)\
**Post date:** [May 9, 2022, 7:20am UTC](https://discourse.datamethods.org/t/internal-validation-imputation-nested-within-bootstrap/5608/1 "2022-05-09T07:20:45Z")

</div>

Background: I would like to internally validate my prediction model using bootstrapping. I would also like to use multiple imputation to deal with missing data and will be nesting the MI within each bootstrap sample. The main model is a proportional hazards regression.

Question: For testing performance, should I test each bootstrap model on the original _incomplete_ data or should I also create a separate “original” _imputed_ dataset that has nothing to do with the bootstrap samples? I saw the latter in [Wahl et. al., 2016](https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-016-0239-7) but haven’t seen any other examples.

Sidenote - if there are any SAS users who have done MI nested in bootstrap, I would love to see what procedures you used. I’m using surveyselect and MI but am not confident whether MIANALYZE is the right procedure to pool my analysis results.

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [May 9, 2022, 11:22am UTC](https://discourse.datamethods.org/t/internal-validation-imputation-nested-within-bootstrap/5608/2 "2022-05-09T11:22:50Z")

</div>

Please read [https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/) and move your post (if it’s not already answered by what’s there) to the bottom of that page as a “reply” to the last post there. Thanks.

---

<div class="post-metadata">

**Author:** ![pravleen](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/p/9f8e36/32.png) [@pravleen](https://discourse.datamethods.org/u/pravleen)\
**Post date:** [May 9, 2022, 7:19pm UTC](https://discourse.datamethods.org/t/internal-validation-imputation-nested-within-bootstrap/5608/3 "2022-05-09T19:19:21Z")

</div>

For anyone who stumbles into this, I found the answer on page 4-48 of Dr. Harrell’s [Regression Modeling Strategies](https://hbiostat.org/doc/rms.pdf) and in the replies of the page linked by him above. I will multiply impute the original (incomplete) data and create my “final model” using that. For internal validation, I will create bootstrap models (draw B bootstrap samples from original incomplete data \> multiply impute each \> create B bootstrap models) and test each on the original (multiply imputed) data. I know we’re supposed to repeat each modeling step when running bootstraps so I believe nesting the MI in bootstrap makes sense (and is how Wahl 2016 does it).