# RMS Describing, Resampling, Validating, and Simplifying the Model

**URL:** <https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790>\
**Category:** modeling strategy\
**Tags:** graphics, bootstrap, validation\
**Created:** [September 14, 2021, 7:24pm UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790 "2021-09-14T19:24:11Z")\
**Posts on this page:** 20\
**Page:** 7

<div class="post-metadata">

**Author:** ![Sanne](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/sanne/32/4244_2.png) [@Sanne](https://discourse.datamethods.org/u/Sanne)\
**Post date:** [October 18, 2023, 12:03pm UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/126 "2023-10-18T12:03:08Z")

</div>

Hi,

If a test split is used, can the variability of the performance measure (say RMSE) be assessed by applying bootstrap to the test set? This would be valuable in case smallish test sets are used.

Cheers,

Sanne

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [October 18, 2023, 2:02pm UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/127 "2023-10-18T14:02:28Z")

</div>

Think of the bootstrap as telling you how volatile a procedure (e.g., variable selection) is or how imprecise an estimate is (e.g., computing standard errors or confidence limits). The bootstrap doesn’t help with small sample sizes as it inherits the small sample size in all that it calculates. And note that splitting data into training and test requires a huge sample size (sometimes 20,000) in order to not be volatile.

---

<div class="post-metadata">

**Author:** ![Sanne](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/sanne/32/4244_2.png) [@Sanne](https://discourse.datamethods.org/u/Sanne)\
**Post date:** [October 18, 2023, 2:05pm UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/128 "2023-10-18T14:05:57Z")

</div>

Thank you! I should have added the goal of the exercise: to show how volatile any performance calculation is on a small test set.

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [October 19, 2023, 12:32pm UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/129 "2023-10-19T12:32:57Z")

</div>

The bootstrap will help with that although it may be more direct to simulate multiple small test datasets and show how results disagree across them.

---

<div class="post-metadata">

**Author:** ![FurlanLeo](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/furlanleo/32/3812_2.png) [@FurlanLeo](https://discourse.datamethods.org/u/FurlanLeo)\
**Post date:** [October 24, 2023, 1:59pm UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/130 "2023-10-24T13:59:56Z")

</div>

Please,

Does anyone know how to extract the values of the error measures (_Mean absolute error, Mean squared error, 0.9 Quantile of absolute error_) displayed on the calibration plot (`plot(calibrate(f))`)?

I want to insert these values into a data frame for ploting.

Thanks.

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [October 24, 2023, 2:37pm UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/131 "2023-10-24T14:37:11Z")

</div>

Start with `str(calibrate(...))` to see what’s in the `calibrate` object.

---

<div class="post-metadata">

**Author:** ![FurlanLeo](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/furlanleo/32/3812_2.png) [@FurlanLeo](https://discourse.datamethods.org/u/FurlanLeo)\
**Post date:** [October 24, 2023, 6:27pm UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/132 "2023-10-24T18:27:52Z")

</div>

Hi Professor, thank you for the suggestion.  
I couldn’t find anything related to the errors with `str(calibrate())`.  
But I think I found a way:

`c <- calibrate()`  
`p <- plot(c)`  
`mean(abs(p))`  
`quantile(abs(p), probs = 0.9)`

The errors are stored in `p`.

---

<div class="post-metadata">

**Author:** ![FurlanLeo](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/furlanleo/32/3812_2.png) [@FurlanLeo](https://discourse.datamethods.org/u/FurlanLeo)\
**Post date:** [October 25, 2023, 8:43pm UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/133 "2023-10-25T20:43:45Z")

</div>

I have been doing some prediction analyses/modeling and could use some feedback.

- The data are displayed on the figure attached, which is a print of `describe()`;
- _x1_, _x2_, _x3_, _x4_, and _x5_ are clinical scales/tests usually performed by a physical therapist;
- _y_ is a functional scale (integer values from 0 to 57);
- _x_s are all performed at baseline and _y_ is measured after ~ 3 months;
- _DOM_, _BAM_, _SEX_, _AGE_ constitute a _Base Model_:

`f1 <- lrm(y ~ DOM + BAM + SEX + rcs(AGE, 3), data = data, x = T, y = T)`

- Goal is to determine which of the clinical scales/tests (the _x_s) add more predicitve power to the _Base Model_;
- I fitted 6 other _Models_ on top of the _Base Model_, 1 for each _x_, with the latter _Model_ featuring a combination of _x1_ and _x2_:

`f2 <- lrm(y ~ DOM + BAM + SEX + rcs(AGE, 3) + x1, data = data, x = T, y = T)`  
`f3 <- lrm(y ~ DOM + BAM + SEX + rcs(AGE, 3) + x2, data = data, x = T, y = T)`  
`f4 <- lrm(y ~ DOM + BAM + SEX + rcs(AGE, 3) + rcs(x3, 3), data = data, x = T, y = T)`  
`f5 <- lrm(y ~ DOM + BAM + SEX + rcs(AGE, 3) + rcs(x4, 3), data = data, x = T, y = T)`  
`f6 <- lrm(y ~ DOM + BAM + SEX + rcs(AGE, 3) + x5, data = data, x = T, y = T)`  
`f7 <- lrm(y ~ DOM + BAM + SEX + AGE + x1 + x2, data = data, x = T, y = T)`

- Then I produced several (optimism-corrected) measures of _Model Performance_ for each _Model_, using the following code:

`validate(f, method = 'boot', B = 500, group = data$y)` _# internal validation using bootstrap_  
`calibrate(f, method = 'boot', B = 500, group = data$y)` _# calibration curve using boostrap_  
`pred <- predict(f, type = 'lp', ycut = 36)` _# predicted probabilities for y\>=36_  
`1-var(predBase)/var(predx)` _# fraction of new information_

- These measures are displayed on the figures attached;
- From these analyses I conclude that _x4_ makes the greatest contribution to increasing predictive power of the _Base Model_;
- _x1_ and _x2_ are much simpler/quicker tests to perform in clinical practice, and they also seem to be important for predicting _y_ (either in isolation or combined), but their performance indexes are a bit conflicting in my view;

Any comments?

![describe](https://discourse.datamethods.org/uploads/default/original/2X/1/1de7521617b4dbba9b66b51947ef5f3b397da514.png)

 ![BaseModel](https://discourse.datamethods.org/uploads/default/original/2X/c/cc69b8eb3ac0abd3fd212ca7c6c0a79fc6f4a809.png)  
 ![x1Model](https://discourse.datamethods.org/uploads/default/original/2X/f/fdbe873f616a8cfc79443148e1bf7ad5109fe8b3.png)  
 ![x2Model](https://discourse.datamethods.org/uploads/default/original/2X/9/91b631cd3e641618a65b8968195823db9f906595.png)  
 ![x3Model](https://discourse.datamethods.org/uploads/default/original/2X/a/a898c0aff909aa88dcbaf512d7ba47c238ae9e71.png)  
 ![x4Model](https://discourse.datamethods.org/uploads/default/original/2X/4/4b2a794d87e5337ec77f9cb22c66e647dafdf869.png)  
 ![x5Model](https://discourse.datamethods.org/uploads/default/original/2X/e/ecf06522f2aa8a6c95b7f380adbe6d029804e6bf.png)  
 ![x1+x2Model](https://discourse.datamethods.org/uploads/default/original/2X/1/19cb738eef3f618bfebb6992f64ccfac5121d4fd.png)  
 ![Somer'sD](https://discourse.datamethods.org/uploads/default/original/2X/d/d6b5f50d4cbeaf363de45afeadd3aa97a3132949.png)  
 ![MAPE](https://discourse.datamethods.org/uploads/default/original/2X/c/cc193f3fd4e7de5941e7b2fcfa4467d8b80819d1.png)  
 ![0.9qAPE](https://discourse.datamethods.org/uploads/default/original/2X/f/f1cd8b3954a9a5043b3eab91d01c63cf1a0958c4.png)  
 ![Histograms](https://discourse.datamethods.org/uploads/default/original/2X/7/723658bbc40e2eb16e5d8609cf29f8fbd37ccbb0.png)  
 ![NewInfo](https://discourse.datamethods.org/uploads/default/original/2X/b/b083f292b49c0b560771badd6a54642b37ab0031.png)

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [October 26, 2023, 1:03pm UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/134 "2023-10-26T13:03:11Z")

</div>

Not sure this is posted in the right topic. Should have probably been in [RMS Multivariable Modeling Strategies - modeling strategy - Datamethods Discussion Forum](http://datamethods.org/rms4)

You are going to way too much trouble for the goal you have. Add all of those variables at once and do a likelihood ratio chunk \chi^2 test for added value. And describe the added value using a type of partial R^2 measure. And compute the proportion of log-likelihood explained by the new variables. See also [Statistical Thinking - Statistically Efficient Ways to Quantify Added Predictive Value of New Measurements](http://fharrell.com/post/addvalue)

---

<div class="post-metadata">

**Author:** ![FurlanLeo](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/furlanleo/32/3812_2.png) [@FurlanLeo](https://discourse.datamethods.org/u/FurlanLeo)\
**Post date:** [October 26, 2023, 2:08pm UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/135 "2023-10-26T14:08:12Z")

</div>

Thank you Professor.  
Ok, I will move the post then.

Actually, my analysis is based exactly on [this](https://www.fharrell.com/post/addvalue/index.html). That’s why I made the _back-to-back histograms_ and the _fraction of new information_ plot.

So it’s better to have a full model with all the _x_s in and then compute e.g., **Proportion of Overall Chi-Square** and then **Relative Explained Variation** using _bootstrap_, as shown in **Section 5.1.3** of **RMS**?

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [October 26, 2023, 2:33pm UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/136 "2023-10-26T14:33:46Z")

</div>

Don’t need to move the post, just keep in mind for next time. Yes to your questions. The upcoming version of `rms` has the `rexVar` function which automatically incorporates bootstrap repetitions if you’ve run `bootcov`, to get uncertainties in a new relative importance measure. The `rms` course notes are already updated to include an example of this.

---

<div class="post-metadata">

**Author:** ![FurlanLeo](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/furlanleo/32/3812_2.png) [@FurlanLeo](https://discourse.datamethods.org/u/FurlanLeo)\
**Post date:** [October 29, 2023, 12:44pm UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/137 "2023-10-29T12:44:03Z")

</div>

Thank you Professor!

Here are the results of my updated analysis.  
Please, any comments or suggestions for improving it?

1. **Chunk test / likelihood ratio chi-square test for added predictive value of the clinical measures altogether** :

```auto
# Chi-Square d.f. P 
# 6.386165e+01 7.000000e+00 2.545708e-11

```

1. **REV (500 bootstrap resamples of the full model)**:  

2. **Fit x4 model** :  
`lrm(formula = y ~ DOM + BAM + SEX + rcs(AGE, 3) + rcs(x4, 3),`  
`data = data, x = T, y = T)`

3. **Anova (Likelihood Ratio Statistics)**:  
Significant association of AGE (_df=2; p=0.0007_) and x4 (_df=2; p\<0.0001_)

4. **Bootstrap model and partial effects plots** :  

5. **Model validation with bootstrap (B=500)**:  
_Somer’s D_ (corrected index) = 0.5455

6. **Model calibration with bootstrap (B=500)**:  
_Mean absolute error_ = 0.014  
_0.9 Quantile of absolute error_ = 0.025  

7. **Nomogram** :

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [October 29, 2023, 12:58pm UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/138 "2023-10-29T12:58:23Z")

</div>

Nice work. I don’t think of any suggestions.

---

<div class="post-metadata">

**Author:** ![FurlanLeo](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/furlanleo/32/3812_2.png) [@FurlanLeo](https://discourse.datamethods.org/u/FurlanLeo)\
**Post date:** [November 4, 2023, 6:54pm UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/139 "2023-11-04T18:54:23Z")

</div>

Thank you Professor, I appreciate the feedback.

I just want to make sure I got this right…

The **REV** approach is much more straightforward, but is the **other** approach wrong ( **Question 1** )?

By **other** approach I mean when I fit a base/ reference model and a separate model for each new predictor (i.e., base model + new predictor) and then compare these models in terms of their predicitve performance, using _discrimination_, _calibration_ and _prediction error_ indexes, e.g., D\_{xy}, E\_{max}, Mean Absolute Prediction Error, etc.

If this approach is correct, then what would be the most appropriate index for choosing the _best_ model ( **Question 2** )? The reason I ask is because sometimes these indexes are in disagreement, e.g., for index _a_, model _w_ is the best, but for index _b_, model _z_ is the best.

Section **4.10** from **RMS** contains a list of criteria for comparing models. If I undestood correctly, _calibration_ seems to be the most important criterion if the goal is to obtain _accurate probabilities_ for future patients, whereas _discrimination_ is more relevant if we want to accurately predict _mean y_.

This matters to me beacuse in my case (_PO models_ fitted with `lrm`) _y_ has 35 intercepts and all the bootstrap validations and calibrations are performed using the middle intercept, but from a clinical perspective I think it is more relevant to predict _mean y_. Does that mean I should focus on _discrimination_ ( **Question 3** )?

Finally, considering the recommendations on section **13.4.5** from **RMS** , when describing my selected _best_ model, besides presenting _partial effects plots_ and a _nomogram_, both on the _mean_ scale, what else could I do ( **Question 4** )? I think it could also be relevant to report predictions (on the _mean_ scale and with _confidence limits_) for new subjects using `expand.grid` and `predict.lrm`. But I’m having trouble with the arguments of `Mean.lrm`:

`new <- expand.grid()`  
`m <- Mean(f, codes = T, conf.int = T)` # what about the design matrix ‘X’ ( **Question 5** )?  
`lp <- predict(f, new)`  
`m(lp)`

Thank you for your time Professor.

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [November 16, 2023, 11:42am UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/141 "2023-11-16T11:42:32Z")

</div>

Concerning model comparison I would not use that approach but would rather use the approach [here](https://fharrell.com/post/addvalue). This emphasizes the gold standard log likelihood-based measures plus measures built on dispersion of predicted values on the linear predictor or probability scale. Don’t fit separate models for “each new predictor” as this is too similar to stepwise regression. Pre-specify 2-3 models for the competition.

This leaves the question of when to consider calibration. Usually I demonstrate good overall calibration in the final, single, model.

---

<div class="post-metadata">

**Author:** ![FurlanLeo](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/furlanleo/32/3812_2.png) [@FurlanLeo](https://discourse.datamethods.org/u/FurlanLeo)\
**Post date:** [November 22, 2023, 2:30pm UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/142 "2023-11-22T14:30:36Z")

</div>

Ok Professor,  
Thank you very much for the valuable suggestions.

---

<div class="post-metadata">

**Author:** ![FurlanLeo](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/furlanleo/32/3812_2.png) [@FurlanLeo](https://discourse.datamethods.org/u/FurlanLeo)\
**Post date:** [February 15, 2024, 10:47am UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/143 "2024-02-15T10:47:55Z")

</div>

Hi Professor @f2harrell,

One final question that remains on this analysis approach ([https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/137?u=furlanleo](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/137?u=furlanleo)):

I understand that the _Relative Explained Variation_ index, computed by `rexVar`, should only be used in a final, pre-specified model, and not for selecting important predictors.

Therefore, I believe there’s a problem with my approach from above, where I first do a chunk test for the added predictive value of 5 new measures (X1, X2, … X5) and then, after confirming that there is indeed added predictive value, I use `rexVar` to decide which of these 5 new measures is more important, and finally I select this more important measure (e.g., X4) to build my final model.

If that is the case, then I think a better approach would be to first select 3 of the 5 new measures and build a separate model for each, and then use AIC to compare these 3 models and select the best one to be my final model:

- Model 1: `y ~ DOM + BAM + SEX + rcs(AGE, 3) + rcs(X4, 3)`
- Model 2: `y ~ DOM + BAM + SEX + rcs(AGE, 3) + rcs(X1, 3)`
- Model 3: `y ~ DOM + BAM + SEX + rcs(AGE, 3) + rcs(X2, 3)`

Is my understanding correct?

I think this is in line with the idea that, if the sample size is not large enough, then selecting winners is a futile task, as shown in the “boostrap ranking of predictors” example [here](https://www.fharrell.com/post/badb/#difficulties-of-picking-winners).

Thank you.

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [February 15, 2024, 1:10pm UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/144 "2024-02-15T13:10:29Z")

</div>

No that is not correct. Just pre-specify the model, allowing for as many parameters as you think _may be needed_, subject to limitations imposed by your effective sample size for Y.

---

<div class="post-metadata">

**Author:** ![FurlanLeo](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/furlanleo/32/3812_2.png) [@FurlanLeo](https://discourse.datamethods.org/u/FurlanLeo)\
**Post date:** [February 15, 2024, 3:33pm UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/145 "2024-02-15T15:33:29Z")

</div>

Sorry, I don’t think I expressed myself well.  
Could you kindly have another look at what I did [here](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/137?u=furlanleo)?

You said afterwards that it was a “nice work”, but I’m confused because after confirming the added predictive value of the 5 new measures altogether, I did a REV analysis using `rexVar` and decided that X4 was _The Best_ measure among the 5, and then I built my prediction model with X4 only (besides the “base” predictors). Isn’t that a bit like variable selection?

That’s why I mentioned in my last post that it was my understanding that `rexVar` should only be used in a final model (but not to specify this final model).

So the big question is: **what to do when you want to select, among a set of new clinical measures, only one measure to include in your prediction model?**

I’m aware that I should preferably be fitting a full model with all the measures in (possibly with some form of penalization) and then using such model as my new prediction model. But the central ideia is to choose among the new measures only one, as using all of them in clinical practice would be too laborious. Is that reasonable?

Thank you!

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [February 16, 2024, 1:31pm UTC](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/146 "2024-02-16T13:31:42Z")

</div>

Just like when running `anova()`, the fact that you want to quantify which predictors are very important doesn’t mean that should lead to changes in the model. You are getting into the classic Grambsch-O’Brien problem discussed in RMS.

[Previous page](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790.md?page=6)

[Next page](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790.md?page=8)