# model validation

**URL:** https://discourse.datamethods.org/c/stat/modval/22.md?page=1

[Latest](https://discourse.datamethods.org/latest.md) · [Categories](https://discourse.datamethods.org/categories.md) · [Tags](https://discourse.datamethods.org/tags.md)

**Page:** 2

---

## [Schoenfeld Residuals](https://discourse.datamethods.org/t/schoenfeld-residuals/5699)

<div class="topic-metadata">

**Author:** [@bbulen](https://discourse.datamethods.org/u/bbulen)\
**Replies:** 1\
**Last updated:** [May 23, 2022, 9:17pm UTC](https://discourse.datamethods.org/t/schoenfeld-residuals/5699 "2022-05-23T21:17:23Z")

</div>

I am assessing proportional hazards using Shoenfeld residuals and have an interesting question as far as interpreting a plot vs. cox.zph() output. The following plot shows my residuals and a red abline using the coef() f…

---

## [Validation of cox model](https://discourse.datamethods.org/t/validation-of-cox-model/4296)

<div class="topic-metadata">

**Author:** [@matthew](https://discourse.datamethods.org/u/matthew)\
**Replies:** 4\
**Last updated:** [May 2, 2022, 7:00pm UTC](https://discourse.datamethods.org/t/validation-of-cox-model/4296 "2022-05-02T19:00:12Z")

</div>

Hi everyone, Currently I am working on an analysis using the cox model. I have about 200 observations. I built a model based on the results of univariate analysis (also cox, with the best p-value results). In the next …

---

## [Bootstrap vs. cross-validation for model performance](https://discourse.datamethods.org/t/bootstrap-vs-cross-validation-for-model-performance/2779)

<div class="topic-metadata">

**Author:** [@Lynild](https://discourse.datamethods.org/u/Lynild)\
**Replies:** 10\
**Last updated:** [April 4, 2022, 2:30pm UTC](https://discourse.datamethods.org/t/bootstrap-vs-cross-validation-for-model-performance/2779 "2022-04-04T14:30:56Z")

</div>

So I have developed some logistic models, and obviously the next step is to validate them. I have no option for external validation, so internal is the way to go for me. On that note, I have been looking into this articl…

---

## [Assessing generalizability in biomarker studies](https://discourse.datamethods.org/t/assessing-generalizability-in-biomarker-studies/5155)

<div class="topic-metadata">

**Author:** [@limitededition](https://discourse.datamethods.org/u/limitededition)\
**Replies:** 5\
**Last updated:** [January 12, 2022, 1:38pm UTC](https://discourse.datamethods.org/t/assessing-generalizability-in-biomarker-studies/5155 "2022-01-12T13:38:10Z")

</div>

In the development of prediction models using e.g. temporal or geographically distinct cohorts, assessing the true external performance (predictive accuracy) of the model developed is apparently very important. Typicall…

---

## [Calibration of predicted full distribution percentiles](https://discourse.datamethods.org/t/calibration-of-predicted-full-distribution-percentiles/5105)

<div class="topic-metadata">

**Author:** [@hanowell](https://discourse.datamethods.org/u/hanowell)\
**Replies:** 1\
**Last updated:** [November 19, 2021, 3:21pm UTC](https://discourse.datamethods.org/t/calibration-of-predicted-full-distribution-percentiles/5105 "2021-11-19T15:21:37Z")

</div>

Suppose you predict the full conditional distribution of some non-zero continuous outcome as a complex non-linear function of predictors, and the predicted distribution does not have a tractable functional form. Suppose …

---

## [Why do people focus on standardized mean difference, K-S tests, etc., to evaluate propensity score models?](https://discourse.datamethods.org/t/why-do-people-focus-on-standardized-mean-difference-k-s-tests-etc-to-evaluate-propensity-score-models/4265)

<div class="topic-metadata">

**Author:** [@hanowell](https://discourse.datamethods.org/u/hanowell)\
**Replies:** 12\
**Last updated:** [November 19, 2021, 4:09am UTC](https://discourse.datamethods.org/t/why-do-people-focus-on-standardized-mean-difference-k-s-tests-etc-to-evaluate-propensity-score-models/4265 "2021-11-19T04:09:06Z")

</div>

The literature on evaluating propensity score model specification tends to focus on the use of distributional difference metrics (e.g., Kolmogorov-Smirnov) and standardized mean differences. Yet when I build a propensity…

---

## [How do we use the calibration optimism in context of the bias correction method?](https://discourse.datamethods.org/t/how-do-we-use-the-calibration-optimism-in-context-of-the-bias-correction-method/4603)

<div class="topic-metadata">

**Author:** [@Rajshree](https://discourse.datamethods.org/u/Rajshree)\
**Replies:** 7\
**Last updated:** [July 12, 2021, 10:00pm UTC](https://discourse.datamethods.org/t/how-do-we-use-the-calibration-optimism-in-context-of-the-bias-correction-method/4603 "2021-07-12T22:00:15Z")

</div>

Good afternoon! I have couple of questions regarding how to use the calibration optimism method in context of the bias correction method for model validation? Just to provide some context to my question - Calibration …

---

## [Optimism correction for hazard ratio?](https://discourse.datamethods.org/t/optimism-correction-for-hazard-ratio/4245)

<div class="topic-metadata">

**Author:** [@bavery](https://discourse.datamethods.org/u/bavery)\
**Replies:** 4\
**Last updated:** [March 31, 2021, 3:50pm UTC](https://discourse.datamethods.org/t/optimism-correction-for-hazard-ratio/4245 "2021-03-31T15:50:09Z")

</div>

I’m using bootstrap internal validation to correct a Cox proportional hazards model that uses principal component scores extracted from gene expression data as the predictor (X) variables. Among other reasons, one reaso…

---

## [Does removing other events from training data amount to data snooping?](https://discourse.datamethods.org/t/does-removing-other-events-from-training-data-amount-to-data-snooping/4119)

<div class="topic-metadata">

**Author:** [@hanowell](https://discourse.datamethods.org/u/hanowell)\
**Replies:** 4\
**Last updated:** [February 10, 2021, 5:06pm UTC](https://discourse.datamethods.org/t/does-removing-other-events-from-training-data-amount-to-data-snooping/4119 "2021-02-10T17:06:57Z")

</div>

Suppose you are interested in probability of death within X days due to cause A, which competes with cause B. Suppose you’ve convinced yourself (wrongly, in my opinion) that you will draw a cleaner comparison between sur…

---

## [Calculate linear predictor for a prediction model using restricted cubic splines](https://discourse.datamethods.org/t/calculate-linear-predictor-for-a-prediction-model-using-restricted-cubic-splines/3975)

<div class="topic-metadata">

**Author:** [@bsurial](https://discourse.datamethods.org/u/bsurial)\
**Replies:** 5\
**Last updated:** [December 11, 2020, 3:32pm UTC](https://discourse.datamethods.org/t/calculate-linear-predictor-for-a-prediction-model-using-restricted-cubic-splines/3975 "2020-12-11T15:32:22Z")

</div>

Hi all. I have a question regarding the use of restricted cubic splines. Let’s assume I would like to externally validate a model that was published in the literature but I have no access to the data. To do so, I would …

---

## [Bias correction for c-statistics](https://discourse.datamethods.org/t/bias-correction-for-c-statistics/3856)

<div class="topic-metadata">

**Author:** [@langsetmo](https://discourse.datamethods.org/u/langsetmo)\
**Replies:** 1\
**Last updated:** [October 18, 2020, 2:33pm UTC](https://discourse.datamethods.org/t/bias-correction-for-c-statistics/3856 "2020-10-18T14:33:40Z")

</div>

I have a question about MULTIVARIABLE PROGNOSTIC MODELS: 1996, in particular the part about estimating the optimism of the C-statistic. Per algorithm one should use bootstrap sample to create model, the calculate differe…

---

## [Question re: classification model evaluation](https://discourse.datamethods.org/t/question-re-classification-model-evaluation/3842)

<div class="topic-metadata">

**Author:** [@js592](https://discourse.datamethods.org/u/js592)\
**Replies:** 4\
**Last updated:** [October 12, 2020, 11:16pm UTC](https://discourse.datamethods.org/t/question-re-classification-model-evaluation/3842 "2020-10-12T23:16:56Z")

</div>

Hi all, I am in the midst of peer review on a paper developing a probabilistic binary classifier for a geoscience application. Here are a few key properties of the research purpose and dataset: The dataset is fairly s…

---

## [Can traditional statistical metrics be used to evaluate whether a prediction model should be used in practice?](https://discourse.datamethods.org/t/can-traditional-statistical-metrics-be-used-to-evaluate-whether-a-prediction-model-should-be-used-in-practice/3755)

<div class="topic-metadata">

**Author:** [@VickersBiostats](https://discourse.datamethods.org/u/VickersBiostats)\
**Replies:** 21\
**Last updated:** [October 8, 2020, 2:47pm UTC](https://discourse.datamethods.org/t/can-traditional-statistical-metrics-be-used-to-evaluate-whether-a-prediction-model-should-be-used-in-practice/3755 "2020-10-08T14:47:32Z")

</div>

A second scenario. Cancer patients routinely have lymph nodes removed during surgerey, but this often causes adverse effects. A model developed on a different data set is tested on a new cohort, with the results below. T…

---

## [Is how we have been evaluating prediction models for years kind of useless?](https://discourse.datamethods.org/t/is-how-we-have-been-evaluating-prediction-models-for-years-kind-of-useless/3765)

<div class="topic-metadata">

**Author:** [@VickersBiostats](https://discourse.datamethods.org/u/VickersBiostats)\
**Replies:** 9\
**Last updated:** [October 5, 2020, 8:31pm UTC](https://discourse.datamethods.org/t/is-how-we-have-been-evaluating-prediction-models-for-years-kind-of-useless/3765 "2020-10-05T20:31:53Z")

</div>

Two models have been proposed to select for biopsy patients at high-risk for aggressive prostate cancer amongst those with elevated PSA. These two models are evaluated on an independent data set. Model A is the standard …

---

## [Residuals of high resolution calibrations curves](https://discourse.datamethods.org/t/residuals-of-high-resolution-calibrations-curves/3384)

<div class="topic-metadata">

**Author:** [@hanowell](https://discourse.datamethods.org/u/hanowell)\
**Replies:** 2\
**Last updated:** [May 29, 2020, 9:18pm UTC](https://discourse.datamethods.org/t/residuals-of-high-resolution-calibrations-curves/3384 "2020-05-29T21:18:06Z")

</div>

It is said that probability predictions are their own error metrics. But what about the areas of the predictive probability domain that are imperfectly calibrated? How do we incorporate what we know about calibration imp…

---

## [More on the sad state of fraud detection](https://discourse.datamethods.org/t/more-on-the-sad-state-of-fraud-detection/3383)

<div class="topic-metadata">

**Author:** [@hanowell](https://discourse.datamethods.org/u/hanowell)\
**Replies:** 0\
**Last updated:** [May 27, 2020, 1:47pm UTC](https://discourse.datamethods.org/t/more-on-the-sad-state-of-fraud-detection/3383 "2020-05-27T13:47:56Z")

</div>

Here is a paper reviewing fraud detection methods. Fraud is a rare event, and there is low signal to noise (since fraudsters try to fly under the radar). Companies are trying to manage the risk of fraud, which implies ri…

---

## [An example of what we're dealing with here](https://discourse.datamethods.org/t/an-example-of-what-were-dealing-with-here/3370)

<div class="topic-metadata">

**Author:** [@hanowell](https://discourse.datamethods.org/u/hanowell)\
**Replies:** 0\
**Last updated:** [May 20, 2020, 9:43pm UTC](https://discourse.datamethods.org/t/an-example-of-what-were-dealing-with-here/3370 "2020-05-20T21:43:20Z")

</div>

Page 5 paragraph 1 of this article lauds the performance of complex non-linear algorithms over poor ole logistic regression with reference to AUC performance improvement. Zero consideration of calibration whatsoever desp…

---

## [Validating elastic-net model with binary outcome](https://discourse.datamethods.org/t/validating-elastic-net-model-with-binary-outcome/2981)

<div class="topic-metadata">

**Author:** [@Elisabetta](https://discourse.datamethods.org/u/Elisabetta)\
**Replies:** 2\
**Last updated:** [February 17, 2020, 3:17pm UTC](https://discourse.datamethods.org/t/validating-elastic-net-model-with-binary-outcome/2981 "2020-02-17T15:17:41Z")

</div>

Dear all, I have data on 140 subjects (70 cases and 70 controls) and 200 features (biomarkers). I do not have independent data for external validation. My aim is to build a prediction model and to internally evaluate i…

---

## [Are synthetic controls tantamount to clinical prediction models?](https://discourse.datamethods.org/t/are-synthetic-controls-tantamount-to-clinical-prediction-models/3007)

<div class="topic-metadata">

**Author:** [@davidcnorrismd](https://discourse.datamethods.org/u/davidcnorrismd)\
**Replies:** 1\
**Last updated:** [February 15, 2020, 10:51pm UTC](https://discourse.datamethods.org/t/are-synthetic-controls-tantamount-to-clinical-prediction-models/3007 "2020-02-15T22:51:40Z")

</div>

I generally avoid reading papers involving propensity scores, but recently made an exception for \[1\] on account of its senior author. While the paper employs some cautious-sounding language, if one takes it at all seriou…

---

## [Model Performance Following Best Subset Selection](https://discourse.datamethods.org/t/model-performance-following-best-subset-selection/2921)

<div class="topic-metadata">

**Author:** [@Lynild](https://discourse.datamethods.org/u/Lynild)\
**Replies:** 4\
**Last updated:** [January 26, 2020, 4:38pm UTC](https://discourse.datamethods.org/t/model-performance-following-best-subset-selection/2921 "2020-01-26T16:38:50Z")

</div>

I was just curious to whether performance measures like R^2, Brier score, calibration slope/intercept etc. are as applicable on Best Subset Selection (BSS)/LASSO-based models compared to a regular GLM based model ? For …

---

## [Split + resampling, or just resampling?](https://discourse.datamethods.org/t/split-resampling-or-just-resampling/2811)

<div class="topic-metadata">

**Author:** [@eric](https://discourse.datamethods.org/u/eric)\
**Replies:** 2\
**Last updated:** [January 4, 2020, 3:03am UTC](https://discourse.datamethods.org/t/split-resampling-or-just-resampling/2811 "2020-01-04T03:03:07Z")

</div>

I’m learning the tidymodels suite of R packages, and I have a question about splitting and resampling. From the {rsample} docs: A typical scheme for splitting the data when developing a predictive model is to create an…

---

## [Information criteria vs cross validation](https://discourse.datamethods.org/t/information-criteria-vs-cross-validation/2375)

<div class="topic-metadata">

**Author:** [@Arman\_Eshaghi](https://discourse.datamethods.org/u/Arman_Eshaghi)\
**Replies:** 3\
**Last updated:** [September 29, 2019, 10:18am UTC](https://discourse.datamethods.org/t/information-criteria-vs-cross-validation/2375 "2019-09-29T10:18:24Z")

</div>

Following Gelman’s 2017 publication entitled “Understanding predictive information criteria for Bayesian models” I understand cross-validation and information criteria (Bayesian information criterion and Akaike’s informa…

---

## [Optimism-correction for Lasso-Cox](https://discourse.datamethods.org/t/optimism-correction-for-lasso-cox/1297)

<div class="topic-metadata">

**Author:** [@deb\_agbe](https://discourse.datamethods.org/u/deb_agbe)\
**Replies:** 7\
**Last updated:** [February 27, 2019, 12:27pm UTC](https://discourse.datamethods.org/t/optimism-correction-for-lasso-cox/1297 "2019-02-27T12:27:59Z")

</div>

Dear All, I am trying to predict resistance to treatment with a Lasso-Cox model trained on 1435 patients follow for 10 years, 181 events and 33 variables (both continuous and categorical for a total of approximately 50 …

---

## [Calibration Question](https://discourse.datamethods.org/t/calibration-question/1286)

<div class="topic-metadata">

**Author:** [@chrisarg](https://discourse.datamethods.org/u/chrisarg)\
**Replies:** 6\
**Last updated:** [January 28, 2019, 9:01pm UTC](https://discourse.datamethods.org/t/calibration-question/1286 "2019-01-28T21:01:40Z")

</div>

Good evening, I was wondering whether someone more experienced in this , could shed some light into a question about model calibration. I have two models (let’s call them model1 and model2) which were developed in two …

[Previous page](https://discourse.datamethods.org/c/stat/modval/22.md)