# Development of prediction models for multi-omic data: Is this a valid approach to follow?

**URL:** <https://discourse.datamethods.org/t/development-of-prediction-models-for-multi-omic-data-is-this-a-valid-approach-to-follow/7041>\
**Category:** machine learning\
**Tags:** logistic-regression\
**Created:** [August 30, 2023, 11:25am UTC](https://discourse.datamethods.org/t/development-of-prediction-models-for-multi-omic-data-is-this-a-valid-approach-to-follow/7041 "2023-08-30T11:25:47Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![MadawaJay](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/madawajay/32/3954_2.png) [@MadawaJay](https://discourse.datamethods.org/u/MadawaJay)\
**Post date:** [August 30, 2023, 11:25am UTC](https://discourse.datamethods.org/t/development-of-prediction-models-for-multi-omic-data-is-this-a-valid-approach-to-follow/7041/1 "2023-08-30T11:25:47Z")

</div>

Hi all,

I’ve recently come across this paper ([Multi-omic machine learning predictor of breast cancer therapy response | Nature](https://www.nature.com/articles/s41586-021-04278-5)) which discusses about development of a prediction model for multi-omic data. What are your thoughts and comments about the model development and validation aspects? Is this a valid approach to follow? Particularly, taking the average of three models to form the predictor.

This link has the supplementary information: [https://static-content.springer.com/esm/art%3A10.1038%2Fs41586-021-04278-5/MediaObjects/41586\_2021\_4278\_MOESM1\_ESM.pdf](https://static-content.springer.com/esm/art%3A10.1038%2Fs41586-021-04278-5/MediaObjects/41586_2021_4278_MOESM1_ESM.pdf)

Thanks,  
Madawa

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [August 30, 2023, 6:07pm UTC](https://discourse.datamethods.org/t/development-of-prediction-models-for-multi-omic-data-is-this-a-valid-approach-to-follow/7041/2 "2023-08-30T18:07:26Z")

</div>

Unless one has tens of thousands of patients, a binary response/no response target variable contains far too little information to yield reliable associations with high-dimensional features.

---

<div class="post-metadata">

**Author:** ![MadawaJay](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/madawajay/32/3954_2.png) [@MadawaJay](https://discourse.datamethods.org/u/MadawaJay)\
**Post date:** [August 30, 2023, 10:03pm UTC](https://discourse.datamethods.org/t/development-of-prediction-models-for-multi-omic-data-is-this-a-valid-approach-to-follow/7041/3 "2023-08-30T22:03:42Z")

</div>

Yes, indeed. That is the major problem.

1. Only 147 patient samples in the training model (but don’t know what is the distribution of outcome across the variables)
2. Using CV will further reduce sample size in the model training step (in a low sample size study like this)
3. Using random forrest/SVM for this data.  
Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints: [Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints | BMC Medical Research Methodology | Full Text](https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-14-137)
4. Proposed ensemble method. Are there any references for these type of model averaging methods?
5. Only reporting C-index as the performance measure and no other methods are provided. No model calibration information provided for the model.

---

<div class="post-metadata">

**Author:** ![R\_cubed](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/r_cubed/32/1518_2.png) [@R\_cubed](https://discourse.datamethods.org/u/R_cubed)\
**Post date:** [September 1, 2023, 10:10am UTC](https://discourse.datamethods.org/t/development-of-prediction-models-for-multi-omic-data-is-this-a-valid-approach-to-follow/7041/4 "2023-09-01T10:10:28Z")

</div>

> [@MadawaJay](#):
>
> Proposed ensemble method. Are there any references for these type of model averaging methods?

I posted some references to older but still relevant papers, in these threads:

1. Bayesian Model Averaging:  
[RMS Describing, Resampling, Validating, and Simplifying the Model - #63 by R\_cubed](https://discourse.datamethods.org/t/rms-describing-resampling-validating-and-simplifying-the-model/4790/63)

2. Frequentist Model Averaging:  
[Should one derive risk difference from the odds ratio? - #447 by R\_cubed](https://discourse.datamethods.org/t/should-one-derive-risk-difference-from-the-odds-ratio/4403/447)

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [September 1, 2023, 12:28pm UTC](https://discourse.datamethods.org/t/development-of-prediction-models-for-multi-omic-data-is-this-a-valid-approach-to-follow/7041/5 "2023-09-01T12:28:00Z")

</div>

General coment: ensemble methods and model averaging don’t usually offer any performance gain and are much harder to interpret, as compared to a flexible single model. Exception: when there is uncertainty about model form, e.g., you want to average Cox models with accelerated failure time models.

---

<div class="post-metadata">

**Author:** ![giuliano-cruz](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/giuliano-cruz/32/4666_2.png) [@giuliano-cruz](https://discourse.datamethods.org/u/giuliano-cruz)\
**Post date:** [September 1, 2023, 4:09pm UTC](https://discourse.datamethods.org/t/development-of-prediction-models-for-multi-omic-data-is-this-a-valid-approach-to-follow/7041/6 "2023-09-01T16:09:26Z")

</div>

I think the main challenge with high-D model development is calibration. You can still get at models with good discrimination but these need to be re-calibrated - something the omics community seems to overlook. As I see it, there is a need for developing sample size criteria specifically for predictive modelling using high-D data, at least under a workflow that assumes, e.g., recalibration in a new sample or some form of conformalized prediction. BTW, this was part of my MSc but then life happened and it ended up postponed to the PhD.