Nature Sleep Prediction Study

[trumanfrancis]

This study on sleep and disease prediction is being heavily promoted A multimodal sleep foundation model for disease prediction | Nature Medicine

Could there be selection and detection bias since the subjects were all referred or sought a sleep clinic analysis—sleep clinic patients likely have higher healthcare utilization, leading to more diagnoses being captured.

They acknowledge this in the Discussion:their cohort comprises referral patients with suspected sleep disorders, not the general population. The model’s generalizability to screening populations is uncertain

No calibration plots or metrics are presented. This seems to be a major omission for a prognostic model. They report only discrimination metrics (C-Index, AUROC). Shouldnt you want to see calibration in the large, calibration slope, or at minimum observed vs. predicted risk plots across deciles.

[f2harrell]

Binning risk is a no-no so I highly recommend against the use of deciles. But we need to see calibration in the small and large, with no binning, accounting for overfitting. Any claim to be a clinical prediction model should be dismissed outright without a rigorous analysis of continuous calibration. Authors need to care about absolute predictive accuracy!

[stephenrho]

100% agree - the absence of calibration is a major omission. The editors/reviewers should have required following TRIPOD-AI.

Also given the hype about this it is remarkable to see how marginal the increases in discrimination are over the baseline model, which it is important to note only included age, gender, BMI, and race/ethnicity.

Of course, datamethods readers don’t need reminding that comparing c-statistics is not how to look at added predictive value…