I was wondering if anyone had guidance about including certain candidate predictors in a prognostic model that are potentially part of the outcome definition.
For example, the diagnosis of Alzheimer’s disease is based in part on cognitive scores such as the MMSE. If we were developing a model to predict diagnosis of Alzheimer’s disease in the future, is there anything wrong with including baseline MMSE as part of the model? PROBAST has a section on how this might lead to optimistic estimates of model performance for diagnostic studies (see below). However, it feels that we would want to include baseline MMSE score because it is such a powerful predictor of going on to develop Alzheimer’s disease.
Another example might be if we wanted to develop a model to predict heart failure with reduced ejection fraction (HFrEF). HFrEF is commonly diagnosed by symptoms of heart failure + EF <40% on echo. Is there anything wrong with including baseline EF as part of the model? Someone with an EF of 40-50% at baseline is likely to be at higher risk for developing HFrEF compared to someone with a baseline EF that is normal. This isn’t always the case as there are many ways to get to an EF of 40-50% such that having a baseline EF of 40-50% doesn’t necessarily mean that you are at higher risk for having an EF <40% compared to someone with a normal EF (if for example the person with a normal EF has other factors that put them at extremely high risk). However, on the whole, having a baseline EF 40-50% indicates a higher risk state.
Are there any potential issues with including these as predictors? Should they be included in the model since they are powerful predictors and then acknowledge that this could lead to bias in model estimates?
Thanks for any guidance!
In PROBAST, they discuss in section 3.3 the case of including baseline troponin among the predictors for predicting non-fatal MI since troponin value is used to define an MI.
3.3 Were predictors excluded from the outcome definition?
Outcomes should ideally be determined without information about predictors (see signaling question 3.5), but in some cases it is not possible to avoid including predictors—for example, when outcomes require determination by a consensus panel using as much information as is available. If a predictor in the model forms part of the definition or assessment of the outcome that the model predicts, the association between predictor and outcome will likely be overestimated, and estimates of model performance will be optimistic; in diagnostic research, this problem is generally called incorporation bias (104, 111, 115, 117, 119, 131–134).
Example. Aslibekyan and colleagues (86) aimed to develop a cardiovascular risk score based on the ability of predictors (such as dietary components, physical activity, smoking status, alcohol consumption, socioeconomic status, and measures of overweight and obesity) to predict nonfatal MI. The study reported that MI was defined according to World Health Organization criteria, including cardiac biomarkers, electrocardiography, imaging, or autopsy confirmation. Because the lifestyle and socioeconomic predictors Aslibekyan and colleagues used for modeling do not form any part of this definition of MI, the study would be rated as Y for this signaling question. If the study had included a cardiac biomarker (such as troponin T at initial hospital presentation) among the predictors assessed, this signaling question would likely be rated as N. This is because the initial troponin T measurement may have formed part of the information used to determine the outcome (MI).