I am working on validating a 1-year risk of an event in EMR data. The previously developed model (which is binary logistic) is already integrated in the EMR, which calculates 1-year risk of the event every time a patient has a visit (using info from that visit and previous 180 days). But a patient could have several visits in a year and each time the risk is calculated. So, what I have is a patient-visit data with likely different risk scores at different visits. I am basically censoring the patient-visit observation whenever next visit happens and a new predicted risk is added. How do I go about validating these one-year risks in this time-to-event type of setting with censoring? Do I:
Follow the external validation techniques for survival model (following for example, this paper: https://www.acpjournals.org/doi/10.7326/M22-0844). So, converting 1-year risk to risk at each follow-up time for each observation (assuming constant hazard), and estimating O/E, calibration slope/plot, Harrell’s c-statistic..; OR
Do I use inverse probability of censoring weights (IPCW) and apply the approaches for discrimination, calibration for binary prediction with the IPCW weights?
I am struggling to figure out what the right approach is in such situations.
I would like to explore background issues first. For a binary logistic model to work in the first place, no patient could have been lost to follow-up before one year. Is that definitely the case?
Second, binary logistic models are very inefficient in this setting, and freeze the user to use only a one-year prognostic horizon. Time-to-event models would have allowed prognostication about any time frame that was within the study’s follow-up window. They would have also handled variable censoring times.
These risks come from EMR vendor-proprietary models. I think it was developed simply retrospectively following patients for one year in one healthcare system (ignoring loss to follow-up). I am exploring ways to go about at least evaluate it in our health system. But these risks are calculated and added to patients’ records at each visit, which leads to issues of varying time between visits (and multiple visits and recalculation of the risk within one year itself).
The naivest approach for external validation in this case may be to index on one risk (maybe the first) and follow-up for one year for outcomes, ignoring other visits and other estimated risk, and evaluate discrimination/calibration. But this seems inefficient to me, as we are ignoring lots of clinical info from the follow-up.
This entire exercise is based on an invalid analysis that will result in a major calibration problem. This project should be abandoned. Note that one censoring is ignored, what is being predicted is a function of the enrollment date of a patient and is not one year survival.