I’m struggling with this specific challenge in my work and I wonder if anyone has a good advice.
I’m trying to fit a prediction model in temporal validation framing, i.e 2023-train, 2024-validation, 2025-test.
In conventional prediction model workflow I try to escape overfitting a specific functional forms, but what can I do when the train / val / test sets are different in terms of outcomes?
I thought about using IPWs for making the train / val look more like the test set but I’m not sure if I can articulate the “propensity” in a meaningful way.
What if there’s some kind of macro event that makes the cohorts look different?
I also thought about some fancy rolling fit that will allow me to use just slices of the longitudinal outcome and use more updated sets, but it still feels kind of hacky.
I bet @f2harrell would preach for moving to bayesian modeling but I’m not sure if I understand why.
The approach you are entertaining is a complex workaround that avoids the subject, which is time trends. I recommend against data splitting by time. Why call a model a failure if it is inaccurate in a later time period than it was trained on? It’s best to model calendar time using for example a restricted cubic spline in year + fraction of a year since 2000-01-01. 100 repeats of 10-fold (at random) cross-validation or a few hundred bootstraps is a preferred way to validate the (time-unified) model and to estimate the amount of overfitting.
Perhaps just to augment Frank’s comments, if you do not take time into account as a covariate, possibly with relevant interactions, you are making at least three assumptions:
Your underlying sample characteristics do not change over time.
The outcome that you are trying to predict is also stable over time.
The relevant practice of medicine, in terms of treatments and related matters, which may affect the relationships between your covariates and outcome, is stable over time.
One or more of these can be false, resulting in an increasing divergence in the predictions from reality over time.
So, while you want a production system to be accurate, within some acceptable amount of error, depending upon how dynamic the assumptions above may be, you will find yourself updating the model(s) with a level of frequency thats reasonably reflects current experience to make future predictions acceptable.
Those model updates can even involve different covariates and/or their representations in the model over time to reflect those temporal dynamics.
Would temporal splits combined with explicit modeling of time still be useful? Otherwise you’re relying on the linear extrapolation from the RCS ‘in production’ and this is not really reflected in the optimism estimate?
