# Robust approach for outcome differences in temporal validation framing

**URL:** <https://discourse.datamethods.org/t/robust-approach-for-outcome-differences-in-temporal-validation-framing/28915>\
**Category:** general\
**Tags:** prediction, design, interpretation\
**Created:** [November 2, 2026, 6:35pm UTC](https://discourse.datamethods.org/t/robust-approach-for-outcome-differences-in-temporal-validation-framing/28915 "2026-11-02T18:35:51Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Uriah](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/uriah/32/1280_2.png) [@Uriah](https://discourse.datamethods.org/u/Uriah)\
**Post date:** [November 2, 2026, 6:35pm UTC](https://discourse.datamethods.org/t/robust-approach-for-outcome-differences-in-temporal-validation-framing/28915/1 "2026-11-02T18:35:51Z")

</div>

Hi all!

I’m struggling with this specific challenge in my work and I wonder if anyone has a good advice.

I’m trying to fit a prediction model in temporal validation framing, i.e 2023-train, 2024-validation, 2025-test.

In conventional prediction model workflow I try to escape overfitting a specific functional forms, but what can I do when the train / val / test sets are different in terms of outcomes?

I thought about using IPWs for making the train / val look more like the test set but I’m not sure if I can articulate the “propensity” in a meaningful way.

What if there’s some kind of macro event that makes the cohorts look different?

I also thought about some fancy rolling fit that will allow me to use just slices of the longitudinal outcome and use more updated sets, but it still feels kind of hacky.

I bet @f2harrell would preach for moving to bayesian modeling but I’m not sure if I understand why.

By the way, it’s a PPO OLM 🙂

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [November 2, 2026, 11:36pm UTC](https://discourse.datamethods.org/t/robust-approach-for-outcome-differences-in-temporal-validation-framing/28915/2 "2026-11-02T23:36:35Z")

</div>

The approach you are entertaining is a complex workaround that avoids the subject, which is time trends. I recommend against data splitting by time. Why call a model a failure if it is inaccurate in a later time period than it was trained on? It’s best to model calendar time using for example a restricted cubic spline in year + fraction of a year since 2000-01-01. 100 repeats of 10-fold (at random) cross-validation or a few hundred bootstraps is a preferred way to validate the (time-unified) model and to estimate the amount of overfitting.

BTW I wouldn’t call your time splits “cohorts”.

---

<div class="post-metadata">

**Author:** ![Uriah](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/uriah/32/1280_2.png) [@Uriah](https://discourse.datamethods.org/u/Uriah)\
**Post date:** [November 3, 2026, 6:22am UTC](https://discourse.datamethods.org/t/robust-approach-for-outcome-differences-in-temporal-validation-framing/28915/3 "2026-11-03T06:22:38Z")

</div>

Thanks! I’ll give it a try.

Isn’t it kind of the point of prediction model to be accurate in “production”?

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [November 3, 2026, 11:58am UTC](https://discourse.datamethods.org/t/robust-approach-for-outcome-differences-in-temporal-validation-framing/28915/4 "2026-11-03T11:58:40Z")

</div>

Yes, but when the effect of calendar time is not flat and you ignore time in the model your predictions are inaccurate by definition.

---

<div class="post-metadata">

**Author:** ![MSchwartz](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/mschwartz/32/2579_2.png) [@MSchwartz](https://discourse.datamethods.org/u/MSchwartz)\
**Post date:** [November 3, 2026, 1:41pm UTC](https://discourse.datamethods.org/t/robust-approach-for-outcome-differences-in-temporal-validation-framing/28915/5 "2026-11-03T13:41:44Z")

</div>

Perhaps just to augment Frank’s comments, if you do not take time into account as a covariate, possibly with relevant interactions, you are making at least three assumptions:

1. Your underlying sample characteristics do not change over time.
2. The outcome that you are trying to predict is also stable over time.
3. The relevant practice of medicine, in terms of treatments and related matters, which may affect the relationships between your covariates and outcome, is stable over time.

One or more of these can be false, resulting in an increasing divergence in the predictions from reality over time.

So, while you want a production system to be accurate, within some acceptable amount of error, depending upon how dynamic the assumptions above may be, you will find yourself updating the model(s) with a level of frequency thats reasonably reflects current experience to make future predictions acceptable.

Those model updates can even involve different covariates and/or their representations in the model over time to reflect those temporal dynamics.

---

<div class="post-metadata">

**Author:** ![stephenrho](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/s/4bbf92/32.png) [@stephenrho](https://discourse.datamethods.org/u/stephenrho)\
**Post date:** [November 3, 2026, 3:48pm UTC](https://discourse.datamethods.org/t/robust-approach-for-outcome-differences-in-temporal-validation-framing/28915/6 "2026-11-03T15:48:29Z")

</div>

Would temporal splits combined with explicit modeling of time still be useful? Otherwise you’re relying on the linear extrapolation from the RCS ‘in production’ and this is not really reflected in the optimism estimate?

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [November 3, 2026, 7:41pm UTC](https://discourse.datamethods.org/t/robust-approach-for-outcome-differences-in-temporal-validation-framing/28915/7 "2026-11-03T19:41:52Z")

</div>

Splitting on time will make estimation of time effects have much wider confidence intervals.