# Good reading on when "doubly robust" is worth the effort?

**URL:** <https://discourse.datamethods.org/t/good-reading-on-when-doubly-robust-is-worth-the-effort/17033>\
**Category:** general\
**Created:** [July 3, 2024, 10:52pm UTC](https://discourse.datamethods.org/t/good-reading-on-when-doubly-robust-is-worth-the-effort/17033 "2024-07-03T22:52:16Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![timdisher](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/t/3d9bf3/32.png) [@timdisher](https://discourse.datamethods.org/u/timdisher)\
**Post date:** [July 3, 2024, 10:52pm UTC](https://discourse.datamethods.org/t/good-reading-on-when-doubly-robust-is-worth-the-effort/17033/1 "2024-07-03T22:52:17Z")

</div>

Hello everyone,

Every couple of years it seems there is a wave of doubly robust boosters in my field of interest (evidence synthesis/comparative effectiveness supporting HTA submissions primarily). Does anyone have some nice recommended reading on cases where a doubly robust estimator might actually save me/be worth the extra effort in inheriting all of the workflow difficulties of treatment assignment and outcome modelling?

My knee-jerk reaction is:

1. It doesn’t protect me from variable selection issues since variables needed for adjustment are the same in a treatment assignment/outcome modelling paradigm
2. The applications I’ve generally seen are something like a logistic regression IPTW with a bunch of simple linear terms and then the same formula in the outcome model.
3. I’m worried people will just do a bad version of both models (see number 2).

Is there something I’m missing, or is the argument just that it’s not really that much extra effort, might have a benefit, and probably won’t cause harm?

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [July 4, 2024, 12:14pm UTC](https://discourse.datamethods.org/t/good-reading-on-when-doubly-robust-is-worth-the-effort/17033/2 "2024-07-04T12:14:44Z")

</div>

I have exactly the same reservations, and hope that others can provide relevant references. I know there is a paper studying targeted maximum likelihood estimation, a related method, which found that the standard errors provided by that method have a major bias (too low) and need correction by an impossibly computationally expensive bootstrap process. There is another paper out there comparing one multi-step method with ordinary covariate adjustment, finding no advantage. I think the multi-step method was a “doubly robust” method.

What has not been well understood by the doubly robust community is that ordinary covariate adjustment is pretty darn robust to model misspecification. And it tries to do the right thing in explicitly handling outcome heterogeneity within exposure groups.

---

<div class="post-metadata">

**Author:** ![tspeidel](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/t/ac8455/32.png) [@tspeidel](https://discourse.datamethods.org/u/tspeidel)\
**Post date:** [July 15, 2024, 5:03pm UTC](https://discourse.datamethods.org/t/good-reading-on-when-doubly-robust-is-worth-the-effort/17033/3 "2024-07-15T17:03:21Z")

</div>

I’ve seen several in the ML community linking the new book [CausalML Book (causalml-book.org)](https://causalml-book.org/) from Chernozhukov et al. containing several doubly robust methods. It is from the point of view of the econometric community and the ML label is frustratingly useless in my view. The authors are solid, although I must make the disclaimer that this is not my field and I’ve only skimmed through the book.

Scanning through the book, it’s apparent that the main novelty is in adding the ML label to the title, because the methods and approaches have been around for quite some time (e.g. SEM, penalization via LASSO etc.).

On the other hand, there’s a funny paragraph on [p. 256](https://causalml-book.org/assets/chapters/CausalML_book.pdf) regarding overfitting and certain practices which, together with other remarks scattered throughout the book (e.g. the sidenote on [p.298](https://causalml-book.org/assets/chapters/CausalML_book.pdf)), cast a welcome critical view of certain ML practices.

I don’t know if this is common in econometrics, but I was surprised at the staggering amount of interactions in some of the modelling approaches (and yet no mention of non-linearities in these same models, which in my mind ought to be prioritized over interactions). [Here’s an example in their notebooks](https://colab.research.google.com/github/CausalAIBook/MetricsMLNotebooks/blob/main/PM2/r_ml_for_wage_prediction.irnb).

I have not seen a lot of focus on some of the dominant challenges like selection of variable (which seems to be addressed mostly in the context of LASSO/Ridge/ElasticNet), and evaluation of non-linearities is very narrow going from simple transformations straight into models like RF or ANN. Related to this, I have not seen sample size considerations being addressed. In light of the excellent paper by Riley et al. on model instability ([Stability of clinical prediction models developed using statistical or machine learning methods - Riley - 2023 - Biometrical Journal - Wiley Online Library](https://onlinelibrary.wiley.com/doi/full/10.1002/bimj.202200302)), I would say that’s _one of_ the elephants in the room in these applications.

I’d love to hear from others on this topic.

---

<div class="post-metadata">

**Author:** ![sscogges](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/sscogges/32/4647_2.png) [@sscogges](https://discourse.datamethods.org/u/sscogges)\
**Post date:** [August 18, 2024, 7:36am UTC](https://discourse.datamethods.org/t/good-reading-on-when-doubly-robust-is-worth-the-effort/17033/4 "2024-08-18T07:36:29Z")

</div>

@timdisher  
I agree with you that the usefulness of doubly robust estimators is very questionable if you are just fitting basic parametric models for both the outcome regression and the propensity score model.

Based on my (admittedly limited) reading into the doubly robust literature, the actual utility of doubly robust estimators arises when you want to use machine learning algorithms as part of your estimation procedure. The reason for this is that, because of the way they are constructed, a doubly robust estimator that uses ML algorithms for the outcome regression/propensity score can have standard asymptotic behavior even if the individual ML algorithms used to construct the doubly robust estimator do not. See this paper [https://arxiv.org/pdf/2004.10337](https://arxiv.org/pdf/2004.10337) and in particular this paragraph from the discussion (emphasis mine):

> The **need** for doubly-robust estimators with cross-fitting when using data-adaptive machine learning for nuisance function estimation arises from two terms in the Von Mises expansion of the estimator […] The second term is the second-order remainder, and it converges to zero as the sample size increases. **For valid inference, it is desirable for this remainder term to converge as a function of n−1/2, referred to as as root-n convergence**. Convergence rates are not a computational issue, but rather a feature of the estimator itself. Unfortunately, data-adaptive algorithms often have slower convergence rates as a result of their flexibility. However, **because the second-order remainder term of doubly-robust estimators is the product of the approximation errors of the treatment and outcome nuisance models, doubly-robust estimators only require that the product of the convergence rates for nuisance models be n−1/2**. To summarize, cross-fitting permits the use of highly complex nuisance models, while doubly-robust estimators permit the use of slowly converging nuisance models. Used together, these approaches allow one to use a wide class of data-adaptive machine learning methods to estimate causal effects.

A source of confusion is that many papers on doubly robust estimators start from the premise that double robustness is some super desirable property in and of itself. The reasoning then seems to go that _if_ we want to achieve double robustness, _then_ we should use flexible ML algorithms to ensure correct specification of both the outcome regression/propensity score. From the paper above, we can see that this line of reasoning should almost be reversed. That is, _if_ we are committed to using ML algorithms in our estimation procedure for either the outcome regression or the propensity score, _then_ we must use something like a doubly robust estimator in order to get standard asymptotic behavior (and hence all the standard ways of constructing p-values and confidence intervals). Seen from this perspective, double robustness is not an **end goal** to be pursued, but rather a **necessary condition** for using ML/“data adaptive” procedures.

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [August 18, 2024, 11:49am UTC](https://discourse.datamethods.org/t/good-reading-on-when-doubly-robust-is-worth-the-effort/17033/5 "2024-08-18T11:49:10Z")

</div>

But beware of underestimated standard errors.

---

<div class="post-metadata">

**Author:** ![Uriah](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/uriah/32/1280_2.png) [@Uriah](https://discourse.datamethods.org/u/Uriah)\
**Post date:** [February 17, 2026, 9:12am UTC](https://discourse.datamethods.org/t/good-reading-on-when-doubly-robust-is-worth-the-effort/17033/6 "2026-02-17T09:12:11Z")

</div>

I enjoyed this one - Doubly Robust estimation for loss-based measures under covariate shift:

> **[Robust Estimation of Loss-Based Measures of Model Performance under Covariate...](https://pmc.ncbi.nlm.nih.gov/articles/PMC11636945/)**
>
> We present methods for estimating loss-based measures of the performance of a prediction model in a target population that differs from the source population in which the model was developed, in settings where outcome and covariate data are ...

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [February 17, 2026, 9:49pm UTC](https://discourse.datamethods.org/t/good-reading-on-when-doubly-robust-is-worth-the-effort/17033/7 "2026-02-17T21:49:10Z")

</div>

Nice. Am I reading it correctly that doubly robust estimators were not worth the trouble?

---

<div class="post-metadata">

**Author:** ![Uriah](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/uriah/32/1280_2.png) [@Uriah](https://discourse.datamethods.org/u/Uriah)\
**Post date:** [February 18, 2026, 9:11am UTC](https://discourse.datamethods.org/t/good-reading-on-when-doubly-robust-is-worth-the-effort/17033/8 "2026-02-18T09:11:17Z")

</div>

Not exactly 😅

They try to estimate performance metrics of a prediction model under covariate shift and when the outcome is not available, kind of reminds me de-groot work:

> **[Adjusting for differential-verification bias in diagnostic-accuracy studies:...](https://pubmed.ncbi.nlm.nih.gov/21228702/)**
>
> In studies of diagnostic accuracy, the performance of an index test is assessed by verifying its results against those of a reference standard. If verification of index-test results by the preferred reference standard can be performed only in a...

I’m not sure if it is really likely to have a misspecified conditional loss or a misspecified weights for the shift, it seems more likely to me to have unstable performance metrics because of the weights.

Low propensity → high ipw → unstable performance metric. Ideally I would like to see all 3 variations of the correction (conditional loss, weighting, and doubly robust) and judge for my self.

Sounds reasonable?

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [February 18, 2026, 12:32pm UTC](https://discourse.datamethods.org/t/good-reading-on-when-doubly-robust-is-worth-the-effort/17033/9 "2026-02-18T12:32:47Z")

</div>

It does to me at least.

I would like to always see a comparison with an optimal method that doesn’t use weights in any way. I find that weights are always ad hoc and there’s often a full modeling approach that relies on the log likelihood.