Should one derive risk difference from the odds ratio?

data analysis bayes
207

I am still thinking about some of the papers that you suggested, and I will get back to you about about multistage carcinogenesis after I have processed it a little bit more.

But I want to thank you sincerely for pointing my attention to Weinberg (AJE 1986). I wrote in my newsletter (linked above) that Sheps’ insights have been repeatedly rediscovered independently across several academic fields. This is another instance of this, one which I was previously not aware of.

Weinberg suggests using models that are additive in log risk (risk ratio) when the risk factor is “preventive”, models that are additive in log of the complementary risk (survival ratio) when the risk factor is causal, and that the survival ratio can be approximated by the risk difference with a rare-disease assumption.

This is exactly what I have been struggling to convince the epidemiologic community about for the last 6 years, so it is surprising to see the same point being made in AJE in 1986 by a highly regarded and influential epidemiologist.

While I am still open to the possibility of being proven wrong by other papers, Weinberg (1986) is not a counterexample to my claim that there is no plausible data generating mechanism that leads to stability of the risk difference (except when intervention increases risk and the risk difference it is an approximation of the survival ratio) or of the risk ratio (unless intervention reduces risk and other conditions are met)

1 Like
208

I read several of your “old” papers in Murray Mittleman’s course Epi247 (otherwise known as Epi 24/7). They are excellent papers which taught me a lot, both in terms of the content of the paper itself, and also in terms of how to write convincing methods papers (which is something I still have a lot to learn about).

But in general, I find that it it is often easier to learn these ideas from secondary sources than from the original papers. Methods papers are optimized for convincing skeptical reviewers, for precision and for completeness of the argument. This means it isn’t really possible to focus fully on the insight that might make the idea “click” in the head of a student.

Papers which have been around for more than 10-20 years have had a chance to percolate through the academic zeitgeist, they have been processed by several thinkers who may have simplified the central insights and how they can best induce the “click” in their students. This speeds up learning, and makes it greatly more enjoyable.

I wouldn’t try to learn calculus from Newton or Leibniz’ original writings, and likewise, I wouldn’t try to learn causal inference from Robins (1986).

1 Like
209

Good you finally read Weinberg. Still, you love to jump to conclusions before studying too much. Please read p. 74-83 of Modern Epidemiology 3 before shooting off a reply again.

210

Once again you make up straw-man directives. No one said you needed to initially learn from old sources or go back in history to primordial beginnings (Newton’s era is sometimes presented as the origin of epidemiology, with studies by Graunt, Halley and Petty on mortality). Although I will say those that do go back to origins often see (as I did) how smart and nuanced some of the founders were, sometimes in ways that have been lost over time. For me this was especially so in statistics, with “An Essay towards solving a Problem in the Doctrine of Chances” by Thomas Bayes (posthumous 1764).

Again, as you saw with Sheps long before you were born (and not long after I was) and now with Weinberg, there is plenty of importance much more recently that is worth revisiting. Which reminds me, regarding Sheps whatever became of your reading of Khoury et al.? (“On the measurement of susceptibility in epidemiologic studies”, Am J Epidemiol 1989;129:183–190).

212

My apologies for posting this twice, I accidentally hit “post” on the last version before I finished writing it.

(1) Khoury et al (1989) contains the same basic argument for the case where an intervention increases risk, but as far as I can tell, does not consider the situation where the intervention decreases risk. I have added it as a reference in our paper and will add it to the newsletter list of others who have considered closely related ideas

(2) Modern Epidemiology contains a lot of good, correct discussion about the relationship between response types and effect measure stability. This is fundamental to the argument structure of the points I am making.

On page 78, there is an argument that when we are considering two interacting exposures, the absence of “interaction types” implies stability of the risk difference. This is a strong assumption, but I will acknowledge that it works in theory.

The first thing I asked myself after seeing this, is why it doesn’t lead to a paradox in situations where effect stability would lead to risks above 100%. I think the answer to that, is that it is set up for causal interaction instead of effect modification. That setup factors out the possibility that risk due to background factors varies between groups; any baseline risk difference is due to the causal effect of one of the interacting factors. I am therefore not convinced that this result would allow you to establish stability of an additive parameter between groups whose baseline risk differs due to background factors.

I realize I never explicitly said that this was my goal, so I can see that it may have been reasonable to interpret me as making a broader claim than what I had intended. I would like to note that for the purpose of the generalizability problems that motivate me, what matters is effect modification rather than causal interaction: Doctors generally cannot intervene on the observable markers of background risk that they use to guide their decisions.

213

Well how about you try and convince yourself one way or another by plugging some numbers in for the proportions of noninteraction types allowed (which have only one constraint: they must to sum to 1) to get the risks from the formulas on p. 77 of ME3 and see if you can vary the baseline risk (I presume you mean R00) while holding the RDs constant? I think you will find that yes the RDs can be constant while R00 varies, limited only by the fact that the RDs must sum to equal or less than 1-R00; if as typical R00 is not very large then in RD terms this is a vast range for additive effects to live in. In other words, your original claim is wrong…

You can research it if you want, but I seem to recall this was known in the bioassay literature in the 1920s and is one of the reasons product terms in linear models came to be called “interactions” (much later, statisticians in their unbounded carelessness began calling all product terms “interactions” regardless of the link function for the linear predictor, even though a connection of those terms to actual biologic interactions is absent from all but a few models).

P.S. Added belatedly: For those with a historical bent, in a paper invited by an engineering toxicology journal (“Elementary models for biological interaction”, Journal of Hazardous Materials 1985;10:449-454) I attempted to provide a connection between bioassay and epidemiologic models for interaction.

1 Like
214

At the risk of adding even more to everyone’s Sander Greenland reading list, I submit the following:

It goes into a number of issues discussed in this thread, especially the distinction between “main” effects and “causal” effects, which depends on the context.

Blockquote
An effect is a coefficient of study exposure in a generalized linear model for the outcome of
interest … The parametric definition arose in the context of randomized experiments … Given randomization, the definition is not very misleading.

Read the entire open access paper for details. The ellipses skip over a lot of important fundamentals.

While this thread has a wealth of information on the details of a data analysis that deserve an independent thread, I wanted to clarify where the essential area of disagreement is.

From what I can interpret, the dispute is whether Y is viewed as inherently ordinal (ie. a patient reported outcome, a functional capacity assessment, etc) or a proxy for a parametric variable in a causal model (ie. going from a cluster of qualitative observations to the discovery of a biophysical cause – ie. Alzheimer’s disease, physiology of pain syndromes, etc.)

As an example, the dominant perspective in chronic pain (maybe fashion is the more honest word) is the “new” biopsychosocial model (which isn’t all that new) while “biomechanical” models are criticized (often for erroneous reasons).

The way I interpret the disagreement between @Sander and @f2harrell (using pain syndromes as an example):

  1. At the initial onset of an episode of pain and decreased function, efforts to discover a causal mechanism (reducing ordinal observations to physiological mechanisms) have the best chance of success, so absolute measures of effect (RD, RR) should be computed from logistic models (unless some other model is more appropriate). This is a question of basic science that might provide guidance on treatment.

  2. But at later stages in the evolution of pain syndromes, when the basic question of causation is much more difficult to answer, and questions of treatment efficacy still remain. As these outcome measures of functional improvement are inherently ordinal, the covariate adjusted distribution derived from the OR is the most appropriate. Here, relative measures of effect adjusted by individual risk factors address the essential research question of direct interest to clinicians, patients, and other stakeholders.

Is this a reasonable interpretation of the dispute above? I infer this from my familiarity with the published writings of both Sander and Frank. In all of the examples in Sander’s papers, there was a reasonable prior that the pathophysiology of the disease was connected to chemical exposure and could be reduced to some (parametric) biochemical model.

Frank’s writings have intersected with my field of rehabilitation, where he has been listed as one of the authors on papers that deal with low back pain. In addition, his most recent project has been the promotion of ordinal outcome measures, where his recommendations have substantial merit (in my opinion).

215

For readers who are jumping into the discussion at this stage, I want to be clear that this part of the discussion relates only to a claim I made in this thread that no mechanism has been proposed that would guarantee stability of the risk difference, and not to any argument we make in the preprint. I am less certain about the claims I am making in this post than anything we wrote in the preprint; if it turns out I’m making a mistake in this part of the discussion, it is highly unlikely to affect the argument in manuscript.

Maybe I am being thick, but after thinking about this for a while now, I still believe that at least some form of the argument I made in this thread is correct (though possibly phrased imprecisely).

Modern Epidemiology considers two interacting factors, which I will relabel as V and A. Suppose the outcome is Y. I am interested in whether the effect of A on Y differs between people who have V=0 and people who have V=1.

There is then a result which easily can be restated to tell you that if there are no “interacting response types”, it follows that Pr(Y^(a=1, v=1)) - Pr(Y^(a=0, v=1)) = Pr(Y^(a=1, v=0)) - Pr(Y^(a=0, v=0)). This is what Tyler refers to as absence of causal interaction on the additive scale.

I claim that what usually matters is not whether there is no causal interaction, but rather whether there is no effect modification, i.e. whether Pr(Y^(a=1) | V=1) - Pr(Y^(a=0) | V=1) = Pr(Y^(a=1) | V=0) - Pr(Y^(a=0) | V=0) . Absence of causal interaction does not imply absence of effect modification, and doctors will generally find themselves in a situation where it is impossible to intervene on V. Effect modification is therefore usually a much more useful concept for clinical decision making than causal interaction.

Even if doctors are assigning two interventions simultaneously, scale-stability relative to variation in baseline risk due to background causes is much more central to the decision-making problem than scale-stability between the two interventions relative to each other

You might be able to argue that deviation from effect measure additivity implies the existence of some past covariate U which V is a marker for, and which has “interaction response types” with Y. But I am not sure this solves any practical problems unless you can condition on U. I note that this is likely to be a very large set of covariates, my intuition is that it will functionally require adjustment for every cause of Y.

If you wanted to set up the argument from Modern Epidemiology in terms of effect modification, I think it would be preferable to instead of considering 16 joint response types, to consider 4 response types for A in V=0, and separately, 4 response types for A in V=1. But I think it would then be very hard to come up with a plausible biologically interpretable argument (based on these abstractions) that leads to stability of the risk difference for A between strata of V…

For a concrete example (for readers of this discussion who are less familiar with the distinction between effect modification and interaction than Sander), suppose A is alcohol and V is smoking. If there is no causal interaction, this means that in trial where you randomize both alcohol and smoking, the effect of alcohol will be equal between V=1 and V=0. However, when smoking is assigned randomly, it follows that if there is a difference between the two conditional baseline risks for the alcohol intervention, Pr(Y^(a=0) | V=0) and Pr(Y^(a=0) | V=1), then this is entirely due to the causal effect of smoking, not due to smoking being a predictor of risk. This observation is essential for why joint response types can get you the rest of the way towards an argument for stability when considering causal interactions.

(Edited to add: Maybe it is possible to interpret the model as implying no effect modification because any interaction with a past covariate U which predicts V will be reflected in the response types between V and A. But then it becomes really challenging to give a meaningful interpretation to what an “interaction response type” is, and in order to rule out their existence, you need to rule out interaction with everything in the past…)

216

Anders: While I’m all for exploring practical limits of simple causal models and their implications, right now it looks to me that you are also doing what you complained about Doi did to defend his claims: Adding unnecessary and even irrelevant diversions that only obscure your mistake in issuing too sweeping a denial of the existence of simple mechanistic causal population structures under which the causal RD can be stable (unmodified) despite variation in background risk (risk under absence of both factors).

For those not having a copy of Modern Epidemiology (Ch. 18 in 2nd ed, Ch. 5 in 3rd ed) where the results at issue are covered, they were taken from Greenland S, Poole C, “Invariants and noninvariants in the concept of interdependent effects”, Scand J Work Environ Health 1988;14:125–129 (as usual, available on request). These citations delineate an entire class of mechanisms which can produce constant, additive RDs across a large range of baseline risk variation. In these mechanisms, for every individual the x effect does not depend on the z level and the z effect does not depend on the x level. A consequence is that there would be no modification of the RD (no effect-measure modification) for either x across z or z across x, i.e., perfect additivity of separate-treatment RDs to get joint treatment effects.

Variation in the baseline risk across populations need not destroy this additivity. I understand that such mechanistic RD additivity seems “paradoxical” given the range restrictions, but that just shows how your intuition suffers the same sort of limits as does Doi’s and Harrell’s (mine would too but for the fact that I encountered these results 40 years ago). As with any causal model, whether such noninteractive, non-modifying structures are plausible or realistically transportable is context dependent and hence largely in the eye of the beholder, so is a separate topic that we won’t resolve here.

It’s fine to pursue the source of an incorrect initial intuition to see what can be learned from it, but randomization and prediction are irrelevant to the present case: The above cites and my points are about the true causal RDs computed directly from the full x*z potential-outcome vector of each population member under every exposure combination. Whether we are accurately estimating effects or predicting risks are vast topics that do not bear on the results.

3 Likes
217

I’ll get back to you in a week’s time after having thought through this thoroughly. For now, I will just point out that your read on my psychology is wrong. I try very hard to always acknowledge my mistakes when someone convincingly points them out. Moreover, I am confident that I have a track record to back up that claim.

I don’t know whether what’s happening is that I’m not smart enough to understand your argument, or that I’m not communicating precisely enough for you to understand what I’m saying. But in either case, it is not because I am making any attempts to obscure my mistakes or defend a lost position.

2 Likes
218

If we indicate different baseline risks by a, b, c… etc then the implication is that:
a(RR-1) = b(RR-1) = c(RR-1) …
Which means that, by your own admission, in essence there are an entire class of mechanisms suggesting non-constancy of the RR.

219

“By your own admission”? Where in this thread has anyone claimed the RR is constant or nearly so outside of special models? You and coauthors are the ones who made erroneous assertions about constancy of the OR based on a confusion of statistical models with mechanistic causal models, combined with a misanalysis of meta-analytic data; those mistakes are what kicked off this thread and the exchanges in JCE. What is your point now? Have you finally learned that the OR is not constant under simple mechanistic models?

My position all along has been that in epidemiologic practice any assumption of constancy is just a statistical convenience which rarely has any credible basis in data or biology (I can’t speak for other fields, like bioassay which has a century-long literature on the topic). It’s been known for generations and obvious from the math that (with rare exceptions) constancy of one measure means nonconstancy of the rest - so if the RD is constant, we get nonconstancy of the OR, RR, relative difference, AF, and so on. And it’s obvious from correct meta-analyses that it’s rare indeed that one can correctly claim even one of them is constant (as opposed to mistakenly claiming constancy based on failure to “reject” using nearly powerless statistical tests).

When (as usual) the direction of heterogeneity is unclear, the best we can hope from assuming constancy is that it removes enough noise to more than compensate for the bias it creates - the old “bias vs. variance” tradeoff in minimizing mean-squared error. But that hope is not guaranteed and depends entirely on the target of estimation. Better still is to allow some heterogeneity controlled by penalties (priors) that can average the extremes between unconstrained heterogeneity and complete homogeneity, as allowed by hierarchal modeling of product terms. That became computationally feasible 40 years ago and by the 1990s could be found in textbooks, e.g. p. 431 of Modern Epidemiology 2nd ed. 1998 (p. 438 of the 3rd. ed.).

6 Likes
220

A few days ago, I wrote this in this thread:

Sander then correctly pointed out an example from Modern Epidemiology page 76, table 5.2, in which they show that if there are no “interaction response types" (which roughly means that for every individual, at least one of the interacting factors A and V must have no effect, for both values of the other interacting factor) then there will be stability of the causal risk difference for A, between the setting where we intervene to set V to 0, and the setting where we intervene to set V to 1. If we additionally assume exchangeability (independence between V and Y(a,v)), the causal risk difference for A will be equal between groups defined conditional on the observed value of V.

I do not contest this, and I am sorry about the imprecision in my claim. I do however maintain that this result is not very relevant to the kinds of decision problems that I am considering, in which a clinical decision maker needs to individualize the causal effect of the intervention (A) to a patient with V=v

Just so everyone is on the same page, let us differentiate between three subtly different phenomena:

  • Statistical interaction/Product term in observational model: Both interacting factors are observational
  • Effect modification: Primary intervention variable is counterfactual, the other factor is observational
  • Causal interaction: Both intervention variables are counterfactual

The reason I consider effect modification as more relevant to the decision problem than causal interaction, is that in order to individualize treatment to a person with V=v, doctors cannot intervene on V, and therefore need to find some aspect of reality that is invariant over observed V.

I am fairly sure you will agree that if there exists a common cause of V and Y, which we will call U, then even if there are no interaction response types between V and A, there is no guarantee that there will be no additive effective modification. To see why, you can stratify table 5.2 by U, and work out Pr(Y^(a) |​ V=v) for all a and v (using law of total probability over U). I am not going to type up the maths, but I would be very surprised if this claim is incorrect.

I note that if we are considering causal interaction, then any differences in baseline risk are due to the causal effect of V. This model therefore cannot give you a reason to expect additive stability between groups whose baseline risks differ because of common causes of V and Y (at least not without expanding the model to also claim no interaction types between A and all predictors of V ).

So in summary, What I should have written is something like this:

“Would you be able to formalize the biological story for a data generating process that leads to absence of effect modificaiton, i.e. stability of the causal risk difference across groups constructed based on observational variables, and whose baseline risk differs arbitrarily?.

3 Likes
221

Thanks Anders for the concession and clarifications. I certainly agree that there is no guarantee of observing additivity if there exists a common cause U of V and Y, because we now have confounding of the V effect on Y by U. To be more precise, we should not expect additivity of the observed RD(AY|V) and RD(VY|A) even if the causal AY and VY RDs do add, because in your set-up the observed RD(VY|A) is confounded by U.

It appears that you want to use V to guide decisions about treatment A without concern about confounding of V effects; that’s fine. But as far as I can see, all you are saying is that you are concerned with a case in which we think RD(AY|V) is unbiased enough for the AY effect to provide to clinicians, but we may have failed to control enough confounding of RD(VY|A) to make valid inferences about the V effect on Y. In that case it should be no surprise that we cannot make valid inferences about the interaction of A and V effects.

In sum, the one point I see coming out of your arguments is that if you want to study the causal interactions of A and V on Y, you have to control confounding of both A and V. More precisely and generally: To study causal interactions among components Xj of a vector of exposures X = (X1,…,XJ), you have to control confounding of X, e.g., by blocking all back-door paths from X to Y. A corollary is that to ease correct deductions about confounding control for studying causal interactions from a DAG, we ought to examine the graph that replaces the potentially interacting exposures with a single vector of them.

Given that your concern translates into a higher-order confounding problem, my answer to your query would be Yes: I can easily formalize a biological story for the data generating process that leads to absence of RD modification (i.e. stability of the causal risk difference across groups constructed from observational variables, with baseline risks differing arbitrarily within logical constraints): All I need for that is (1) no AV-interaction response types and (2) sufficient control of confounding of effects of the exposure vector (A,V) on Y; then there will be no modification of the observed RD(AY|V) across V within confounder levels and also after marginal adjustment (or after averaging using any shared weighting scheme). Note that (2) is no more stringent a requirement than that for mediation analysis, in which we replace the baseline variable V with an mediator M between A and Y.

Do you agree (in which case this subthread should be done) or can you exhibit a mistake in my reasoning?

2 Likes
222

This discussion on causal effect modification seems interesting so lets take an example. If we run some GLMs on Sanders example in the recent non-collapsibility paper in JCE and accept that there is neither confounding nor a sample artifact, the product term (EXP) in the log-risk model is 0.5 (sub-multiplicative) and in the logistic model is 1.6 (super-multiplicative). Thus male gender (M) modifies treatment (Rx) related death risk (decreased by 50% in M compared to females (F)) while simultaneously M also modifies treatment related death odds (increased by 55% in M compared to F). However in both M and F, Pr(death) increases with Rx (M from 0.6 to 0.9 and F from 0.1 to 0.3). In summary the association measures are (compared to baseline of noRxF):

Rx RR=3 OR=3.9
M RR=6 OR=13.5
RxM RR=9 ↓ OR=81 ↑

Obviously the results are conflicting because these measures are measuring association differently and our point has been that no heterogeneity of treatment effects that are meaningful should be derived from measures of association that measure that association poorly.
The inequality of the association of treatment with death in strata of gender (or the non-null product term) can be called association modification. This, of course, happens commonly even if gender is not a cause of death or not associated with a cause of death e.g. because of artifacts of the sample. Because association modification is mathematically reciprocal – If gender modifies the Rx-death association then Rx modifies the gender-death association so it suffices to just call this association modification or a statistical product term and avoid use of the term interaction.
The unfortunate reality is that even when Rx and gender are both causes of death and even when confounders of their marginal association with death are absent and even when there is no artifact of the sample, there is still no guarantee that the association modification between Rx and gender with respect to death corresponds to causal effect modification if the measure of association is poor (i.e. measuring association poorly due to interference by baseline risk).
The last point above is what we are discussing in this thread. We have three common choices in medical decision making around association measures for binary outcomes – RD, RR or OR. If any of these measure the intended association poorly, then the association modification will correspond poorly to effect modification aka heterogeneity of treatment effects even if confounding or artifacts are absent. This is why we see different results from different measures. In the example above, it is quite clear that the RR is spurious for the RxM comparison to baseline (noRxF) as examination of the data clearly demonstrates that belonging to the RxM group gives the greatest increase in mortality. This sort of non-monotone relationship of measures with true association (many ways this can be measured) seems to be a hallmark of collapsible measures of association.

224

Anders, I see you posted this:

Your site finally let me post the following comment which seems relevant here as well:
Tell us Anders, where did you find out about Abbott’s paper and why did you obtain and read it? After all, here at the Datamethods blog on July 4 you repeatedly dismissed my comments that mechanistic models and risk results go back to the 1920s, replying
“It therefore isn’t at all obvious to me that reading the old literature is generally the best use of time” and “I wouldn’t try to learn calculus from Newton or Leibniz’ original writings, and likewise, I wouldn’t try to learn causal inference from Robins (1986)”,
to which I replied at length, ending with:
“For those with a historical bent, in a paper invited by an engineering toxicology journal (“Elementary models for biological interaction”, Journal of Hazardous Materials 1985;10:449-454) I attempted to provide a connection between bioassay and epidemiologic models for interaction.” Among other things this 1985 paper cites W.S. Abbott, A method of computing the effectiveness of an insecticide, J. Econ. Entomol., 18 (1925) 265–267.

Also, you never responded to my deduction of risk additivity from the assumptions of no interaction response types and of no confounding of either single or joint factor effects.

225

To take this one step further, lets take the same example again (Sander’s JCE paper example). Lets denote X & Z as treatment & gender and denote death as Y. If we were to interpret these results in terms of marginal effects, we would typically look at the effect of the explanatory variables on the probability of death. However let us start with the logistic model and in the table below odds are presented and not probabilities and they are just the expected number of people who die for every person who lives. For a male the ratio is 9/1.5 = 6 and for a female that is 0.43/0.11 = 3.9. The interaction effect between X and Z is 1.55 which tells us (on the multiplicative scale) how much the effect of treatment differs between males and females – the effect of treatment for men is 6/3.9=1.55 times that for women.

We can compute the marginal effect as the difference in odds (which is what the fuss is all about) and from the table the marginal effect of treatment for women is 0.43-0.11 = 0.32 and for men is 9-1.5 = 7.5. As soon as we change to differences in odds we see that this is different from the multiplicative effect.

Predictive margins Number of obs = 300
Model VCE : OIM
Expression : exp(xb())
over : z x
Delta-method
Margin Std. Err. z P>z [95% Conf. Interval]
z#x
0 0 0.1111111 0.037037 3 0.003 0.0385199 0.1837024
0 1 0.4285714 0.093522 4.58 0 0.2452718 0.6118711
1 0 1.5 0.4330127 3.46 0.001 0.6513107 2.348689
1 1 9 4.242641 2.12 0.034 0.6845771 17.31542

We can do the same for the log risk model and here we get RR 3 in females, 1.5 in males - the effect of treatment for men is 1.5/3 = half of that for women. Obviously the discussion in this thread uses this as the standard against which the OR is judged and based on this the OR results are somehow biased. Of course, one could easily argue the opposite, but in order to resolve this debate we need to understand which argument is correct.

Predictive margins Number of obs = 300
Model VCE : OIM
Expression : exp(xb())
over : z x
Delta-method
Margin Std. Err. z P>z [95% Conf. Interval]
z#x
0 0 0.1 0.03 3.33 0.001 0.0412011 0.1587989
0 1 0.3 0.0458258 6.55 0 0.2101832 0.3898168
1 0 0.6 0.069282 8.66 0 0.4642097 0.7357903
1 1 0.9 0.0424264 21.21 0 0.8168458 0.9831542

In support of the log-risk argument has been one simple observation: non-collapsibility. This is not limited to medical researchers but if we look at Social Science (Mood (2010) cited 2665 times) the same argument is given as was echoed by Edward Norton (which was discussed in one of the threads in datamethods):
a) Somehow the OR reflects unobserved heterogeneity so it is not a substantive effect
b) Models with different independent variables cannot be compared with the OR because the unobserved heterogeneity varies across models
c) Even with the same samples, unobserved heterogeneity means non-comparable ORs across samples, groups or time.
As Buis has pointed out, one characteristic of logistic models that many of those that discussed the OR (e.g. Norton) find problematic is that if we add a variable to a logistic regression model, the effects of the other variable(s) is very likely to change, even if this additional variable is uncorrelated with the other explanatory variables. This is essentially the argument laid out by Sander in the JCE paper. The OR(XY) was 2.75 and addition of the gender variable (with (OR(XY)=3 or without (OR(XY)=4.5 the product term) to the model changes OR(XY) despite gender and treatment being uncorrelated. All that is required is that this new variable influences the outcome, it does not need to be correlated with the other explanatory variables and of course this is what Sander means by non-collapsibility. This does not happen in a log-risk model (with the product term) and hence a conclusion was drawn – non-collapsibility is bad because control variables are added to a model to adjust for possible confounders and if a variable is not a confounder we would like the effects of our explanatory variable of interest to not change when adding non-confounders. This has been the argument in the last 40 years – in a logistic regression the scale of the latent dependent variable changes when a new variable is added even when uncorrelated with the others or if the same variables are modeled in different samples etc.
What Buis has suggested previously and what Frank has been pointing out for a decade and what Doi et al paper suggests is that a logistic regression models linear effects of explanatory variables on the ln-odds of experiencing the event i.e. the log of the expected failures per successes (or vice versa). This is a fixed scale and nothing changes when uncorrelated variables are added to the model. So the effect on death in this example is influenced by many variables (even if we only knew two) and in any model this effect is conditional on what we know about the patient and is not an absolute property of the patient and learning more about the patient (gender) must change our perception of the chance that the patient dies attributable to treatment. As Buis has said, we are not uncertain about the patients in our dataset as we have already observed whether or not they die, we are using the model to make an assessment of how likely it is that someone with these characteristics will die. This is conditional on everything mentioned before – sample, variable choice, etc and this non-collapsibility is a desirable property of a logistic model because it reacts to new information as we would expect it to do. So when we added gender its not just the change in plausibility of the outcome due to adding gender but change in the effect of treatment due to adding extra information. Of course we can use any metric to measure this “expectation” – RR, RD, OR – but only the OR will have this desirable property because it is non-collapsible. This means that adding unconfounding prognostic variables should increase the effect of other variables, which is exactly what happens in logistic regression. As I have demonstrated in one of the posts above – if Z is a random variable not correlated with Y then the logistic model will still act as expected (no change in OR(XY) unless we add the product term because in this case we are more likely than not creating an artifact of the sample and therefore we have to be very careful to start from the position of homogeneity unless we have strong reasons not to.
Finally, yes, models with different variables added will have different OR(XY) as described above and yet Doi et al suggested that the OR is the preferred association measure for meta-analysis across different studies where the variables in the model and the sample are different. This in my view is akin to systematic error – so long as the different ORs vary on a magnitude less than the effect of treatment of interest, the meta-analytical OR will have clinical relevance otherwise the outcome is largely a function of these other variables. Can we use the RR in meta-analysis? Probably not and apart from the reasons above, there are other reasons suggested by Bakbergenuly , Hoaglin & Kulinskaya.

2 Likes
226

For those interested in a part of the current stat literature far better informed about causality and much more relevant for patient-centered treatment-effect estimation than some of the comments still being posted here, have a look at Gao Hastie Tibshirani: Assessment of heterogeneous treatment effect estimation accuracy via matching
https://onlinelibrary.wiley.com/doi/full/10.1002/sim.9010
-It’s good to see that some veteran senior statisticians are capable of incorporating modern causal concepts into their models and methodology.

1 Like
227

I’d be more at ease with that approach if someone develops a way to calculate the needed sample size for reliability of results and the applications have at least that sample size.

228

The point is to see what kind of model it takes to approach such problems. Besides, they had to start somewhere and no one paper can do everything. Also, it’s 2021 so why not e-mail them your concern? Wouldn’t take long and not likely they will see it here.