Sequel to the odds ratio story

@ngreifer posted a story here

This is my sequel to that story:

The patient, still frozen mid-flinch, opens one eye.

“Wait. Wait wait wait. Don’t stick me yet. I think I actually understand this now.”

The doctor lowers the needle again, visibly annoyed but curious despite herself.

“The odds ratio is the same in both subgroups because it’s not being distorted by anything,” the patient says slowly, working it out loud. “It’s the outcome that’s heterogeneous, not the treatment effect. My gene doesn’t change how the drug works. It changes how sick I already was to begin with.”

“Go on,” says the doctor, arms crossed.

“The overall OR of 2 is a mixture. It’s contaminated by the mixing proportions, eighty percent of the trial has no variant, twenty percent has it. Mix two very different baseline risks together and you get a number that describes nobody. The subgroup OR of 4, that’s the real, portable parameter. It doesn’t care how many people in the room have the variant. It applies to me, specifically, regardless of which group I fall into. That’s not a paradox, that’s the whole point of conditioning on a prognostic variable that isn’t a confounder.”

“I’m listening,” says the doctor, though she is smiling now.

“So there’s no interaction on the odds scale. Same OR, four and four, in both subgroups. But…” the patient’s eyes narrow, “…let me see those percentages again.”

He grabs the tables back.

“Without the variant: eighty-five percent recovery on placebo, ninety-six percent on treatment. Risk difference, eleven points. With the variant: nine percent on placebo, twenty-nine percent on treatment. Risk difference, twenty points. Those aren’t the same. Are you telling me now that there IS an interaction, just on a different scale?”

“That was going to be my next question for you,” says the doctor.

The patient sits with it. “No. No, I don’t think so. I think that’s not a second finding, it’s the same finding wearing a different coat. Probability is stuck between zero and one. My no-variant self is already sitting up near ninety-six percent, there’s barely any room left for the drug to push me any higher, the ceiling eats the effect. My variant self is way down at nine percent, there’s tons of room, so the same underlying multiplicative boost turns into a much bigger jump in raw percentage points. The risk difference isn’t disagreeing with the odds ratio. It’s just what a constant odds ratio looks like once you force it through a bounded scale at two very different starting points.”

The doctor raises an eyebrow. “So which one do you trust?”

“The odds ratio is the model,” the patient says, sounding more certain now. “It’s unconstrained, it goes from zero to infinity, and once you have it plus my baseline risk, you can generate the exact risk difference for me or anyone else without losing any information. The risk difference is just that model’s output, translated into a number I can feel in my gut. It’s not competing evidence. It’s arithmetic downstream of the real thing. If I’d looked at the risk differences first and concluded there was some special interaction going on between my genes and the drug, I’d have been fooled by the ruler, not the reading.”

“So,” says the doctor, needle hovering again, “no confounding, no interaction, one consistent multiplicative effect, and a genetic variant that’s telling you about your prognosis, not about how the drug works on you specifically.”

“Correct.”

“And you have the variant.”

“Correct.”

“Which means your real expected jump is from nine percent to twenty-nine percent recovery, not the eleven-point bump the no-variant group gets.”

“Also correct.”

“So,” the doctor says, raising the needle one final time, “shot or no shot?”

The patient rolls up his sleeve without flinching.

“Odds ratio of four, applies to me regardless of anyone else in the room. Yeah. Give me the shot.”

3 Likes

Threequel

The needle is a centimeter from his arm when the door bangs open.

“Sorry. Lab.” A tech, holding a printout. “Extended panel. He’s also positive for GRP99. Prognostic, not a modifier, not a confounder. Same story as the first one.”

“Fine,” says the patient. “What’s the odds ratio if you split on that one?”

The doctor is already retabulating. “3 if he has GRP99. And 3 if he doesn’t.”

“Three? It’s 4 for bRAT365.”

“And 3 for GRP99. Equal in both strata, both times.”

“They can’t both be the real one.”

“You had a criterion,” says the doctor. “The numbers came out equal. They do for both. Also for the overall table, if you’re wondering — one stratum, nothing to disagree with. The 2 has been passing your test this whole time.”

The patient thinks. “Then I’m not conditioning enough. Split on both.”

Retabulating. “8. All four cells.”

“And that’s the real one?”

“Depends,” says the tech, flipping pages. “There’s two hundred loci on the full panel.”

“What happens to it then?”

“Bigger. Keeps getting bigger. Know everything about you and we know whether you recover, and then it’s not a ratio of anything.”

The patient sits with that. “So the 4 was a mixture too.”

“Of everything we hadn’t sequenced. So is the 8.”

“But I said it applies to me regardless of who’s in the room.”

“Which room? You keep getting moved to smaller ones.”

He picks up the tables again, slower now. “Why were the 4s equal in the first place?”

“Because I wrote the table.”

“You what?”

“I fixed a constant odds ratio and filled in the cells. Then you looked at the risk differences and explained, beautifully, that they only differed because the ceiling ate your effect. Fix a constant survival ratio instead, fill in the cells, and you’d have told me the odds ratios only differed because the floor let it run away. Same speech. Different ruler. The table doesn’t know which one I used.”

Long pause.

“So none of them are real? The 2, the 4, the 3, the 8?”

“All of them are real, given what we’d conditioned on. None of them is the real one. That was the part you had wrong — not the arithmetic. The bit where one number was a mixture and the next one was the truth.”

He rolls his sleeve back up. “Then give me the shot, and stop telling me which parameter is the real one.”

(For full disclosure: This threequel was written in collaboration with Claude/Fable 5.1, with extensive input from me)

1 Like

The threequel’s twist only works by conflating two different phenomena: noncollapsibility under genuinely prognostic covariates (real, monotonic, well-understood) with arbitrary table construction (not real, not monotonic, just algebra). The doctor’s “I wrote the table” reveal proves nothing except that you can fabricate any pattern you like if you’re allowed to invent the data. That’s not a discovery about odds ratios, it’s a tautology about curve-fitting.

The actual mathematics says something the story never lets anyone say: when you condition on a variable that is truly prognostic for the outcome and not a confounder, the conditional OR moves monotonically toward the true individual-level causal effect, and it does so specifically because OR is noncollapsible in that direction. This isn’t a coin flip between “2, 3, 4, or 8, take your pick.” It’s a predictable escalation with a destination: as you add more prognostic information, you’re removing more of the mixing-over-heterogeneity that waters down the marginal 2. The 8 isn’t “another arbitrary ruler reading.” It’s closer to the truth than the 4, which was closer than the 2.

The “same speech, different ruler” line is the sleight of hand. You cannot construct the mirror-image RR story with the same monotone convergence, because RR and OR are not symmetric under this operation, that’s the whole reason OR-noncollapsibility under prognostic covariates is a named, cited phenomenon (Gail 1986; Robinson & Jewell) and not a rhetorical trick. The tech’s mysticism, “none of them is the real one”, is just refusing to notice that the sequence is walking somewhere.

This is a good way of interpreting the process of forward information flow / progressive revelation of information. Although I think that there is an estimand pair that is better than a single number for this story (absolute personalized risk under treatment A, absolute risk under B) odds and hazard ratios computed using incomplete information are the best we can do at the moment. The fact that there is a perfect odds ratio out there doesn’t mean that an imperfect one (that doesn’t know about the mixture in effect) isn’t OK.

1 Like

How do you define this “true individual level causal effect”? At the limit of conditioning, when we know everything about the individual, we should have enough information to reduce the outcome under each treatment condition to a binary “yes” or “no”, so the odds ratio is undefined?

Perhaps there is some irreducible randomness at the individual level, such that we need stochastic counterfactuals, Then we can define the individual odds ratio as the ratio of the counterfactual individual odds under each treatment condition. If this is what you have in mind: Are we assuming homogeneity of the effect at the individual level? What is the justification for this? And where is your implied proof that the odds ratio converges to this under increasingly fine stratification?

Fully agree with you, though I wonder why the causal inference community still make statements such as “…the odds ratio is rarely, if ever, the parameter of interest for causal inference…”? Seems very counterintuitive to me since they can easily convert this to their preferred parameter of interest.

Two claims are being conflated here:

Claim 1: the odds ratio is not of interest in itself; what matters is risk under each treatment condition. That’s the sentence you quoted, and you agree with it: that’s what “convert it” means.

Claim 2: the odds ratio should not be the quantity used to carry an effect from one baseline risk to another. That is not the position of any “causal inference community”. It’s mine; I am not here to crusade for a pre-existing consensus among people I used to work with, please stop attributing my claims to them. Most of them have no position on this.

I argued for this claim on specific grounds: a measure is stable across baseline risk only if some mechanism makes it so, there is a mechanism under which the switch relative risk is stable, and none under which an odds ratio is. If you want to disagree, that’s what to disagree with.

Converting an OR to this patient’s treated risk is valid only if the OR is what stays fixed as baseline risk changes; that is, only if claim 2 is wrong. So “we can easily convert” isn’t a neutral observation about claim 1. It’s claim 2’s negation, assumed without justification .

You’ve misrepresented my post by reframing my quote into something trivial while claiming personal ownership of something that isn’t yours. Claim 1 was never ‘risk matters more than odds’, that’s not what the quote said. And claim 2 is in Hernán and Robins’ book ‘What If’, not something you originated.

I’m not claiming ownership. I’m trying to avoid dragging my former supervisors into a discussion they have no interest in and have taken no position on.

Show me where they claimed that the OR should not be used to carry an effect from one baseline risk to another. As far as I can tell they claimed no such thing, only that the OR has no intrinsic interest. They frequently use logistic regression as a prediction device and have never stated any objection to this practice.

The question about the individual-level odds ratio is still open.

Editorial note: Every time we discuss odds ratios lots of people have lots of opinions, and rightfully so. Let’s keep it cool.

I too see no fundamental reason for the OR to be constant across baseline risks… But given our finite resources don’t we often just have to assume it to be constant because otherwise we blow up the variance and end up making worse predictions for the individual patient?

1 Like

We have to assume that something is constant. Whether that something should be the odds ratio, is up for discussion. I have written extensively on an alternative (the switch relative risk), and my coauthors and I introduced a new regression framework in part to enable using parameters closely related to SRR in regression models. See https://academic.oup.com/jrsssb/advance-article/doi/10.1093/jrsssb/qkag112/8725584 for details.

1 Like

We have discussed this sufficiently on this site so let’s talk about something different. I will note my personal view that odds ratios are easier to interpret and empirically shown to be as or more constant than other measures. And they don’t need a new regression framework. But it is easy to compare deviance of a logit link model with that of probit, log-log, or complementary log-log to gauge goodness of fit.

In the methodological literature, two parameters are variation independent if either may take any value in its own parameter space regardless of the value taken by the other; they are variation dependent if the permissible range of one parameter is constrained by the value of the other. No effect size has constancy across populations and variation dependence or independence (where OR wins hands down) should be distinguished from the separate empirical question of whether an effect measure is actually stable or constant across populations with different baseline risks. Empirical constancy is a data-driven property that may or may not hold in a particular setting. Variation independence, in contrast, is a mathematical property of the parameter space and makes no claim that an effect measure will be empirically constant across populations. We are discussing outcome heterogeneity here in light of the variation independence property, not constancy.

The property often called variation independence is the same property that my coauthors and I have referred to as “closure” in several papers. I believe “closure” is a better term, as it prevents certain misunderstandings about what the property is: A measure is closed if, for every baseline risk p₀ and every admissible value of the measure, the implied p₁ lies in [0,1].

When we discuss effect heterogeneity, what matters is whether effects are actually heterogeneous: Whether the measure takes the same value across groups with different baseline risks. That is an empirical question about the world, settled by data and by whatever theory gives us reason to expect stability in the first place.

Closure bears on this only as a weak filter, many measures pass the filter, and it cannot be used to choose among them. The odds ratio passes. So does the switch relative risk, which for any baseline risk always implies a probability between 0 and 1. Others do too, and more can be constructed. Passing tells you that an effect measure is a candidate. It does not tell you it is the right one. That has to be settled on other grounds.

The term “variation independence” invites the reading that the effect does not vary with baseline risk, which is exactly the empirical claim the property does not make. Closure/variation independence only says that the implied risk stays in [0,1].