Causal Formalism and RCTs

[f2harrell]

Iván Díaz and I have engaged in a long discussion on X.com about the role of causal formalism in randomized clinical trials. This was motivated somewhat by my blog article Causal by Design where I argued that straight-up RCTs, i.e., those in which all covariates are measured at time zero or earlier and there are no interruptions in the outcome flow (e.g., no truncation by death) allow us to do causal inference without the need for causal formalism. I also clearly wrote that there are more complex situations where randomization is used but post-time-zero things happen that makes interpretation difficult and calls for formal causal inference.

Díaz has posted a long thread laying out challenges to those like me who feel that causal formalism has a more narrow usefulness in RCTs. It is here on X and on BlueSky and is reproduced below. This topic page is intended to be a place for responses to Iván’s thread, and hopefully for him to respond to the responses.

To frame the discussion it is important to first state the main purpose of a parallel-group RCT: to make sound decisions about licensing a product for sale, or about using an available therapy in practice, by quantifying evidence for a benefit for those assigned to get the treatment in comparison to those assigned to a comparator. Like clinical practice, RCTs cannot control actual use of a therapy in a patient, and there is a distribution of adherence patterns over time. Another framing of the discussion is understanding why I use the term causal by design when referring to a straight-up RCT. This refers to a situation where by design there are only two explanations for a treatment difference: randomness in data (plain old bad luck, addressed by using probabilities to handle uncertainties), and the experimental intervention.

Now to Iván Díaz’s posts.

---

1. Over the last few days I have been discussing the role of formal causal inference in RCTs, and in general the role of formalism and rigor in statistics in twitter with @f2harrell. Thread with an example for why both are fundamental for biomedical research:
2. @f2harrell and others claim that no formal causal models are needed in RCTs.
3. He rejected even the potential outcomes framework, the first formalization (100 y/o) for why “RCTs are causal by design”. To delve into that position, I posed a problem to him:
4. Consider a randomized antidepressant trial examining the effect on weight Y at month m. Some participants die before m. Let A be treatment and S survival. A student proposes comparing arms only among those with S=1.
5. Can Frank give the student conditions under which E[Y|A=1,S=1]−E[Y|A=0,S=1] identifies a causal effect?
6. I did not get a straight answer, but one of his responses was that any assumptions needed to solve the problem would necessarily be untestable.
7. He couldn’t have been certain about that without formalism, but it is false. Below I show a simple solution using potential outcomes.
8. In the same way, no one can be certain why RCTs “are causal by design” without the formalism Neyman gave us 100 years ago.
9. This lack of rigor in a mathematical discipline is concerning. The consequences of rejecting formalism extend beyond this point on RCTs.
10. As the student in this example, I would be disappointed to learn that I had been taught something incorrect because my training lacked sufficient rigor.
11. As a collaborator, I would be disappointed if a possible solution to my problem was missed because the formalism was rejected.
12. As a patient, I might be denied important information that could help me make a treatment decision because an analysis was dismissed without a rigorous causal argument.
13. Here is the solution: let S(a) and Y(a) be survival and weight under assignment a. Y(a) exists only if S(a)=1.
14. An observed survivor could die under the other treatment, so one of Y(1),Y(0) may be undefined. The causal effect may not even exist.
15. We therefore first need to say what “causal effect” means. I use “average causal effect” to mean the mean difference between the same individuals’ potential outcomes, in a population where both outcomes are defined.
16. Now assume S(1)=S(0) almost surely. In words, treatment changes no individual’s survival at m. By consistency, S=S(A)=S(1)=S(0). Both potential weights therefore exist for everyone with S=1.
17. It follows that E[Y(1)−Y(0)|S=1] is a well-defined average causal effect among observed survivors.
18. For each a, consistency gives E[Y|A=a,S=1]=E[Y(a)|A=a,S=1]. Since S=S(a), randomization gives E[Y(a)|A=a,S=1]=E[Y(a)|S=1].
19. Therefore, E[Y|A=1,S=1]−E[Y|A=0,S=1]=E[Y(1)−Y(0)|S=1]. Under S(1)=S(0), the proposed survivor contrast identifies a well-defined causal effect. That is the theorem.
20. S(1)=S(0) is Fisher’s sharp null and has testable implications. In particular, it implies P(S=1|A=1)=P(S=1|A=0), so the assumption is falsifiable.
21. Partial-identification bounds on P[S(1)=S(0)] can also be used. So assumptions capable of solving this problem are not necessarily untestable. The formalism makes that easy to see.
22. Intercurrent events do not create the need for a formal definition of causality or a formal identification argument. They expose a need that was already present in the uncomplicated trial.

---

[isagonas]

Thanks for this, it feels better reading it here on a long format, rather than trying to read each tweet separately.

I am gonna offer some thoughts as a physician with a biostatistics background but not an expert by a long mile.

For actually estimating the ordinary treatment-assignment effect in a pristine parallel-group RCT, one does not need to draw a DAG or invoke do-calculus, fit a g-formula and the likes. Randomization plus an as-randomized analysis does the identification work. In that operational sense I largely agree with Frank. But to say precisely what randomization identifies and why, one is inevitably relying on causal concepts “equivalent” to potential outcomes, whether written down or left implicit. Frank’s position that the remaining explanations are random variation and experimental intervention is at least to my eyes essentially an informal design-based causal argument.

Ivan’s example with the truncation illustrates the value of making that implicit structure explicit and it is what formalism buys you: it separates the definition of the estimand, well-definedness of the potential outcomes, and identification assumptions.

One comment that I have on Ivan’s example is that yes S(1)=S(0) has observable implications. Unequal arm-specific survival probabilities falsify it in the population and the individual-level sharp null for survival is testable by randomization. But equality of survival distributions does not establish S(1)=S(0). There could be “protected” and “harmed” individuals whose effects cancel marginally. Thus the crucial cross-world condition remains vastly stronger than P{S=1∣A=1}=P{S=1∣A=0}. It is falsifiable but not empirically verifiable. And although partial-identification bounds can tell us something about the principal strata, they generally don’t make the SACE point-identified without additional assumptions.

Back to Frank’s argument I think that the “causal by design” formulation needs a qualification. Randomization eliminates baseline confounding of assignment but it does not by itself guarantee that every apparent treatment contrast has the simple causal interpretation that we want. One still requires an adequately defined intervention, appropriate handling of interference, outcome ascertainment that has not become treatment-dependent in a problematic way and so on. Most of those disappear under Frank’s RCT example which is why his practical conclusion at least for me works. But I think the potential-outcome language is an efficient way of seeing exactly which assumptions are being tucked into “well-controlled.”

I cannot wait for the experts to weigh in.

[MichiganWater]

One question I’d like the causal influencers to answer is whether or not a formalism is required in this situation: I have a dozen eggs and throw them at the ground where they break.

Do I need a formalism to declare that my throwing the eggs at the ground caused them to break? If I don’t need a formalism for this example, why not? Seriously. What is it about this example that would not require a formalism to claim causality? When it comes to clinical trials it seems the causal influencers implicitly (or explicitly?) claim that NOTHING can be claimed causal without formalism.

Discusses can get quite deep about clinical trials, the complications of dropout, etc, but I’d like a straightforward answer to this simple question. Can the causal influencers even answer it?

[samw235711]

At one time, causal inference was called “observational data analysis.” I wonder if it would be helpful to revert back to this.

The antidepressant example is nice - formalism does not create complexity, it reveals it. However, it is notable that finding an example that does not involve unmeasured confounding requires that one reach outside of what would typically be considered observational data analysis…

Overall, I think the major issue is that causal inference when considered a “tool” for observational data is misleading, as it makes people think it’s achievable, when really the unobserved confounder assumption is a major bane. This said, as I think is shown, the potential outcomes notation (and do-calculus) are not “tools” for inference, they are ways of describing what already is.

Overall, though, I wonder if a rose by another name might smell as sweet.

[f2harrell]

I don’t think that causal inference was called observational data analysis. I think it was more the case that the original emphasis of causal inference was to be able to solve complex problems that exist in observational research, while not excluding experimental research. The early hope was to make observational research have a higher chance of providing trustworthy causal inference. That hasn’t really worked out, and we now see a stream of misleading studies under under the name “target trial emulation” and lots of other observational studies relying on DAGs for which pathway uncertainties are not acknowledged, much less quantified. Otherwise you made some good points.

[llynn]

I am pleased that the causal inference community is now focusing on causal formalism in randomized controlled trials rather than on OS. I have been trying to convince them to show such formalism can help solve the decades old patient safety problem of false RCT transport and the cause mixture paradox in critical care.

That Diaz chose to assail missing or inaccessible data is unfortunate because that’s not where the primary and dangerous RCT structural flaw commonly missed by the lack of RCT causal formalism resides.

RCT are “causal by design” but may not be “functionally causal”. In other words, because of the cause-mixture paradox, internal causality in a trial designed without structural formalism may render an estimand that is dangerous to transport.

I am speaking at an upcoming critical care conference on failure mode analysis of RCT. Here is one of the slides. The point of the slide is that an RCT is an engineered patient safety structure (with integrated structure and math). Formally considering engineered integration of math with structure with formal documentation is required. Statisticians or CI experts are the engineers who know the math and can integrate it with structure, they should embrace that role. The trialists are clinicians, they are not presently trained in that integration and it is my goal, using the CI community (because statisticians reject such formalism) to “get them trained”.

[f2harrell]

My Response to the Challenge from Iván Díaz

This is What Happened

• Researcher R poses an unconditional (on the future) question about response Y
• Deaths before Y is measured prevent the question from being answered
• Causal formalism asks the researcher to change the question to a conditional one with unverifiable assumptions and a reduced sample size
• R should have been asked to change the question to a different unconditional one that is consonant with available data
• The new unconditional question meshes with available data and intent-to-treat and requires no causal formalism

My Argument

Formalizing a physical or thought process can expose logic and assumptions. It can lead to better understanding and to process improvement. For formalization to be helpful, I submit that the following conditions must hold.

1. If the process is intended to involve data, the formalization should be concordant with the data generating process.
2. The original question must be cohesive and answerable.
3. The formalization must answer the question in a way that is useful for time zero decision-making.
4. Assumptions made by the process must be visible.
5. The assumptions must be verifiable by data even if the sample size needed to check assumptions is quite large.

When formalization requires changing the question, requires making assumptions that could never be verified, or is discordant with the likely data generating process, it is not very helpful.

For straight-up RCTs with no interruptions to the outcome flow, where the analysis includes all randomized patients, I have asserted that no causal formalism is needed. This is because the design excludes all but two explanations for a treatment difference: bad luck (data randomness) and the experimental manipulation. Data randomness is addressed by having probability distributions for statistics (in the frequentist world) or parameters (with Bayes). I also stated in the linked blog article that the best place for formal causal inference may be for RCTs where secondary non-intent-to-treat questions are asked when there is interruption to the outcome flow, resulting in comparisons of non-random samples from the original RCT cohort. Examples of such settings include estimating the treatment benefit were everyone assigned to the treatment actually 100% adhering to it, comparison of longitudinal trends when there are dropouts, and the example to be discussed below

Iván Díaz posed a good example of a non-straight-up RCT in which the target outcome Y = weight at month m may not be observed because some of the patients may die between randomization (time zero) and month m. The clinical question is to estimate the effect of an antidepressant (vs. control) on weight Y at month m. The naive approach of comparing Y among m-month survivors is usually wrong. Using Iván’s notation with assignment to antidepressant denoted by A=1 and to control by A=0, this naive comparison when stated as a difference in non-baseline-weight-adjusted means (not a great idea) is E(Y|A=1,S=1) - E(Y|A=0, S=1) where S=1 means survival until at least time m. The selection process involved here is based on a different outcome (death) that is assessed post time zero. The post-time-zero exclusion represents non-random sampling which creates a bias of unknown magnitude. The researcher might be extremely lucky and the bias=0 but because the bias is not restricted in any way, the contrast is a biased one and is frowned upon in the RCT world.

Going back to my 5-item list, item 1 is only somewhat satisfied in the example. The data generating process needs to include a huge amount of variation in Y that can be explained by baseline weight, and probably needs to account for baseline height. More important is whether item 2 is satisfied, i.e., is the researcher’s question answerable? Her question was “what is the effect of treatment on weight”. In the presence of death, a truncating, terminating event, this question cannot possibly be answered without unverifiable assumptions (more about that later). An informal poll conducted in 2021 found that researchers have great difficulty interpreting an outcome interrupted by death, and far less difficulty interpreting an outcome that includes death. Thus, at its heart the original question is defective and could be replaced by the following questions that deal only with observables:

1. Do patients having baseline weight b who are on the antidepressant tend to have a higher probability of being alive and having Y < y than control patients with baseline weight b?
2. Do antidepressant patients have higher chance of being dead or having weight > y than control patients? (the complement of 1)
3. If death as considered to be the worst outcome what is the difference in median death-penalized weight?
4. Best: Make a 3-dimensional plot showing continuously varying baseline weight b, month m weight threshold y, and P(Y > y or death | A=1, b) - P(Y > y or death, A=0, b).

These questions can all be answered with a multi-state transition model that treats Y as ordinal and has death as an absorbing state. Once a true prospective question is posed by the researcher, we see that the causal formalization does not help to answer it. The RCT once again becomes “straight up.”

Why have I brought up all these other points? Because in drug development, sponsors have had great difficulty operationalizing causal formalizations of intercurrent events, and have more frequently adopted compound endpoints as a solution. They have seen the real advantages of reliance only on observables.

Díaz claims that I would never want to use the potential outcomes framework. He is correct. It doesn’t matter to me what great statisticians’ names are attached to this framework.

Getting back to formalization, consider a randomized crossover study. Unlike the classic parallel-group RCT design we have been discussing, in a crossover study one actually observes Y(0) and Y(1), the outcomes under control (A=0) and treatment (A=1). That means that data are available for computing the variance of the within-patient difference no matter what the correlation between the A=1 response and the A=0 response. Now return to the parallel-group study in which Y(0) and Y(1) can never both be observed on the same patient. Causal inference provides a way to estimate the sample average Y(1)-Y(0) were that to be interesting without accounting for the baseline distribution. The problem is that to be able to operationalize this causal estimand to get causal estimates we must also be able to get uncertainties about the treatment differences. As AP Dawid has elegantly shown, a cohesive analysis that involves “what might have happened had a patient received the treatment they didn’t receive” requires one to know or well-estimate the variance of Y(1)-Y(0). The joint distribution between a patient’s two potential outcomes can never be estimated or checked, even in principle, and even for infinite sample sizes.

So although the marginal causal contrast has a point estimate available, it has no uncertainty estimator, unlike crossover designs. The unobservable within-patient contrast would need to be known to compute a variance. Neyman derived bounds for the variance of the contrast, but conservative bounds reduce the effective sample size; patients are too precious to be wasted in an RCT.

Regarding Iván’s gratuitous point 12, saying that a patient is poorly served when a rigorous causal argument is absent, when the causal argument requires the patient to play the “if I live” game when patients already have enough to consider at baseline, is problematic.

Regarding Iván’s point 16, the assumption that treatment changes no individual’s survival at m is not obviously plausible. And this uses incomplete conditioning since it makes no use of the timing or cause of death. It may be desirable to simply condition on survival until month m, but if there is a mixture of causes of death, anything can happen. One treatment may have more of a certain cause of death than the other, and even if the total number of deaths is the same, the mixture of patients having weight measured may be different across treatment arms in ways that make the conditional estimand impossible to interpret.

On point 17, a “well-defined average causal effect among observed survivors” is of little help to the time zero decision maker. So I see that this is a theorem, but question its utility and the plausibility of its assumptions. For item 20 these “testable implications” do not have much relevance unless the power of the survival comparison is extremely high. With few total deaths this cannot happen. And I am still not clear on whether there is a hidden problem of “different kinds of patients dying on depressant than dying on control therapy” even if overall survival is the same.

The well-defined causal effect among survivors turns the study into a landmark analysis where observation starts (and ends) at time m.

Overall I can see that causal formalism sheds positive light on the “death getting in the way of weight assessment” issue, but find the research question ill-defined, assumptions implausible, and uncertainty calculations especially tenuous. I don’t find the notion of intercurrent events very clear, and would rather just talk about events. A multi-state transition model with death as an absorbing state would cut through a lot of complexity and would involve only observables. There is a trend in drug development with sponsors moving towards ordinal outcomes and multi-state models and away from complexities of intercurrent events.

---

Notes

• A minor quibble with Iván’s point 15: “population” should be replaced with “RCT cohort” (or approximately “sample”).
• Causal inference formalists who have attempted to take this further into the realm of individual patient decision-making have created a true mess.
• The antidepressant study of weight gain would have been better designed as a longitudinal study with frequent weight measurements. The more frequent the measurements the less damage done by dropouts and death and the more likely a missing-at-random assumption can be true.
• General thoughts about RCT estimands and the role of statistical models in RCTs are here.

[SpiekerStats]

IMHO, embedding this in a formal framework is helpful because it forces you to clarify the contrast. Did throwing the eggs cause them to break? Well, relative to what? Relative to the scenario that the ground were made of some plush soft material? Relative to had you not thrown them at all? Relative to the scenario that you dropped them from half the height? What is your A=0 and what is your A=1? The example invokes a prototypical image of an unfortunate kitchen mishap that we’ve all experienced. We don’t have that luxury in medicine when we’re trying to push beyond the boundary of the known.

[SpiekerStats]

@samw235711

formalism does not create complexity, it reveals it

I really like this phrase “formalism does not create complexity, it reveals it.”

[SpiekerStats]

My understanding of what you mean when you say it hasn’t really worked out is that people have been applying the framework too liberally and being careless about “assuming their problems away” whether the assumptions are reasonable or not. That’s more a problem with the researchers than the framework, IMHO. A good causal inference analysis will defend assumptions and, where possible, conduct sensitivity analysis. There are whole hosts of methods out there aimed at assessing sensitivity to untestable assumptions, and I think that’s just the right way to do it. Sometimes things can’t be randomized but a good causal analysis can give a really good snapshot into what’s happening.

[MichiganWater]

Could it be helpful? Sure. That’s not what the causal influencers seem to be stridently claiming, though. I’ll quote my previous post:

When it comes to clinical trials it seems the causal influencers implicitly (or explicitly?) claim that NOTHING can be claimed causal without formalism.

Is a formal framework required for my egg-dropping scenario? My question isn’t whether it can be helpful. It’s whether it is required to be able to make causal claims regarding the eggs. Those are very different claims.

I’m open to the possibility that I’ve completely misunderstood, and that Pearl and the others are only claiming ‘Our tools and methods are helpful for understanding experiments, though not necessary.’ - however that is definitely not my interpretation of what I’ve read of them.

[llynn]

In medical science we don’t deal in absolutes and we don’t have to look far to find sequential trials where the eggs broke and then they didn’t. We have seen many RCT based guidelines later reversed due to harm without formal failure mode analysis which would be required in other fields so no one determined why the failure happened or how to prevent it. We do see pages of ambiguous dialog describing “ the limitations of the trial” as if that’s a substitute for formal failure mode analysis. That wouldn’t fly elsewhere.

Formal causal modeling is coming to RCT science just as safety focused formalism has come to most public safety endeavors. The question really is how long will it take to drag the stat community, kicking and screaming, into the modern world of formalized public safety science.

[ESMD]

Acceptance/non-acceptance of the potential outcomes framework aside, is there at least consensus that causal diagrams can be helpful in the design of RCTs in more complex therapeutic areas? I’m thinking, specifically, about @Pavlos_Msaouel’s point in this post. He has found that constructing causal diagrams in the design phase of oncology RCTs is helpful in anticipating the type of covariate information that will be important to collect at various transition times, for cancer therapies that are given sequentially. Is there any reason to dispute the necessity of constructing causal diagrams in anticipation of developing “state transition models” in this particular context?

[SpiekerStats]

Maybe I’m interpreting your angle incorrectly, but I’m feeling like the argument is that if it seems intuitively obvious that dropping the egg caused it to break, then that’s good enough. It boils down to whether you think a theoretical framework is necessary to justify our procedures. I’d argue that in everyday life, maybe not—but in science, yes. The theoretical framework itself can morph over time and there may be differences in the philosophy surrounding the framework, but I do think it’s reasonable to require the science to have a theoretical framework.

(I’d love to get philosophy folks involved in this conversation because they think about causality very differently and they would have an absolute field day :))

[DylanArmbruster]

"At one time, causal inference was called “observational data analysis.” "

When was this and by who?

[SpiekerStats]

@MichiganWater

that I’ve completely misunderstood, and that Pearl and the others are only claiming ‘Our tools a

Another way I’m thinking about this: This would be like saying “We don’t need Einstein’s theory of relativity to tell us that apples fall down from the trees.” My feeling is that having a theory that is consistent with what feels intuitively right in basic cases is a more credible starting point for building to the more complex cases. So to the question of whether it’s necessary, I guess it depends on what you mean. If you mean “the theory leads us to the same place as what basic heuristics lead us to in obvious cases,” then I can’t really dispute that. But I guess all I’m pointing out is that it doesn’t get you much further. I could be misunderstanding, though.

[ESMD]

I’m not sure that there’s an objection, per se, to the injection of causal formalism into discussions of RCT design. Rather, I sense that experts who object to its injection are so mortified by the widespread misunderstandings of RCTs displayed by many of those who promote causal formalism, that they don’t trust them to inject it either correctly or responsibly. Fancy notation can obscure some pretty egregious misconceptions and assumptions.

[llynn]

@ESMD

Rather, I sense that experts who object to its injection are so mortified by the widespread misunderstandings of RCTs displayed by many of those who promote causal formalism,

What misunderstandings are those? Causal formalism interrogates the structure and provides for failure mode analysis and formal design collaboration between the clinician and the statistician, it does not change the design, it exposes structural weakness/flaws/benefits and the integrated math.

Statisticians have argued that transport is a biological question (in other words, not their problem), reviewing CONSORT , that appears to be their view of interrogation of the gate also. Neither is true. You can’t do the math and covariate adjustment without interrogating the gate unless it is a very simple Bradford Hill type trial. A RCT is one integrated structure with integrated math at the central core.

The first questions to ask is: Are there two species of RCT on a spectrum? Is there a third layer estimand ( E3-cause mixture) and a cause mixture paradox? Statisticians avoid answering those politically sensitive questions by declining causal formalism.

[DylanArmbruster]

For a informal conversation I think it’s fine. But for a serious conversation… Well, have you heard of Hume? SpiekerStats makes a good point here.

[llynn]

The discussion keeps going to OS (one classical role of causal inference) but the instant topic is “causal formalization of RCT design” another important role of causal inference.

How many expert statisticians could do a formal causal failure mode analysis of this linked cause agnostic RCT? Probably none. That would require collaboration with a knowledgeable expert in critical care pathophysiology as well as knowledge of a method of causal interrogation the “gate to guideline causal structure” of an RCT. But that’s the point isn’t it. “Assumption reveal” is the sweet nectar of causal inference.

Every field resists deeper oversight when it comes but the resistance only serves to delay until it’s forced upon them and they lose control. Physicians followed that useless resistance path, It is time for RCT science to rise to the challenge and embrace the inevitable causal “gate to guideline” formalization and documentation”.

https://pubmed.ncbi.nlm.nih.gov/39982179/