RCT with missing follow-up outcomes: ANCOVA + MI vs Mixed-Effects Modelling?

Thanks for the ideas, @MSchwartz. This still isn’t exactly what I’m looking for, but it’s the best compromise I’ve seen.

2 Likes

@SolomonKurz @f2harrell @MSchwartz Thank you all for the spirited but collegial discussion, from which I have greatly benefitted!

6 Likes

Solomon if you can re-articulate the ultimate need let’s try to figure out how conditioning on baseline can get you there, or not.

3 Likes

I want a model where the baseline measure of the primary outcome variable is part of the likelihood so that I can make population inferences about baseline, and I can use those population inferences to plot the pre/post trajectories, as in the right panel of the plot I showed, above. One solution is the multilevel ANCOVA, as discussed in my blog post. Another solution, as discussed above, would be allowing the residual variance of the multilevel ANCOVA to vary by time point, which would partially, though imperfectly, address possible truncation concerns. Over on the Stan forums, we’re further exploring whether it’s possible to have a Gaussian likelihood with conditional truncation, where the lower bound a is the cut-off point at baseline, and is -\infty otherwise. At the moment, it’s unclear if this is feasible, but it’d be really cool if if was. A fourth option, also mentioned above, is to fit a bivariate model where pre is modeled in an intercept-only truncated Gaussian model, and post is modeled with the conventional Gaussian ANCOVA. There may well be other options I haven’t considered. But regardless, I want pre as part of the likelihood. And yes of course I want post to be part of the likelihood, too, so I can compute the ATE.

1 Like

Could you please elaborate why this is important to your research?

Because from my perspective, I can’t understand the post-treatment outcome until I understand the pretreatment baseline. The two are linked.

Do you need pretreatment baseline in the likelihood to understand it? I quite liked @MSchwartz plots.

My preference is something of a generalized and simplified version of what I’d do with any therapy client. [I’m not currently a therapist, but I have 8 years of experience giving therapy.] I’d take down their outcome measure at our first meet-and-greet session, and record it in subsequent sessions. Progress is viewed not only in terms of absolute values, though it is viewed in that way too, but also with reference to the baseline. Oftentimes we don’t have access to the intermediary values in therapy research, but we do at least have the pre and post measures. And thus, I view the post-treatment group means, and their contrast (ATE and all that), in the context of the pre-treatment mean. The pre-treatment assessment isn’t just a control; it provides its own inherent meaning, and thus it belongs in the likelihood. It’s part of what I’m here for.

As to what I “need,” I wouldn’t use that language. This is what I want, and if I can indeed have it, I’m going to take it. I’m a greedy researcher; I want all the estimands (so to speak).

1 Like

After reading this exchange I am more convinced than ever that you need to condition on baseline to reach your goals. It does not seem logical to me to go to extraordinary efforts (e.g., truncated distributions) to solve a problem of your own creation. Think of baseline as an investigator-manipulated quantity, which would include the degenerate case where you have to have a baseline level equal to some constant to qualify for the study. Some studies may qualify patients with very low or very high values, creating a bimodal distribution to try to make sense of.

Modeling Y(t) | Y(0), X allows you to estimate absolute Y(t) as well as Y(t)-Y(0), the change from baseline.

You raised the issue of population inference about the baseline. That takes the problems I’ve described to a whole new level of severity, and doesn’t recognize how subjects are accrued for studies. Population inference would only be valid if you did a probability sample from the population and you knew all the sampling probabilities.

1 Like

Every version of the ANCOVA I’ve proposed, including the multilevel ANCOVA in my original blog post, achieves this.

I understand, but that’s the same issue in any other study with population inference, which is the supermajority of psychology studies, observational, experimental, and anywhere between.

2 Likes

I wonder what are everyone’s thoughts about plotting the trajectories with unadjusted data and only present the coefficients of the model (with baseline as a covariate) in a table?