Heterogeneity in RCTs with continuous outcome

[R_cubed]

@lachlan

I can imagine the value of presenting heterogeneity in an RCT would be that, like meta-analysis, it would shift the goals of your analysis. If your trial found low heterogeneity this would support the validity of reporting an ‘average treatment effect’. The average treatment effect is roughly what clinicians can expect to see in future patients.

It might help to clarify what type of meta-analyses (MA) you are referring to. Retrospective MA do not have enough control over the data generation process to put much credibility in low heterogeneity statistics. Low Q or I statistics could always be a symptom of publication bias.

More persuasive would be to control for sources of heterogeneity at the design phase – ie. synthesize future experiments to be performed by the same team or agent, with specification of what most likely sources of heterogeneity need to be accounted for.

I’ve collected a number of journal articles in this thread. Start with Senn’s articles, and then look up what seems relevant to your problem.

Nonparametric Effect Size estimation, Likelihood Methods, and Meta Analysis

The aggregation of effect sizes on anything other than interval and ratio scaled variables is much more complicated than I initially anticipated. You would not know this from most of the readily available texts on meta-analysis, as far as I can tell. This discussion hinted at the problems: Relevant article by Stephen Senn on Effect size measurements, particularly non-parametric ones. Stephen Senn (2011) U is for Unease: Reasons for Mistrusting Overlap Measures for Reporting Clinical Tri…

Also study a few posts by @f2harrell on the correct analyses of this type of data. I’ve reluctantly come to the sad conclusion that most of the published work using this type of data is not reliable, given how it is analyzed.

Exemplary QOL analyses that avoid change-from-baseline blunders?

The ISCHEMIA study QOL analyses led by John Spertus are role models for how to do this, avoiding change from baseline and using Bayesian models to get more interpretable and accurate results. We’ll have to wait for a paper or preprint to see analysis details. I have a perfect example but of how a proper analysis yields a much different (and more correct result) but it uses proprietary data so I can’t show it. But here’s a description. The Hamilton-D depression scale is often used in antidepr…

The 3 comments are also related to the issues surrounding analysis of PRO data:

Quality of remdesivir trials

I think there are various efforts to produce some guidance on the topic. The most recent one I saw was this one, which is a pretty good review and highlights for example the need to adjust for baseline covariates. The one part I really disagree with is that in cases where a PRO is evaluated at a specific time point they recommend linear regression over ANCOVA. As mentioned above, in parallel group randomized clinical trials, ANCOVA on the PRO score adjusted for baseline score is a highly reliabl…