Thanks for all the pointers to potential limitations. I would however prefer if you could instead focus on what do you think can be learned from the study as is, despite the flaws (I am open to the possibility that the answer is “very little”). Clinicians unfortunately don’t have the option to pause treatment choices until a better study is conducted. Thanks a lot!
3 Likes
I think that the conclusion should indicate that there it is still plausible that L-R is beneficial - look at the CI for the Hazard Ratio for the primary outcome. (by the way, the Primary outcome HR in the Abstract and in the Text are slightly different:
*ABSTRACT: hazard ratio for clinical improvement, 1.24; 95% confidence interval [CI], 0.90 to 1.72
TEXT: hazard ratio for clinical improvement, 1.31; 95% confidence interval [CI], 0.95 to 1.85; P=0.09
*
I’ve asked NEJM to clarify.
3 Likes
Sure; but we could learn more from it with better analysis that used the information more efficiently.
I’d say we don’t learn a huge amount from it. The data are consistent with the intervention having some benefit, so I wouldn’t write it off based on these data alone. I’ve seen it said elsewhere that these drugs are unlikely to be harmful so if that’s right, maybe we should put more credence on the potential benefits. The very black and white interpretations in the NEJM (and in lots of commentary - a “negative trial”) seem unhelpful. A trial of this size was never going to definitively tell us whether there was any benefit, so it’s important to look at the results with that in mind.
Unfortunately, I’ve heard some messages from doctors that have stopped treating with these drugs based on this specific trial as of yesterday. So regrettably, some clinicians are changing their behaviour over this already.
4 Likes
Another aspect that I find problematic is that they do not adjust their analysis on any patient covariate. And given the results they have it is likely that such adjustments would make the results significant. Just from listening to the media, age and comorbidities seem like a safe bet as important prognostic covariates. And I’m sure that there are more appropriate covariates to consider. They do not even adjust for their stratification factors:
randomization was stratified on the basis of respiratory support methods at the time of enrollment: no oxygen support or oxygen support with nasal duct or mask, or high-flow oxygen, noninvasive ventilation, or invasive ventilation including ECMO.
and it is known that stratifying on a covariate and then not taking it into account in the analysis leads to deflated type 1 error and an underpowered analysis.
4 Likes
I abhor this sentiment. We need small, theoretically substantive trials capable of advancing medical science rapidly in a mode of strong inference. We do not need large, theoretically shallow trials that grope blindly* for small-effect-size metaphorical ‘oomphs’ that lack realistic theoretical connections to physiology.
- Cf. #amoEBMism:
3 Likes
Hell, we need large observational datasets NOW for hypothesis generation.
4 Likes
In Israel , two tertiary hospitals have stopped using L-R due to interpreting the results as " no added benefit". I hope some official reply or letter to the editor maybe published so that clinicians would not stop a potentially helpful treatment based on wrong interpretation.
2 Likes
We have digitized the curve, and reanalyzed it with a Bayesian Cox model, under a weakly informative prior. The posterior probability of effect (clinical improvement with HR>1.15) is 73%, while the probability that the therapeutic effect is in the region of practical equivalence (ROPE) is 17%. I would still use lopinavir-ritonavir in my patients, until more evidence is available.
11 Likes
We have digitized the curve, and reanalyzed it with a Bayesian Cox model, under a weakly informative prior. The posterior probability of effect (clinical improvement with HR>1.15) is 73%, while the probability that the therapeutic effect is in the region of practice equivalence (ROPE) is 17%. I would still use lopinavir-ritonavir in my patients, until more evidence is available.
Would love to see this in medRxiv.
4 Likes
I have submitted a short letter to the editor, as an urgent plea to the researchers to use Bayesian methods in these critical times. I don’t think they’ll accept it but I had to try.
6 Likes
Any update on this? I feel like this would be a good opportunity to learn how to do this.
2 Likes
The original curve from the Lotus-China trial may be scanned using free software such as:
https://apps.automeris.io/wpd/
The database is then reconstructed according to the method by Guyot et al.
This method is available in the R survHE package.
It’s simple but it takes a little practice.
Then, you have an approximate database very similar to that of the original study. Below I show my approximate plot.
Quite similar to the original plot here:
In fact, my frequentist analysis yields the same result as the original.
Once you’ve done this, you can do a Bayesian reanalysis. In this case I fitted a Bayesian Cox model with the R brms package. Below I show the resulting half-eye plot. You can see that most of the probability mass in favor of the therapeutic effect tilts to the right.
Therefore, the posterior probability that the effect is in the region of practical equivalence (+/- 10%) is 17%. There is a clear directionality, and according to the frequentist statistical plan used by the authors, the study was underpowered, as it was stopped prematurely.
7 Likes
That’s for their time to event primary outcome, right? It’s a bit of a strange outcome. It would be interesting to analyse the 7-category ordinal outcome (at 7 and 14 days) using an ordinal regression (preferably Bayesian) - I think there is enough information in the paper to do that…
1 Like
I agree. But I can’t see the information for reanalysis…
It’s in Table 3, isn’t it?
I’ve taken Table 3 from the article:
https://www.nejm.org/doi/full/10.1056/NEJMoa2001282
I applied a cumulative link model with the R brms package. According to this model, the use of lopinavir/ritonavir reduces the risk of outcomes or worse outcomes with odds ratio [OR] 0.72 (95% CrI, 0.43-1.20) under a non-informative prior. The posterior probability of obtaining an effect size >15% (OR<0.85) is 73%, which coincides with the Bayesian Cox model. Below is the halfeye plot:
and the marginal effects according to the fitted model.
I also show the code:
lot<-data.frame(group=c( rep(2,43+28), rep(3,24+8),rep(4,20+25),rep(5,6+5), rep(6,5+3),rep(7,17+15)),
treat=c(rep(1,43),rep(0,28),rep(1,8),rep(0,24),rep(1,25),rep(0,20),rep(1,5),rep(0,6),rep(1,3),rep(0,5),rep(1,15),rep(0,17)))
lot$group<-factor(lot$group, ordered = T)
table(lot$group,lot$treat)
library(brms)
fit3 <- brm(
formula = group ~ 1 + treat,
data = lot,
family = cumulative("logit")
)
hypothesis(fit3, "treat1 < log(0.85)",alpha=.05)
marg<-marginal_effects(fit3, "treat", categorical = TRUE)
6 Likes
For univariate proportional odds Bayesian models using Stan I am about to update the R rms package with a host of easy-to-use functions for plotting, estimation, credible intervals for indexes of predictive performance, nomograms, etc.
5 Likes