Article: JAMA meta-analysis of Placebo response and cannabis based therapies for pain

I came across a recently published meta-analysis that got some attention in the mainstream media:

From the abstract:

Main Outcomes and Measures Change in pain intensity from before to after treatment, measured as bias-corrected standardized mean difference (Hedges g).

Despite generating lots of attention, virtually nothing can be learned from this meta-analysis because of the naive use of textbook methods (ie. means, standardized mean differences).

After studying this issue in depth, I have come to the unfortunate conclusion that the default use of parametric models on patient reported outcomes at the individual study level makes any meta-analysis based on aggregate data a pointless waste of time.

:new: The preferable way to examine patient reported outcomes is with ordinal regression at the individual
study level. Various Bayesian or Frequentist approaches could be used for synthesis.

I have not looked at supplemental materials yet, but I saw no discussion related to the various pain scales in use, nor the complexities of aggregating them.

There is no discussion of the computation of pain change scores in the trials that were not cross-over designs.

I’ve collected a long list of references on meta-analysis in the context of patient reported outcomes in this thread.

Examination of the study characteristics table reports sample sizes, which ranged from 9 - 339; only 4 studies had more than 100 participants.

With very small sample sizes like the ones reported here, adaptive allocation (while maintaining blinding) is worth serious consideration from a decision theory POV.


A valuable lesson. Great post.

Four points in this meta-analysis
a) The pre and post measurements are correlated and the effect size used should have been standardized mean gain not standardized mean difference, assuming that the pain scores indeed needed standardization.
b) Doing a moderator analysis by RoB itself introduces bias
c) There was no " bias-corrected standardized mean difference" analyzed (as indicated in the abstract only) - bias adjusted meta-analysis have a few methods none of which were implemented here
d) The random effects model has poor error estimation and thus there is a problem of overdispersion here. Also, given the one-way distribution of weights (from large to small studies) the pooled estimate is essentially an arithmetic mean (Figure 3 boxes are all the same size)

1 Like