Article: JAMA meta-analysis of Placebo response and cannabis based therapies for pain

[R_cubed]

I came across a recently published meta-analysis that got some attention in the mainstream media:

Placebo Response and Media Attention in RCTs of Cannabis-Based Pain Therapies

From the abstract:

Blockquote
Main Outcomes and Measures Change in pain intensity from before to after treatment, measured as bias-corrected standardized mean difference (Hedges g).

Despite generating lots of attention, virtually nothing can be learned from this meta-analysis because of the naive use of textbook methods (ie. means, standardized mean differences).

After studying this issue in depth, I have come to the unfortunate conclusion that the default use of parametric models on patient reported outcomes at the individual study level makes any meta-analysis based on aggregate data a pointless waste of time.

Preprint: Analysis of Likert Type Data Using Metric Methods

Dennis Boos and Judy Chen (2022). Analysis of Likert-Type Data Using Metric Methods. I edited this to bring the main points earlier, and the decision theoretic justification later in the post. Introduction The authors defend the common practice of using parametric models on ordinal data on the basis of pragmatics (ie.“easier” to for researchers understand), and perceived robustness in that parametric methods on empirical or simulated data sets are not sufficiently inaccurate that skepti…

Journal Article: Which is more generalizable, powerful and interpretable in meta-analyses, mean difference or standardized mean difference?

I thought this would be rather interesting, considering some of the things I’ve learned over the past year or so doing research on meta-analysis. It took awhile, but much of what I believed (ie. meta-analysis is a relatively straightforward way to synthesize results) is simply not so. Here are a number of methodology articles I’ve found helpful, particularly the papers by Stephen Senn later in the thread. With that background, I post the following link on the use of mean difference vs. stan…

The preferable way to examine patient reported outcomes is with ordinal regression at the individual
study level. Various Bayesian or Frequentist approaches could be used for synthesis.

Statistical Thinking - Resources for Ordinal Regression Models

I have not looked at supplemental materials yet, but I saw no discussion related to the various pain scales in use, nor the complexities of aggregating them.

There is no discussion of the computation of pain change scores in the trials that were not cross-over designs.

I’ve collected a long list of references on meta-analysis in the context of patient reported outcomes in this thread.

Nonparametric Effect Size estimation, Likelihood Methods, and Meta Analysis

The aggregation of effect sizes on anything other than interval and ratio scaled variables is much more complicated than I initially anticipated. You would not know this from most of the readily available texts on meta-analysis, as far as I can tell. This discussion hinted at the problems: Relevant article by Stephen Senn on Effect size measurements, particularly non-parametric ones. Stephen Senn (2011) U is for Unease: Reasons for Mistrusting Overlap Measures for Reporting Clinical Tri…

Examination of the study characteristics table reports sample sizes, which ranged from 9 - 339; only 4 studies had more than 100 participants.

With very small sample sizes like the ones reported here, adaptive allocation (while maintaining blinding) is worth serious consideration from a decision theory POV.

Stability of results from small RCTs

Blockquote Randomization does not guarantee exchangeability. The very definition of random implies imbalances in prognostic factors could happen, even if they are more and more rare as the sample size increases. This very important point is carefully elaborated in the following: So that leads me to the following question: Are small randomized trials worth conducting? In a previous post @f2harrell wrote: Blockquote Most statisticians I’ve spoken with who frequently collaborate with …

[f2harrell]

A valuable lesson. Great post.

[s_doi]

Four points in this meta-analysis
a) The pre and post measurements are correlated and the effect size used should have been standardized mean gain not standardized mean difference, assuming that the pain scores indeed needed standardization.
b) Doing a moderator analysis by RoB itself introduces bias
c) There was no " bias-corrected standardized mean difference" analyzed (as indicated in the abstract only) - bias adjusted meta-analysis have a few methods none of which were implemented here
d) The random effects model has poor error estimation and thus there is a problem of overdispersion here. Also, given the one-way distribution of weights (from large to small studies) the pooled estimate is essentially an arithmetic mean (Figure 3 boxes are all the same size)