Rationale for the high level of evidence for meta-analyses

[R_cubed]

I’ve collected a large number of papers on meta-analysis in this thread. But I especially found the following articles (2 by Stephen Senn) helpful understanding the issues.

Nonparametric Effect Size estimation, Likelihood Methods, and Meta Analysis

The aggregation of effect sizes on anything other than interval and ratio scaled variables is much more complicated than I initially anticipated. You would not know this from most of the readily available texts on meta-analysis, as far as I can tell. This discussion hinted at the problems: Relevant article by Stephen Senn on Effect size measurements, particularly non-parametric ones. Stephen Senn (2011) U is for Unease: Reasons for Mistrusting Overlap Measures for Reporting Clinical Tri…

That will place the following paper in a better context:

The Missing Medians: Exclusion of Ordinal Data from Meta-Analyses

The main things I’ve gotten from them:

1. Retrospective meta-analysis is a very limited technique if we desire to come up with an aggregate effect. It almost certainly does not deserve to be at the top of any “evidence pyramid.”

2. If the outcome does not have an interval or ratio units of measure, it is preferable to use the log odds as the measure of effect in most cases. Meta-analyses that aggregate standardized effects are not likely to be reliable.

There have been number of enlightening threads here on Data Methods about the real challenges of evidence synthesis and meta-analysis that you won’t find in the journals. This comment (as well as the entire thread) stands out as one of them:

Question arising from JAMA Guide article "Odds Ratios—Current Best Practice and Use"

Totally agree that unadjusted are no better (and almost certainly worse). My impression from these data are that efforts to synthesize data from observational and randomized trials is even harder than the current literature base makes it out to be. Not sure about the fields you work in most commonly, but everyone in my end of neonatology seems to have their own pet characteristics to adjust for. Could this be addressed at all by working from an assumed baseline risk and then converting ORs to R…

Where does that leave the practical clinician who would very much like to learn from the experience of others? There hasn’t been much written in an accessible fashion on this.

After study of a Sander Greenland paper [1], I started to think about p value aggregation techniques described in table 12.2.a of [2].

The classical interpretation of p value aggregation methods does leave a lot to be desired. But the relationship between p values and Bayes factors, described in [3] could provide a very useful cognitive technique to help make the qualitative discussions about directions of effect, a bit more rigorous.

I haven’t seen a Bayesian p value meta-analysis technique explicitly described, but I believe I have derived one based on a number of papers I’ve read (some of them I found on this discussion board!).

I really should write it up in a separate post just to make sure I am not missing something obvious.

1. Sander Greenland, Invited Commentary: The Need for Cognitive Science in Methodology, American Journal of Epidemiology, Volume 186, Issue 6, 15 September 2017, Pages 639–645, (link)

2. Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions, Ch. 12, version 6.0 (updated July 2019). Cochrane, 2019. (link)

3. Goodman, S. Toward evidence-based medical statistics. 1. The P value fallacy and 2. Bayes Factors. Ann Intern Med.1999 Jun 15;130(12):995-1004. (PDF)