# Doubts on data extraction meta-analysis **URL:** \ **Category:** meta-analysis\ **Created:** [May 4, 2021, 8:45pm UTC](https://discourse.datamethods.org/t/doubts-on-data-extraction-meta-analysis/4355 "2021-05-04T20:45:03Z")\ **Posts on this page:** 1\ **Showing post:** 2

**Author:** ![R\_cubed](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/r_cubed/32/1518_2.png) [@R\_cubed](https://discourse.datamethods.org/u/R_cubed)\ **Post date:** [May 5, 2021, 2:06am UTC](https://discourse.datamethods.org/t/doubts-on-data-extraction-meta-analysis/4355/2 "2021-05-05T02:06:43Z")

Credibly aggregating a heterogeneous collection of studies from a retrospective literature review isn’t as easy as the textbooks or journals make it out to be. You might want to read the following threads, and follow up on a few of theses references. I may have become too skeptical, but I’ve come to the conclusion that if you can’t do a meta-regression to make an attempt to explain heterogeneity (too few studies), and cannot control for heterogeneity before collecting data (ie a prospective meta-analysis), just combine the p-values of the individual studies to indicate there is indirect evidence of a possible effect. Effect size MA with very small numbers of studies are misleading in many (maybe even the majority) cases, although I can think of a few examples were effect size aggregation with as few as 5 studies was valuable. Care needs to be taken with effect size combination methods, as the following threads will show. I recommend starting with the Senn articles, and then look up the articles by @Sander on the issue of standardized effects. > [@Nonparametric Effect Size estimation, Likelihood Methods, and Meta Analysis](https://discourse.datamethods.org/t/nonparametric-effect-size-estimation-likelihood-methods-and-meta-analysis/1525/9): > > The aggregation of effect sizes on anything other than interval and ratio scaled variables is much more complicated than I initially anticipated. You would not know this from most of the readily available texts on meta-analysis, as far as I can tell. This discussion hinted at the problems: Relevant article by Stephen Senn on Effect size measurements, particularly non-parametric ones. Stephen Senn (2011) [U is for Unease: Reasons for Mistrusting Overlap Measures for Reporting Clinical Tri…](https://amstat.tandfonline.com/doi/abs/10.1198/sbr.2010.10024) > [@Question arising from JAMA Guide article "Odds Ratios—Current Best Practice and Use"](https://discourse.datamethods.org/t/question-arising-from-jama-guide-article-odds-ratios-current-best-practice-and-use/483/7): > > Totally agree that unadjusted are no better (and almost certainly worse). My impression from these data are that efforts to synthesize data from observational and randomized trials is even harder than the current literature base makes it out to be. Not sure about the fields you work in most commonly, but everyone in my end of neonatology seems to have their own pet characteristics to adjust for. Could this be addressed at all by working from an assumed baseline risk and then converting ORs to R… > [@Journal Article: Which is more generalizable, powerful and interpretable in meta-analyses, mean difference or standardized mean difference?](https://discourse.datamethods.org/t/journal-article-which-is-more-generalizable-powerful-and-interpretable-in-meta-analyses-mean-difference-or-standardized-mean-difference/3978): > > I thought this would be rather interesting, considering some of the things I’ve learned over the past year or so doing research on meta-analysis. It took awhile, but much of what I believed (ie. meta-analysis is a relatively straightforward way to synthesize results) is simply not so. Here are a number of methodology articles I’ve found helpful, particularly the papers by Stephen Senn later in the thread. With that background, I post the following link on the use of mean difference vs. stan… --- _[View the full topic](https://discourse.datamethods.org/t/doubts-on-data-extraction-meta-analysis/4355)._