I assume you are referring to a retrospective analysis of results published in journals. Please correct me if I am mistaken.
Your question reminded me of this article from 1999, published in the American Journal of Epidemiology by @Sander_Greenland and Charles Poole. It examines some of the considerations in conducting a meta-analysis that you mentioned above.
Their primary recommendations:
- If you are going to calculate any summary stat, compute both a fixed effect and a random effects estimate (unless you have a really good reason to prefer one or the other).
- If there is a meaningful difference between the two estimates, there is heterogeneity among the studies, and any summary statistic is likely to be misleading.
- If possible, use meta-regression methods to explore how study design affects the results.
- Random effects methods can be susceptible to plausible types of publication bias, making them less conservative than is ordinarily believed.
- Very small samples of studies make any quantitative method very limited. They would prefer narrative description in this scenario.
While I have no doubt you would be aware of the limitations of this type of meta-analysis, for completeness, I also recommend readers interested in this topic to study the @Sander_Greenland 2005 paper on multiple bias modelling for observational data (which would include meta-analysis by any reasonable definition). It is not only relevant to this topic, but it provides an excellent example of how to think like a good, skeptical statistician.