I’ve been thinking quite a bit about the challenges of meta-analysis for clinically heterogeneous conditions in the context of neurological rehabilitation (with CVA as a prime example) that I am organizing for a separate post for expert input. Much should apply towards your project.
Pay particular attention to the problem Dr. Harrell mentioned regarding change scores. At least in that case, you should be able to rescue the data that the primary researchers reported and correct them.
But the statistical errors in the published literature is much higher than you might expect. These errors are more likely to make a paper publishable, because using an alternative methods of analysis are less well-known. This will lead to an important source of bias in the studies you do find.
The Missing Medians: Exclusion of Ordinal Data from Meta-Analyses
Blockquote
These reporting considerations have important implications for meta-analysis. Where ordinal data are reported appropriately in individual studies, they are often excluded from meta-analysis due to the difficulty in pooling them. Alternatively, where study authors report means and standard deviations, often inappropriately, these data can be included in meta-analysis but the validity of the pooled results is questionable. Meta-analytical results are heavily influenced by treatment of outliers and by parametric versus non-parametric estimation [5]. The Cochrane collaboration acknowledge the problem with meta-analysis of ordinal or non-parametric data in their handbook (“difficulties will be encountered if studies have summarised their results using medians”, section 9.2.4[2]), but do not propose a solution. In practice, investigators often dichotomise data from shorter ordinal scales, and treat data from longer ordinal scales as continuous. Both of these approaches are sub-optimal. Dichotomising scales necessitates a loss of detail, and participants close to but on opposite sides of the split are characterised as very different rather than very similar. Statistical power is lost: a median split has been equated to discarding one-third of the data [6]. Treating data as continuous implies a consistent relationship between each level of the scale, which is not true of ordinal scales, and assumptions of normality are often violated. In the context of meta-analysis, it may be argued that, due to central limit theorem, mean values across a group of studies (and hence mean differences) will be approximately normally distributed, rendering any concerns about violation of normality invalid. Although this may be true, [for estimates with a finite variance – my emphasis] it fails to acknowledge that it is inappropriate to use means as a measure of central tendency for scales where we know only the order of levels on the scale, and not the distance between them.
This poses scholars without access to the individual patient data a tough question. To what extent can the clinical observations (summarized by inappropriate use of parametric effect sizes) be converted to a form where a defensible analysis can be performed?
While all of the references in a thread I’ve started on this topic should be helpful, the links in this post should be examined first, so you know what you are facing. A good meta-analysis is not easy.