Understanding Sr Results
A meta-analysis can be one of the components of a systematic review. In a meta-analysis the results of several individual studies are combined statistically. A meta-analysis can only be performed if the study design, participants, interventions, and outcomes in the individual studies are similar. The overall effect estimate will be calculated as a weighted average of the treatment effects estimated in the individual studies. The weighted average is based on the treatment effect and the standard error of the results; larger and/or more precise studies have more influence than the smaller ones. By combining the results of several individual studies in a meta-analysis the presence of relatively small effects can be more easily detected.
In general there are two statistical models by which to perform a meta-analysis: the fixed effects model and the random effects model3. They differ in the way the variability of the results between the individual studies is treated. The fixed effects model assumes that the true effect of treatment (in both magnitude and direction) is the same value in every study (i.e. fixed across studies) and that the observed variability in the meta-analysis is exclusively due to random variation. The random effects model assumes a different underlying effect for each study and takes this into consideration as an additional source of variation, which leads to somewhat wider Confidence Intervals, a more even distribution of study weights, and a slightly different interpretation in the combined estimate compared to the fixed effects model. A substantial difference in the overall effect estimate (and corresponding Confidence Interval) calculated by the fixed and random effects models will be seen only if studies are markedly heterogeneous.
What is heterogeneity?
Inevitably, studies brought together in systematic reviews will differ. Variability among the individual studies is called heterogeneity. There are different types of heterogeneity1: clinical heterogeneity may be caused by variability in the participants, interventions and outcomes studied, and methodological heterogeneity may be caused by variability in study design and quality.
A consequence of clinical and/or methodological heterogeneity, sometimes referred to as diversity, is the occurrence of statistical heterogeneity, which is variability in the treatment effects being evaluated in the different studies. Statistical heterogeneity manifests itself in the observed treatment effects being more different from each other than one would expect due to chance alone. From now on, we will refer to statistical heterogeneity as simply heterogeneity.
Assessment of the consistency of results across studies is an essential part of meta-analysis. Unless we know how consistent the results of studies are, we cannot determine the generalizability of the findings of the meta-analysis. We will describe three methods for identifying heterogeneity.
The first method is the visual inspection of the graphical display of the results (the forest plot or meta-graph). If the Confidence Intervals for the results of the individual studies have poor overlap, this generally indicates the presence of heterogeneity.
The second method is the chi-squared test; however, this test has low power to detect heterogeneity when studies have small sample size or are few in number, resulting in misleading results.
At this moment, the best available measurement is the third method, the I2 statistic. Unlike the chi-square test, the I2 statistic attempts to quantify heterogeneity (taking for granted its existence) rather than testing for it. It does not inherently depend on the number of studies in the meta-analysis. The I2 statistic describes the percentage of the total variability in effect estimates that is due to heterogeneity rather than within study variation. A value of 0% indicates no observed heterogeneity, and larger values show increasing heterogeneity. A value above 50% is often considered substantial heterogeneity.
What if heterogeneity is identified?
A number of options are available if heterogeneity is identified while conducting a meta-analysis. In The Cochrane Handbook seven strategies for addressing heterogeneity are described1:
- 1.Check again if the data are correct. For example, unit of analysis errors like mistakenly entering Standard Errors as Standard Deviations may cause apparent heterogeneity.
- 2.Do not perform a meta-analysis. Particularly if there is inconsistency in the direction of effect or if only two or three studies are available that differ largely in their results it may be misleading to quote an average value for the treatment effect.
- 3.Explore heterogeneity. Look for apparent differences between studies. This can be done using the PICOD framework: patients, interventions, control, outcome and design of the study. Subgroup meta-analyses or more sophisticated and complicated meta-regression can be conducted. Cochrane reviews are expected to pre-specify investigations of characteristics of studies that may be associated with heterogeneity in the protocol—design—of the review.
- 4.Ignore heterogeneity. This is done by performing a fixed effects meta-analysis. However, we do not think this is a good strategy, because the assumptions of a fixed effect model imply that the observed differences among individual study results are due solely to chance, i.e. that there is no heterogeneity.
- 5.Perform a random effects meta-analysis. This may be used to incorporate heterogeneity among individual studies and is intended for heterogeneity which can not be explained. When using a random effects model the presence of heterogeneity is still an issue.
- 6.Change the effect measure. For example, choice of effect measure (like Odds Ratio, Relative Risk or Risk Difference) may affect the degree of heterogeneity among results.
- 7.Exclude some studies. This may introduce bias; however, if an obvious reason for an outlying result of a study is apparent, the study might be excluded from the meta-analysis with more confidence.