SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. Understanding Sr Results
  4. Using Sr Results
  5. Bottom Line
  6. References

Tips and Tricks for understanding and using SR results in Evidence-Based Child Health is aimed at helping to understand the results of systematic reviews and to use the results in clinical practice. This time we focus on the concepts of meta-analysis and heterogeneity. The information in this article is based on earlier papers, The Cochrane Handbook, and the collective experience of the authors in teaching evidence-based medicine1–5. Copyright © 2010 The Cochrane Collaboration. Published by John Wiley & Sons, Ltd. The Cochrane Collaboration


Understanding Sr Results

  1. Top of page
  2. Abstract
  3. Understanding Sr Results
  4. Using Sr Results
  5. Bottom Line
  6. References

Meta-analysis

A meta-analysis can be one of the components of a systematic review. In a meta-analysis the results of several individual studies are combined statistically. A meta-analysis can only be performed if the study design, participants, interventions, and outcomes in the individual studies are similar. The overall effect estimate will be calculated as a weighted average of the treatment effects estimated in the individual studies. The weighted average is based on the treatment effect and the standard error of the results; larger and/or more precise studies have more influence than the smaller ones. By combining the results of several individual studies in a meta-analysis the presence of relatively small effects can be more easily detected.

In general there are two statistical models by which to perform a meta-analysis: the fixed effects model and the random effects model3. They differ in the way the variability of the results between the individual studies is treated. The fixed effects model assumes that the true effect of treatment (in both magnitude and direction) is the same value in every study (i.e. fixed across studies) and that the observed variability in the meta-analysis is exclusively due to random variation. The random effects model assumes a different underlying effect for each study and takes this into consideration as an additional source of variation, which leads to somewhat wider Confidence Intervals, a more even distribution of study weights, and a slightly different interpretation in the combined estimate compared to the fixed effects model. A substantial difference in the overall effect estimate (and corresponding Confidence Interval) calculated by the fixed and random effects models will be seen only if studies are markedly heterogeneous.

What is heterogeneity?

Inevitably, studies brought together in systematic reviews will differ. Variability among the individual studies is called heterogeneity. There are different types of heterogeneity1: clinical heterogeneity may be caused by variability in the participants, interventions and outcomes studied, and methodological heterogeneity may be caused by variability in study design and quality.

A consequence of clinical and/or methodological heterogeneity, sometimes referred to as diversity, is the occurrence of statistical heterogeneity, which is variability in the treatment effects being evaluated in the different studies. Statistical heterogeneity manifests itself in the observed treatment effects being more different from each other than one would expect due to chance alone. From now on, we will refer to statistical heterogeneity as simply heterogeneity.

Identifying heterogeneity

Assessment of the consistency of results across studies is an essential part of meta-analysis. Unless we know how consistent the results of studies are, we cannot determine the generalizability of the findings of the meta-analysis. We will describe three methods for identifying heterogeneity.

The first method is the visual inspection of the graphical display of the results (the forest plot or meta-graph). If the Confidence Intervals for the results of the individual studies have poor overlap, this generally indicates the presence of heterogeneity.

The second method is the chi-squared test; however, this test has low power to detect heterogeneity when studies have small sample size or are few in number, resulting in misleading results.

At this moment, the best available measurement is the third method, the I2 statistic. Unlike the chi-square test, the I2 statistic attempts to quantify heterogeneity (taking for granted its existence) rather than testing for it. It does not inherently depend on the number of studies in the meta-analysis. The I2 statistic describes the percentage of the total variability in effect estimates that is due to heterogeneity rather than within study variation. A value of 0% indicates no observed heterogeneity, and larger values show increasing heterogeneity. A value above 50% is often considered substantial heterogeneity.

What if heterogeneity is identified?

A number of options are available if heterogeneity is identified while conducting a meta-analysis. In The Cochrane Handbook seven strategies for addressing heterogeneity are described1:

  • 1.
    Check again if the data are correct. For example, unit of analysis errors like mistakenly entering Standard Errors as Standard Deviations may cause apparent heterogeneity.
  • 2.
    Do not perform a meta-analysis. Particularly if there is inconsistency in the direction of effect or if only two or three studies are available that differ largely in their results it may be misleading to quote an average value for the treatment effect.
  • 3.
    Explore heterogeneity. Look for apparent differences between studies. This can be done using the PICOD framework: patients, interventions, control, outcome and design of the study. Subgroup meta-analyses or more sophisticated and complicated meta-regression can be conducted. Cochrane reviews are expected to pre-specify investigations of characteristics of studies that may be associated with heterogeneity in the protocol—design—of the review.
  • 4.
    Ignore heterogeneity. This is done by performing a fixed effects meta-analysis. However, we do not think this is a good strategy, because the assumptions of a fixed effect model imply that the observed differences among individual study results are due solely to chance, i.e. that there is no heterogeneity.
  • 5.
    Perform a random effects meta-analysis. This may be used to incorporate heterogeneity among individual studies and is intended for heterogeneity which can not be explained. When using a random effects model the presence of heterogeneity is still an issue.
  • 6.
    Change the effect measure. For example, choice of effect measure (like Odds Ratio, Relative Risk or Risk Difference) may affect the degree of heterogeneity among results.
  • 7.
    Exclude some studies. This may introduce bias; however, if an obvious reason for an outlying result of a study is apparent, the study might be excluded from the meta-analysis with more confidence.

Using Sr Results

  1. Top of page
  2. Abstract
  3. Understanding Sr Results
  4. Using Sr Results
  5. Bottom Line
  6. References

Heterogeneity absent

Example 1 (Figure 1): See the review titled, ‘Prophylactic intravenous indomethacin for preventing mortality and morbidity in preterm infants’, in this issue. Although two small studies estimate a non-statistically-significant increase of severe intraventricular hemorrhage in the prophylactic indomethacin vs control comparison, the overall picture is one of reduced hemorrhage risk. In this case the I2 statistic equals 0.

thumbnail image

Figure 1. Prophylactic indomethacin vs. control: effect on severe intraventricular hemorrhage

Download figure to PowerPoint

Heterogeneity present

Example 2: In the Cochrane review ‘Vaccines for preventing malaria (SPf66)’ by Graves et al. considerable heterogeneity is detected for different comparisons6. We look more closely at the comparison of SPf66 vaccine vs placebo to reduce the risk of a new malaria episode caused by P. falciparum. The overall Relative Risk of a new malaria episode is 0.90 [0.84, 0.96] when a fixed effect model is used (Figure 2a); a statistically significant result. However, the I2 statistic is 73%. When a random effect model is used the relative risk of a new malaria episode is 0.89 [0.77, 1.02] (Figure 2b). In their analyses the authors used the random effects model, following The Cochrane Handbook's recommendation #5.

thumbnail image

Figure 2. SPF66 vaccine versus placebo: effect on new malaria episode by P. falciparum. (Figure 2a) meta-analysis performed using a fixed effects model; (Figure 2b) meta-analysis performed using a random effects model

Download figure to PowerPoint

thumbnail image

Figure 3. Education versus No Intervention: effect on booster seat use (reported or observed). (Figure 3a) meta-analysis performed using a fixed effects model; (Figure 3b) meta-analysis performed using a random effects model

Download figure to PowerPoint

When using a fixed effects model the obtained results differ from the ones obtained when using a random effects model. The random effects model contains a more conservative, i.e. moving toward the 1.0 value of no-difference, point estimate of the Risk Ratio with a much wider Confidence Interval. This results from the more balanced weighting used by the random effects method and an increased variance resultant from taking into account the large between study variance. It is important to keep in mind that when review authors are using a random effects model, the presence of heterogeneity is still an issue.

Using The Cochrane Library (http://www.thecochranelibrary.com), readers can recalculate the meta-analyses using a different statistical model. To this aim, download the data of the review, and do the analysis using the RevMan 5 software.

Example 3: In the review ‘Interventions for promoting booster seat use in four to eight year olds traveling in motor vehicles’ by Ehiri et al.7 considerable heterogeneity is detected in the comparison of ‘education’ to ‘no intervention’ in the outcome booster seat use: the I2 statistic is 82%. In their analysis, the authors use the fixed effects model and statistically significant differences between the intervention group and control group were identified.

In this case, the random effects model contains a much higher point estimate of the Risk Ratio, i.e. moving away from the 1.0 value of no-difference but with a much wider Confidence Interval that overlaps the 1.0 value. With the fixed effects model a statistically significant difference between the intervention group and control group is found, yet this seems not to be the right model given this obvious heterogeneity. Although both single trials suggest a benefit for education compared to no education, we conclude that no firm conclusion could be made based on a pooled analysis. We recommend not to interpret this meta-analysis as evidence of effect, not to rely on a misleading pooled result. An example of ‘do not perform a meta-analysis’ (The Cochrane Handbook's recommendation #2).

Bottom Line

  1. Top of page
  2. Abstract
  3. Understanding Sr Results
  4. Using Sr Results
  5. Bottom Line
  6. References

A meta-analysis can be a component of a systematic review. Variability among the individual studies is called diversity or heterogeneity. The I2 statistic describes the percentage of variability in effect estimates that is due to heterogeneity rather than within study variation. If statistical heterogeneity is identified a random effects model can be used to perform the meta-analysis, although random effects should not be used to totally explain away heterogeneity. Sometimes it is best not to perform a meta-analysis, and, as a user of the information, not to rely on a pooled result since it can be misleading.

References

  1. Top of page
  2. Abstract
  3. Understanding Sr Results
  4. Using Sr Results
  5. Bottom Line
  6. References
  • 1
    Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions, version 5.0.2 [updated 2010]. The Cochrane Collaboration, 2010. Available from: http://www.cochrane.org/training/cochrane-handbook.
  • 2
    Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ 2003; 327: 557560.
  • 3
    Egger M, Smith GD, Phillips AN. Meta-analysis: principles and procedures. BMJ 1997; 315: 15331537.
  • 4
    Smith GD, Egger M, Phillips AN. Meta-analysis: beyond the grand mean? BMJ 1997; 315: 16101614.
  • 5
    Egger M, Smith GD. Meta-analysis: potentials and promise. BMJ 1997; 315: 13711374.
  • 6
    Graves PM, Gelband H. Vaccines for preventing malaria (SPf66). Cochrane Database of Systematic Reviews 2006; Issue 2. Art. No.: CD005966. DOI: 10.1002/14651858.CD005966.
  • 7
    Ehiri JE, Ejere HOD, Magnussen L, Emusu D, King W, Osberg JS. Interventions for promoting booster seat use in four to eight year olds traveling in motor vehicles. Cochrane Database of Systematic Reviews 2006; Issue 1. Art. No.: CD004334. DOI: 10.1002/14651858.CD004334.pub2. Edited (no change to conclusions), published in Issue 1, 2009.