Potential conflict of interest: Nothing to report.
Reply: Methodological issues in a meta-analysis†
Article first published online: 25 JUN 2010
Copyright © 2010 American Association for the Study of Liver Diseases
Volume 52, Issue 1, pages 396–397, July 2010
How to Cite
Thorlund, K. and Awad, T. (2010), Reply: Methodological issues in a meta-analysis. Hepatology, 52: 396–397. doi: 10.1002/hep.23764
- Issue published online: 25 JUN 2010
- Article first published online: 25 JUN 2010
- Manuscript Accepted: 10 MAY 2010
- Manuscript Received: 8 MAY 2010
We thank Dr. Crémieux for bringing this issue to our attention, and we agree that sensitivity analysis is an important and necessary component of all systematic reviews. We are, however, concerned about the validity of the method proposed by Tian et al.1 that Crémieux has chosen for his re–meta-analysis of sustained virological response. Currently, the method by Tian et al. has only been sparsely validated for meta-analysis scenarios (>30 trials with event rates < 10%) that are far from representative of the sustained virological response meta-analysis (8 trials with event rates ranging from 20% to 80%).2 Although the method by Tian et al. may in time be proven statistically superior, at the current stage, preferring this method over the conventional meta-analysis methods is analogous to preferring a phase 1 drug over a Food and Drug Administration–approved drug in clinical practice.
Crémieux points out that the lack of evidence for heterogeneity (I2 = 0%) makes the fixed-effects model appropriate. In other words, Crémieux suggests that the choice of model (fixed versus random) may be determined by the estimate of heterogeneity (I2). This is highly inappropriate meta-analytic conduct that the Cochrane Collaboration abandoned some years ago.3 Findings from empirical studies and simulation studies also testify to the inappropriateness of this conduct.4-6 However, since we do not have evidence of heterogeneity, we cannot preclude the possibility that a fixed-effects model is appropriate. Using the conventional Mantel-Haenszel fixed-effects model, we obtained a pooled relative risk of 1.10 (95% confidence interval = 1.03-1.18). Excluding low-dose peginterferon alfa-2b from the Individualized Dosing Efficacy Versus Flat Dosing to Assess Optimal Pegylated Interferon Therapy (IDEAL) trial, we obtained a pooled relative risk of 1.10 (95% confidence interval = 1.03-1.19).
In his commentary, Crémieux interprets his findings as discrepant with the original findings of our systematic review. Although we do agree that some caution should be exercised, the word discrepant seems too strong to describe the observed statistical difference. In fact, this claim of discrepancy is based on an unfair comparison. In our original analyses, we took precautions to interpret the statistical inference according to the strength of the evidence. In this vein, we constructed adjusted thresholds for statistical significance by using an approach (trial sequential monitoring boundaries) analogous to approaches used for interim analysis in clinical trials.7-10 Roughly speaking, we can state that this analysis translates into an adjusted P value less than 5% but not less than 1% (the unadjusted meta-analyzed P value was 0.8%). In comparison, the confidence intervals proposed by Tian et al.1 would translate into a P value of roughly 6%. Several authors have previously warned against relying on the conventional criterion for statistical significance.7, 8, 11-14 In this vein, a P value of 1% to 5% and a P value of 6% should be interpreted similarly.
We thank Crémieux for bringing up issues that may have been hidden to the statistically inexperienced reader. However, even if one were to take a leap of faith and believe that Crémieux's analyses were based on a valid method, the conclusions of our analysis and his analysis still do not differ: there is statistical evidence, albeit moderate, that peginterferon alfa-2a is superior to peginterferon alfa-2b in achieving sustained virological response.
- 1Exact and efficient inference procedure for meta-analysis and its application to the analysis of independent 2x2 tables with all available data but without artificial continuity correction. Biostatistics 2009; 10: 275-281., , , , .
- 2Peginterferon alpha-2a is associated with higher sustained virological response than peginterferon alfa-2b in chronic hepatitis C: systematic review of randomized trials. HEPATOLOGY 2010; 51: 1176-1184., , , , , .
- 3Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions, Version 5.0.1 [updated September 2008]. The Cochrane Collaboration, 2008. Available from www.cochrane-handbook.org.
- 4Uncertainty in heterogeneity estimates in meta-analyses. BMJ 2007; 335: 914-916., , .
- 5Undue reliance on I(2) in assessing heterogeneity may mislead. BMC Med Res Methodol 2008; 8: 79., , , .
- 6A comparison of heterogeneity variance estimators in combining results of studies. Stat Med 2007; 26: 1964-1981., .
- 7Cumulating evidence from randomized trials: utilizing sequential monitoring boundaries for cumulative meta-analysis. Control Clin Trials 1997; 18: 580-593., .
- 8Can trial sequential monitoring boundaries reduce spurious inferences from meta-analyses? Int J Epidemiol 2009; 38: 276-286., , , , , , et al.
- 9Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis. J Clin Epidemiol 2008; 61: 64-75., , , .
- 10Estimating required information size by quantifying diversity in a random-effects meta-analysis. BMC Med Res Methodol 2009; 9: 86., , , .
- 11Uncertainty of the time of first significance in random-effects cumulative meta-analysis. Control Clin Trials 1996; 17: 357-371., , .
- 12The fading of reported effectiveness. A meta-analysis of randomised controlled trials. BMC Med Res Methodol 2006; 6: 25., , .
- 13Sifting the evidence—what's wrong with significance tests? Br Med J 2001; 322: 226-231., .
- 14Effect sizes in cumulative meta-analyses of mental health randomized trials evolved over time. J Clin Epidemiol 2004; 57: 1124-1130., , , , , , et al.
Kristian Thorlund M.Sc.*, Tahany Awad M.Sc.*, * Cochrane Hepatobiliary Group, Copenhagen Trial Unit, Center for Clinical Intervention Research, Rigshospitalet, Copenhagen University Hospital, Copenhagen, Denmark.