In random-effects model meta-analysis, the conventional DerSimonian–Laird (DL) estimator typically underestimates the between-trial variance. Alternative variance estimators have been proposed to address this bias. This study aims to empirically compare statistical inferences from random-effects model meta-analyses on the basis of the DL estimator and four alternative estimators, as well as distributional assumptions (normal distribution and t-distribution) about the pooled intervention effect. We evaluated the discrepancies of p-values, 95% confidence intervals (CIs) in statistically significant meta-analyses, and the degree (percentage) of statistical heterogeneity (e.g. I²) across 920 Cochrane primary outcome meta-analyses. In total, 414 of the 920 meta-analyses were statistically significant with the DL meta-analysis, and 506 were not. Compared with the DL estimator, the four alternative estimators yielded p-values and CIs that could be interpreted as discordant in up to 11.6% or 6% of the included meta-analyses pending whether a normal distribution or a t-distribution of the intervention effect estimates were assumed. Large discrepancies were observed for the measures of degree of heterogeneity when comparing DL with each of the four alternative estimators. Estimating the degree (percentage) of heterogeneity on the basis of less biased between-trial variance estimators seems preferable to current practice. Disclosing inferential sensitivity of p-values and CIs may also be necessary when borderline significant results have substantial impact on the conclusion. Copyright © 2012 John Wiley & Sons, Ltd.

The print component of this fourth ‘Article Alerts’ installment comprises 100 articles published in 2010. More than 2500 items have been added to the archive component since the preceding installment. Of these new archive items, more than 1500 were disseminated in 2009; the remainder, between 1994 and 1998, inclusive. Copyright © 2012 John Wiley & Sons, Ltd.

There is an increasing interest in the conduct of qualitative evidence syntheses (QES), particularly in the field of health care. Approaches to QES vary in the way they conduct a search, a critical appraisal or the data-analysis. To date, the use of multi-context versus context-specific QES has not yet been fully considered. In a multi-context, QES exhaustive searches are used that retrieve studies from a broad variety of geographical, socio-cultural, political, historical, economical, health care, linguistic, or other context relevant to the review. Authors of a context-specific QES would generally have a particular end user in mind, therefore, using a selective search strategy with a focus on one particular context in order to provide lines of actions or theories that are sensitive to a local setting. We used the insights from a recently conducted, context-specific QES to map out potential strengths and weaknesses of these two approaches and make recommendations regarding the future conduct of QES. We propose two ways of combining the best of both: the production of umbrella reviews of context-specific syntheses and/or the trans-cultural modification and trans-contextual adaptation of findings from multi-context syntheses. Copyright © 2012 John Wiley & Sons, Ltd.

Methods to calculate bias-adjusted estimates for meta-analyses are becoming more popular. The objective of this paper is to use the structural approach to bias and causal diagrams to show that (i) the current use of the bias-adjusted estimating tools may sometimes introduce bias rather than reduce it and (ii) the Cochrane collaboration risk of bias tool, which was designed for randomized studies, is also applicable to non-randomized studies with only minimal changes. Causal diagrams are used to illustrate each of the items in the current risk of bias tool and how they apply to both randomized and non-randomized studies. With the exception of confounding by indication, the structure of all potential biases present in non-randomized studies may also be present in randomized studies. In addition, causal diagrams demonstrate important limitations to the methods currently being developed to provide bias-adjusted estimates of individual studies in meta-analyses. Finally, causal diagrams can be helpful in deciding when it is appropriate to combine studies in a meta-analysis of non-randomized studies even though the studies may use different adjustment sets. Copyright © 2012 John Wiley & Sons, Ltd.

Because of insufficient evidence from direct comparison trials, the use of indirect or mixed treatment comparison methods has attracted growing interest recently. We investigated the views and knowledge of Cochrane systematic review authors regarding the use of indirect comparison and related methods in the evaluation of competing healthcare interventions.

Meta-analysis is widely used to synthesise results from randomised trials. When the relevant trials collected time-to-event data, individual participant data are commonly sought from all trials. Meta-analyses of time-to-event data are typically performed using variants of the log-rank test, but modern statistical software allows for the use of maximum likelihood methods such as Cox proportional hazards models or interval-censored logistic regression. In this paper, the different approaches to the analysis of time-to-event data are examined and compared with show that log-rank test approaches are in fact first-order approximations to the maximum likelihood methods and that some methods assume proportional hazards, whereas others assume proportional odds. A simulation study is performed to compare the different methods, which shows that log-rank test approaches give biased estimates when the underlying hazard ratio or odds ratio is far from unity. It also shows that proportional hazards methods give biased results when hazards are not proportional, and proportional odds methods give biased results when odds are not proportional. Maximum likelihood models should, therefore, be preferred to log-rank test based methods for the meta-analysis of time-to-event data and any such meta-analysis should investigate whether proportional hazards or proportional odds assumptions are valid. Copyright © 2011 John Wiley & Sons, Ltd.

Meta-analyses of individual patient data (IPD) provide a strong and authoritative basis for evidence synthesis. IPD are particularly useful when the outcome of interest is the time to an event. Methodological developments now enable the meta-analysis of time-to-event IPD using a single model, allowing treatment effect and across-trial heterogeneity parameters to be estimated simultaneously. This differs from the standard approaches used with aggregate data, and also predominantly with IPD. Facilitated by a simulation study, we investigate what these new ‘one-stage’ random-effects models offer over standard ‘two-stage’ approaches. We find that two-stage approaches represent a robust, reliable and easily implementable way to estimate treatment effects and account for heterogeneity. Nevertheless, one-stage models can be used to provide a deeper insight into the data. Software for fitting one-stage Cox models with random effects using Restricted Maximum Likelihood methodology is made available, and its use demonstrated on an IPD meta-analysis assessing post-operative radio therapy for patients with non-small cell lung cancer. Copyright © 2011 John Wiley & Sons, Ltd.

Within the systematic review process, the searching phase is critical to the final synthesis product, its use and value. Yet, relatively little is known about the utility of different search strategies for reviews of complex, inter-disciplinary evidence. This article used a recently completed programme of work on cultural and sporting engagement to conduct an empirical evaluation of a comprehensive search strategy. Ten different types of search source were evaluated, according to three dimensions: (i) effectiveness in identifying relevant studies; (ii) efficiency in identifying studies; and (iii) adding value by locating studies that were not identified by any other sources. The study found that general bibliographic databases and specialist databases ranked the highest on all three dimensions. Overall, websites and journals were the next most valuable types of source. For reviewers, these findings highlight that general and specialist databases should remain a core component of the comprehensive search strategy, supplemented with other types of sources that can efficiently identify unique or grey literature. For policy makers and other research commissioners, this study highlights the value of methodological analysis for improving the understanding of, and practice in, policy relevant, inter-disciplinary systematic reviews. Copyright © 2011 John Wiley & Sons, Ltd.

Meta-analytic structural equations modeling is increasingly used in theory testing. There has been much debate when meta-analyzed correlation matrices are used in structural equations modeling on whether to use mean observed correlations (i.e., corrected only for sampling error) or correlations corrected for study artifacts such as unreliability in measures. This paper investigates whether the fit indices are affected by the corrections and if the stability of the paths (i.e., changes in significance, magnitude, and relative strengths or rank order) is affected by the corrections. Results suggest that substantive model conclusions are generally unaffected by study artifacts and related statistical corrections as long as the variables included in the path analyses had typical levels of reliability as found in the psychological literature. More specifically, all models examined exhibited similar model fit and pathway stability. Copyright © 2011 John Wiley & Sons, Ltd.

Background

In any scientific discipline, the ability to portray research patterns graphically often aids greatly in interpreting a phenomenon. In part to depict phenomena, the statistics and capabilities of meta-analytic models have grown increasingly sophisticated. Accordingly, this article details how to move the constant in weighted meta-analysis regression models (viz. “meta-regression”) to illuminate the patterns in such models across a range of complexities. Although it is commonly ignored in practice, the constant (or intercept) in such models can be indispensible when it is not relegated to its usual static role. The moving constant technique makes possible estimates and confidence intervals at moderator levels of interest as well as continuous confidence bands around the meta-regression line itself. Such estimates, in turn, can be highly informative to interpret the nature of the phenomenon being studied in the meta-analysis, especially when a comparison with an absolute or a practical criterion is the goal. Knowing the point at which effect size estimates reach statistical significance or other practical criteria of effect size magnitude can be quite important. Examples ranging from simple to complex models illustrate these principles. Limitations and extensions of the strategy are discussed. Copyright © 2011 John Wiley & Sons, Ltd.