In this note, we explain how the method proposed by Hartung and Knapp provides a compromise between conventional meta-analysis methodology and ‘unconstrained averaging’, as used by the Particle Data Group. Copyright © 2013 John Wiley & Sons, Ltd.

Graphs are an essential part of scientific communication. Complex datasets, of which meta-analyses are textbook examples, benefit the most from visualization. Although a number of graph options for meta-analyses exist, the extent to which these are used was hitherto unclear. A systematic review on graph use in meta-analyses in three disciplines (medicine, psychology, and business) and nine journals was conducted. Interdisciplinary differences, which are mirrored in the respective journals, were revealed, that is, graph use correlates with external factors rather than methodological considerations. There was only limited variation in graph types (with forest plots as the most important representatives), and diagnostic plots were very rare. Although an increase in graph use over time could be observed, it is unlikely that this phenomenon is specific to meta-analyses. There is a gaping discrepancy between available graphic methods and their application in meta-analyses. This may be rooted in a number of factors, namely, (i) insufficient dissemination of new developments, (ii) unsatisfactory implementation in software packages, and (iii) minor attention on graphics in meta-analysis reporting guidelines. Using visualization methods to their full capacity is a further step in using meta-analysis to its full potential. Copyright © 2013 John Wiley & Sons, Ltd.

The current paper describes and illustrates a Bayesian approach to the meta-analysis of coefficient alpha. Alpha is the most commonly used estimate of the reliability or consistency (freedom from measurement error) for educational and psychological measures. The conventional approach to meta-analysis uses inverse variance weights to combine information from independent studies to provide an overall estimate. The Bayesian approach provides similar estimates to the conventional approach if a diffuse prior is used. However, the Bayesian approach also provides ‘shrunken’ local estimates of reliability in each context. The amount of shrinkage depends upon both the variability in the underlying populations and the sampling variance of the local estimates. Advantages of the approach are the estimation of individual studies adjusted for sampling error and the application of meta-analytic results to new local studies in which the local study ‘borrows strength’ from the meta-analysis. The ability to borrow strength for the new local studies is particularly useful in applied work in which the estimate of the local parameter is of primary interest. The approach is illustrated by the analysis of studies of the reliability of the General Ethnicity Questionnaire – Abridged, a measure of identification with the culture of one's heritage and the culture of one's host country. Copyright © 2013 John Wiley & Sons, Ltd.

Dependent effect size estimates are a common problem in meta-analysis. Recently, a robust variance estimation method was introduced that can be used whenever effect sizes in a meta-analysis are not independent. This problem arises, for example, when effect sizes are nested or when multiple measures are collected on the same individuals. In this paper, we investigate the robustness of this method in small samples when the effect size of interest is the risk difference, log risk ratio, or log odds ratio. This simulation study examines the accuracy of 95% confidence intervals constructed using the robust variance estimator across a large variety of parameter values. We report results for both estimations of the mean effect (intercept) and of a slope. The results indicate that the robust variance estimator performs well even when the number of studies is as small as 10, although coverage is generally less than nominal in the slope estimation case. Throughout, an example based on a meta-analysis of cognitive behavior therapy is used for motivation. Copyright © 2013 John Wiley & Sons, Ltd.

Weighting of studies in meta-analysis is usually implemented by using the estimated inverse variances of treatment effect estimates. However, there is a possibility of one study dominating other studies in the estimation process by taking on a weight that is above some upper limit. We implement an estimator of the heterogeneity variance that takes advantage of dominance adjusted weights. The performance of this estimator is compared with that of the commonly used estimator in meta-analysis, the DerSimonian–Laird estimator. Two test procedures for the overall treatment effect are proposed that are based on the quadratic form associated with the proposed heterogeneity variance estimator. An example is given to illustrate the application of these procedures. Copyright © 2013 John Wiley & Sons, Ltd.

Meta-analysis of time-to-event data has proved difficult in the past because consistent summary statistics often cannot be extracted from published results. The use of individual patient data allows for the re-analysis of each study in a consistent fashion and thus makes meta-analysis of time-to-event data feasible. Time-to-event data can be analysed using proportional hazards models, but incorporating random effects into these models is not straightforward in standard software.

Regression methods are widely used by researchers in many fields, yet methods for synthesizing regression results are scarce. This study proposes using a factored likelihood method, originally developed to handle missing data, to appropriately synthesize regression models involving different predictors. This method uses the correlations reported in the regression studies to calculate synthesized standardized slopes. It uses available correlations to estimate missing ones through a series of regressions, allowing us to synthesize correlations among variables as if each included study contained all the same variables. Great accuracy and stability of this method under fixed-effects models were found through Monte Carlo simulation. An example was provided to demonstrate the steps for calculating the synthesized slopes through sweep operators. By rearranging the predictors in the included regression models or omitting a relatively small number of correlations from those models, we can easily apply the factored likelihood method to many situations involving synthesis of linear models. Limitations and other possible methods for synthesizing more complicated models are discussed. Copyright © 2012 John Wiley & Sons, Ltd.

An individual diagnostic accuracy study rarely provides enough information to make conclusive recommendations about the accuracy of a diagnostic test; particularly when the study is small. Meta-analysis methods provide a way of combining information from multiple studies, reducing uncertainty in the result and hopefully providing substantial evidence to underpin reliable clinical decision-making. Very few investigators consider any sample size calculations when designing a new diagnostic accuracy study. However, it is important to consider the number of subjects in a new study in order to achieve a precise measure of accuracy.

The use of meta-analysis in medicine and epidemiology really took off in the 1970s. However, in high-energy physics, the Particle Data Group has been carrying out meta-analyses of measurements of particle masses and other properties since 1957. Curiously, there has been virtually no interaction between those working inside and outside particle physics. In this paper, we use statistical models to study two major differences in practice. The first is the usefulness of systematic errors, which physicists are now beginning to quote in addition to statistical errors. The second is whether it is better to treat heterogeneity by scaling up errors as do the Particle Data Group or by adding a random effect as does the rest of the community. Besides fitting models, we derive and use an exact test of the error-scaling hypothesis. We also discuss the other methodological differences between the two streams of meta-analysis. Our conclusion is that systematic errors are not currently very useful and that the conventional random effects model, as routinely used in meta-analysis, has a useful role to play in particle physics. The moral we draw for statisticians is that we should be more willing to explore ‘grassroots’ areas of statistical application, so that good statistical practice can flow both from and back to the statistical mainstream. Copyright © 2012 John Wiley & Sons, Ltd.

Background

Non-randomized studies may provide valuable evidence on the effects of interventions. They are the main source of evidence on the intended effects of some types of interventions and often provide the only evidence about the effects of interventions on long-term outcomes, rare events or adverse effects. Therefore, systematic reviews on the effects of interventions may include various types of non-randomized studies. In this second paper in a series, we address how review authors might articulate the particular non-randomized study designs they will include and how they might evaluate, in general terms, the extent to which a particular non-randomized study is at risk of important biases. We offer guidance for describing and classifying different non-randomized designs based on specific features of the studies in place of using non-informative study design labels. We also suggest criteria to consider when deciding whether to include non-randomized studies. We conclude that a taxonomy of study designs based on study design features is needed. Review authors need new tools specifically to assess the risk of bias for some non-randomized designs that involve a different inferential logic compared with parallel group trials. Copyright © 2012 John Wiley & Sons, Ltd.

Background

Background

The terms applicability, generalizability, external validity and transferability are related, sometimes used interchangeably and have in common that they lack a clear and consistent definition in the classic epidemiological literature. However, all of these terms generally describe one overarching theme: whether or not available research evidence can be directly utilized to answer the healthcare questions at hand, ideally supported by a judgment about the degree of confidence for this utilization. This concept has been called directness. The objectives of this paper were to delineate how non-randomized studies (NRS) inform judgments in relation to directness and the concepts that it encompasses in the context of systematic reviews. We will briefly review what is known and describe the theoretical and practical issues as well as offer guidance to those tackling the challenges of judging directness and using research evidence to answer healthcare questions with evidence from NRS.

Background

A problem that is frequently encountered during the systematic review process is when studies that meet the inclusion criteria do not provide the appropriate numerical estimates to include in a meta-analysis. For dichotomous outcomes, a method has been suggested by Di Pietrantonj for reconstructing the 2 × 2 table when the Odds Ratio (OR), the Standard Error (SE(lnOR)) and the sample sizes are provided. The method produces two possible 2 × 2 tables; and to select the correct one, the Control Group Risk (CGR) is used. As CGR is typically unknown and only rounded figures of the OR and SE(lnOR) are provided, the accuracy of the reconstruction method varies. In this paper, we evaluate the performance of the method using simulated and empirical data. Small studies with large OR and CGR away from 50% are reconstructed satisfactorily, and the use of SE(lnOR) rounded to the third decimal rather than the second one improves the performance of the method. However, when CGR is unknown, its estimation from other studies is problematic as it exhibits high heterogeneity. Inclusion of an incorrectly reconstructed table in the meta-analysis may result in different summary effects. Reviewers that consider applying the method should be cautious about its impact in the meta-analysis. Copyright © 2012 John Wiley & Sons, Ltd.

Visual display of reported impacts is a valuable aid to both reviewers and readers of systematic reviews. Forest plots are routinely prepared to report standardised effect sizes, but where standardised effect sizes are not available for all included studies a forest plot may misrepresent the available evidence. Tabulated data summaries to accompany the narrative synthesis can be lengthy and inaccessible. Moreover, the link between the data and the synthesis conclusions may be opaque.

In many scientific disciplines, systematic reviews and meta-analyses are increasingly indispensable as summaries of the evidence in relation to a particular phenomenon, making it ever more important for scientists to know how best to review evidence. Khan, Kunz, Kleijnen, and Antes's Systematic Reviews to Support Evidence-Based Medicine (2nd edition, 2011, CRC Press, ISBN-13: 9781853157943) provides a fine, brief introduction to the subject, in particular for those in medicine and public health fields. Our review details strengths of this book, such as its conciseness and clarity and its close match to current conventions in systematic reviewing. We also discuss nuances of the subject that might augment a future edition or lead readers to other resources. In doing so, these discussions also address how present practices in systematic reviewing might improve. Copyright © 2013 John Wiley & Sons, Ltd.