META-ANALYSIS OF ECONOMICS RESEARCH REPORTING GUIDELINES
Meta-regression analysis (MRA) can provide objective and comprehensive summaries of economics research. Their use has grown rapidly over the last few decades. To improve transparency and to raise the quality of MRA, the meta-analysis of economics research-network (MAER-Net) has created the below reporting guidelines. Future meta-analyses in economics will be expected to follow these guidelines or give valid reasons why a meta-analysis must deviate from them.
We live in a remarkable era of economics research and growth in our collective empirical knowledge. There continues to be an explosion of empirical economics research. Inexpensive computers and hand-held devices allow us to download relevant economic data in seconds, more data than was available to past generations over entire careers. With widely available software, any researcher can run millions of regressions before lunch (Sali-i-Martin, 1997). As with any advancement, there are some costs.
Our scientific journals contain so much information, including noise and misinformation, that it threatens to drown out genuine scientific knowledge and informed policy action (Heckman, 2001). Within this flood of research output, we always find a wide spread of the reported results. Because economic phenomena are the result of human interactions in different and evolving historical, cultural and institutional environments (Hodgson, 1998; Rosenberger and Johnston, 2009), economic researchers will often find large disparities among estimates of important economic parameters, typically many times greater than what is implied by the reported uncertainty (e.g. standard errors) of these estimated parameters. Thus, without systematic, unbiased and rigorous summaries and analyses of empirical literatures in economics, the true consequences of research are likely to remain hidden, and we will be unable to see how key parameters vary.
A collection of econometric methods have been developed to fill this void – meta-regression analysis (MRA) (Stanley and Jarrell, 1989; Roberts, 2005). Applications of MRA, like their data sources, have grown at a remarkable rate – 18% per year over the last couple of decades (Stanley and Doucouliagos, 2012). Perhaps as many as 200 meta-analyses are conducted per year on economic topics. However, just as the studies on which MRAs are based are heterogeneous, the conduct and reporting of these MRAs are also diverse. While diversity in available methods is necessary and appropriate, it can also obscure potential ‘abuse’ of MRA in economics (Nelson and Kennedy, 2009). To avoid such abuse and the misleading conclusions that might result, we believe that it is time to codify minimum quality standards for this important and rapidly growing field.
MRA is the systematic review and quantitative synthesis of empirical economic evidence on a given hypothesis, phenomenon, or effect. MRA is a type of meta-analysis that is explicitly designed to integrate econometric estimates, typically regression coefficients or transformations of regression coefficients. It seeks both to summarize and to explain the wide, often disparate, variation routinely found among reported econometric results. Although guidelines for conducting and reporting meta-analyses have been offered before (Stroup et al., 2000; Higgins and Green, 2008), none have explicitly considered the type of empirical evidence typically found in applied econometric research. Moreover, existing MRA guidelines in the economics literature focus primarily on methodological aspects of econometric estimation and interpretation (e.g. Nelson and Kennedy, 2009), rather than on broader standards of MRA practice and reporting. Because MRA is widely accepted throughout our scientific literature, members of the Meta-Analysis of Economics Research Network (MAER-Net) believe that it is appropriate to offer guidelines for reporting meta-regression analyses and to serve as minimal standards for academic journals. The editorial board of the Journal of Economic Surveys will expect that any meta-analysis submitted to JoES fulfill these reporting requirements or give valid reasons/justifications why meta-analysts deviate from them.
MAER-Net recommends that all meta-analyses in economics should comply with the following reporting protocols.
2. Reporting Guidelines for Meta-Regression Analyses in Economics
Research papers that conduct MRAs in economics should include:
2.1 Research Questions and Effect Size
- A clear statement of the specific economic theories, hypotheses or effects studied.
- A precise definition of how effects are measured (the ‘effect size’), accompanied by any relevant formulas.
- An explicit description about how measured effects are comparable, including any methods used to standardize or convert them to a common metric.
2.2 Research Literature Searching, Compilation and Coding
- A full report of how the research literature was searched. This report should include:
- the exact databases or other sources used;
- the precise combination of keywords employed and
- the date that the search was completed.
- A full disclosure of the rules for study (or effect size) inclusion/exclusion. It is also useful to provide a list of all studies included and a description of why others were excluded.
- A statement addressing who searched, read and coded the research literature. Two or more reviewers should code the relevant research.
- A complete list of the information coded for each study or estimate. At a minimum, we recommend that reviewers code:
- the estimated effect size;
- its standard error, when feasible, and the degrees of freedom (or sample size);
- variables that distinguish which type of econometric model, methods and techniques were employed;
- dummy (i.e. 0/1) variables for the omission of theoretically relevant variables in the research study investigated;
- empirical setting (e.g. region, market, industry);
- data types (panel, cross-sectional, time series, etc.);
- year of the data used and/or publication year;
- type of publication (journal, working paper, book chapter, etc.) and
- the primary study, publication and/or dataset from which an observation is drawn.
2.3 MRA Modeling Issues
- A table of descriptive statistics of the variables that are coded (means and standard deviations) and graph(s) displaying the effect sizes (e.g. funnel graphs, forest plots, bar charts).
- A fully reported multiple MRA, along with the exact strategy used to simplify it (e.g. general-to-specific, Bayesian).
- An investigation of publication, selection, and misspecification biases. When suspected, these should be controlled for in subsequent MRA models.
- Methods to accommodate heteroscedasticity and within-study dependence.
- Results from MRA model specification tests, robustness checks or sensitivity analyses.
With one possible exception, MAER-Net has come to a clear consensus about these reporting guidelines. The requirement to have two reviewers code all the relevant research has received the most comment and discussion. As economists, we all are acutely aware of the tradeoff between the improved quality that the second coder will likely add (through catching mistakes and resolving ambiguities) and the increased cost (in weeks of highly skilled professional labour). We understand that the highest standards of scientific rigor demand at least two highly knowledgeable researchers code the relevant research base. Nonetheless, MAER-Net does not wish to prohibit Ph.D. students and researchers at resource-challenged institutions from employing this important tool to understand their areas of research. To finesse these opposing concerns, the above statement is sufficiently broad to encompass a second reviewer randomly checking a substantial proportion of the research literature if their coding protocol is stated explicitly and justified.
These guidelines are not meant to express the last words about how best to conduct MRAs in economics. Rather, we support all efforts to raise the quality of MRAs. Nelson and Kennedy (2009) and Stanley and Doucouliagos (2012) offer further useful guidance for what is best practice in applying meta-analysis to economics and environmental economics. The above reporting protocols represent a floor for scientific rigor, replicability and quality that we hope will be surpassed by most MRAs in economics.