The checklist for methods reporting is available at: http://www.wiley-docs.com/Checklist_for_reporting_experimental_details.pdf
Reporting standards in experimental studies
Article first published online: 30 SEP 2013
© 2013 John Wiley & Sons Ltd/CNRS
Volume 16, Issue 12, pages 1419–1420, December 2013
How to Cite
Ecology Letters (2013) 16: 1419–1420
- Issue published online: 28 OCT 2013
- Article first published online: 30 SEP 2013
Hypothesis tests in ecology and evolution rely strongly on experimental studies. As is well known, a major advantage of experiments is that they allow the manipulation of factors of interest and thus may allow separating confounded covariates (i.e. explanatory factors). Empirical studies in ecology and evolution contribute to more general knowledge through repeated tests of a hypothesis in a variety of contexts.
Unfortunately, it is very common for published studies in ecology to be incomplete with regard to reporting many basic and essential details of the experiment. Missing information on the size and timing of an experiment occur frequently, although many experimental results are scale-dependent (Englund 1997; Hillebrand 2009), and field experiments sometimes even lack details on the exact location of the experiments. The gaps in characterising experiments also comprise details of measurements and sample processing. There are two major reasons to correct this problem. First, this lack of context limits the ability to correctly interpret the results of these studies. Second, these limitations make it much more difficult to correctly integrate such studies in research syntheses.
The success of Ecology Letters as a journal is based on its reputation for publishing innovative and important new work in the discipline. The value of this work depends on the completeness and accuracy of these publications. In addition, Ecology Letters also benefits from its ability to attract innovative quantitative syntheses on highly relevant topics in ecology and evolution. Therefore, the editorial board of Ecology Letters aims to raise awareness of reporting details, including those concerning how data were collected, with the goal of enhancing the value of the study itself as well as the ability to use the results of the study in synthesis efforts. To this aim, we have added a checklist to the Guide for Authors to encourage authors to check the level of detail reported for each study. In other disciplines, the minimum necessary information and other requirements for publication are becoming more standardised across journals: for instance, see the Equator network (http://www.equator-network.org/index.aspx?o=1032) and the Biosharing resource (http://biosharing.org/standards/reporting_guideline) for checklists and guidelines for publication in medicine and other fields.
An increasingly valuable and important use of both experimental and observational ecological studies is their incorporation in research syntheses. Experiments in ecology are often small and local, limiting our ability to reach general inferences from single studies. Quantitative synthesis in the form of meta-analysis offers the promise of overcoming some of the limitations to generalising from individual studies by detecting central tendencies across groups of studies and by testing hypotheses on a set of experiments conducted in different settings and with different organisms. If a sufficient number of studies exist, meta-analysis offers a formal framework for testing generalisability, but also allows additionally testing more subtle effects that are not easily detectable in individual studies (Francoeur 2001).
Modern methods for research synthesis, including meta-analysis, have been developed to address the need for reaching generalisations across studies. Patterns have been elucidated, major paradigms in ecology and evolution have been tested or reanalysed, and new hypotheses have been formulated. In conducting a modern meta-analysis, one examines whether data from individual studies are comparable by modelling the influence of different covariates (explanatory factors) on the studies' outcomes. Meta-analysis methodology includes the development of statistical tools to model the contribution of covariates to heterogeneity among studies. Sometimes also called meta-regression, these techniques allow testing of how different study outcomes are influenced by characteristics of experiments such as location, size or duration, or the characteristics of the organisms studied. For this reason, each study entered in a meta-analysis has to be carefully characterised with regard to the study details so that the influence of such covariates can be evaluated (Cote et al. 2013; Rothstein et al. 2013).
As was the case in the early years of modern research synthesis in other disciplines, some ecologists have faulted the use of meta-analyses in specific and in general, criticising, for example, the discrepancies between studies and offering the view that this heterogeneity impairs their comparability in a common analysis (Whittaker 2010). The criticism that meta-analysis “combines apples and oranges” has long ago been addressed in the literature of meta-analysis and research synthesis (Borenstein et al. 2009), and in fact, it was recently pointed out that without heterogeneity in the literature being combined, it is impossible to reach general conclusions (‘merely combining Jonathan apples is undesirable,’ R. Rosenthal, Olkin Award Speech, SRSM meeting, Providence RI June 28, 2013). Better reporting of data will facilitate higher quality meta-analyses because heterogeneity can be better modelled and taken into account when covariates are accurately reported.
To remedy the limitations in reporting contextual information above, before submitting a paper, authors should check that their studies adhere to this checklist of reporting standards and provide these essential details in their ‘Materials and Methods’ section. The methods in the main body of an Ecology Letters paper need to contain sufficient detail for a general reader to understand the study, and for a scientist to successfully re-conduct the study. Additional details can go into online supplements. The checklist provides categories of relevant information which must or should be reported. ‘Must report’ applies to the issues of treatments, replication, location, timing, spatial and temporal scale, and ‘should report’ to context-specific information on the background of the experiment.
Improved standards for data reporting will enhance the value of primary studies and will greatly facilitate the ability to carry out higher quality research syntheses, including meta-analyses, in ecology and evolutionary biology.
- 2009). Criticisms of Meta-Analysis. In: Introduction to Meta-Analysis. John Wiley & Sons, Ltd, Chichester, U.K, pp. 377–387. , , & . (
- 2013). Gathering Data: Searching Literature & Selection Criteria. In: Handbook of meta-analysis in ecology and evolution (eds Koricheva, J., Gurevitch, J. & Mengersen, K.). Princeton University Press, Princeton USA, pp. 37–51. , , & . (
- 1997). Importance of Spatial Scale and Prey Movements in Predator Caging Experiments. Ecology, 78, 2316–2325. (
- 2001). Meta-analysis of lotic nutrient amendment experiments: detecting and quantifying subtle responses. J. N. Am. Benthol. Soc., 20, 358–368. (
- 2009). Meta-analysis of grazer control of periphyton biomass across aquatic ecosystems. J. Phycol., 45, 798–806. (
- 2013). Quality Standards for Research Syntheses. In: Handbook of meta-analysis in ecology and evolution (eds Koricheva, J., Gurevitch, J. & Mengersen, K.). Princeton University Press, Princeton USA, pp. 323–338. , , , & . (
- 2010). Meta-analyses and mega-mistakes: calling time on meta-analysis of the species richness-productivity relationship. Ecology, 91, 2522–2533. (