SEARCH

SEARCH BY CITATION

Keywords:

  • climate manipulation experiments;
  • elevated CO2;
  • literature surveys;
  • meta-analysis;
  • publication bias;
  • soil respiration;
  • statistical synthesis

Literature surveys are a prerequisite to enhance scientific knowledge as they allow us to separate systemic from idiosyncratic mechanisms and processes, and thus provide insight at a higher level than can be gained from individual studies. In ecological research, statistical synthesis of literature surveys, using meta-analysis, has become a powerful tool to quantify global mean responses to a changing climate (Curtis & Wang, 1998; Medlyn et al., 1999; Rustad et al., 2001; Treseder, 2004, 2008; Knorr et al., 2005; de Graaff et al., 2006; Luo et al., 2006; Janssens et al., 2010). Such analyses have greatly improved our understanding of ecosystem functioning and the parameterization of models. Unfortunately, there is also a downside to quantitative review methods such as meta-analysis, as they can easily be affected by publication bias (Møller & Jennions, 2001). Publication bias can be defined as the selective publication of articles showing certain types of results over those showing other types of results. The most commonly suspected publication bias is the tendency for authors and journals to only publish studies with statistically significant results, which has been termed the ‘file-drawer problem’ (Rosenthal, 1979). Moreover, researchers are under increasing pressure to publish frequently, and it is much easier to publish results that can easily be explained or support widely accepted hypotheses (Jarvis et al., 2001), than having to fight a time-consuming battle with conservative and suspicious referees (as they perhaps should be).

A clear example of such a publication bias occurred in a European network of CO2-enrichment experiments on trees (ECOCRAFT, 1999). From the 19 experiments that measured the response of soil respiration to elevated CO2, only one-third were published. Across these 19 experiments the mean stimulation of soil respiration was 9% (Fig. 1a). In elevated CO2, more C is allocated belowground to fine roots and mycorrhizae (Ceulemans et al., 1999; Pendall et al., 2004), and therefore an increase in soil respiration is the expected response (Zak et al., 2000). However, we found that elevated atmospheric CO2 concentrations enhanced soil respiration only in 11 out of 19 experiments (Fig. 1a), despite enlarged root systems in 15 experiments. Surprisingly, among the six published studies on soil respiration in this network, five showed that the expected increase in soil respiration and the mean overall response across these six experiments was a statistically significant 45% increase in soil respiration (Fig. 1b). Of the eight experiments with a negative response, only one was published in a conference proceedings book (Le Dantec et al., 1997). The other negative responses were either never written up and submitted for publication because the researchers had difficulties believing or explaining their results, or were rejected by reviewers who did not accept such unexplainable, negative responses. Therefore, it is likely that many similar, negative responses from outside the ECOCRAFT network have died a silent death in some researcher’s drawer.

image

Figure 1. Frequency distributions of experiments studying the responses of soil respiration to atmospheric CO2 enrichment. (a) All studies performed in the ECOCRAFT network. (b) Only those studies of (a) that were published. (c) All studies of (a) that were not published. In each panel, the meta-analysis mean effect size, the 95% CI and the number of studies (n) are indicated.

Download figure to PowerPoint

There are a number of indirect methods to indicate whether a data set could be influenced by publication bias (Borenstein, 2005). Here, we tested these available methods to find out whether they can detect the clear case of publication bias in the ECOCRAFT data set. We then applied them on a comprehensive literature survey data set of soil-respiration responses to CO2 enrichment (Dieleman et al., 2010; Supporting Information Tables S1–S3) to test for indications of publication bias in this much larger data set. The prerequisites for the data to be included in the database were: the availability of a measure of variation on the data (e.g. SD), and a minimum of two replicates in the experimental design. Statistical tests were performed using MetaWin 2.1 (Rosenberg et al., 2000) and Comprehensive Meta Analysis (Borenstein, 2005).

Tests for publication bias

  1. Top of page
  2. Tests for publication bias
  3. Can the publication bias in the ECOCRAFT network be detected?
  4. Is publication bias detectable in a comprehensive database?
  5. Conclusion
  6. Acknowledgements
  7. References
  8. Supporting Information

First, a visual sense of the data should be considered using funnel plots. A funnel plot is a scatter plot of the effect sizes obtained in individual studies vs some measure of size of the experiments (usually related to sample size, here 1/standard error). This scatter plot was coined ‘funnel plot’ as it often has a funnel shape because studies with a larger sample size tend to have smaller sampling error. In the absence of publication bias, the studies will be dispersed symmetrically about the combined mean effect size. However, the smaller the sample size, the smaller the precision and the greater the estimated effect sizes must differ from zero to be statistically significant. Thus, if publication success is related to statistical significance, one would expect that the bottom of the plot would show a higher concentration of studies on one side of the mean than on the other (e.g. see Møller & Jennions, 2001; Borenstein, 2005).

Second, a numerical form of evidence can be provided by the rank correlation test (Begg & Mazumdar, 1994), Egger’s regression procedures (Borenstein, 2005) and fail-safe numbers (Borenstein, 2005). Both the Begg and Mazumdar’s rank correlation test and the Egger’s regression method are statistical analogues for the funnel plot. While the rank correlation test reports the correlation of the individual effect sizes and their variance (or standard errors), Egger’s regression reports the linear regression between effect size and study precision (1/standard error). In both cases, a statistically significant relationship indicates that studies with results in one direction are preferentially published, which may indicate publication bias (Borenstein, 2005). The rank correlation test has low power unless there is severe bias, or when the number of data points included in the analysis exceeds 75. The power of the Egger’s regression test is believed to be higher than the power for the rank correlation test, but is still relatively low unless there are a substantial number of studies. Fail-safe numbers indicate the number of nonsignificant, unpublished studies that would need to be added to the meta-analysis to shift the overall effect from statistically significant to nonsignificant. High numbers indicate that publication bias is unlikely (5n + 10 (n, number of studies) is considered high enough according to Rosenthal (1979)).

Last, a more nuanced perspective can be provided by the trim and fill method (Duval & Tweedie, 2000). If publication bias is not affected by sample size, but because of a tendency not to publish unexpected or unexplainable results, the trim and fill method may provide a more correct indication. The trim and fill method is based on the condition of a symmetrical distribution of the data around the overall mean effect in a funnel plot. Using this method, the studies needed to achieve total symmetry in the funnel plot are added to the analysis, and a new overall mean is calculated (Duval & Tweedie, 2000). This method thus provides an estimate of the effect size after the bias has been taken into account, and can indicate whether the treatment effect is, or is no longer, statistically significant.

As the various statistical procedures approach the problem of bias from a number of directions and each has its own drawbacks, the goal should be to compile the different pieces of information and make a decision based on the weight of evidence.

Can the publication bias in the ECOCRAFT network be detected?

  1. Top of page
  2. Tests for publication bias
  3. Can the publication bias in the ECOCRAFT network be detected?
  4. Is publication bias detectable in a comprehensive database?
  5. Conclusion
  6. Acknowledgements
  7. References
  8. Supporting Information

There were too few experiments to test the ECOCRAFT data set for publication bias by means of funnel plots. Hence, these plots are not shown here. Based on the rank correlation and regression analysis, neither the complete ECOCRAFT data set, nor the separated published and unpublished data sets demonstrate evidence for a publication bias problem (Table 1). This lack of evidence for publication bias using the correlation and regression tests probably arises from the low statistical power of the test with a low number of studies (Begg & Mazumdar, 1994; Borenstein, 2005). The Rosenthal fail-safe number is not relevant for the total and unpublished data set as the mean effect is already not statistically significant (Fig. 1a). For the published data set, the fail-safe number indicates publication bias, as < 5+ 10 studies need to be added to the meta-analysis to shift the 45% significant increase to being nonsignificant (Table 1).

Table 1.   Correlation tests, regression tests and fail-safe numbers for the ECOCRAFT data set on responses of soil respiration to atmospheric CO2 enrichment
 Rank correlation1Egger’s regression2Rosenthal fail-safe3Calculated threshold
  1. 1The rank correlation test gives a P-value for correlation between individual effect sizes and their variance. A significant correlation indicates possible publication bias.

  2. 2Egger’s regression gives a P-value for a relationship between effect size and their precision (1/standard error). A significant correlation indicates possible publication bias.

  3. 3Rosenthal’s fail-safe number gives the number of nonsignificant, unpublished or missing studies needed to shift the overall effect from statistically significant to nonsignificant. High numbers (5+ 10 (n, number of studies) is considered high enough, see the calculated threshold) indicate that publication bias is unlikely. Numbers in italics indicate values for comparison with Rosenthal fail-safe numbers.

Total ECOCRAFT database0.510.35105
ECOCRAFT published database0.570.317.9 40
ECOCRAFT unpublished database0.90.38 75

Based on the fail-safe numbers, we can state that there was at least an indication for publication bias in the published ECOCRAFT data set. However, the number of studies was low, resulting in limited power in some of the tests. The results of the trim and fill procedure suggest that neither the complete ECOCRAFT data set, nor the split-up published and unpublished data sets demonstrate evidence for a publication bias problem. Taken together, as with most other tests, the number of available studies was probably too low to detect any sign of publication bias, although in reality there clearly was (compare Fig. 1b with 1c).

Is publication bias detectable in a comprehensive database?

  1. Top of page
  2. Tests for publication bias
  3. Can the publication bias in the ECOCRAFT network be detected?
  4. Is publication bias detectable in a comprehensive database?
  5. Conclusion
  6. Acknowledgements
  7. References
  8. Supporting Information

We then applied these tests to evaluate a data set of 49 published and 19 unpublished studies on the responses of soil respiration to elevated atmospheric CO2 concentrations under trees (the ECOCRAFT data included) (Dieleman et al., 2010, = 68, Fig. 2).

image

Figure 2. Frequency distributions of experiments studying responses of soil respiration to atmospheric CO2 enrichment. (a) Only the published studies included in a comprehensive database of soil respiration responses. (b) All (published and unpublished) studies included in this comprehensive database. In each panel, the meta-analysis mean effect size, the 95% CI and the number of studies (n) are indicated.

Download figure to PowerPoint

The overall distribution of the data in the funnel plots is quite symmetrical (Fig. 3a,b) and fail-safe numbers do not indicate a possible publication bias (Table 2). By contrast, the correlation tests and the regression tests do indicate a possible publication bias (Table 2). These significant correlations and regressions might result from the lack of ‘medium’ precision studies with negative effect sizes, promoting the significant correlations and regressions. The trim and fill procedure suggests a slight adjustment of the overall effect size (Fig. 3a,b).

image

Figure 3. Funnel plot of experiments studying responses of soil respiration to atmospheric CO2 enrichment. (a) Only the published studies included in a comprehensive database of soil respiration responses. (b) All (published and unpublished) studies included in this comprehensive database. The calculated meta-analysis mean effect size, the trim and fill corrected mean effect size, and the number of studies added in the trim and fill analysis are indicated. In the absence of publication bias, studies are distributed symmetrically about the mean calculated effect size.

Download figure to PowerPoint

Table 2.   Correlation tests, regression tests and fail-safe numbers for a comprehensive literature data set on the responses of soil respiration to atmospheric CO2 enrichment
 Rank correlation1Egger’s regression2Rosenthal fail-safe3Calculated threshold
  1. 1The rank correlation test gives a P-value for correlation between individual effect sizes and their variance. A significant correlation indicates possible publication bias.

  2. 2Egger’s regression gives a P-value for a relationship between effect size and their precision (1/standard error). A significant correlation indicates possible publication bias.

  3. 3Rosenthal’s fail-safe number gives the number of nonsignificant, unpublished or missing studies needed to shift the overall effect from statistically significant to nonsignificant. High numbers (5+ 10 (n, number of studies) is considered high enough, see the calculated threshold) indicate that publication bias is unlikely.

Total database< 0.0010.003732.3350
Total published database< 0.0010.003610.7255

In conclusion, we can state that there is probably a small publication bias and that the overall stimulatory effect of elevated CO2 on soil respiration is slightly overestimated. However, we can still have confidence that under elevated atmospheric CO2 concentrations there typically is a stimulation of soil respiration, because the trim and fill-corrected effect size is still statistically significant.

Conclusion

  1. Top of page
  2. Tests for publication bias
  3. Can the publication bias in the ECOCRAFT network be detected?
  4. Is publication bias detectable in a comprehensive database?
  5. Conclusion
  6. Acknowledgements
  7. References
  8. Supporting Information

Although we were not able to indicate a publication bias with the available tests, the ECOCRAFT data set indicates how literature means can converge towards expected, and perhaps strongly biased, values. As we showed here, a bias may sometimes be difficult to detect, and although such a strong bias probably does not occur too often, care should be taken when interpreting results from literature reviews. Quantitative syntheses, such as meta-analyses, can have a direct impact on our understanding of terrestrial responses to environmental changes as they may provide data to validate global models. Therefore, accurate estimations are of utmost importance. We are now living in a world with virtually unlimited possibilities of communication, and we should make use of technologies such as Google Code for submitting and compiling data sets (e.g. Bond-Lamberty & Thomson, 2010, although their data set is limited to published data only). Establishing traceable and citable data portals where everyone can post their (un)published data could be a major step forward. Alternatively, sending out inquiries with specific research questions electronically to a broad part of the specialized scientific community could be a better way of collecting data than literature surveys. We might not be able to gather data from every related experiment, but we could at least overcome the bias towards previously hypothesized results that may now be found occasionally in the literature.

Acknowledgements

  1. Top of page
  2. Tests for publication bias
  3. Can the publication bias in the ECOCRAFT network be detected?
  4. Is publication bias detectable in a comprehensive database?
  5. Conclusion
  6. Acknowledgements
  7. References
  8. Supporting Information

The authors want to thank the researchers of the ECOCRAFT network (Framework programmes of the EC (EC contracts within 5FP and 6FP, Environment and Research)), who provided the data that formed the foundation for this paper. Ivan A. Janssens holds a Flemish Science Foundation (FWO) research grant. This research was supported by the University of Antwerp Research Centre of Excellence ECO.

References

  1. Top of page
  2. Tests for publication bias
  3. Can the publication bias in the ECOCRAFT network be detected?
  4. Is publication bias detectable in a comprehensive database?
  5. Conclusion
  6. Acknowledgements
  7. References
  8. Supporting Information
  • Begg CB, Mazumdar M. 1994. Operating characteristics of a rank correlation test for publication bias. Biometrics 50: 10881101.
  • Bond-Lamberty B, Thomson A. 2010. Temperature-associated increases in the global soil respiration record. Nature 464: 579583.
  • Borenstein M. 2005. Software for publication bias. In: Rothstein HR, Sutton AJ, Borenstein M, eds. Publication bias in meta-analysis – prevention, assessment and adjustments. New York, USA: John Wiley & Sons Ltd, 193220.
  • Ceulemans R, Janssens IA, Jach ME. 1999. Effects of CO2 enrichment on trees and forests: lessons to be learned in view of future ecosystem studies. Annals of Botany 84: 577590.
  • Curtis PS, Wang XZ. 1998. A meta-analysis of elevated CO2 effects on woody plant mass, form, and physiology. Oecologia 113: 299313.
  • Dieleman W, Luyssaert S, Rey A, de Angelis P, Barton CVM, Broadmeadow MSJ, Broadmeadow SB, Chigwerewe KS, Crookshanks M, Dufrêne E. 2010. Soil [N] modulates soil C cycling in CO2 fumigated tree stands: a meta-analysis. Plant, Cell & Environment. doi: 10.1111/j.1365-3040.2010.02201.x.
  • Duval S, Tweedie R. 2000. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56: 455463.
  • ECOCRAFT, 1999. Predicted impacts of rising carbon dioxide and temperature on forests in Europe at stand scale. In: Jarvis PG, ed. Final Report of the Environment R&D programme contact EC: ENV4–CT95–0077. Edinburgh, UK: Institute of Ecology and Resource Management, University of Edinburgh.
  • de Graaff MA, van Groenigen KJ, Six J, Hungate Bvan Kessel C. 2006. Interactions between plant growth and soil nutrient cycling under elevated CO2: a meta-analysis. Global Change Biology 12: 20772091.
  • Janssens IA, Dieleman W, Luyssaert S, Subke JA, Reichstein M, Ceulemans R, Ciais P, Dolman AJ, Grace J, Matteucci G et al. 2010. Reduction of forest soil respiration in response to nitrogen deposition. Nature Geoscience 3: 315322.
  • Jarvis PG, Dolman AJ, Schulze ED, Matteucci G, Kowalski AS, Ceulemans R, Rebmann C, Moors EJ, Granier A, Gross P et al. 2001. Carbon balance gradient in European forests: should we doubt ‘surprising’ results? A reply to Piovesan & Adams. Journal of Vegetation Science 12: 145150.
  • Knorr M, Frey SD, Curtis PS. 2005. Nitrogen additions and litter decomposition: a meta-analysis. Ecology 86: 32523257.
  • Le Dantec V, Dufrêne E, Saugier B. 1997. Soil respiration in mini-beech stands under elevated CO2. In: Mohren GMJ, Kramer K, Sabaté S, eds. Impacts of global change on tree physiology and forest ecosystems. Dordrecht, the Netherlands: Kluwer Academic Publishers, 179186.
  • Luo YQ, Hui DF, Zhang DQ. 2006. Elevated CO2 stimulates net accumulations of carbon and nitrogen in land ecosystems: a meta-analysis. Ecology 87: 5363.
  • Medlyn BE, Badeck FW, De Pury DGG, Barton CVM, Broadmeadow M, Ceulemans R, De Angelis P, Forstreuter M, Jach ME, Kellomaki S et al. 1999. Effects of elevated [CO2] on photosynthesis in European forest species: a meta-analysis of model parameters. Plant, Cell & Environment 22: 14751495.
  • Møller AP, Jennions MD. 2001. Testing and adjusting for publication bias. Trends in Ecology and Evolution 16: 580586.
  • Pendall E, Bridgham S, Hanson PJ, Hungate B, Kicklighter DW, Johnson DW, Law BE, Luo YQ, Megonigal JP, Olsrud M et al. 2004. Below-ground process responses to elevated CO2 and temperature: a discussion of observations, measurement methods, and models. New Phytologist 162: 311322.
  • Rosenberg MS, Adams DC, Gurevitch J. 2000. Metawin: statistical software for meta-analysis. Sunderland, MA, USA: Sinauer Associates, Inc.
  • Rosenthal R. 1979. The “File Drawer Problem” and tolerance for null results. Psychological Bulletin 86: 638641.
  • Rustad LE, Campbell JL, Marion GM, Norby RJ, Mitchell MJ, Hartley AE, Cornelissen JHC, Gurevitch J. 2001. A meta-analysis of the response of soil respiration, net nitrogen mineralization, and aboveground plant growth to experimental ecosystem warming. Oecologia 126: 543562.
  • Treseder KK. 2004. A meta-analysis of mycorrhizal responses to nitrogen, phosphorus, and atmospheric CO2 in field studies. New Phytologist 164: 347355.
  • Treseder KK. 2008. Nitrogen additions and microbial biomass: a meta-analysis of ecosystem studies. Ecology Letters 11: 11111120.
  • Zak DR, Pregitzer KS, King JS, Holmes WE. 2000. Elevated atmospheric CO2, fine roots and the response of soil microorganisms: a review and hypothesis. New Phytologist 147: 201222.

Supporting Information

  1. Top of page
  2. Tests for publication bias
  3. Can the publication bias in the ECOCRAFT network be detected?
  4. Is publication bias detectable in a comprehensive database?
  5. Conclusion
  6. Acknowledgements
  7. References
  8. Supporting Information

Table S1 General information about the sites included in the database

Table S2 All experiments included in the database, with information on the database and the publication status at the time of this analysis

Table S3 Source references for the experiments used in the analysis

Please note: Wiley-Blackwell are not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing material) should be directed to the New Phytologist Central Office.

FilenameFormatSizeDescription
NPH_3499_sm_TablesS1-S3.xls57KSupporting info item