Explicit consideration of critical effect sizes and costs of errors can improve decision-making in plant science
Article first published online: 8 JUL 2013
© 2013 The Author. New Phytologist © 2013 New Phytologist Trust
Volume 199, Issue 4, pages 876–878, September 2013
How to Cite
Mudge, J. F. (2013), Explicit consideration of critical effect sizes and costs of errors can improve decision-making in plant science. New Phytologist, 199: 876–878. doi: 10.1111/nph.12410
- Issue published online: 2 AUG 2013
- Article first published online: 8 JUL 2013
- costs of errors;
- critical effect size;
- null hypothesis significance test;
- policy decisions;
- significance threshold;
- Type I error;
- Type II error
Ensuring that sound science informs policy decisions has been suggested as one of the most important societal issues faced by plant scientists in the twenty-first century (Grierson et al., 2011). Plant scientists therefore are responsible for ensuring that data are analyzed and presented in a way that facilitates good decision-making. Good statistical practices can be an important tool for ensuring objective and transparent and data analysis. Despite frequent criticism over the last few decades (Cohen, 1994; Gigerenzer, 2004; Rinella & James, 2010), null hypothesis significance testing (NHST) remains widely used in plant science. While others have argued that the persistence of NHST is due to ignorance of alternative approaches or resistance to change (Fidler et al., 2004; Gigerenzer et al., 2004), its persistence is likely at least partly due to its usefulness as a decision-making tool (Robinson & Wainer, 2002; Mogie, 2004; Stephens et al., 2007). Are different plant species equivalent in their ability to remediate toxic land? Are different management actions equivalent in their ability to control invasive plant species? Are there differences in antibiotic activity among different plant metabolites? Do more diverse plant communities provide greater levels of a particular ecosystem service than less diverse plant communities? Each of these questions could be appropriately answered using null hypothesis significance tests. Other tests might also be appropriate, but it would not be wrong to use NHST. The utility of NHST as a decision-making tool does not, however, warrant ignoring the problems that are associated with these statistical tests. The problems associated with NHST need to be addressed in order for it to better inform policy decisions that involve plant science.
The major flaws of NHST primarily surround the use of an arbitrary threshold for judging statistical significance. Although the use of α = 0.05 does represent a common criterion for Type I errors that everyone must adhere to, consistent use of this threshold results in Type II error rates that vary wildly among studies. The value of 0.05 as a significance criterion has no logical foundation, nor does the practice of holding Type I errors consistent while allowing Type II errors to vary (Cowles & Davis, 1982). Consistent use of an arbitrary significance threshold also causes statistical significance to frequently differ from biological significance (Martínez-Abraín, 2008) and makes significance heavily influenced by sample size (Johnson, 1999), with high sample sizes tending to make even trivial effects statistically significant (Nakagawa & Cuthill, 2007) and low sample sizes leading to even large effects being considered non-significant (Sedlmeier & Gigerenzer, 1989). Use of a consistent significance threshold regardless of sample size has contributed to frequent misinterpretations of P-values as being ‘highly significant’ or ‘marginally significant’, and/or as measures of how likely the alternate hypothesis is to be true (Hubbard & Bayarri, 2003).
If using a consistent arbitrary significance level is problematic, how should significance levels be set? I argue that the significance threshold for a null hypothesis significance test should be set to achieve the goal of the statistical test. One (and perhaps the only) reasonable goal of NHST is to minimize the chances and/or costs of making wrongful conclusions concerning a set of collected data. If the goal of NHST is to minimize the chances and/or costs of errors, then the decision-making threshold (α) should be set to minimize the combined probabilities and/or combined costs of Type I and Type II errors. Mudge et al. (2012a) describe a general approach for calculating study-specific optimal significance levels that minimize the combined probabilities and/or costs of Type I errors under the null hypothesis and Type II errors under the alternate hypothesis. The optimal α approach for null hypothesis significance tests is paralleled by signal detection theory in electrical engineering (Peterson et al., 1954) and psychology (Green & Swets, 1966) and has recently been applied in environmental monitoring (Mudge et al., 2012b) and physiology (Mudge et al., 2012c).
The calculation of an optimal significance level requires estimates of the same parameters that are needed to calculate statistical power for a null hypothesis significance test (sample size and a critical effect size relative to variability) and also requires an estimate of the relative costs of Type I vs Type II error, to minimize the combined costs of Type I and Type II error (the relative prior probabilities of null and alternate hypotheses, if known, can also be incorporated into the calculation of an optimal significance level, however studies with prior probability estimates typically employ Bayesian statistical methods). Although the need to explicitly consider and specify a critical effect size relative to variability and the relative costs of Type I vs Type II error may appear to constitute a barrier to calculating optimal significance levels in plant science, it should not be viewed as such. Critical effect sizes, costs of errors and prior probabilities of null and alternate hypotheses are important to consider for any research question, and failure to incorporate them into the statistical decision-making threshold leads to implicit and unexamined assumptions about them when using α = 0.05.
A critical effect size is the smallest size of effect that would be considered biologically meaningful if it were to exist. What is the minimum meaningful magnitude of antibiotic activity for a plant metabolite? How much variability among different management actions is enough to warrant a change in management policy for controlling invasive plant species? What is a biologically meaningful strength of relationship between plant species diversity and levels of particular ecosystem services? Currently in plant science, consideration of biological relevance is often discussed after the presentation of results, creating pressure to justify statistical significance as being biologically relevant (i.e. arguments that statistically significant results are meaningful and arguments that nonsignificant results are not of a meaningful observed effect size). Using α = 0.05, it is common to have statistical significance without biological significance and vice versa. Setting critical effect sizes a priori and incorporating them into the statistical decision-making process removes a component of subjectivity in results interpretation (the post hoc evaluation of biological relevance), and the subjectivity that remains (the selection of the critical effect size itself) is at least transparently stated. There may also be some objective methods for setting critical effect sizes in plant science. Previous studies can sometimes be helpful when setting a critical effect size for new study. The magnitude of antibiotic activity of compounds used in current drugs may inform the level that would be considered meaningful for a newly studied plant metabolite. Setting critical effect sizes such that they fall outside some specified range of natural variability may also be appropriate. The amount of variability in effectiveness of any single management action for controlling invasive plant species may inform the amount of variability tolerated among different management approaches. When there is no strong justification for any single critical effect size, significance thresholds can be calculated for multiple potential effect sizes representing, small, intermediate and large effects, enabling the reader to evaluate results based on the effect size that they feel is most appropriate. Ultimately, although specifying a critical effect size for a research question may be a difficult task, the authors themselves (as experts in their study subject) are nearly always among the most qualified to make this important estimate and responsibility should fall on them for doing so. Presenting data but shifting the entire burden of judging biological significance onto the reader in an attempt to circumvent the possibility of criticism discourages discussion about biological relevance among plant scientists.
Regardless of research question, statistical decision errors in plant science research always have associated costs. These costs of errors may be financial and/or any combination of other nonmonetary costs such as wasted time, loss of scientific progress, or damage to the environment. Some costs, such as wasted time, can be more easily translated into monetary terms than others, such as loss of scientific progress or environmental damage. Despite their frequent difficulties with quantification, the consideration of costs of errors is inherent in good decision-making, and plant science is no exception. Accepting a greater probability of a more costly type of error when it is possible to shift some of this probability to a less costly type of error is only rational if a small reduction of the more costly error requires accepting a sufficiently large increase in the likelihood of the less costly error such that the increased chance of error offsets the gain associated with avoiding more costly errors. There have been many attempts to quantify costs of environmental degradation (see Costanza et al., 1997 for a review of > 100 attempts to quantify the value of ecosystem goods and services). For ecosystem management, costs of errors represent potential degradation of ecosystem value vs potential unnecessary management costs and potential lost economic opportunities. For basic plant science research, costs of errors are more difficult to quantify because implications of Type I and Type II errors are often not easily foreseen. Plant scientists are not, however, totally unaccustomed to considering the relative costs of errors, as judgments concerning relative costs of errors are often made when ‘conservative’ estimates are made for particular unknown parameters. Choosing to make a ‘conservative’ estimate implies that any errors associated with underestimation are less serious than errors associated with overestimation. For cases where costs of errors are truly unknown, setting equal costs of errors assumes any errors are equally serious and results allows for the calculation of an optimal significance level that minimizes the combined probabilities of Type I and Type II error. Setting relative costs of Type I vs Type II error equal may, therefore, often be the best practice for purely scientific questions.
While determining an optimal decision-making threshold separately for each study through explicit consideration of biological relevance and relative costs of Type I vs Type II errors is a good thing, pressures to publish and self-delusion may sometimes tempt researchers to choose an inappropriate critical effect size or relative error cost ratio to produce a desired statistical outcome. However, the practice of using α = 0.05 is also not immune to the possibility of manipulating of statistical significance. When using α = 0.05, one can virtually guarantee statistical significance by taking many samples and focusing on endpoints with low variability. Similarly, nonsignificance can be nearly assured by taking few samples and introducing measurement error to increase variability. Use of α = 0.05 only creates the illusion of consistent statistical rigor among studies while hiding potential manipulation of significance behind combinations of sample size and endpoint variability. By contrast, it is more difficult to disguise the manipulation of significance when using the optimal α approach, for two reasons. First, authors would have to explicitly state and provide a rationale for their critical effect size and relative error cost estimate, and unusual critical effect sizes or error cost estimates would (hopefully) raise eyebrows of both reviewers and readers. Second, researchers would essentially ‘shoot themselves in the foot’ by attempting to increase the chance of a significant conclusion by claiming inappropriately small effect sizes would be important to be able to detect, because it would result in a higher α level, which would weaken the confidence in the statistical outcome. Also, if sample sizes and P-values are stated in a paper where a reader disagrees with the authors’ choice of critical effect size or relative error cost estimate, there is nothing preventing the reader from calculating another optimal significance level for their own critical effect size and/or error cost estimate and re-evaluating the significance accordingly.
Null hypothesis significance tests remain commonly used as a statistical decision-making technique in plant science, but consistently relying on the traditional α = 0.05 significance level results in unnecessarily high combinations of Type I and biologically relevant Type II errors and results in conclusions not tied to biological relevance or the relative costs of Type I vs Type II errors. Critical effect sizes and costs of errors are imperative to making good decisions in plant science and there is no rationale for continuing to use a consistent but arbitrary statistical decision-making threshold in plant science when an alternative exists that incorporates critical effect sizes and relative error costs to set study-specific significance thresholds that minimize the combined probabilities or costs of Type I and biologically relevant Type II errors. I believe that incorporation of explicitly considered and transparently stated estimates of critical effect sizes and relative costs of errors into statistical decision-making thresholds will improve the relevance of plant science research for policy-makers in the twenty-first century.
- 1994. The earth is round (p < .05). American Psychologist 49: 997–1003. .
- 1997. The value of the world's ecosystem services and natural capital. Nature 387: 253–260. , , , , , , , , , et al.
- 1982. On the origins of the .05 level of statistical significance. American Psychologist 37: 553–558. , .
- 2004. Statistical reform in medicine, psychology and ecology. The Journal of Socio-Economics 33: 615–630. , , , .
- 2004. Mindless statistics. The Journal of Socio-Economics 33: 587–606. .
- 2004. The null ritual: what you always wanted to know about significance testing but were afraid to ask. In: Kaplan D, ed. The Sage handbook of quantitative methodology for the social sciences. Thousand Oaks, CA, USA: Sage Publications, 391–408. , , .
- 1966. Signal detection theory and psychophysics. New York, USA: John Wiley & Sons. , .
- 2011. One hundred important questions facing plant science research. New Phytologist 192: 6–12. , , , , , , , , , et al.
- 2003. Confusion over measures of evidence (p's) versus errors (α's) in classical statistical testing. The American Statistician 57: 171–182. , .
- 1999. The insignificance of statistical significance testing. The Journal of Wildlife Management 63: 763–772. .
- 2008. Statistical significance and biological relevance: a call for a more cautious interpretation of results in ecology. Acta Oecologica 34: 9–11. .
- 2004. In support of null hypothesis significance testing. Proceedings of the Royal Society of London, Series B: Biological Sciences 271: S82–S84. .
- 2012a. Setting an optimal α that minimizes errors in null hypothesis significance tests. PLoS ONE 7: e32734. , , , .
- 2012b. Negative consequences of using α = 0.05 for environmental monitoring decisions: a case study from a decade of Canada's Environmental Effects Monitoring Program. Environmental Science and Technology 46: 9249–9255. , , , .
- 2012c. Optimizing α for better statistical decisions: a case study involving the pace-of-life syndrome hypothesis. BioEssays 12: 1045–1049. , , .
- 2007. Effect size, confidence interval and statistical significance: a practical guide for biologists. Biological Reviews 82: 591–605. , .
- 1954. The theory of signal detectability. Proceedings of the IRE Professional Group on Information Theory 4: 171–212. , , .
- 2010. Invasive plant researchers should calculate effect sizes, not p-values. Invasive Plant Science and Management 3: 106–112. , .
- 2002. On the past and future of null hypothesis significance testing. The Journal of Wildlife Management 66: 263–271. , .
- 1989. Do studies of statistical power have an effect on the power of studies? Psychological Bulletin 105: 309–316. , .
- 2007. Inference in ecology and evolution. Trends in Ecology and Evolution 22: 192–197. , , .