SEARCH

SEARCH BY CITATION

Keywords:

  • Ecotoxicology;
  • Statistics;
  • NOEC;
  • LOEC;
  • ECx

Abstract

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. WHY THE NOEC AND LOEC ARE STILL USED
  5. THE NEED FOR DEFINITIVE INTERNATIONAL GUIDANCE
  6. CONCLUSIONS
  7. Acknowledgements
  8. REFERENCES

The use of the no observed effect concentration (NOEC) and lowest observed effect concentration (LOEC) in ecotoxicology has been consistently criticized for over 30 years. A search of the literature from the past 30 years found 22 articles challenging the validity and/or appropriateness of NOEC/LOEC data compared to only one in defense of such data. Notwithstanding this compelling weight of evidence, the NOEC and LOEC remain commonly published measures of toxicity from ecotoxicological studies. In this article we argue that the major reason for the continued generation and publication of NOEC/LOEC data is that key government and intergovernmental organizations have been “sitting on the fence” on the issue for more than a decade. Although most key environmental quality guideline, toxicity testing, and associated guidance documents have now recognized the limitations of NOEC/LOEC data, to date no such document or standard toxicity test method has formally ceased recommending or providing guidance on the generation of such data. This is a problem because it is these very guidance documents and test methods that regulatory agencies demand be used by industry for regulatory activities, and on which commercial testing facilities attain and maintain their testing accreditation. Consequently, there will be little impetus for change to statistical analysis practices unless changes to the key guidance documents and test methods necessitate it. Although some progress on this has been made (e.g., in Canada, Australia and New Zealand), there needs to be stronger and universal action to ensure NOEC/LOEC data are no longer generated. Integr Environ Assess Manag 2012; 8: 242–245. © 2012 SETAC


INTRODUCTION

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. WHY THE NOEC AND LOEC ARE STILL USED
  5. THE NEED FOR DEFINITIVE INTERNATIONAL GUIDANCE
  6. CONCLUSIONS
  7. Acknowledgements
  8. REFERENCES

It is normal scientific practice that when a convention is successfully challenged, the next step is for an alternative, superior convention to be agreed on, adopted, and implemented. It is interesting then, that after 3 decades of the no observed effect concentration (NOEC) and lowest observed effect concentration (LOEC) being consistently challenged, a shift to an accepted alternative convention has not yet occurred. The inertia at play here defies logic and reflects poorly on the science of ecotoxicology. In this article, we argue that it is the failure of governments and international organizations to explicitly rule out the use of NOEC/LOEC data from their guidance documents on ecotoxicity testing and analysis, environmental quality guidelines, and/or ecological risk assessment that has perpetuated the inappropriate analysis of ecotoxicity data and prevented a more robust convention from being established. Our argument is not dissimilar to that made very recently by Jager (2012), and we offer our views in the hope that they might add to the momentum to effect this change that is long overdue.

WHY THE NOEC AND LOEC ARE STILL USED

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. WHY THE NOEC AND LOEC ARE STILL USED
  5. THE NEED FOR DEFINITIVE INTERNATIONAL GUIDANCE
  6. CONCLUSIONS
  7. Acknowledgements
  8. REFERENCES

Published scientific opinion and analysis of the limitations of the NOEC/LOEC as estimates of toxicity can be tracked back at least 30 years, to Skalski (1981). In a recent literature search (using Google Scholar and our own archive of relevant references), we found that, since 1981, there have been at least 22 publications critical of the NOEC/LOEC and only 1 defending their use (Table 1). The overall lack of strong rebuttal to the consistent criticisms of the use of NOEC/LOEC data is compelling evidence that the approach is indeed flawed. Interestingly, there are several publications, not included in either of the above categories, which have acknowledged the flaws of NOEC/LOEC data, but continue to defend or allow their use for 1 reason or another (Table 1). Critically, most of these documents are among the most influential in terms of their ability to change the way many ecotoxicologists go about their work. They represent some of the key national and international guidance documents on the conduct of ecotoxicity testing and/or statistical analysis of ecotoxicity data, including guidance from the Organisation for Economic Co-operation and Development (OECD 2006), Australia and New Zealand (ANZECC/ARMCANZ 2000) and Canada (CCME 2007).

Table 1. Number of papers and reports since 1981 that criticize and/or defend the use of the NOEC and LOECa,b
 Only criticizeOnly defendBoth criticize and defend
  • LOEC = lowest observed effect concentration; NOEC = no observed effect concentration.

  • a

    Based on a Google Scholar literature search and the co-authors' existing archive of relevant publications. The literature search did not include 1) published standard toxicity test methods (e.g., those of OECD and USEPA), or 2) publications evaluating alternative methods with only cursory criticism of the use of NOEC/LOEC data.

  • b

    Some of these publications have been cited in the text. There is no need to cite all of them.

Number2214
DetailsMix of published research papers, editorials or opinion pieces, and unpublished reportsbDhaliwal et al. (1997)

de Bruijn and Hof (1997

); ANZECC and ARMCANZ (2000); CCME (2007

); OECD (2006)

The acknowledgment by governments and international organizations of the limitations of NOEC/LOEC data demonstrates they have at least heard the compelling arguments of the scientific community. However, instead of getting off the fence and formally discrediting and even abandoning NOEC/LOEC data for an alternative approach, they have been unmoved for over a decade. Accepting that statistical conventions cannot be replaced overnight, especially when they are bound in legal and regulatory frameworks (see de Bruijn and Hof 1997), it is still inconceivable that 30 years of scrutiny and legitimate criticism has been insufficient for the NOEC/LOEC to be replaced. This is especially so given the efforts by the OECD and the Society for Environmental Toxicology and Chemistry (SETAC) in the 1990s to lay this issue to rest (see Chapman et al. 1996; OECD 1998). Despite these efforts stimulating a large body of published and unpublished work evaluating the NOEC/LOEC and possible alternatives, the “killer blow” has never been landed.

By way of example, a short chronology of the key OECD ecotoxicology statistics guidance documents makes the point in general. OECD (1998) reported on a technical workshop on statistical analysis of aquatic ecotoxicity data held in October 1996. The workshop was founded on several reports from the early 1990s (Pack 1993; Noppert et al. 1994), and concluded that the NOEC was inappropriate for a number of reasons and should, therefore, be phased out and replaced by regression-based estimation techniques. The workshop made a series of additional recommendations, ranging from further research and investigation projects to the establishment of steering groups, to progress this issue. By 2003, a Draft Guidance Document for the Statistical Analysis of Ecotoxicity Data (OECD 2003) appeared. However, by the time the final document was published in 2006 (OECD 2006), it no longer represented a formal OECD Guidance Document, rather it had become an overview of current approaches in the statistical analysis of ecotoxicity data. It is difficult to know what impact this demotion of the report's official status had on its influence in changing how ecotoxicity data are analyzed, but it certainly was not helpful. On the issue of NOEC/LOEC data, OECD (2006) stated the following:

“…There was an ISO resolution (ISO TC147/SC5/WG10 Antalya 3) as well as an OECD workshop recommendation (OECD 1998) that the NOEC should be phased out from international standard. However, the NOEC is still required in many regulatory standards from many countries and in some cases where a detailed determination of an ECx is not relevant and the alteration of the study design is too costly to fulfill the requirements for regression models. Therefore, guidance will be provided on the statistical methods for the determination of the NOEC….”

Thus, the 2006 document acknowledged the problems associated with NOEC/LOEC data (or at least acknowledged the conclusions of OECD [1998]) but failed to draw a line in the sand with respect to their continued generation and use. The justifications for this, i.e., 1) constraints of existing regulatory requirements, 2) determination of the ECx values are not relevant, and 3) high costs of altering experimental designs, are somewhat perplexing. Addressing these each in turn: 1) regulatory requirements can be changed but often will require higher-level authoritative guidance and advice (such as could have been provided by the OECD) to precipitate such a change; 2) it is very difficult to think of a situation where a NOEC/LOEC may be relevant but a detailed determination of a low ECx will not be equally if not more relevant; and 3) adjustment of experimental designs to accommodate regression-based approaches should not be a costly exercise, as additional costs associated with testing more treatment concentrations can be largely offset by the ability to reduce or eliminate within-treatment replication (Moore and Caux 1997). Moreover, costs of altering experimental designs are likely to be insignificant compared to the potential costs associated with inappropriate management decisions based on poor data.

The equivocal guidance on appropriate statistical analysis methods is also reflected in standard toxicity testing protocols (e.g., those by the OECD, US Environmental Protection Agency [USEPA]). These protocols represent the primary test methods that regulatory agencies demand be used by industry for regulatory purposes, and on which commercial testing facilities attain and maintain their testing accreditation. Consequently, there will be little impetus for change “on the ground” unless there is a directive from the relevant national and international organizations that results in the guidance documents and test methods being revised. Indeed, Jager (2012) found that the publication of NOEC data over the past decade has not diminished, which serves to emphasize the need for stronger action on this issue as in addition to just ongoing education, as has been promoted by some (Fox 2012).

It is worth noting at this point, that the push away from statistical significance testing is also occurring in other scientific disciplines (e.g., conservation biology, ecology, economics, psychology) and, in some cases is being met with as much resistance as in ecotoxicology (Fidler et al. 2006; Newman 2008; Warne and van Dam 2008). Landis and Chapman (2011) acknowledged the efforts of the USEPA in promoting concentration-response modeling for human health risk assessments. However, again, relevant authorities appear to have not yet updated the relevant guidance documents or, where the guidance has, or is in the process of, being updated, it does not explicitly recommend ceasing the generation of the lowest observed adverse effect level and no observed adverse effect level.

THE NEED FOR DEFINITIVE INTERNATIONAL GUIDANCE

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. WHY THE NOEC AND LOEC ARE STILL USED
  5. THE NEED FOR DEFINITIVE INTERNATIONAL GUIDANCE
  6. CONCLUSIONS
  7. Acknowledgements
  8. REFERENCES

Some progress in the field of ecotoxicology does seem to have been made. CCME (2007) represents the most recent revision to Canada's national guidance on the derivation of water quality guidelines and probably goes the furthest in terms of discouraging the generation of NOEC/LOEC data. It has placed NOEC data toward the bottom of a list of preferred toxicity estimates and has clearly stated a preference for toxicity estimates derived from regression-based approaches (i.e., ECx). After having acknowledged the limitations of NOEC/LOEC data in the late 1990s (Warne 1998), a similar approach to that of Canada is being adopted by Australia and New Zealand as part of a current revision of the ANZECC/ARMCANZ (2000) water quality guidelines, which will also include updated detailed guidance on the design of experiments for concentration-response modeling (FMWQ 2010). Moreover, it has been recommended that once preferred toxicity estimates (i.e., ECx, no-effect-concentration [NEC]) are available for ≥8 species (from 4 taxonomic groups), all NOEC data will be excluded from the data set (FMWQ 2010). This trend to a stronger position on the non-use of NOEC/LOEC data should be continued by subsequent revisions of other national and international guidance documents. It is disappointing then, that the European Commission's most recent guidance documentation on the derivation of environmental quality standards (EC 2011), completely failed to recognize the limitations of using the NOEC/LOEC data, let alone attempting to address the problem.

We need to finally push governments and international organizations off the fence on which they have been sitting. Perhaps it is past the time for diplomacy and pragmatism. Recently, 2 opinion pieces have called for specific scientific journals to ban the publication of NOEC/LOEC data unless there is sound scientific reasoning otherwise (e.g., for comparison with historical data) and they are accompanied by ECx data (Warne and van Dam 2008; Landis and Chapman 2011). Jager (2012) has supported this. Again, the scientific community is having to take the lead on the issue. However, Jager (2012) suggested the scientific community as a whole is also to blame. He is probably correct to some extent, but at least there is a core group within this community pushing for change. Specific bans by individual journals will raise awareness of the need to effect change, but those alone will not solve the problem. Only when the key guidance documents provide definitive guidance on this issue will we begin to see wholesale change in the analysis of ecotoxicity data. Landis and Chapman (2011) have also recognized this. Further highlighting this, at a recent international conference (Environmental Quality Standards for the Protection of Aquatic Ecosystems [EQSPAE], Hong Kong, 3–7 December 2011) on the derivation of environmental quality guidelines for aquatic ecosystems, it was evident that developing countries embarking on the process of developing procedures for deriving country and/or region specific guidelines are currently generating vast amounts of NOEC data for local species because they are basing their procedures on existing documentation that recommends or allows this (i.e., OECD and USEPA test methods).

As a final point, we have argued previously that criticism of a method must be accompanied by alternative options (Warne and van Dam 2008). The popular alternative to the NOEC/LOEC is the ECx. However, there has also been debate about aspects of the ECx, often accompanied by promotion of alternative approaches such as the NEC (Fox 2008, 2009, 2010) and mechanistic toxicity models that include exposure time as a variable (e.g., Jager 2011). Somewhat perversely, this useful and necessary debate has actually hindered the step away from the NOEC/LOEC. On this, Jager (2012) noted “…even a journey of a thousand miles begins with a single step….” That single step needs to be taken and, thus, we continue to support the shift to a greater focus on concentration-response modeling, with low ECx values being the focal statistical measure of toxicity in the first instance.

There is likely to be little need for additional prescriptive instruction on the appropriate analytical procedures (e.g., appropriate experimental design, model-fitting considerations, reporting requirements including quality assurance and quality control, method, and analytical details) for deriving ECx values than is currently offered by existing guidance documents (OECD 2006). Assuming the use of appropriate experimental designs and analytical methodologies, all that is needed is agreement on an appropriate range of low “x” values constituting a “nil or acceptably low effect” (e.g., 0%–10%). Different (applicable) methods of ECx estimation (whether frequentist or Bayesian) should yield sufficiently similar results as long as the test design is appropriate for the purposes of concentration-response modeling. In parallel to this first step, coordinated research is required on the broad applicability and practicality of new analytical methods (Grist et al. 2006; Billoir et al. 2008; Fox 2010) and toxicity models (Jager et al. 2006; Ashauer and Escher 2010), which is critical to progressing the future applications of statistics in ecotoxicology.

CONCLUSIONS

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. WHY THE NOEC AND LOEC ARE STILL USED
  5. THE NEED FOR DEFINITIVE INTERNATIONAL GUIDANCE
  6. CONCLUSIONS
  7. Acknowledgements
  8. REFERENCES

In conclusion, we reiterate our belief that the major reason for the continued generation and use of NOEC/LOEC data has been the failure of governments and international organizations to formally discredit and cease recommending such approaches, despite compelling scientific argument for this to occur. We agree there are other measures that can and should be taken (e.g., journals banning the publication of NOEC/LOEC data, formal forums for debating statistical issues among, and educating, the ecotoxicology community). However, the main action needs to be universal and taken from the top (although we need to continue to push from the bottom). Significant change will only occur once the key guidance documents and standard toxicity testing protocols provide definitive guidance and no longer provide the option for ecotoxicologists to generate NOEC/LOEC data.

Acknowledgements

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. WHY THE NOEC AND LOEC ARE STILL USED
  5. THE NEED FOR DEFINITIVE INTERNATIONAL GUIDANCE
  6. CONCLUSIONS
  7. Acknowledgements
  8. REFERENCES

The authors would like to thank Peter M. Chapman (Golder Associates), Brian Priestly (Monash University), and Claire Costello (Environmental Research Institute of the Supervising Scientist) for their assistance and advice in the preparation of this manuscript.

REFERENCES

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. WHY THE NOEC AND LOEC ARE STILL USED
  5. THE NEED FOR DEFINITIVE INTERNATIONAL GUIDANCE
  6. CONCLUSIONS
  7. Acknowledgements
  8. REFERENCES
  • [ANZECC/ARMCANZ] Australian and New Zealand Environment Conservation/Agriculture and Resource Management Council of Australia and New Zealand. 2000. Australian and New Zealand guidelines for fresh and marine water quality. National Water Quality Management Strategy Paper No 4. Australian and New Zealand Environment and Conservation Council and Agriculture and Resource Management Council of Australia and New Zealand, Canberra, Australia.
  • Ashauer R, Escher BI. 2010. Advantages of toxicokinetic and toxicodynamic modeling in aquatic ecotoxicology and risk assessment. J Environ Monit 12: 20562061.
  • Billoir E, Delignette-Muller ML, Péry ARR, Charles S. 2008. A Bayesian approach to analyzing ecotoxicological data. Environ Sci Technol 42: 89788984.
  • [CCME] Canadian Council of Ministers of the Environment. 2007. A protocol for the derivation of water quality guidelines for the protection of aquatic life 2007. In: Canadian Environmental Quality Guidelines. Winnipeg, MB, Canada.
  • Chapman PF, Crane M, Wiles J, Noppert F, McIndoe E. 1996. Improving the quality of statistics in regulatory ecotoxicity tests. Ecotoxicology 5: 169186.
  • de Bruijn JHM, Hof M. 1997. How to measure no effect. Part IV: How acceptable is the ECx from an environmental policy point of view? Environmetrics 8: 263267.
  • Dhaliwal BS, Dolan RJ, Batts CW, Kelly JM, Smith RW, Johnson S. 1997. Warning: Replacing NOECs with point estimates may not solve regulatory contradictions. Environ Toxicol Chem 16: 124125.
  • [EC] European Communities. 2011. Technical guidance for deriving environmental quality standards. Guidance Document No. 27, Common Implementation Strategy for the Water Framework Directive (2000/60/EC), Technical Report-2011-055, European Commission, Brussels, Belgium.
  • Fidler F, Burgman MA, Cumming G, Buttrose R, Thomason N. 2006. Impact of criticism of null-hypothesis significance testing on statistical reporting practices in conservation biology. Conserv Biol 20: 15391544.
  • [FMWQ] Fresh and Marine Water Quality Working Group 4. 2010. Draft FMWQ Working Group 4 Workshop Minutes. Fresh and Marine Water Quality Working Group 4—Toxicants and sediments, 14–16 April 2010. Unpublished report from Phase I of the revision of the Australian and New Zealand Guidelines for Fresh and Marine Water Quality, Lucas Heights, NSW, Australia.
  • Fox DR. 2008. NECs, NOECs and the ECx. Australas J Ecotox 14: 79.
  • Fox DR. 2009. Is the ECx a legitimate surrogate for a NOEC? Integr Environ Assess Manag 5: 351353.
  • Fox DR. 2010. A Bayesian approach for determining the no effect concentration and hazardous concentration in ecotoxicology. Ecotoxicol Environ Saf 73: 123131.
  • Fox DR. 2012. Response to Landis and Chapman (2011) Integr Environ Assess Manag 8: 4.
  • Grist EPM, O'Hagan A, Crane M, Sorokin N, Sims I, Whitehouse P. 2006. Bayesian and time-independent species sensitivity distributions for risk assessment of chemicals. Environ Sci Technol 40: 395401.
  • Jager T. 2011. Some good reasons to ban ECx and related concepts in ecotoxicology. Environ Sci Technol 45: 81808181.
  • Jager T. 2012. Bad habits die hard: The NOEC's persistence reflects poorly on ecotoxicology. Ecotox Toxicol Chem 31: 228229.
  • Jager T, Heugens EHW, Kooijman SALM. 2006. Making sense of ecotoxicological test results: Towards application of process-based models. Ecotoxicology 15: 305314.
  • Landis WG, Chapman PM. 2011. Well past time to stop using NOELs and LOELs. Integr Environ Assess Manag 7: viviii.
  • Moore DRJ, Caux PY. 1997. Estimating low toxic effects. Environ Toxicol Chem 16: 794801.
  • Newman MC. 2008. What exactly are you inferring? A closer look at hypothesis testing. Environ Toxicol Chem 27: 10131019.
  • Noppert E, Leopold A, van der Hoeven N. 1994. How to measure no effect: Towards a measure of chronic toxicity. Unpublished workshop report. Delft, the Netherlands: BKH Consulting Engineers.
  • [OECD] Organisation for Economic Co-operation and Development. 1998. Report of the OECD Workshop on statistical analysis of aquatic toxicity data. OECD Environmental Health and Safety Publication, Series on Testing and Assessment, No. 10, Environment Directorate, ENV/MC/CHEM(98)18, Paris, France.
  • [OECD] Organisation for Economic Co-operation and Development. 2003. Draft guidance document for the statistical analysis of ecotoxicity data. OECD Environmental Health and Safety Publication, Series on Testing and Assessment, Environment Directorate, Paris, France.
  • [OECD] Organisation for Economic Co-operation and Development. 2006. Current approaches in the statistical analysis of ecotoxicity data: A guide to application. OECD Environmental Health and Safety Publication, Series on Testing and Assessment, No. 54, Environment Directorate, ENV/JM/MONO (2006) 18, Paris, France.
  • Pack S. 1993. A review of statistical data analysis and experimental design in OECD aquatic toxicology test guidelines. Report to the Organisation for Economic Cooperation and Development, Paris, France.
  • Skalski JR. 1981. Statistical inconsistencies in the use of no-observed-effect-levels in toxicity testing. In: Branson DR, Dickson KL, editors. Aquatic Toxicology and Hazard Assessment: Fourth Conference ASTM STP 737. Philadelphia (PA): American Society for Testing and Materials. pp. 377387.
  • Warne MStJ. 1998. Critical review of methods to derive water quality guidelines for toxicants and a proposal for a new framework. Supervising Scientist Report 135, Supervising Scientist. Canberra (AU). ISBN 0 642 24338 7.
  • Warne MStJ, van Dam R. 2008. NOEC and LOEC data should no longer be generated or used. Australas J Ecotox 14: 15.