Correspondence site: http://www.respond2articles.com/MEE/
Structured elicitation of expert judgments for threatened species assessment: a case study on a continental scale using email
Article first published online: 3 JUL 2012
© 2012 The Authors. Methods in Ecology and Evolution © 2012 British Ecological Society
Methods in Ecology and Evolution
Volume 3, Issue 5, pages 906–920, October 2012
How to Cite
McBride, M. F., Garnett, S. T., Szabo, J. K., Burbidge, A. H., Butchart, S. H. M., Christidis, L., Dutson, G., Ford, H. A., Loyn, R. H., Watson, D. M. and Burgman, M. A. (2012), Structured elicitation of expert judgments for threatened species assessment: a case study on a continental scale using email. Methods in Ecology and Evolution, 3: 906–920. doi: 10.1111/j.2041-210X.2012.00221.x
- Issue published online: 5 OCT 2012
- Article first published online: 3 JUL 2012
- Received 9 August 2011; accepted 17 April 2012 Handling Editor: Daniel Faith
- cognitive bias;
- Delphi process;
- expert elicitation;
- expert panel;
- IUCN Red List;
- subjective knowledge;
- virtual teams;
- 4-point estimation method
1. Expert knowledge is used routinely to inform listing decisions under the IUCN Red List criteria. Differences in opinion arise between experts in the presence of epistemic uncertainty, as a result of different interpretations of incomplete information and differences in individual beliefs, values and experiences. Structured expert elicitation aims to anticipate and account for such differences to increase the accuracy of final estimates.
2. A diverse panel of 16 experts independently evaluated up to 125 parameters per taxon to assess the IUCN Red List category of extinction risk for nine Australian bird taxa. Each panellist was provided with the same baseline data. Additional judgments and advice were sought from taxon specialists outside the panel. One question set elicited lowest and highest plausible estimates, best estimates and probabilities that the true values were contained within the upper and lower bounds. A second question set elicited yes/no answers and a degree of credibility in the answer provided.
3. Once initial estimates were obtained, all panellists were shown each others’ values. They discussed differences and reassessed their original values. Most communication was carried out by email.
4. The process took nearly 6 months overall to complete, and required an average of 1 h and up to 13 h per taxon for a panellist to complete the initial assessment.
5. Panellists were mostly in agreement with one another about IUCN categorisations for each taxon. Where they differed, there was some evidence of convergence in the second round of assessments, although there was persistent non-overlap for about 2% of estimates. The method exposed evidence of common subjective biases including overconfidence, anchoring to available data, definitional ambiguity and the conceptual difficulty of estimating percentages rather than natural numbers.
6. This study demonstrates the value of structured elicitation techniques to identify and to reduce potential sources of bias and error among experts. The formal nature of the process meant that the consensus position reached carried greater weight in subsequent deliberations on status. The structured process is worthwhile for high profile or contentious taxa, but may be too time intensive for less divisive cases.