How useful is expert opinion for predicting the distribution of a species within and beyond the region of expertise? A case study using brush-tailed rock-wallabies Petrogale penicillata

Authors

  • Justine V. Murray,

    Corresponding author
    1. The Ecology Centre, School of Biological Sciences, The University of Queensland, Brisbane, Qld 4072, Australia
      *Correspondence author. E-mail: justine.murray@uqconnect.edu.au
    Search for more papers by this author
  • Anne W. Goldizen,

    1. The Ecology Centre, School of Biological Sciences, The University of Queensland, Brisbane, Qld 4072, Australia
    Search for more papers by this author
  • Rebecca A. O’Leary,

    1. School of Mathematical Sciences, Queensland University of Technology, Brisbane, Qld 4000, Australia
    Search for more papers by this author
  • Clive A. McAlpine,

    1. The Ecology Centre, School of Biological Sciences, The University of Queensland, Brisbane, Qld 4072, Australia
    2. School of Geographical Sciences, Planning and Environmental Management, The University of Queensland, Brisbane, Qld 4072, Australia
    Search for more papers by this author
  • Hugh P. Possingham,

    1. The Ecology Centre, School of Biological Sciences, The University of Queensland, Brisbane, Qld 4072, Australia
    Search for more papers by this author
  • Samantha Low Choy

    1. School of Mathematical Sciences, Queensland University of Technology, Brisbane, Qld 4000, Australia
    Search for more papers by this author

*Correspondence author. E-mail: justine.murray@uqconnect.edu.au

Summary

1. Species’ distribution modelling relies on adequate data sets to build reliable statistical models with high predictive ability. However, the money spent collecting empirical data might be better spent on management. A less expensive source of species’ distribution information is expert opinion. This study evaluates expert knowledge and its source. In particular, we determine whether models built on expert knowledge apply over multiple regions or only within the region where the knowledge was derived.

2. The case study focuses on the distribution of the brush-tailed rock-wallaby Petrogale penicillata in eastern Australia. We brought together from two biogeographically different regions substantial and well-designed field data and knowledge from nine experts. We used a novel elicitation tool within a geographical information system to systematically collect expert opinions. The tool utilized an indirect approach to elicitation, asking experts simpler questions about observable rather than abstract quantities, with measures in place to identify uncertainty and offer feedback. Bayesian analysis was used to combine field data and expert knowledge in each region to determine: (i) how expert opinion affected models based on field data and (ii) how similar expert-informed models were within regions and across regions.

3. The elicitation tool effectively captured the experts’ opinions and their uncertainties. Experts were comfortable with the map-based elicitation approach used, especially with graphical feedback. Experts tended to predict lower values of species occurrence compared with field data.

4. Across experts, consensus on effect sizes occurred for several habitat variables. Expert opinion generally influenced predictions from field data. However, south-east Queensland and north-east New South Wales experts had different opinions on the influence of elevation and geology, with these differences attributable to geological differences between these regions.

5.Synthesis and applications. When formulated as priors in Bayesian analysis, expert opinion is useful for modifying or strengthening patterns exhibited by empirical data sets that are limited in size or scope. Nevertheless, the ability of an expert to extrapolate beyond their region of knowledge may be poor. Hence there is significant merit in obtaining information from local experts when compiling species’ distribution models across several regions.

Introduction

Management of threatened species relies on knowledge of their distributions and abundances. This has led to an explosion of techniques for modelling species’ distributions (Guisan & Zimmermann 2000; Ferrier et al. 2002). The cost of collecting field data, in terms of time, expense and necessary resources, can be large and may substantially reduce the budget available for management (Field et al. 2004; Seoane, Bustamante & Diaz-Delgado 2005). While objective confirmation through appropriately designed field sampling is desirable, it is not always possible within the time and budget constraints of a project. Where there is insufficient field data, an efficient source of less expensive information can be expert knowledge gained from extensive experience. Expert knowledge has been increasingly incorporated into management recommendations and practices in a wide variety of fields (Yamada et al. 2003; Martin et al. 2005; Seoane et al. 2005).

Expert knowledge can be incorporated into species’ distribution models using a Bayesian statistical framework (Low Choy, O’Leary & Mengersen 2009). Information from experienced experts, including their uncertainty, can be quantified to construct a ‘prior’ probability distribution for model parameters (e.g. regression coefficients). Using Bayes’ theorem we can update prior estimates of model parameters using other data to form ‘posterior’ estimates (Ellison 2004; O’Hagan et al. 2006). Prior estimates have more effect when data are limited and highly variable. Priors reflecting vague or no previous knowledge are ‘non-informative’ and can prove problematic with limited data. ‘Informative priors’, based on elicited expert knowledge, can also be used so that results utilize all available relevant species information (Morgan & Hemrion 1990; Ellison, Gregory & Hardcastle 1998; Ellison 2004).

Expert opinion has successfully been used as a priori information for developing ecological models (Yamada et al. 2003; Martin et al. 2005; O’Leary et al. 2008). McCarthy & Masters (2005) found combining expert opinion and data avoided the need for 2 years of additional sampling. In contrast, Pearce et al. (2001) and Seoane et al. (2005) found little improvement in the predictive power of models when expert opinion was used at different stages of model-building within a classical statistical approach, including modifying species’ distribution models a posteriori. Although many studies demonstrate valuable contributions by expert opinion to data analysis, only a few have discussed the sourcing of suitable experts for informing ecological models or compared expert opinions from different regions (O’Neill et al. 2008; Hurley, Rapaport & Johnson 2009). In particular, we pose the questions: (i) How does combining expert opinion with empirical data improve species’ distribution models? (ii) Does the source of expertise make a difference to expert assessments?

We addressed these questions by modelling the distribution of a nationally threatened marsupial from eastern Australia, the brush-tailed rock-wallaby Petrogale penicillata Gray 1827. The species is declining and is listed as threatened across its range. Hence, there is an urgency to implement management actions to stop and reverse the decline, utilizing all information currently available. Managers need to identify potential habitat to protect the species from common threats, such as introduced predators and wildfires. Brush-tailed rock-wallabies live in small colonies in naturally fragmented habitat in rugged terrain incorporating cliffs or boulder piles. A dominant male usually defends a number of females that occupy suitable rock refuges during the day to escape from extremes of weather and potential threats (Short 1982; Jarman & Bayne 1997; Murray et al. 2008). Collecting adequate field data in such challenging terrain is costly and time-consuming. Expert knowledge provides an alternative source of data for this species to supplement traditional field sampling, which is limited due to time and monetary constraints.

In this study, we further the work performed by O’Neill et al. (2008) in assessing uncertainties in expert opinions. We use an extensive field data set and a number of habitat variables to compare species’ distribution predictions of brush-tailed rock-wallaby distributions derived from field data alone together with predictions derived from Bayesian models using the same field data combined with expert knowledge obtained over the same area and quantified as priors. We used a newly designed elicitation tool with readily available geographical information system (GIS) data to interview multiple experts from two regions and evaluate how their assessments affect predictions in the adjoining less familiar region. Furthermore, we offer suggestions for conservation of the brush-tailed rock-wallaby and for choosing experts best suited to assessment in the region of interest.

Materials and Methods

Conceptual model

A Bayesian statistical modelling framework was chosen to enable the combination of expert knowledge formulated as priors with observed data. We addressed our first question by fitting three models: (i) experts only (informative priors); (ii) field data only (posteriors with non-informative priors); and (iii) combined expert and field data (posteriors with informative priors). We addressed our second question by considering four scenarios: Region I experts or Region II experts combined with Region I field data, and Region I experts or Region II experts with Region II field data.

Study area

The study area was located on the Great Dividing Range in eastern Australia (Fig. S1, Supporting Information), and separated into two adjoining regions, south-east Queensland (Region I) and north-east New South Wales (NSW; Region II). These regions span five biogeographical regions supporting a diversity of landforms and ecosystems ranging from steep narrow ridgelines and plateau-topped mountains to river valleys and gorges.

Remnant vegetation ranges from subtropical dry rainforest on steep sheltered slopes and gullies, to dry sclerophyll forests and woodlands on open slopes and flat areas (Young & Dillewaard 1999). The agricultural landscapes west of the escarpment are extensively cleared apart from small fragments of native vegetation. The geology of south-east Queensland is dominated by metamorphic and sedimentary rock (45%), intrusive igneous rock (8%), and volcanic rock (30%). In north-east NSW, the geology comprises sedimentary and metamorphic rock (54%), intrusive igneous rock (18%) and volcanic rock (20%). Remaining geological areas, including alluviums and colluviums, were classed as non-habitat for the brush-tailed rock-wallaby and excluded from analysis.

Sampling design

Diurnal habitat of the brush-tailed rock-wallaby was sampled using presence/absence surveys within the two study regions. We sampled 200 sites in south-east Queensland (113 presences, 87 absences) and 266 sites in north-east NSW (163 presences, 103 absences; Fig. 1) between September 2004 and August 2006, a period with below average rainfall and moderate (north-east NSW) to severe drought (south-east Queensland) conditions. Sites included locations known to contain rock-wallabies, supplemented by additional sites selected using a stratified random sampling design (Murray et al. 2008). Sites were located a minimum of 300 m apart based on previous studies on diurnal colony separation (median distance 308 m) and nocturnal home range size (<3 ha) of individual animals (Jarman & Bayne 1997; Laws & Goldizen 2003). Each site represented up to hundreds of metres along linear rock-face features. At each site, we assessed presence/absence of rock-wallabies and collected habitat data. Presence required sightings of individuals or fresh faecal pellets.

Figure 1.

 Elicitation tool interface for expert assessment at each site. The interactive boxplot and probability density curve reflect assessed probability of presence at a site. Text boxes allow input and reflection of: upper and lower bounds, quartiles, mode and overall confidence in assessment.

To obtain landscape-scale data for this analysis, we applied a buffer of 1000 m for each site. If buffered sites overlapped, they were merged and called site-clusters. If part of a site-cluster was occupied, the whole site-cluster was recorded as occupied at the landscape-scale because 1000 m is within the dispersal range of rock-wallabies (Hazlitt, Eldridge & Goldizen 2004). This provided an empirical data set consisting of 229 site-clusters extending up to several kilometres along linear features. Rock-wallabies were present at 59 of 108 site-clusters in south-east Queensland, and 56 of 121 clusters in north-east NSW. Landscape information was acquired from GIS layers for the site-clusters for each region (Table 1). Field sampling methods, site and landscape predictor variables relevant to both regions and preliminary analysis of south-east Queensland data are detailed in Murray et al. (2008).

Table 1.   Explanatory variables used to predict the presence–absence of brush-tailed rock-wallabies at the landscape-scale
VariablesDescription of variable (all based on digital layers)
Geology0, non-habitat; 1, volcanic; 2, intrusive igneous; 3, sedimentary/metamorphic
Elevation (m)50 m digital elevation model (DEM). Areas with elevation <150 m were omitted
Slope (°)Derived from 50 m DEM (above). Slopes <10° were omitted
Landcover0, non-habitat; 1, forest/regrowth; 2, Pasture or cleared land
Remnant vegetation1, woodland or open forest with a grassy or shrubby understorey; 2, rainforest, vine-thicket or wet sclerophyll forest with or without a rainforest understorey; 3, grassland or cleared areas not classified as non-habitat

Experts

Nine experts were chosen from the limited pool of 20 experts with knowledge of brush-tailed rock-wallaby ecology and habitat in either region. All experts had a conservation management background. Following the definition of O’Hagan et al. (2006), we deemed an ‘expert’ to be a person with sound knowledge of the brush-tailed rock-wallaby’s habitat, defined by them exploring at least 10 different occupied sites. Information collected at the beginning of the elicitation process provided an understanding of the source of each expert’s knowledge of the species, statistics and GIS (Appendix S1, Supporting Information). There were four experts from Region I and five from Region II. Seven experts had in-depth local knowledge of specific areas within one region. Four experts had greater than 10 years of experience with rock-wallabies and only one had less than 5 years of experience. Their expertise was gained mainly from undertaking fieldwork or as a supervisor of research, but was also gained from the literature by six experts and by developing a recovery programme for the species by two experts. Statistical knowledge of each expert was limited to undergraduate level or beginner courses with very little recent exposure. Five experts were GIS-proficient and regular users whilst others were ad hoc users with no formal training. The interviewer (J.M.) was the same for all experts and had extensive knowledge of both regions.

Elicitation process

This study motivated development, and is the first application, of a new software tool to elicit expert knowledge, based on the prototype of Denham & Mengersen (2007), but with substantial modifications to the underlying elicitation model and encoding. Instead of directly asking experts questions about regression coefficients which can be difficult to conceptualize (Low Choy et al. 2009), we applied an indirect elicitation method, asking easier questions on assessing the probability of presence at sites with known habitat attributes. The tool performs the complex calculations required to translate this expert knowledge into prior distributions for regression coefficients of each habitat factor with output suitable for Bayesian analysis using win bugs (Imperial College & Medical Research Council 2003).

The tool was built to accompany a GIS interface, allowing experts to manipulate spatial layers representing habitat factors relevant to the target area and species. Each expert was given a short tutorial on the elicitation tool. Most experts gained their knowledge from site-scale observations. To ensure experts were confident with landscape-scale influences, we showed them photographs of different landforms, such as undulating land and steep escarpments, together with the corresponding GIS layers.

Experts were individually asked to assess the probability of presence at a set of elicitation sites; habitat factors were listed in the tool and mapped in the accompanying GIS. Twenty sites were previously selected by the interviewer, using random sampling stratified by vegetation and geology within a small area of one region. All experts were unfamiliar with this area. Elicitation was performed for the same sites, but in different randomized orders for each expert. Thus opinions of experts could be compared within and between regions, site selection bias was minimized and experts were not pressured to choose sites.

In the tool, when experts clicked on a designated site, a pop-up window showed different graphics to help elicit their opinion on plausible values for probability of presence at that site (Fig. 1). Experts were asked to specify the outer limits of this probability (95% credibility interval), then an interval with a 50–50 chance of containing this probability of presence (50% credibility interval), then their best estimate (mode; Appendix S2). This sequence avoids anchoring bias (Kynn 2008). The mode is easier to elicit and more useful for encoding a skewed quantity such as probability (Low Choy, Mengersen & Rousseau 2008). The tool also records expert confidence in estimates (Fig. S2). The expert could record this distribution by manipulating a Beta density function, a boxplot or else numerically (Fig. 1). The expert could interact with these plots by moving key features (such as lines representing the mode or intervals), with changes reflected instantaneously in plots and text. This information effectively provided an expert ‘data set’ of estimated quantiles for the probability of presence at various sites with known covariates. Fitting a logistic regression using the expert data provided prior estimates and standard errors, defining the prior distribution, for habitat model coefficients.

The elicitation tool provided several forms of feedback on the regression fit to expert data (Denham & Mengersen 2007). Univariate species response graphs reflected patterns in an expert’s assessments across sites describing the impact of each habitat factor on the probability of species occurrence. These impacts were summarized using boxplots for categorical habitat factors, such as geology, and curves for continuous habitat factors, such as slope. Experts could click on sites on the species response curves to revisit or change assessments for those sites.

Statistical analysis

Priors obtained from each expert were combined within their region using unweighted moment averaging (Martin et al. 2005). Informative priors obtained from the tool defined the expert-only models in each region (Model 1). Non-informative priors were assigned Gaussian distributions with zero mean (favouring neither positive nor negative impacts) and wide variance (0·1–0·01) then combined with field data from Region I or II (Model 2). Informative priors based on expert opinions from either region were combined with field data from either region to produce posterior estimates (Model 3). Bayesian logistic regression was used to fit posterior Models 2 and 3 using WinBUGS 1.4, via Markov chain Monte Carlo (MCMC) simulation. For each model, MCMC chains comprising 100 000 iterations with a burn-in of 1000 were found sufficient to achieve convergence. MCMC diagnostics were calculated using the ‘CODA’ library (Plummer et al. 2007) in the R 2.4.0 statistical package (R Development Core Team 2006).

Standard convergence diagnostics (Raftery & Lewis 1992) helped ensure sufficient burn-in, thinning and iterations. Thinning by 10 saved space and reduced dependence within chains. In addition, the Heidelberger and Welch diagnostic determined if chain means had stabilized (Plummer et al. 2007).

Models 1–3 were assessed by inspecting the prior and posterior distributions, including means and variances, of regression coefficients. The Deviance Information Criterion (DIC) is the standard Bayesian measure of goodness-of-fit to data. By definition any model with an informative prior will have a lower DIC than the same model with a non-informative prior (Spiegelhalter et al. 2002). Hence, although inappropriate for a direct comparison of non-informative and informative posterior models fit to the same empirical data set, the DIC reflects how empirical data has updated expert-defined priors. Further predictive model performance was tested using predictive performance checks, and misclassification error rates for false positives and false negatives within threshold plots and receiver operating characteristic (ROC) curves. These predictive test methods are described in the Supporting Information.

Results

Elicitation results

The elicitation process took 2–6 h, depending on the expert’s familiarity with GIS. All experts chose to manipulate the interactive boxplot, gaining familiarity within three elicitations. Experts were not required to understand the statistical computations embedded in the tool and easily understood the questions asked. They used the feedback provided by species response curves (Fig. 3) to confirm whether their assessments reflected their understanding of the relationship between species occurrence and specific habitat factors.

Figure 3.

 Example of varying expert opinion. Plots show mean (logit) responses (with ±1 SE) for each expert on effects of (a) closed forest and cleared land; and (b) intrusive igneous and sedimentary/metamorphic rock. The line at zero indicates the type of habitat is no different to the baseline, open forest for (a) and volcanic rock for (b), in its effect on presence of the species. Bars above the line indicate an increased effect on the species, compared to the baseline, whilst bars below indicate a decrease.

Based on priors averaged across experts within each region (Table S1, Supporting Information; Fig. S3), experts estimated the overall probability of presence to be c. 26% (95% CI 3–5% to 69–75%) in baseline habitat, which was volcanic rock with open forest on a minimum 10° slope and 150 m elevation. Experts from both regions unanimously agreed that closed forest slightly lowered (70–81% chance of decrease) this baseline probability of presence, whilst cleared land greatly lowered it (94–97% chance). Experts in Region I indicated that elevation increased this baseline (64% chance) whereas Region II experts supported a decrease (76% chance). Region I experts supported slope as having a positive effect on the baseline (89% chance), with Region II experts being more certain (99·9% chance of positive effect) and nearly tripling the size of the effect. Expert results were averaged, following Kuhnert et al. (2005), and are justified by the experts’ high confidence in their answers (Fig. S2).

Univariate response curves enabled experts to visualize how their assessments determined the shape of the curves and reflected their beliefs regarding the impacts of habitat factors (Fig. 2). In this example, the expert thought sedimentary rock was linked to a higher probability of presence. There was uncertainty about the impact of intrusive igneous rocks, portrayed by a boxplot with large error bars (Fig. 2a). Closed forest correlated with a high probability of presence (60%), closely followed by open forest (50%) but there was a low chance of presence (10%) in cleared land (Fig. 2b). The probability of presence was believed to increase with increasing elevation (Fig. 2d) and slope (Fig. 2e) and a north to north-west aspect (Fig. 2f). Prior distributions encoded from elicitations at 20 sites, and logistic regression results, are available for each expert (see Tables S1 and S2, Supporting Information).

Figure 2.

 An example of feedback plots in the form of univariate species response plots created by the elicitation tool encoding assessments from one expert for each site using a logistic regression model. Response is shown for habitat factors: (a) geology, (b) remnant vegetation, (c) landcover, (d) elevation, (e) slope and (f) aspect. This feedback allowed experts to visualize how probability of presence related to each habitat factor as suggested by their encoded opinions provided site-by-site. Boxplots (a–c) show plausible range of responses (probability of presence) to categorical variables with one bar per elicited site. Other plots (d–f) represent continuous variables with one point per elicited site. Boxplots indicate consistency of the expert; i.e. the wider the boxplot, the wider the range of values given; the narrower the boxplot, the more sure and consistent the expert was of their answer. This feedback allowed experts to confirm patterns or to click on sites (bars or points) to change assessments at sites providing any unusual values, such as outliers.

When comparing individual opinions of all nine experts on vegetation and geology, all except two agreed that cleared land decreased rock-wallaby presence, although there was mixed opinion on the effect of closed forest (Fig. 3a). However, overall differences of opinion on the effect of vegetation were minor (Figs S3 and S4). Compared with volcanic rock, experts from the two regions had different opinions on the impacts of intrusive igneous and sedimentary/metamorphic rock on rock-wallaby presence. Region I experts suggested that these rock types reduced presence in contrast to the increase suggested by Region II experts (Fig. 3b).

Bayesian posterior analysis

Non-informative priors provided only a vague indication of the possible extent of the impact of each habitat factor. Posteriors based on non-informative priors differed from those based on expert-informed priors, indicating that experts contributed information beyond the data (Table 2). Prior values are fully reported in Table S2. Posterior analysis of data originating from different regions, with non-informative priors, differed considerably in some cases, namely geology and elevation.

Table 2.   Posterior distributions calculated in WinBUGS for three different priors together with field data from Region I or Region II
PosteriorsNon-informative priors/Region I field dataUsing Region I expert priors/Region I field dataUsing Region I expert priors/Region II field data
Region I expertsMeanSDMeanSDMeanSD
Intercept−1·6201·305−0·4410·649−0·3980·631
Intrusive igneous−1·6820·733−1·5700·475−0·3730·427
Sedimentary−1·5960·564−1·1700·3700·2540·371
Closed forest−0·6821·058−0·3690·390−0·6000·361
Cleared forest−1·4370·667−0·9600·373−0·6760·364
Landcover forest−0·5481·021−1·1310·474−1·2950·523
Landcover cleared−0·9152·644−0·7770·727−1·5050·478
ElevationN/AN/A0·0000·001−0·001−0·001
Slope0·1040·0390·0490·0160·0480·013
DIC121·554118·948122·474
PosteriorsNon-informative priors/Region II field dataUsing Region II expert priors/Region I field dataUsing Region II expert priors/Region II field data
Region II expertsMeanSDMeanSDMeanSD
Intercept−3·1301·333−1·2460·671−1·0860·696
Intrusive igneous0·8470·712−0·8430·4650·2230·445
Sedimentary1·7470·671−0·7710·3560·6530·382
Closed forest−1·5290·866−0·2500·413−0·5520·396
Cleared forest0·1220·694−0·9830·379−0·4340·389
Landcover forest−1·9532·309−0·7040·539−0·7830·639
Landcover cleared−2·2050·796−1·2520·857−1·6430·539
ElevationN/AN/A0·0000·001−0·0010·001
Slope0·0780·0250·0770·0160·0670·014
DIC119·817119·758118·263
PosteriorsNon-informative priors/combined field dataUsing combined expert priors/combined field data
Combined regionsMeanSDMeanSD
  1. SD, standard deviation; DIC, Deviance Information Criterion; Not Available.

Intercept1·6140·386−1·1170·613
Intrusive igneous−2·0390·624−1·0620·452
Sedimentary−1·4360·468−0·6680·337
Closed forest−0·4820·988−0·3350·393
Cleared forest−0·9660·558−0·9390·350
Landcover forest−1·4330·908−0·9400·495
Landcover cleared−2·7391·144−1·2770·650
ElevationN/AN/A0·0000·001
Slope0·092020·01950·0700·015
DIC251·240247·856

When expert opinion from either region was incorporated using informative priors, regional differences diminished considerably for most covariates regardless of the source of expert opinion. In general, posterior analyses informed by Region I experts attributed slightly lower impacts to habitat factors, and this effect was larger due to the offset by higher estimates for the baseline in Region I (Fig. 4). Differences in regional opinion resulted in different inferences on the intercept and effects of geology. A low baseline probability of presence was estimated with Region I informed priors compared with an estimate near 50% for Region II. Compared with volcanic rock, expert knowledge from Region I pulled posterior estimates for sedimentary and intrusive igneous rock towards more negative effects while Region II experts pulled posteriors towards more positive effects. Posteriors reflected differences in opinion regarding the effect of geology, with non-volcanic rock leading to increased plausibility of presence when informed by Region II experts, in contrast to decreases for Region I (Fig. 4).

Figure 4.

 Comparison of four scenarios. Each plot shows the posterior estimated distribution of the effect of a variable, here intrusive igneous or sedimentary rock, on the plausibility of presence of brush-tailed rock-wallabies compared to the baseline of volcanic rock. Zero reflects no difference compared to the baseline. Curves to the left (or right) of zero indicate evidence for a decreased (or increased) effect, relative to the baseline.

The DIC values show that expert-informed priors combined with field data from the same region gave a close goodness-of-fit for Region II (DIC = 118·3) relative to non-informative posteriors (DIC = 119·8) and a reduced goodness-of-fit for expert priors from a different region (DIC = 122·5). Region I experts did not show this pattern (Table 2). The results of the predictive performance checks and ROC curves show that the models built on expert opinion and data sourced from the same regions both performed consistently well in terms of predictive performance. Models of both sets of experts with data from the other region showed slightly lower performance with under-prediction occurring for likely absences and over-prediction from Region I experts for likely presences. In general, predictive performance in the ROC curves was around 0·7–0·8 for all model scenarios, indicating good model discrimination ability (Figs S5 and S6).

Discussion

Comparing expert opinion to field data

We found that expert knowledge, formulated as informative priors, enhanced posterior estimates based solely on field data. For both regions, the expert assessments on brush-tailed rock-wallaby habitat were consistent with the field data regarding the effects of general or widely known variables, such as remnant vegetation. This is consistent with previous studies where expert data and independent field data were included in models (Johnson, Seip & Boyce 2004; Kuhnert et al. 2005). However, it contrasts with Pearce et al. (2001), who found that incorporating expert opinion when selecting variables for generalized additive models did not significantly change the predictive accuracy. Here all experts had definite opinions on what did not comprise brush-tailed rock-wallaby habitat, indicated by the negative effect of cleared land. In this case, the DIC values show that the Region II experts had a greater impact on the empirical data and closer goodness-of-fit from their region and the neighbouring region than the Region I experts. This may be because Region II experts had increased familiarity with GIS layers and the landscape-scale. However, combining all expert opinion with empirical data (DIC = 247·9) did not give on improved goodness-of-fit compared with a model based on non-informative priors and empirical data (DIC = 251·2).

Comparing experts from different regions

In general, experts tended to underestimate the probability of presence in different regions from their own knowledge source. The experts gave varying opinions on the habitat requirements for a species depending on where their knowledge was acquired. Experts from Region I had different opinions on the effect of intrusive igneous rock than experts in Region II. These regional differences in opinions on the effects of geology reflect the different use of landscape by the rock-wallaby in each region. Region II is mostly composed of sedimentary/metamorphic rock with some large intrusive igneous beds appearing in some areas. Most rock-wallaby sites in this region are found on this rock type. Volcanic rock is sparse in Region II and not well represented in steep complex escarpment areas. In contrast, Region I has a large area of volcanic rock, which was a strong predictor of rock-wallaby habitat in the region (Murray et al. 2008).

By considering differences across regions, this study addresses the knowledge gap noted by Johnson & Gillingham (2004) on the accuracy of expert opinion. Yamada et al. (2003) also found inconsistencies in opinions collected on the habitat of sambar deer Cervis unicoloras experts had only partial familiarity of the study area. Our study undertook a quantitative approach to the mandate of Fazey et al. (2006) to understand the geographical extent of an expert’s knowledge and its relevance. We illustrate how misleading it can be to use expert knowledge not sourced from the same region where it is applied.

Eliciting information

Using a GIS framework to capture expert knowledge enables experts to express opinions at landscape scales consistent with those used by conservation practitioners, explore spatial data within an environmental context and directly apply knowledge gained in the field (Denham & Mengersen 2007). GIS-informed expert elicitations on habitat suitability are affected by the quality of the source of spatial data (Store & Kangas 2001; Carter, Stolen & Breininger 2006) and are constrained by its spatial resolution. In this case 50 m pixels may hide small local variations in the landscape.

We gave experts the opportunity to check their responses through a feedback loop via univariate response curves from the logistic regression fit to their elicitations. This provided experts with a foundation for exploring and communicating their knowledge, as it is not just personal experiences but also the cognitive ability of experts to correctly structure their experiences which ensures accurate elicitation (Bradley, Paul & Seeman 2006). Experts are more likely to give justifiable probabilities if feedback is given to reduce uncertainties and develop elicitation skills (Walls & Quigley 2001). This study addressed common biases such as anchoring, over-conservatism and representativeness (Morgan & Hemrion 1990) by design (order of questions, feedback) and training experts before elicitation.

Limitations in expert elicitation

Considerable research has been conducted into the best approaches for eliciting accurate information from an expert (Walls & Quigley 2001; Grist et al. 2006; O’Hagan et al. 2006). Two common mechanisms used in conservation planning for eliciting information are expert panels and interviews with individual experts (Ferrier et al. 2002; Martin et al. 2005). The former allows for general consensus but can be affected by dominant personalities and other group dynamics. Conversely, individual elicitations are problematic when experts are reluctant to expose opinions alone, leading to a tendency to under- or over-predict (Kynn 2005).

In general, collating expert knowledge is cheaper than extensive field sampling. Nevertheless, we found elicitation required significant time investment together with the presence and continual input of the interviewer to ensure experts understood their probability assessments for each elicited site. A mailed survey (e.g. Martin et al. 2005) might be less time-consuming and expensive. However, this would have reduced the quality and consistency of information elicited due to less targeted training on the tool and elicitation process. Our study used many experts and many variables in the model together. Most studies have a major deficiency in one of these areas. For example, Kuhnert et al. (2005) used a number of experts but only univariate regression. Other authors used only one or two experts (e.g. Kynn 2005; Denham & Mengersen 2007; O’Leary et al. 2008).

Management implications

This research has wider implications beyond the conservation of the brush-tailed rock-wallaby. There are trade-offs to using expert opinion. One trade-off is with the number of experts chosen. Choosing one or two experts keeps costs down, is less time-consuming and gives some ecological knowledge but our study shows the variability that can occur in expert answers. The nine experts we interviewed individually did exhibit some variation in elicited means. There was an obvious difference between experts from different regions as well as within regions, hence the danger of using only one or two experts. Using multiple experts allows for identification of outliers and the ability to average or weight experts according to experience, effectively minimizing extreme opinions. Hogarth (1978) and Kynn (2005) argue against this due to the difficulty of judging an expert’s knowledge and independence of thought.

Another trade-off is scale when sourcing data from field sampling or from expert opinion. Collecting empirical data at different scales for species’ distribution modelling gives a greater chance of identifying habitat associations at different levels (Murray et al. 2008). However, data collection is an expensive activity, with time and budgetary constraints that increase with each investigated scale. Expert knowledge is a source of information that can be collected cheaply compared to designing and conducting robust field sampling. However, expert knowledge is limited by the source of the expert’s knowledge. Experts often gain their knowledge through field-based sampling. It is difficult for field-based experts to make comments on another scale without extrapolating beyond their expertise. While it is a trade-off, our study approached this by training the expert to interpret site-scale features into representative landscape patterns, such as steep slopes, and we used feedback methods during elicitation to check the revealed landscape patterns with experts.

Management of the threatened brush-tailed rock-wallaby is already occurring at national and state levels and generally involves expert panels determining the distributional status and risks for the species (NSW Department of Environment and Conservation 2005; Department of Environment and Heritage 2006). Habitat conditions and threats faced by the species are site-specific and differ across regions. While expert involvement ensures some measure of ecological knowledge is incorporated into decision-making, the source of the knowledge is infrequently addressed. This study demonstrates the necessity for choosing suitable experts. Habitats also differ across regions; therefore, this species would benefit from being managed at a regional ecosystem or finer scale suited to expert knowledge input, rather than by a blanket management scheme across its geographical range.

Our results show that expert opinion is useful for collecting general knowledge on a species to inform data collection and may contribute to improving species’ distribution models. Our study also highlights that conservation practitioners should choose experts carefully and ascertain that their experience is within the target area of interest, especially where habitat factors have considerable regional variation. Therefore, experts should only inform management plans for the relevant region.

Acknowledgements

This work was supported by a University of Queensland Postgraduate Research Scholarship and funds from the Applied Environmental Decision Analysis (AEDA) research facility for J.V.M. Generous funding was also provided by the M. Carroll Animal Welfare Fund, the Wildlife Preservation Society of Australia and the Foundation for National Parks and Wildlife. We greatly appreciate the time and effort volunteered by experts who participated in the elicitation.

Ancillary