Mapping uncertainty: sensitivity of wildlife habitat ratings to expert opinion

Authors

  • CHRIS J. JOHNSON,

    Corresponding author
    1. Ecosystem Science and Management Program, University of Northern British Columbia, 3333 University Way, Prince George, British Columbia, Canada V2N 4Z9
    Search for more papers by this author
  • MICHAEL P. GILLINGHAM

    1. Ecosystem Science and Management Program, University of Northern British Columbia, 3333 University Way, Prince George, British Columbia, Canada V2N 4Z9
    Search for more papers by this author

Chris Johnson, Ecosystem Science and Management Program, University of Northern British Columbia, Prince George, BC, Canada V2N 4Z9 (e-mail: johnsoch@unbc.ca).

Summary

  • 1Expert opinion is frequently called upon by natural resource and conservation professionals to aid decision making. Where species are difficult or expensive to monitor, expert knowledge often serves as the foundation for habitat suitability models and resulting maps. Despite the long history and widespread use of expert-based models, there has been little recognition or assessment of uncertainty in predictions.
  • 2Across British Columbia, Canada, expert-based habitat suitability models help guide resource planning and development. We used Monte Carlo simulations to identify the most sensitive parameters in a wildlife habitat ratings model, the precision of ratings for a number of ecosystem units, and variation in the total area of high-quality habitats due to uncertainty in expert opinion.
  • 3The greatest uncertainty in habitat ratings resulted from simulations conducted using a uniform distribution and a standard deviation calculated from the range of possible scores for the model attributes. For most ecological units, the mean score, following 1000 simulations, varied considerably from the reported value. When applied across the study area, assumed variation in expert opinion resulted in dramatic decreases in the geographical area of high- (−85%) and moderately high-quality habitats (−68%). The majority of habitat polygons could vary by up to one class (85%) with smaller percentages varying by up to two classes (9%) or retaining their original rank (7%). Our model was based on only four parameters, but no variable consistently accounted for the majority of uncertainty across the study area.
  • 4Synthesis and applications. We illustrated the power of uncertainty and sensitivity analyses to improve or assess the reliability of predictive species distribution models. Results from our case study suggest that even simple expert-based predictive models can be sensitive to variation in opinion. The magnitude of uncertainty that is tolerable to decision making, however, will vary depending on the application of the model. When presented as error bounds for individual predictions or maps of uncertainty across landscapes, estimates of uncertainty allow managers and conservation professionals to determine if the model and input data reliably support their particular decision-making process.

Introduction

Quantitative habitat models and predictive distribution maps are now important tools for the conservation and management of animals and plants (Guisan & Zimmerman 2000; Raxworthy et al. 2003; Jeganathan et al. 2004; Johnson, Seip & Boyce 2004). The widespread application of such tools is a function of the availability of geographical information system (GIS) data and the popularization of generalized linear models (GLM) and other computationally intensive numerical techniques (Rushton, Ormerod & Kerby 2004). There are still many instances, however, where in the absence of empirical data, inference and decision making are guided by expert opinion. Examples from the natural sciences range from predicting forest succession to evaluating soil quality (Sparling et al. 2003; Zhou, Mills & Teeter 2003).

Since the 1970s, expert opinion has served as the primary information source for wildlife habitat evaluations framed within a general set of methods known as habitat suitability indices (HSI; USFWS 1981). In its simplest form, HSI is an equation of an additive, multiplicative or logical form with coefficients representing the relative value of environmental features. Typically, coefficients are scaled between 0 and 1 and are estimated using best available knowledge as surveyed from experts or published literature. Depending on the definition of habitat suitability, model predictions can represent environmental carrying capacity (as reflected by population density), biomass per unit area or more simply patch occupancy (Schroeder & Vangilder 1997; Oldham et al. 2000; Loukmas & Halbrook 2001). In conjunction with a GIS and data representing the spatial distribution of model inputs, HSI equations can be used to generate maps of ranked habitat units (Li et al. 2002; Store & Jokimaki 2003).

Although expert-based models often are the best and sometimes the only information available to develop, assess and meet conservation and management objectives, there is no inherent assurance that model results portray reality. A model that poorly reflects perceived or actual conditions will not only fail as an evaluation or guidance tool, but may result in misplaced resources or harmful conservation and management actions (Loiselle et al. 2003). HSI models, as an example, are ubiquitous in the management and conservation arenas yet they are infrequently validated and the criteria and approaches for validation may be questioned (Roloff & Kernohan 1999). Furthermore, validation is dependent on the availability of reliable empirical data. Considering that expert-based approaches are typically a response to no or poor-quality data, it is not surprising that HSI models are infrequently validated immediately following development and application. Where validation is conducted, results are appropriate only for a small set of circumstances (Rothley 2001). Given the wide spatial and temporal scales of some HSI models and the possible range of environmental perturbations models are meant to represent, validation of model predictions for all possible conditions is intractable.

Complementary to validation are uncertainty (UA) and sensitivity analyses (SA). UA and SA allow one to quantify the range and distribution of predictions and identify data, model structure or parameters that require improvement (Crosetto, Tarantola & Saltelli 2000). Failing to quantify and understand the variation in model predictions due to uncertainty can lead to assumptions about data accuracy and output that are not valid and ultimately impact upon management practices and decisions (Regan, Colyvan & Burgman 2002). To date, there have been few evaluations of the magnitude of uncertainty inherent in expert opinion or the impact of that uncertainty on the products of expert-based habitat models.

We used Monte Carlo simulations to assess the degree of uncertainty and identify sensitive parameters for habitat classifications and associated maps generated from expert opinion. As a case study, we applied simulation-based uncertainty and sensitivity analyses to wildlife habitat ratings mapped for a 2400 km2 area of British Columbia (BC), Canada, currently subject to development planning. We focused on quantifying the magnitude and sources of uncertainty for ratings of one species during one season, not an error assessment of the ecological base mapping or an evaluation of the predictive capacity of the final maps. Simulation results revealed the most sensitive parameters in the ratings model, the precision of wildlife habitat ratings for a number of ecosystem units, and variation in the total area of high-quality habitats due to uncertainty in expert opinion. We discuss implications of uncertainty for conservation and management of plants and animals, application of sensitivity and uncertainty analyses to predictive habitat models and products, and suggest ways in which uncertainty in expert opinion can be minimized and best collected to integrate with statistical evaluation procedures such as Monte Carlo analyses.

background

Ecosystem mapping and wildlife habitat ratings

Ecosystem mapping (EM) is the most current source of ecological and habitat information for resource development planning across BC. Maps portray ecological units, which are unique combinations of hierarchically ordered environmental factors that are rated as habitat for regionally important wildlife species (Predictive Ecosystem Mapping (PEM) Data Committee 2000). Maps are used at both the site and regional level to meet the information requirements of higher-level planning initiatives and to position individual developments. Despite the increasing use of EM for managing BC's natural resources, there have been few efforts to determine the degree of uncertainty in ecosystem unit designations or wildlife habitat ratings.

Methods for ecosystem mapping vary across project areas according to the availability of data. In most cases, a hierarchical approach is used where coarse-scale terrain and landform units are first identified and mapped. Following the identification of small-scale bioterrain units, the study area is further stratified by climatic conditions. At the finest scales, vegetation inventory data are used to identify ecological units, but site-specific factors such as aspect, soil texture or terrain can further modify the description of a unit.

Ecosystem units represent vegetative ecological associations across a project area, but also serve as the ecological and spatial framework for habitat ratings. Following completion of the EM, conceptual habitat accounts are generated for each identified wildlife species. Expert biologists use their experience, published research findings and site visits to correlate life-history and habitat requirements with the constituent attributes (e.g. climate, topography, vegetation and soil attributes) of the ecological units. Those correlations are summarized as index scores within ratings tables. Scores for each table range from 0 to 1 and serve as a relative index of an attribute's contribution to the value of an ecosystem unit as seasonal habitat for a particular species; when combined as an equation, scores represent an overall resource suitability index (RSI). RSIs are similar to HSI except the former accommodates a greater range of potential environmental attributes.

There is no standard for combining index scores; however, a linear multiplicative model is common. As the final step in the habitat rating process, index scores are classified for mapping purposes; typically, a six-, four- or two-class scheme is used. Although EM polygons are mapped at a scale of 1:20 000–1:50 000, they are not spatially discrete. Polygons can contain labels for up to three ecological units where the percentage area of each unit is specified as a decile. When mapping habitat rating classes, one can choose to represent the average score, the highest score irrespective of decile, or the score that occupies the greatest percentage of the polygon.

Uncertainty and sensitivity analysis

Uncertainty analysis (UA) differs from error assessment. Where error assessment relates model predictions to truth, uncertainty analysis reveals the potential range of values around a predicted outcome (Lodwick, Monson & Svoboda 1990). Typically, UA is conducted as a simulation, where one runs a model multiple times and recalculates the predicted outcome for each systematic perturbation of the input variables. Input can vary in many ways, but is usually sampled from a distribution of values with known properties (Fig. 1). Following the simulation, the variation in outcomes indicates the level of uncertainty in model predictions one might expect given a known or assumed distribution of scores for the input data. Uncertainty analyses allow us to consider all sources of uncertainty simultaneously and determine if the model and input data reliably support the decision process. Sensitivity analysis (SA) works in the opposite direction, revealing model components or data with the greatest influence on the variation in predictions. A range of statistical techniques (e.g. linear regression, correlation analyses, sensitivity indices) are available for performing SA (Saltelli, Chan & Scott 2000). Although UA is more prominent in the field of GIS-based modelling, UA and SA are complementary approaches that provide support for model predictions and highlight areas where assumptions need to be addressed and source data improved or augmented (Crosetto & Tarantola 2001).

Figure 1.

Framework for uncertainty and sensitivity analysis of an ecosystem mapping resource suitability model for woodland caribou during spring.

Methods

study area

Ecological mapping and associated wildlife habitat ratings assisted a planning process designed to minimize the impacts of oil and gas exploration on 11 regionally and provincially significant wildlife species (EBA Engineering 2002a). The EM project was conducted between February 2000 and March 2002 for four geographically distinct planning areas that covered approximately 1·2 million ha of the Muskwa-Kechika Management Area (MKMA; EBA Engineering 2002a). The study areas are located east of the Continental Divide approximately 60 km west of Fort Nelson, BC (58°50′N, 125°3′W) and are diverse in terrain, vegetation and wildlife. Ecosystem units range in elevation from 420 m across valley bottoms to a maximum elevation of 2840 m across alpine areas. A wide variety of forested, wetland, non-forested and alpine vegetation communities are found across the study area (EBA Engineering 2002a). For mapping and wildlife habitat ratings purposes, the Biogeoclimatic Ecosystem Classification (BEC) system was used to hierarchically stratify vegetation associations according to progressively finer scales of climate, soils, and site conditions (Meidinger & Pojar 1991). Four BEC zones (Alpine Tundra (AT), Spruce Willow Birch (SWB), Boreal White and Black Spruce (BWBS) and Engelmann Spruce Subalpine Fir (ESSF), the coarsest unit of ecological stratification, occurred across the study area. Wildlife habitat ratings were developed for woodland caribou, Rangifer tarandus caribou Gmelin, grizzly bear, Ursus arctos L., moose, Alces alces L., Rocky Mountain elk, Cervus elaphus nelsoni L., plains bison, Bison bison L., Stone's sheep, Ovis dalli stonei Nelson, mountain goat, Oreamnos americanus Blainville, American marten, Martes americana Turton, fisher, Martes pennanti Erxleben, three-toed woodpecker, Picoides tridactylus L., and bay-breasted warbler, Dendroica castanea Wilson.

uncertainty and sensitivity analyses

We used a Monte Carlo simulation to perform uncertainty and sensitivity analysis for wildlife habitat ratings from a sample of ecological units found across the MKMA ecosystem mapping project. We selected three units representing low, mid and high RSI scores for each of the four BEC zones found across the study area. Although a total of 25 species by season models were developed for the project, we focused our analyses on the ratings for caribou habitat during the spring season. Attributes defining that model included BEC, site series, structural stage and site modifier (EBA Engineering 2002b; equation 1). In a hierarchical fashion, BEC represents a multi-ecosystem unit description of climate, site and soil conditions; site series describes climax vegetation for a particular ecosystem unit; structural stage represents the successional stage of the ecosystem unit; and site modifier describes atypical occurrences of the site series with respect to variation in topography, moisture, soil and soil characteristics (PEM Data Committee 2000):

image( eqn 1)

Perturbations introduced during a Monte Carlo UA should represent the range of reasonable assumptions about the nature of uncertainty expected from the model or source data. Those assumptions are explicitly defined by a statistical probability distribution from which the source data are sampled. The analyst must choose the appropriate distribution and define parameters such as the mean and standard deviation that shape the distribution. For this project, only a single score was reported for each attribute in a ratings table. The wildlife habitat rating process did not allow an evaluation or provide an estimate of divergence in expert opinion. Therefore, we could not empirically define the distribution of index scores or the variance in scores. As an estimate, we used the index score reported for each ecological unit to define the hypothetical mean value of the distribution of scores. To cover the sampling space (i.e. range of opinion) from which index scores may have occurred, we performed the UA using two probability distributions and three different calculations of variation in ratings.

For each of the 12 ecological units subjected to UA, we sampled scores for the Monte Carlo simulations from a triangular and uniform distribution. A triangular distribution is defined by a minimum, mid and maximum value with sampled index scores having a higher probability of selection as they approach the mid score (Fig. 1). The uniform distribution is defined by the minimum and maximum extent with all scores between those two points having an equal probability of being sampled. The parameters for each distribution were taken from the reported data. The midpoint for the triangular distribution was the reported score and the extents of both distributions were calculated as ±1 standard deviation from the midpoint. Standard deviations were calculated in one of three ways for each RSI model attribute: from the range of scores contained within a ratings table; from the ratings for the attribute across all ecological units; and from the ratings for the attribute across ecological units found within each of the four BEC zones. In the latter case, the value of the standard deviation was specific to BEC zone, whereas in the former two cases the standard deviation was calculated from and applied across all zones.

We developed two approaches to quantifying and exploring uncertainty and sensitivity in wildlife habitat ratings. First, we performed an aspatial analysis constructing 72 individual Monte Carlo simulations from the 12 ecological units, two distributions and three calculations of the standard deviation. Each simulation involved 1000 samples applied to equation 1 from which we calculated and plotted the mean rating ± 1 standard deviation. We used the extended Fourier amplitude sensitivity test (FAST) to quantify the influence of each model attribute on the final predicted rating. FAST is a variance-based or anova-like method which produces a sensitivity index representing the fractional contribution of each model parameter to the total variance (Saltelli, Tarantola & Chan 1999). Extended FAST is capable of coping with interactions among model parameters and reports a total sensitivity index of the individual factors and their interaction effects. We used the simulation software SimLab to conduct the initial UA and SA (SimLab 2003).

In addition to the rigorous exploration of UA and SA for a select set of ecological units, we constructed a simulation program in Visual Basic to produce a second set of analyses in a metric that was more easily related to land use decisions: change in the area and ranking of habitats. For each of the 4736 polygons found across the most southern planning area the programme simulated 100 RSI scores. It also estimated the variation in area of moderate and high-quality habitat across model runs and the potential range of ratings for each polygon (mean ± 1 standard deviation). Area of habitats and ratings overlap were considered in the context of a six-class wildlife habitat ratings scheme. For simulation parameters, we looked to the first set of UA and used the combination of distribution and standard deviation that generated the smallest and largest levels of uncertainty. Ecosystem mapping polygons potentially represent three ecological units; here, we performed the UA considering the average and the largest RSI score as well as the decile with the greatest area.

Results

Mean RSI scores summarized from 1000 Monte Carlo simulations for 12 sample ecological units demonstrated considerable divergence from the expert scores (Fig. 2). In general, uncertainty was greatest for simulations conducted using a uniform distribution and a standard deviation defined by the range of possible scores for the model attribute. Alternatively, we observed the smallest variance for simulations conducted with a triangular distribution and a standard deviation defined by the observed scores within each BEC zone. The initial expert's attribute score influenced the magnitude of uncertainty and the mean simulated score. Typically, ecological units with scores near 1 were consistently biased toward 0 (Fig. 2).

Figure 2.

Uncertainty in estimates of resource suitability indices (•) for three ecological units qualitatively ranked as low-, medium and high-quality habitats found within the Alpine Tundra (AT), Spruce Willow Birch (SWB), Boreal White and Black Spruce (BWBS) and Engelmann Spruce Subalpine Fir (ESSF) Biogeoclimatic Ecosystem Classification (BEC) zones. A Monte Carlo simulation was used to estimate uncertainty given a triangular (▴) and uniform (▪) distribution of estimates and variances calculated from the range of possible scores for the model attribute (□), observed scores across all BEC zones (inline image), and observed scores within each zone (▪).

The FAST analyses revealed that the sensitivity of RSI scores to individual attributes varied across the study area and that uncertainty was not consistently associated with one model attribute (Fig. 3). Across all combinations of distribution and standard deviation, site series was the most influential parameter for 36 simulations followed by BEC at 25, structural stage at 9 and site modifier at 2. For the Alpine Tundra, Spruce Willow Birch and Engelmann Spruce Subalpine Fir BEC zones the biogeoclimatic attribute was the most influential parameter, but for the Boreal White and Black Spruce BEC zone, site series had the largest impact on total model uncertainty.

Figure 3.

Frequency that Biogeoclimatic Ecosystem Classification zones, site series, structural stage and site modifier were the most sensitive model parameters for simulated RSI scores of 12 ecological units stratified by four BEC zones. AT, Alpine Tundra; BWBS, Boreal White and Black Spruce; ESSF, Engelmann Spruce Subalpine Fir; SWB, Spruce Willow Birch.

Relying on the findings of the UA for select ecological units, we performed a second set of polygon-specific analyses with parameters representing the maximum and minimum observed uncertainty. Congruent with our initial findings, the introduction of uncertainty in expert opinion led to variation in the ranking and geographical area of polygons falling within one of the six habitat classes. Across all permutations, a uniform distribution with a RSI defined by the largest decile resulted in the greatest uncertainty in polygon rating. Using a six-class rating system, the mean RSI score ± 1 standard deviation indicated that 4007 polygons could vary by one wildlife habitat rating class and 407 polygons could vary by two classes (Table 1). Results were less extreme for the triangular distribution where 1877 polygons varied by one rating class.

Table 1.  Number of habitat polygons with a sufficient level of uncertainty to fall within one or more adjacent rating classes. Uncertainty in expert opinion was represented by two distributions (uniform, triangular) and three methods for combing polygon deciles: the weighted average RSI score across deciles, the score from the largest decile, and the highest score from among the three deciles. For each polygon we used the mean RSI score ± 1 standard deviation to determine overlap with adjacent rating classes
Class overlapAverage RSI scoreLargest decileLargest RSI score
Uniform distributionTriangular distributionUniform distributionTriangular distributionUniform distributionTriangular distribution
0 5593040 3222859 3302929
1387316964007187740091807
2 304   0 407   0 397   0

Method of incorporating RSI scores across deciles did not have a large influence on variation in the total amount of high- and moderately high-quality habitats (Table 2). In contrast, the introduction of uncertainty in expert opinion resulted in dramatic changes in the percentage area of class 1 and 2 habitats. Relative to the area of habitats calculated using the unperturbed model, we observed an 85% and 68% reduction in high- and moderately high-quality habitats after introducing uniformly distributed uncertainty averaged across deciles (Table 2). Results were less extreme following application of the triangular distribution: area of class 1 and 2 habitats differed by 17 and 32%, respectively, when compared with the RSI habitat ratings for the published model (Table 2).

Table 2.  Variation in area (km2) of high- (HQ) and moderately high-quality (MHQ) habitat due to simulated uncertainty in expert opinion. Uncertainty was represented with uniform and triangular distributions and three methods for combining polygon deciles: the weighted average RSI score across decile, the score from the largest decile and the highest score from among the three deciles
  areaSD areaMinimum areaMaximum area% change
HQMHQHQMHQHQMHQHQMHQHQMHQ
Average score
No uncertainty39·3444·5NANANANANANANANA
Uniform distribution 5·8141·82·611·2 1·0119·312·1169·4−85·2−68·1
Triangular distribution32·6301·34·012·222·8272·244·2329·1−16·9−32·2
Largest decile
No uncertainty37·2472·2NANANANANANANANA
Uniform distribution 7·8160·53·011·7 2·2129·319·5194·6−78·9−66·0
Triangular distribution33·7337·04·311·424·3307·942·2370·5−9·4−28·6
Highest score
No uncertainty45·3518·7NANANANANANANANA
Uniform distribution 8·8189·73·114·1 2·9152·919·1224·2−80·6−63·4
Triangular distribution40·9375·75·213·229·1323·553·5402·8−9·7−27·6

Discussion

Expert opinion is an important source of information for conservation and resource management decision making. In contrast to the inferences from specific empirically based research studies, experts can provide a synthesis perspective drawing on their own observations and those presented as published data. The costs of monitoring wide-ranging or rare species also can be time consuming and prohibitively expensive (Johnson, Heard & Parker 2002; Rushton et al. 2004). In some cases, we have only the knowledge from experts to guide conservation and management initiatives (Pearce et al. 2001). Furthermore, conservation biology is a crisis discipline. Initiatives designed to halt the decline, extirpation or extinction of a species often cannot wait for the development, funding, implementation and conclusion of empirically based research or monitoring studies.

Process and sampling variation are inherent and widely recognized properties of empirically measured data (White 2000). Variation can result in imprecise estimates and a failure to identify statistically meaningful differences between groups. When propagated through a predictive model, such as a RSI or HSI, variation can lead to a range of estimates that may have considerable implications for model uncertainty and informed decision making (Bender, Roloff & Haufner 1996; Burgman et al. 2001). Opinion and best estimates, when solicited from a number of experts, will also vary. Variation may arise from simple disagreement on a value or ranking; however, other sources of divergence such as vague concepts and imprecise terms, perceived, but actual lack of expertise, or interpersonal dynamics during group surveys could also lead to divergence in opinion. Although there are numerous avenues for variation, there is often little understanding or consideration of how differences in expert-based answers may affect outcomes. This is in sharp contrast to decision-making processes founded on empirical data where uncertainty analyses are relatively common especially in the area of risk analysis (Emmi & Horton 1995; Zerger et al. 2002).

The lack of UA and SA for expert-based models may partially be a function of how expert opinion is solicited. Inherent within a Monte Carlo or other simulation approach is an estimate of variability in model parameters, in this case, stemming from differences in expert opinion. If only one expert is consulted or a process is used that builds consensus among experts without recording differences, we must assume the shape and type of probability distribution. For our analyses, only point estimates were reported for each model parameter for each ecological unit. Lack of measured variation forced us to assume a range of plausible distributions. Uncertainty and SA are more realistic and defensible when simulated values are drawn from distributions defined by a sample of repeated observation. In most cases, however, it is unlikely that enough experts would be available for identifying the frequency distribution of opinions on any one question. Non-parametric bootstrapping is an alternative to Monte Carlo simulations where statistical parameters are difficult to identify (Efron & Tibshirani 1993). Researchers have championed the iterative and interactive modified Delphi approach as a method for soliciting and defining levels of agreement between experts (Uhmann, Kenkel & Baydack 2001; Hess & King 2002). We are uncertain, however, if divergence in opinion should be considered after the first or last round of expert consultation.

In situations where uncertainty in expert opinion cannot be quantified we encourage researchers to test a range of possible uncertainties. Repeating analyses for a full range of plausible distributions reveals the sensitivity of the UA to underlying statistical parameters. For each ecological unit, we calculated the standard deviation in index scores in three ways and applied that parameter to two statistical distributions. Our choice of methods was a function of the available data. We assumed that the variance in expert opinion and thus uncertainty increased with the range of possible scores for each attribute and the diversity of ecological units across BEC zones. Selection of distribution was largely arbitrary; however, our guiding criterion was distributions constrained to generate values between 0 and 1. A triangular distribution is more conservative and assumes that expert opinion is centred on the reported score. Alternatively, a uniform distribution assumes that we have no assurances that the reported rating is correct and that scores from multiple experts could range freely within the bounds set for the index score.

Although we could not empirically define the statistical parameters representing variation in expert opinion for each RSI variable, our estimates of uncertainty for habitat ratings were probably conservative. We modelled only variation in expert opinion and ignored other sources of uncertainty such as thematic misclassification of ecological units and imprecision or error in boundary definition of habitat polygons (Davis & Keller 1997). The MKMA EM project involved a complex protocol with numerous spatial data sources all with inherent and introduced error (EBA Engineering 2002a). When expert-based wildlife models are applied to uncertain spatial data we should expect reduced certainty from model predictions.

Subjective judgement and vague concepts are not amenable to probability theory, but can contribute to model uncertainty (Regan et al. 2002). Wildlife habitat ratings are a relative measure of a particular ecological unit's capacity to support a species compared with the best available habitat for that species across the province (Ministry of Environment, Lands and Parks (MELP) 1999). The concept is inherently vague, given a reliance on an incomplete list of subjectively identified provincial benchmarks instead of a well-defined measurable parameter. Also, the scale of mapping and assessment is arbitrary and introduces further uncertainty to the rating process. We do not know how animals scale environments, but given the range of life histories for species represented within EM projects (e.g. grizzly bear and American marten) it would seem inappropriate to apply the grain and extent of ecological units consistently to all species. Methods are available for constructing multiscale expert-based habitat suitability models (Store & Jokimaki 2003).

In the case of EM wildlife habitat ratings, we question the metrics against which RSI index scores are assigned. Past HSI projects have developed functional relationships between model variables and the life history of the focal species (e.g. Prosser & Brooks 1998; Uhman et al. 2001). Shrub height, for example, might be included as a model component because it provides security cover or nesting habitat. Biogeoclimatic ecosystem classification, site series, structural stage and site modifier may be useful for identifying plant associations, but they serve only as vague proxies for factors that dictate the distribution and population dynamics of caribou. Published habitat studies can provide guidance with the identification of ecologically relevant RSI variables.

The results of our work suggest that variation in expert opinion can have dramatic effects on model predictions and ultimately conservation and management actions. Assuming that variation in expert opinion was uniformly distributed, we recorded a maximum 85% reduction in the area of high-quality habitat. Differences were less extreme using the triangular distribution, but still notable. Uncertainty and SA are rarely applied to habitat suitability models; however, in agreement with our findings Bender et al. (1996) reported high uncertainty and overlapping confidence intervals for an HSI of forest types occupied by the grey squirrel, Sciurus carolinensis Gmelin. They assumed static HSI values and instead considered uncertainty in ecological inputs. A logical extension of their simulations would have been an SA to identify input parameters with the strongest impact on model uncertainty. Such post hoc analyses are essential for model and data improvement. For our ecosystem unit analyses, BEC and site series were the most influential parameters. Collapsing the number of BEC and site series classes, would reduce variation in index scores and model uncertainty.

Study-wide UAs suggest that in the absence of uncertainty, experts consistently overestimated the area of high-quality habitats (Table 2). Apparent bias is an artefact of the truncated range of possible scores, 0–1, and the multiplicative model. A low value for any one of the four constituent variables (BEC, site series, structural stage, site modifier) dictates the final RSI and a maximum value of 1 prevents the inclusion of a compensatory score. Given the extreme sensitivity of the final RSI to just one low score, results suggest that wildlife habitat ratings for high-quality habitats are potentially under-represented. Furthermore, the probability of misclassification will increase with model complexity (i.e. the number of variables). Combining scores with a geometric mean would reduce the overall influence of a single low value, but continue to render a polygon unsuitable if an ecological condition necessary for animal occupancy was not satisfied.

Maps are powerful tools for displaying and assessing spatial information including the products of HSI or other predictive distribution or habitat models (Carroll, Noss & Paquet 2001; Johnson et al. 2004). Typically, however, there is little recognition of uncertainty in spatial data and associated model predictions (Khagendra & Bossler 1992; Elith, Burgman & Regan 2002). Applying our methods, we could develop maps to illustrate the level of uncertainty associated with each habitat polygon across a broad study area. Change in habitat ranking after uncertainty is a relatively intuitive metric that is easily presented as a single map with direct relevance to the decision-making process. Alternatively, we could generate maps with a more precise realization of uncertainty: the mean polygon value ± 1 standard deviation or 95% confidence intervals.

Uncertainty in RSI scores revealed that following a six-class rating system most habitat polygons could degrade or improve in ranking by one class. Regardless of distribution or method of combining decile, relatively few habitat polygons maintained their initial ranking following the inclusion of uncertainty. The magnitude of development impact and conservation objectives for the focal species will determine the significance of a one- or two-class change in ranking.

Conclusions

There is evidence to suggest that at some scales of management expert-based habitat models are inferior to those developed using empirical data and statistical approaches (Pearce et al. 2001). Although debate around the relative value of each system continues, we are confident that formalized expert opinion will remain an important information source for some conservation and management problems (Johnson et al. 2004). Our emphasis was not the comparison of empirical and expert-based models. Regardless of how coefficients are generated, model evaluation should be an integral component of the process. Evaluation may include validation relative to some criteria, such as successful prediction, but would benefit greatly from UA and SA (Fielding & Bell 1997). Even where models are considered accurate, UA and SA can reveal situations under which prediction may be unreliable, aid with the identification and visualization of quantitative bounds for potential model outcomes, and identify flaws in model structure or areas of improvement for input data. As demonstrated here, such applications of UA and SA have relevance for the model developer charged with providing reliable information tools and the planner or manager asked to consider inherent uncertainties when making decisions.

Acknowledgements

This research was supported by a grant from the Muskwa-Kechika Trust Fund. We thank Jeff Matheson for constructive comments on an early version of this manuscript.

Ancillary