Diversity and Distributions

Introduction

Species distribution models (SDM) relate incomplete information about the occurrence of species (and sometimes about their absence) with environmental predictors, to generate spatially explicit predictions about their geographic distributions (Araújo & Guisan, 2006; Franklin & Miller, 2009; Peterson et al., 2011; Hortal et al., 2012). Despite their increasing use during the last 15 years (Thuiller et al., 2009; Lobo et al., 2010; Varela et al., 2014), SDMs pose many conceptual problems (Araújo & Guisan, 2006; Soberón, 2007, 2010; Jiménez‐Valverde et al., 2008; Colwell & Rangel, 2009; Soberón & Nakamura, 2009; Hortal et al., 2012) and encompass a number of methodological uncertainties (Barry & Elith, 2006; Heikkinen et al., 2006; Rocchini et al., 2011; Beale & Lennon, 2012).

One of the assumptions of SDMs is that the data used for model calibration (i.e. the samples of presence, or presence and absence) are free of bias. When this assumption is violated by biases in the collection of data, the accuracy of model predictions can be affected (Hortal et al., 2008; Lobo, 2008; Loiselle et al., 2008; Rocchini et al., 2011). In fact, the design of the surveys can have a large impact on SDM performance (Hirzel & Guisan, 2002; Edwards et al., 2006; Albert et al., 2010; Braunisch & Suchant, 2010). It follows that an adequate spatial design of the surveys increases the value of biodiversity data collections to answer different questions in ecology, evolution and biogeography (Hortal & Lobo, 2005; Albert et al., 2010). Many methods to design surveys have been proposed with the objective of maximizing the amount of biodiversity captured, while incorporating time and cost limitations (Austin & Heyligers, 1989; Pereira & Itami, 1991; Hirzel & Guisan, 2002; Funk et al., 2005; Hortal & Lobo, 2005; Medina et al., 2013). However, even planned sample designs can vary in the efficiency with which they detect biodiversity patterns (Sastre & Lobo, 2009).

Perhaps more importantly, good quality data coming from standardized surveys are rare or even lacking for most regions and species (Rocchini et al., 2011). Rather, biodiversity databases include heterogeneous information coming from inventories developed with a variety of objectives (Hortal et al., 2007). The absence of standardized sampling schemes often generates bias in the resulting distributional data. It is well known that the records of occurrence and/or absence of species are more frequent in more accessible locations (i.e. near major road routes, urban areas or the work centres of the taxonomists) and/or classical localities (e.g. national parks or other protected areas) that are repeatedly sampled over time (Dennis & Thomas, 2000; Kadmon et al., 2004; Romo et al., 2006; Hortal et al., 2007). These geographical biases in the survey effort may often result in historical and climatic biases (Hortal et al., 2007, 2008; Lobo et al., 2007a). Such biases can yield incomplete and potentially truncated characterizations of species realized niches (Hortal et al., 2008; Rocchini et al., 2011), with the consequent critical effect on the ability of SDM to describe environmental limits of species’ distributions (Austin & Heyligers, 1989; Thuiller et al., 2004; Albert et al., 2010; Hortal et al., 2012).

Despite the potential effects of spatial biases in the design of the surveys on SDM prediction ability (Araújo & Guisan, 2006; Phillips et al., 2009), this issue has not been addressed in the literature. Here, we try to overcome this gap by systematically evaluating the influence of different survey design strategies on the performance of SDM predictions, as well as the potential interactions between survey designs and SDM techniques. Further, besides survey design we evaluate the magnitude of influence on predictive accuracy of other factors that may affect SDM performance, namely sample size, modelling technique and species modelled. To do this, we use distribution data for the 34 Iberian endemic terrestrial vertebrate species, simulating calibration datasets with different survey designs and levels of sample size within the Iberian Peninsula, and evaluating the ability of different SDM techniques to interpolate their geographical distributions based on these datasets.

Methods

Selection and characteristics of the sample datasets

We extracted samples of five different sizes (1%, 5%, 10%, 20% and 25%) from the full extent of the data domain (5919 10 × 10 km cells in the Iberian Peninsula), using each one of the seven survey designs described above. We ran 50 simulations of each one of the 35 combinations of survey design and sample size (Fig. 1). To describe the quality of the data provided by each class of survey design, we measured the climatic bias of each sample in relation to the environmental conditions of the study area (Kadmon et al., 2003; Hortal et al., 2008). We calculated two characteristics related to the species’ distribution: prevalence and Relative Occurrence Area (ROA; Jiménez‐Valverde et al., 2008; Lobo, 2008). We also evaluated the number of times that a survey design failed to identify enough occurrences to generate models in each SDM technique. That is, every time that a model was not generated for a given species due to insufficient number of presences. See Appendix S1 for more details.

Results

Factors affecting SDM performance varied across the different evaluation metrics. For five of six evaluation metrics, the species modelled was the most important factor affecting model performance, followed by sample size and SDM technique (Table 1). The contribution of survey design, though significant, is much less important than any of these three other factors (Table 1). Contrary to our expectations, predictions of species potential distributions generated from different survey designs showed only comparatively small differences in their variance across evaluation metrics, compared with other factors. Interestingly, the degree of climatic bias was generally low for all survey designs. The most biased design was the one that follows human density (bias around 22%), while systematic surveys were the least biased (Fig. 2). Perhaps as a consequence, samples based on designs biased by human density performed worse for sensitivity, kappa, TSS and AUC, while those stratified by groups produced the better survey designs according to the same metrics (Fig. 3; Supporting information Fig. S1). Increasing sample size led to increasing SDM performance according to all metrics, although this pattern was clearer for Sensitivity and TSS (Fig. 4; Supporting information Fig. S2). Given that for most factors Sensitivity and Specificity were the most informative we show their results in the main text. The rest are available at the Supporting information (Figs. S1 to S5).

With regard to SDM techniques, consensus predictions, GLM and MaxEnt ranked nearly always among the highest performing techniques. However, consensus predictions included some of the lowest performing predictions according to AUC, and GLM was below average for specificity (Fig. 5; Supporting information Fig. S3). Random Forest performed the best according to sensitivity and the worst for specificity, confirming a tendency for overfitting of the data. The opposite pattern was recorded for BIOCLIM, also confirming a tendency for underfitting (Supporting information Fig. S3). Random Forest was the technique the most affected by sample size, followed by MaxEnt and GLM, while consensus predictions were the least affected according to Specificity and CCI (Supporting information Fig. S4). Interestingly, the performance of some SDM techniques was unaffected by sample size across all metrics; according to Specificity and CCI, the interaction between sample size and technique causes a decrease in the performance of BIOCLIM, Gower distance and ENFA (Supporting information Fig. S4).

Despite the large influence of the species being modelled on model performance, we were not able to identify a particular group of species for which SDM performance was consistently better or worse than for others (Supporting information Fig. S5). We performed additional correlation analyses between Relative Occurrence Area (ROA), prevalence of species and all validation metrics (Table 2). Both ROA and prevalence showed negative correlations with all metrics except AUC, so that species with higher prevalences (i.e. more occurrences within the dataset) generated models with lower performance. Specificity, TSS and CCI detected much stronger negative effects of prevalence and ROA on predictions; this is perhaps not surprising, because these metrics also identify species as the main factor affecting SDM results, thus evidencing that the particular characteristics of each species’ geographical range has large impacts on SDM performance. The number of times that a survey design did not generate sufficient occurrences to allow the modelling decreased with the increase in sample size. In all cases, the values of these failures were highest for the biased human density design, particularly at the smaller sample size; almost 40% of the species did not present enough occurrences to be modelled at such sample size (Supporting information Fig. S6).

Discussion

How data are collected is supposed to be of critical importance for the development of species distribution models (Araújo & Guisan, 2006). The data used to calibrate models are known to lead to differences in SDM predictions thus being expected to affect their performance (e.g. Kadmon et al., 2003; Barry & Elith, 2006; Sánchez‐Fernández et al., 2011). However, as shown here for the first time, the importance of data collection is dwarfed when compared with other factors affecting SDM predictions. Indeed, contrary to our own expectations, variation in survey design had relatively small, though significant, effects on the performance of SDM. Instead, species identity ranked highest among factors affecting the models, for all performance metrics (except AUC), followed by sample size and SDM technique.

The main assumption of SDMs is that species distributions are limited by environmental – typically climatic – factors (Araújo & Peterson, 2012). It follows that restricting the range of environmental variation in which a species occurs could potentially affect the calculation of species‐climate response curves which, in turn, could cause projections to be potentially erroneous (Thuiller et al., 2004). Surveys providing a comprehensive coverage of the environmental conditions matching species distributions, like the P‐median or the stratified sampling by groups are suited for SDM applications because they reduce climatic biases in the data. However, neither the samples taken with these survey strategies showed the lowest values of climatic bias, nor did models based on them present significantly better predictive ability. Rather, most survey designs yielded models with very similar predictive accuracy and showed little differences in climatic bias (except for human density, see below).

One possible explanation for this unexpected result is that the spatial biases simulated in our analyses did not produce large differences in climatic biases. Surveys based in non‐random and non‐stratified designs are often expected to be spatially and environmentally biased (Hortal et al., 2007; Loiselle et al., 2008), but this is not always the case. In our simulated surveys, sampling along roads had values of climatic bias similar to random samples, and only slightly higher than systematic surveys. This implies that, at the scale and extent analysed, the spatial location of roads in the Iberian Peninsula covers a large climatic variability, even with small sample sizes. Thus, the spatial bias in the surveys due to accessibility may not always result in climatic biases (Kadmon et al., 2004; McCarthy et al., 2012).

Importantly, the surveys that followed human density showed substantially higher climatic biases (as well as higher variability in such bias) compared with the other strategies. This is probably due to the fact that, following real situations, this simulation allows the cells that are surveyed at each stage to be located either further away or closer in the climatic space. This increases the variability in the values of climatic bias, and at the same time limits further selections of cells to the neighbours of the already surveyed cells, thus increasing the climatic bias. Not surprisingly, this survey design was the one yielding the worst‐performing SDM according to all metrics but specificity. Here, the contrasting behaviour of the values of specificity for the human density simulation (see Fig. 3) can be due to the characteristics of this metric, which evaluates the capacity of the model to discriminate true absences, so the best models are those presenting less commission errors. Models built with biased climatic data can overfit to the climatic conditions that were effectively sampled (Lobo, 2008; Phillips et al., 2009), hence generating spatially restricted predictions. This increases omission errors (false absences) as well as the number of true absences predicted by the model. Thus, metrics that use the number of true absences to assess SDM performance, such as specificity, will present higher values in climatically biased survey designs, whereas metrics based on the number of omission errors can present low values for the same predictive maps. It is also worth noting that other biased survey designs did not show highly variable climatic bias values because both roads and protected areas are well distributed within the Iberian Peninsula, allowing them to cover the most important environmental gradients within this region. However, in areas where the roads and national parks (or any other spatial attribute influencing the surveys) are not widely distributed, samples following these biased designs can generate larger climatic bias, limiting the predictive ability of SDMs and consequently their utility. Should the climatic biases generated by our surveys be higher, we would have found larger effects of survey design. Having said this, our results point out to that the effects of climatic bias on SDM performance could be limited until a certain level, where model accuracy diminishes dramatically, a potential effect that deserves further investigation.

The importance of the particular species being modelled for SDM performance shown by our results may be due to differences in their geographic distributions. Many studies have highlighted that geographical and ecological species’ characteristics affect SDM accuracy (Berg et al., 2004; Segurado & Araújo, 2004; Guisan et al., 2007). Many spatial and ecological characteristics can affect SDM results (Brotons et al., 2004; Segurado & Araújo, 2004; Hernandez et al., 2006; Guisan et al., 2007; McPherson & Jetz, 2007; Newbold et al., 2009; Chefaoui et al., 2011), from which the proportion of the occupied area over the considered territory (i.e. ROA) and the prevalence are frequently reported (Brotons et al., 2004; Luoto et al., 2005; Lobo et al., 2007b; Chefaoui et al., 2011). These variations in performance due to the species characteristics can occur even when using the same SDM technique (Seoane et al., 2005; Hanspach et al., 2010), thus increasing their influence on SDM results. In our case, species with smaller geographical distributions are predicted better, as evidenced by the negative correlation between ROA and the assessment metrics. This is because it is easier for SDMs to capture the climate‐distribution relationship for species of restricted range, for they occupy a smaller – and thus easier to classify – environmental domain within the study region (Segurado & Araújo, 2004; Jiménez‐Valverde et al., 2008; Lobo, 2008). This implies that relatively small sample sizes but with a fair geographic coverage of surveys can provide data of sufficient quality to avoid spurious effects on SDM accuracy of geographically restricted species. As a consequence, the importance of survey design for SDM performance may be limited in our analysis because it is restricted to endemic species – which often present limited spatial distributions. It is however likely that the importance of survey design will increase for species with larger ranges, given that the more widely distributed is a species, the more likely is that biased survey designs are not able to include all the climatic conditions it inhabits, hence resulting in less accurate models.

Sample size is also known to have strong effects on SDM predictive accuracy (Hirzel & Guisan, 2002; Reese et al., 2005; Jiménez‐Valverde et al., 2009; Chefaoui et al., 2011). For example, Araújo et al. (2009) showed that apparent failure of SDM to characterize European bird species‐climate relationships in a high profile study (Beale et al., 2008) was due to incomplete coverage of available data and, in another study, it was shown that the ability to predict range shifts of British birds under climate change was highly affected by the completeness of the data used to calibrate the models (Araújo et al., 2005). In our work, increasing sample size led to increases in SDM performance and decreases in variability of model performance; predictions reached a large degree of stability with sample sizes of 10% of the studied area (592 cells) or larger. The measured effect of sample size may be in part related to its interaction with prevalence (Jiménez‐Valverde et al., 2009); below a certain threshold (such as the 10% we found), the representation of the distribution of the species may be too poor to produce reliable models. Larger sample sizes would allow more accurate descriptions of the species’ response to the environment, thus yielding better predictions (Wisz et al., 2008). Importantly, in our analyses, larger sample sizes provide better coverage of the study area, rather than just increasing numbers of occurrences. Thus, the stability in the predictions can be due to the fair representation of the environmental variation within the region, a key factor for the accuracy of the description of species–environment relationships (Kadmon et al., 2003; Hortal et al., 2008, 2012; Hortal & Lobo, 2011) and therefore of higher importance for SDM performance than sample size per se (Newbold et al., 2009). Sastre and Lobo (2009) found that biased survey designs were not efficient in covering the true geographic pattern of species richness within a region (see also Hortal & Lobo, 2011). As showed by our analyses, the failures in covering the species distributions can be accentuated when biased survey designs are associated with smaller sample sizes. These failures are especially problematic when aiming to model the whole biodiversity of a region, and consequently for conservation actions based on many data‐intensive strategies.

SDM techniques differ in their capacity to capture the relationship between species distributions and environmental variables. In general, complex techniques generate models with better performance within the training data (Elith et al., 2006; Tsoar et al., 2007; but see Lobo, 2008; Hijmans, 2012). However, the results of model comparisons are conflicting across studies, due to strong variations in the modelling techniques and their possible interaction with data characteristics (Araújo et al., 2005, 2009). Although no SDM technique performs better in all cases, our results show that the ensemble forecasting approach could be a good strategy to enhance SDM performance, for it consistently ranks within the best‐performing technique regardless of the validation metric used. However, in some cases, the consensus predictions yielded models with lower performance than individual SDM, perhaps due to the inclusion of predictions from poorly performing techniques. The accuracy of consensus predictions is highly constrained by the quality of individual predictions (Araújo et al., 2005). Therefore, to improve the performance of ensemble forecasting approaches it is necessary to modify the building‐up of the consensus, either removing poor models or weighting individual models based on a previously selected performance metric (Araújo & New, 2007; Marmion et al., 2009; Garcia et al., 2012).

To summarize, our results support the view that the environmental coverage of the surveys is more important than the spatial structure of the calibration data. Spatially biased surveys do not always yield environmentally biased data and therefore can generate models as satisfactory as those using carefully designed surveys. Nevertheless, we contend that the distributional data used for species distribution modelling should be evaluated to characterize their climatic bias and coverage (see Kadmon et al., 2004; Hortal et al., 2008; Hortal & Lobo, 2011). Whenever possible, contrasting estimated realized niches with estimated analogous dimensions of the fundamental niche could provide useful insights as to the possible biases in the estimates with SDM (Araújo et al., 2013). The largest source of variation in SDM predictions is the species being modelled, and more precisely the extent of its distribution within the studied area. Geographically restricted species (i.e. with lower ROA) yield more accurate results, but better predictions are also obtained with larger sample sizes and from survey designs with low climatic biases. Further studies are necessary to reveal the specific geographic or functional traits that can affect the predictive power of SDM. Additionally, it is important to investigate whether there is a minimum threshold of climatic bias behind which the quality of SDM predictions changes dramatically.

Acknowledgements

GT was funded by CAPES REUNI and PDSE grants no 11842121. TFR has been supported by CNPq research grants (564718/2010‐6, 474774/2011‐2, 310117/2011‐9). MBA was funded through the FCT PTDC/AAC‐AMB/98163/2008 project and the Integrated Program of IC&DT Call no 1/SAESCTN/ALENT‐07‐0224‐FEDER‐00175. JH was supported by a Spanish DGCyT RyC Fellowship, by a Brazilian CNPq Visiting Researcher grant (400130⁄2010‐6), and by the CSIC‐CNPq cooperation project 2011BR0071.

Biosketch

Geiziane Tessarolo is a PhD student based at the Universidade Federal de Goiás, Brazil. Her research focuses on the assessment and evaluation of sources of uncertainty in species distribution modelling, particularly on data quality.

Thiago F. Rangel, Miguel B. Araújo and Joaquín Hortal are biogeographers studying the geographic distributions of species, among many other things.

Author contributions: G.T., J.H. and T.F.R. conceived the study; A.M.B. provided the data; G.T., J.H. and T.F.R. designed the analyses, with A.M.B.; G.T., J.H. and T.F.R. analysed the data; G.T., J.H., T.F.R. and A.M.B. wrote the paper.