Climate change vulnerability for species – assessing the assessments

Climate change vulnerability assessments are commonly used to identify species at risk from global climate change, but the wide range of methodologies available makes it difficult for end users, such as conservation practitioners or policy makers, to decide which method to use as a basis for decision-making. Here, we compare the outputs of 12 such climate change vulnerability assessment methodologies, using both real and simulated species, and we test the methods using historic data for British birds and butterflies (i.e., using historical data to assign risks, and more recent data for validation). Our results highlight considerable inconsistencies in species risk assignment across all the approaches considered and suggest the majority of the frameworks are poor predictors of risk under climate change – two methods performed worse than random. Methods that incorporated historic trend data were the only ones to have any validity at predicting distributional trends in subsequent time periods.

projections of future risk. This approach has already been implemented to different extents 3 3 0 by some frameworks considered here 15,16,18 , although the outputs of these show at best 3 3 1 weak correlations with purely trait-based assessments, suggesting that trait-only 3 3 2 assessments may not adequately capture the exposure component of climate risk. The two 3 3 3 general types of assessment (trait, trend) effectively represent different paradigms, with 3 3 4 combined approaches representing arbitrarily-weighted blends of the two.

5
We have demonstrated that different vulnerability assessment frameworks should not be 3 3 6 used interchangeably when attempting to assess a species' potential future risk to climate 3 3 7 change, because assessments made with either real or simulated species produce 3 3 8 conflicting results. Our validation results suggest there is currently little evidence to support 3 3 9 the use of purely trait-based vulnerability assessments. Trend-based approaches are the 3 4 0 only type of methodology to consistently and significantly assign species to appropriate risk 3 4 1 categories in the validation analysis, particularly when this information is supplemented with 3 4 2 additional species trait data. Whilst we recognise this may restrict the assessment options 3 4 3 available to practitioners (e.g. without long-term monitoring data, trend-based approaches 3 4 4 will not be possible), our results highlight the considerable uncertainty in the results of 3 4 5 approaches not incorporating this type of information. A poorly performing framework should 3 4 6 not be used simply because it is the only one for which adequate data are available. Without 3 4 7 significant investment in long-term monitoring, to study change as it occurs, and in research 3 4 8 to identify exactly what traits make a species' vulnerable to climate change, our ability to 3 4 9 identify the species most in need of conservation attention in the face of climate change will 3 5 0 remain poor. The assessments of exemplar real species and additional British bird species (Table 2) were 3 5 5 carried out based on trait and distribution data within Great Britain, due to the quality and 3 5 6 availability of data for the taxa considered within this region. The 18 exemplar species were 3 5 7 chosen because they were the only species of any taxonomic group with both 3 5 8 comprehensive distribution (in two or more time periods) and traits data and a northern or 3 5 9 southern range margin lying within Great Britain 37 (species with range boundaries in a region 3 6 0 are likely to be of interest when running climate change vulnerability assessments). All 3 6 1 common British breeding bird and butterfly species were considered for the additional 3 6 2 assessment, the 218 species selected being the ones for which future distributions could be 3 6 3 modelled based on data availability.

6 4
Trait data for the real species were collected from a variety of sources including scientific 3 6 5 literature and species atlas data 38,39 . Projected distribution changes were based on existing 3 6 6 bioclimate model data 17 , applying a Bayesian, spatially explicit (Conditional Autoregressive) 3 6 7 GAM 40 to the bird and butterfly distribution data. A medium emissions scenario (UKCP09 3 6 8 A1B) for projected climate change for 2080 was used for future climate data, corresponding 3 6 9 to a 4°C increase in average temperature. The assessments were also run using a low 3 7 0 emissions scenario (UKCP09 B1), corresponding to a 2°C increase in average temperature, 3 7 1 with little difference in overall risk category assignment (Supplementary Table 4). To compare the outputs of the 12 risk assessment frameworks using simulated species, we 3 7 4 generated ranges of values for the 117 unique input variables (Supplementary Table 1),  3  7  5 covering characteristics such as species traits and population trends. We then drew values 3 7 6 for each of these input variables to generate 10,000 combinations of 'trait sets' that were 3 7 7 used as simulated species in the assessments, in lieu of real world data for many species. 3 7 8 Where it has been possible to do so, we applied constraints on the input variables to ensure 3 7 9 logical consistency. For example, in the case of interspecific interactions, some frameworks 3 8 0 ask broadly whether there is a dependence of a species on any interspecific interaction, 3 8 1 whilst other frameworks require inputs relating to multiple, clearly-defined interspecific 3 8 2 interactions. In this situation it would not make sense for the broad interaction to be scored 3 8 3 as absent while specific interactions are scored as present. In this case the broad interaction 3 8 4 is generated first and the scores of more specific interaction variables are influenced by that, 3 8 5 to ensure consistent inputs across frameworks. 3 8 6 For continuously distributed input variables, upper and lower bounds were set based on 3 8 7 reported values from the literature (e.g. body size, generation time) or theoretical minimum 3 8 8 and maximum values. A value for the variable for each simulated species was then drawn 3 8 9 from a uniform distribution bounded by those upper and lower limits. Species current 3 9 0 distributions were simulated using the same approach, sampling a value for area occupied 3 9 1 (in km 2 ) from a uniform distribution with an upper limited based on known real world 3 9 2 distribution limits. For projected changes to species distributions under climate change, a 3 9 3 future distribution was generated using the same process as for current distributions, and the 3 9 4 percentage change in area between the two calculated. 3 9 5 The uniform distribution was chosen for all variables (equal probability for binary and 3 9 6 categorical variables) because, for many input variables, there was little or no data available 3 9 7 on how they might be distributed in reality (and they differ greatly between taxonomic 3 9 8 groups), so an arbitrary selection of distribution would have been needed. Nonetheless, 3 9 9 where there was an a priori expectation of the distribution of a trait based on the literature 4 0 0 (e.g. logarithmic scaling of dispersal distance), the uniform draw was from between the 4 0 1 transformed trait limits. The uniform distribution also allows for generation of traits covering 4 0 2 the full range of the potential parameter space for the input variables, which was one of the 4 0 3 main advantages of generated trait sets rather than a larger sample of real species data. 4 0 4 The results therefore test consistency in framework performances, rather than the 'true' 4 0 5 frequencies of risk (which we do not know, given the differences between framework 4 0 6 methods). 4 0 7 Many of the input variables are categorical, typically scored as low/medium/high or some 4 0 8 similar variation. In some cases it is possible to base these on a continuous variable which is 4 0 9 then split into the different categories (e.g. dispersal distance < 1km scored as low, dispersal 4 1 0 distance > 1km and < 10km scored as medium, dispersal distance > 10km scored as high).

1 1
Where it has not been possible to generate a continuous variable to base the categorical 4 1 2 split on (e.g. impact of climate mitigation measures -scored as low to high), the category 4 1 3 was instead assigned randomly to one of the possible options, with an equal probability of 4 1 4 assignment to each. IUCN Red List conservation status was required as an input to one of 4 1 5 the frameworks and was generated using IUCN criteria A to D, with no projected future 4 1 6 changes considered. This conservation status for each simulated species was also used in 4 1 7 comparisons of Red List risk category against risk category for each framework, and 4 1 8 therefore informs us of the relationship between climatic and non-climatic risks rather than 4 1 9 whether the Red List could adequately take climate change into account. To examine how well the different climate vulnerability assessments performed at projecting 4 2 2 future risk we used the results of assessments based on historic species data to compare 4 2 3 against observed recent trends in species distribution/abundance. For validation of the 4 2 4 frameworks to produce robust results they need to be tested using reliable input data, poor 4 2 5 quality input data will always lead to poor assessments of risk regardless of the method used 4 2 6 for the assessment. We therefore utilized some of the best quality data available globally 4 2 7 and selected British birds and butterflies for the analysis. Validations were carried out by using historically-available data to assign species to low-, 4 2 9 medium-and high-risk categories (for each of the 12 risk assessment frameworks), as 4 3 0 though the assessments were carried out in the past, and then we compared recent 4 3 1 distribution and population changes for species that had been assigned to each risk 4 3 2 category. Assessments for British birds were based on the time period 1988-1991, to match 4 3 3 the breeding bird atlas data 41 . Assessment inputs based on the 'then-current' 4 3 4 distribution/population were calculated from this Atlas data, with historic changes in Although these dates partly overlap with the Millennium Butterfly Atlas 39 , the population data 4 4 9 are collected on fixed transects that are separate from the millions of independent 4 5 0 distribution records that give rise to the Atlas maps. Distribution change data for the 4 5 1 butterflies was not used in the analysis due to a large increase in observer effort in latter 4 5 2 time period, which resulted in increases in distribution that are likely to reflect increased 4 5 3 effort rather than true changes in distribution.

5 4
Statistical analysis 4 5 5 The risk category outputs from each of the frameworks were converted to a set of 4 5 6 standardised categories: Low/Medium/High risk (Supplementary Table 2). Broad agreement 4 5 7 between the frameworks was tested on a pairwise basis using Spearman's rank correlation, to establish how consistently species were assigned to the same Low/Medium/High risk 4 5 9 categories by the different frameworks. 4 6 0 Rank correlation allows for a comparison of how well the different frameworks correspond 4 6 1 across all levels of risk assignment, but a potentially more useful comparison is of how well 4 6 2 they agree in identifying a species as high risk, based on the assumption that assessments 4 6 3 will primarily be run to identify the species most vulnerable to climate change. To compare 4 6 4 agreement on just high risk species, the risk categories were further simplified to a binary, 4 6 5 'low and medium' versus 'high' categorisation. Cohen's kappa, a measure of inter-rater 4 6 6 reliability, was calculated to compare agreement between frameworks. The prevalence and 4 6 7 bias-adjusted Cohn's kappa (PABAK) 44 was used due to the relatively low frequency of 4 6 8 species scoring as high risk.

6 9
Principal component analysis (PCA) was used to examine how much of the variation in risk 4 7 0 assignment was influenced by certain frameworks and to identify whether frameworks of the 4 7 1 same general type (trait, trend) showed similar patterns in risk category assignment. Risk 4 7 2 category outputs from each framework for the 10,000 simulated species were used in this 4 7 3 analysis.

7 4
We predicted that all species at high risk due to climate change should have seen 4 7 5 population/distribution decreases, whilst species identified as low risk may have increased, 4 7 6 decreased or not changed their population/distribution if factors other than climate are 4 7 7 driving the changes. We therefore used quantile regression to validate framework 4 7 8 performance, with change in distribution or abundance as the response variable and 4 7 9 framework risk categorisation (Low/Medium/High) as the predictive factor 45 . This allowed us     A  t  l  a  s  2  0  0  7  -1  1  :  t  h  e  b  r  e  e  d  i  n  g  a  n  d  w  i  n  t  e  r  i  n  g  b  i  r  d  s  o  f  B  r  i  t  a  i  n  a  n  d  5  8  4  I  r  e  l  a  n  d  .  (  B  T  O  T  h  e  t  f  o  r  d  ,  2  0  1  3  )  .  5  8  5  3  9  A  s  h  e  r  ,  J  .  e  t  a  l  .  T  h  e  m  i  l  l  e  n  n  i  u  m  a  t  l  a  s  o  f  b  u  t  t  e  r  f  l  i  e  s  i  n  B  r  i  t  a  i  n  a  n  d  I  r  e  l  a  n  d  .  (  O  x  f  o  r  d  U  n  i  v  e  r  s  i  t  y  5  8  6  P  r  e  s  s  ,  2  0  0  1  )  .  5  8  7  4  0  B  e  a  l  e  ,  C  .  M  .  ,  B  r  e  w  e  r  ,  M  .  J  .  &  L  e  n  n  o  n  ,  J  .  J  .  A  n  e  w  s  t  a  t  i  s  t  i  c  a  l  f  r  a  m  e  w  o  r  k  f  o  r  t  h  e  q  u  a  n  t  i  f  i  c  a  t  i  o  n  5  8  8  o  f  c  o  v  a  r  i  a  t  e  a  s  s  All authors conceived and designed the study; CJW, CMB, CDT and RC collected data; 6 0 4 CJW and CMB analysed the data; all authors interpreted the results; CJW produced the 6 0 5 original draft and all authors contributed to revisions 6 0 6 Figure 1. Frequency distribution of high risk classifications for a} simulated species and b).real species assessed with historic data. The number of risk assessment frameworks under which each simulated or real species was classified as high risk.

Figure 2. Correlation matrix showing spearman rank correlation coefficients (r s ) for each of the 12 frameworks, pairwise against the others and the Red List outputs for the simulated species.
The matrix is a visual representation of the r s value (see x axis for range), with darker blue indicating a stronger positive correlation; using output data for the 10,000 simulated species. The correlations between each of the climate change risk assessment frameworks and the simulated Red List risk category are shown in the bottom row of the matrix. Reference numbers are as in Table 1.      Table 3. Summary validation trends. The direction of the trend in either distribution or abundance change for birds and butterflies from Low risk species to high risk species, with a negative trend indicating the framework is performing as expected and a positive trend indicating poor framework performance. Significant trends are denoted with *. The frameworks are ranked first by number of significant negative trends and then by number of non-significant negative trends.