1. The statistical rigour and interpretability of ecological assessments is strongly affected by how well we predict the biological assemblages expected to occur in the absence of human-caused stress, i.e. the reference condition. In this study, we examined how the specific method used to predict the reference condition affected the performance of two commonly used types of ecological index: RIVPACS-based O/E indices and multimetric indices (MMIs).
2. These two types of index have generally relied on different approaches to predicting the reference condition. For MMIs, some type of regionalisation is typically used to describe the range of metric values among reference sites and hence the expected range at assessed sites. For O/E indices, continuous modelling is used to predict how the biota varies among sites both among and within regions. Because the prediction method differs for these two types of index, it has been impossible to judge if differences in index performance (accuracy, precision, responsiveness and sensitivity) are caused by differences in the way reference condition biota are predicted or by differences in what the indices measure.
3. We used a common data set of 94 reference sites and 255 managed sites and the same potential set of predictor variables to compare the performance of five different MMIs and three O/E indices that were derived from different prediction methods: null models, multiple linear regression (MLR), classification and regression trees, Random Forests (RF) and linear discriminant functions models (LDM). We then calculated values of these indices for samples collected from the managed catchments as well as samples collected from 13 reference sites that were progressively altered in known ways by a simulation programme.
4. Both the type of predictor used and the type of index affected overall index performance. Modelled indices generally had the greatest sensitivity in assessing managed sites as biologically different from reference. Index sensitivity was determined by both an aspect of index precision (10th percentile of reference condition values) and responsiveness. The O/E indices showed the best scope of response to known biological alteration. All three O/E indices decreased linearly in response to simulated alteration in both overall assemblage structure (Bray-Curtis dissimilarity) and taxa loss. The MMIs declined linearly from low to intermediate levels of assemblage alteration but were less responsive between intermediate and high levels of biological alteration.
5. Insights gained from simulations can aid in testing assumptions regarding index response to stress and help ensure that we select indices that are ecologically interpretable and most useful to resource managers.