*Ecology Letters* (2011) **14**: 828–839

**Ecology Letters**

# Species coexistence in a variable world

E-mail: dominique_gravel@uqar.qc.ca

## Abstract

### Abstract

The contribution of deterministic and stochastic processes to species coexistence is widely debated. With the introduction of powerful statistical techniques, we can now better characterise different sources of uncertainty when quantifying niche differentiation. The theoretical literature on the effect of stochasticity on coexistence, however, is often ignored by field ecologists because of its technical nature and difficulties in its application. In this review, we examine how different sources of variability in population dynamics contribute to coexistence. Unfortunately, few general rules emerge among the different models that have been studied to date. Nonetheless, we believe that a greater understanding is possible, based on the integration of coexistence and population extinction risk theories. There are two conditions for coexistence in the presence of environmental and demographic variability: (1) the average per capita growth rates of all coexisting species must be positive when at low densities, and (2) these growth rates must be strong enough to overcome negative random events potentially pushing densities to extinction. We propose that critical tests for species coexistence must account for niche differentiation arising from this variability and should be based explicitly on notions of stability and ecological drift.

## Introduction

The ecological niche is a fundamental mechanism to explain species coexistence, but its importance is currently debated because elevated variability in the field often blurs species differences. Niche differentiation implies that species respond differently to their biotic and abiotic environments (Chase & Leibold 2003). There are thousands of examples of this phenomenon in the ecological literature, starting with the early work of naturalists (e.g. Grinnell, 1917; Whittaker 1956; MacArthur 1958), showing that species do indeed differ in their responses to biotic and abiotic conditions. Coexistence from niche differentiation occurs when species are sufficiently different to reduce interspecific competition below intraspecific competition (Lotka 1932; MacArthur & Levins 1967; see reviews in Chesson 2000a; Adler *et al.* 2007). An important empirical question is to what extent observed differences in interspecific and intraspecific competition actually explain species coexistence and diversity patterns (Hubbell 2001; Silvertown 2004). Field studies commonly show elevated, unexplained variability in species traits (e.g. Pacala *et al.* 1996; Albert *et al.* 2010; Clark *et al.* 2010). Moller & Jennions (2002) conducted a meta-analysis on the variance explained by studies in ecology and evolutionary biology and found that more than 80% of *R*^{2} values were less than 10%, with a mean of 5.42%. If two species were niche-differentiated then, ideally, only a simple statistical comparison of their average traits would be needed to distinguish them. But in reality there are numerous problems associated with this approach because of multiple sources of variability in measured trait values. In many cases, sufficient sample sizes will produce a statistically significant difference between estimated species traits, albeit with a low explained variance and large overlap between species. Do species coexist due to niche differentiation despite high variance and species overlap? Or rather, does such noise contribute to species coexistence?

Classical niche theory predicts that two sufficiently similar species cannot stably coexist due to the process of competitive exclusion (Gause 1934; Hardin 1961; MacArthur & Levins 1967). Over the past decade, the competitive exclusion principle has been challenged by neutral models in which demographic stochasticity (see *Glossary*) plays a predominant role in shaping community structure of ecologically equivalent species. Local coexistence in such neutral models of biodiversity is maintained as a balance between immigration and extinction due to ecological drift (Hubbell 1997, 2001; Bell 2000). Neutral theory has been successful at predicting common diversity patterns such as relative abundance distributions (Bell 2001; Hubbell 2001; Volkov *et al.* 2003, 2007; Etienne 2005), species-area curves (Bell 2001; Hubbell 2001; O’Dwyer & Green 2010), and may explain numerous observations of apparently similar species coexisting (Saez & Lozano 2005; Bickford *et al.* 2007). However, others have argued that stochasticity in some traits mask complex, multi-dimensional trade-offs that are essential for understanding species coexistence and that neutrality is just an artefact of incomplete knowledge (Clark *et al.* 2007, 2010; Clark 2009, 2010).

These conflicting views call for a general interpretation of how stochasticity affects coexistence through its joint effect on niche differentiation and ecological drift. The theoretical literature on coexistence and stochasticity is often ignored by field ecologists because of its impenetrability and difficulties in its application. Our intention is to provide tools to make this body of theory accessible to community ecologists. The article is structured as follows. First, we discuss the concepts and definitions surrounding stochasticity and stable coexistence. We show that different traditions have used the term stochasticity for (slightly) different purposes and we propose a more inclusive approach to the multiple levels of variability affecting niche differentiation and community dynamics. Adler & Drake (2008) were the first to our knowledge to reconcile coexistence theory with extinction risk theory. We build upon this work and adopt two mathematical tools – nonlinear averaging and extinction risk estimation – to better understand the consequences of variability on coexistence. Using these tools, we explain how variability at different levels of organisation can be explicitly integrated into current coexistence theory. We then look at how ecological drift may emerge despite niche differentiation and provide novel insights to the niche-neutrality debate (Tilman 1982; Gravel *et al.* 2006; Adler *et al.* 2007). Our synthesis leads to new challenges for the empirical study of coexistence.

## Definitions

Population dynamics in natural systems are often approximated as a stochastic process due to unpredictable environmental variation (May 1973). Theoreticians typically distinguish two broad sources of stochasticity in population dynamics (Lande *et al.* 2003), but recently, finer distinctions have been proposed and analysed (Fox & Kendall 2002; Kendall & Fox 2003; Robert *et al.* 2003; Melbourne & Hastings 2008). *Environmental stochasticity* refers to fluctuations in time and/or space in birth and/or death rates, owing to environmental variability affecting traits. *Demographic stochasticity* refers to fluctuations arising from the probabilistic nature of individual birth and death. Demographic stochasticity is distinguished from the variability emerging from *demographic heterogeneity* (Melbourne & Hastings 2008), where individuals of different sizes, ages or genotypes experience different local environmental conditions and thus exhibit different traits. Together, demographic heterogeneity and demographic stochasticity contribute to the total *demographic variance* (Melbourne & Hastings 2008).

It is important to identify what makes an ecological phenomenon variable. Clark (2009) has emphasised that what we call ‘stochastic’ masks two essential components of uncertainty: imperfect knowledge of species responses to the environment and inadequate characterisation of the exact environment. The former could be reduced with careful observations and appropriate statistical models. But even if species’ response to a given environment were completely known, then the ecological dynamics would be modelled as a stochastic process should the environment itself be probabilistic (e.g. year to year variability in weather conditions – see Box 1). The resulting variability would be found at all levels of organisation, as we progress from individuals to populations. Some processes are not completely predictable (e.g. chaos), but are not stochastic because the same initial conditions always produce the same temporal trajectories (May 1974; Strogatz 1994). Stochastic dynamics (i.e. where the same starting conditions yield different time course trajectories) may emerge, either because the environment is exogenous and has a variable impact on all individuals, or because demographic events are variable at the level of single individuals. In either case, it is imperfect knowledge of the response to the varying environment that limits predictability. In this review, we will employ ‘variability’ as a general term, encompassing stochastic and deterministic sources of variation in the response to the environment across levels of organisation, from individuals to communities. As we will see below, in many contexts it is not chance *per se* that promotes or impedes coexistence, but rather the deterministic and differentiated responses to a variable environment. We will therefore use the term ‘stochasticity’ to refer specifically to probabilistic sources of variation and ‘deterministic variability’ to refer to deterministic responses to environmental variability (at both the individual and the population levels).

To illustrate these different sources of variability, consider an example based on plant population dynamics (Clark *et al.* 2007). For many sessile organisms such as plants, recruitment into an empty location following the death of an adult could be described as a random draw of one seed from the seed rain. Thus, seedling recruitment is often considered as probabilistic (Chesson & Warner 1981; Hubbell 2001). Following recruitment, seedlings could die from different causes, each with an associated probability. Suppose that for a given time frame, seedlings systematically die should they be attacked by an herbivore. Because attack itself has a random component of chance encounters, even if the herbivore and seedling populations are constant over time, there will be variability in the number of seedlings each herbivore consumes. The variability in the proportion of seedlings attacked is a form of demographic stochasticity and generates stochastic dynamics when population sizes are small (Lande *et al.* 2003). Demographic heterogeneity occurs should seedlings of a species respond differently to the herbivore, for instance because of differential resistance between genotypes, or differential nutritional status across locations. Finally, environmental variability could affect recruitment if for instance the environment (e.g. temperature) drives herbivore population density and this, in turn, adds variability over time and/or space to the proportion of seedlings attacked. Deterministic causes of both demographic and environmental variability contribute to deterministic variability.

To increase the predictability of an ecological phenomenon and the identification of its drivers, it is important to clearly distinguish its stochastic and deterministic components. Consider again the above lottery example of a community of two coexisting plant species. Suppose herbivore density varies over time because of temperature variation and influences the demography of only one plant species. This deterministic variability could be accounted for in two different ways. First, one could incorporate it as an additive random variable to the term describing population growth of each plant species (see Box 1). This will lead to maximal uncertainty in the prediction (Fig. 1). Second, temporal variation in herbivore density could be included as a covariable in the functions describing community dynamics (Box 1), shifting the variability from the error term to the deterministic description of population growth (Clark 2009). With even finer detail, one could account for the deterministic response to environmental variability. Uncertainty can thus be considerably reduced, but (evidently) not the variability in community composition (Fig. 1).

## Box 1 Should Population Dynamics be Approximated as a Stochastic Process?

Clark (2009) provides an interesting perspective on the roles of deterministic biological process vs. stochastic approximations of unknown ecological drivers. He formulated a very general statistical model of an ecological process – here population growth – as:

The first term on the right-hand side is deterministic and explanatory, while the second (error) is responsible for uncertainty and represents variation in the response not accounted for by the first term. Clark (2009) argued that *One way to view progress in science is what occurs when variation moves from the second term (unknown) to the first term (known)*. He also argued that proponents of the neutral theory *advocate the movement in the opposite direction: strip away known processes and focus on rejecting a model that is predominantly stochastic […]*.

Pushing this reasoning to the extreme, where we have perfect knowledge of the complex biotic and abiotic environments of a species, all uncertainty shifts to the first term of the equation. For population growth, we would have the resulting, complex equation (reminiscent of Chesson 1994):

where *C* is a function describing the response to the biotic environment and *E* the response to the abiotic environment, n(x,t) is a vector defining the complex structure of species interactions at location x and time t, and k(x,t) is a vector defining the complex structure of the abiotic environment. With this function and perfect knowledge of C and E, one could predict dynamics and the outcome of competition (e.g. Fig. 1).

In natural systems, such perfect knowledge is obviously impossible. The question then is can we still make enough headway in characterising eqn B1.2 to ‘accurately’ define the niche and disentangle the effects of species traits, environment and their interactions? The goal is to attribute the relative impacts of species traits and environmental forcing on coexistence. Both traits and forcing can have deterministic and stochastic components, and we expect that it is *complexity* in species traits, interactions and the environment that inflates the error term for eqn 1. Complexity, in turn, may be intrinsic to the processes themselves (i.e. higher order dimensionality) and/or emerge from our limited ability to identify and measure the key variables driving population growth rates and coexistence.

Characterising the structure of variability in an ecological process is a first step. The second one is to understand the average behaviour of all populations, given the distribution of environmental conditions. Even if we assess all biological parameters, exogenous environmental parameters remain, at least in part, unpredictable. In this case, the tools of nonlinear averaging and diffusion approximations (May 1973; Turelli 1978; Chesson 1994) become useful to model the average dynamics of the species and eventually interpret species coexistence.

## The effects of variability on community dynamics

The literature is replete with specific community models introducing variability at different levels of organisation. For example, Begon & Wall (1987) studied a discrete Lotka-Volterra model of competition with intraspecific variation in the competition coefficients (demographic heterogeneity) and found that it promotes coexistence. Anderies & Beisner (2000) studied a model of phytoplankton competing for a single fluctuating resource and found that demographic heterogeneity alone is far more important in increasing the propensity for coexistence than environmental variability. Vellend (2006) examined a discrete Lotka-Volterra model of competition and an individual-based model of competition and found that, for both, the number of genotypes per species (demographic heterogeneity) promotes community diversity via slow and transient drift. Lichstein *et al.* (2007) showed how the level of demographic heterogeneity could result in either coexistence, or exclusion, or drift. Due to the specific questions examined by these models, they do not, however, easily lend themselves to generalisation.

A general understanding is nonetheless now possible, based on the integration of coexistence and population extinction risk theories (Adler & Drake 2008). Traditional coexistence theory states that two species coexist stably when each is able to invade the other at equilibrium, or in other words all species have a positive per capita growth rate when at low abundance. If for whatever reason one species is driven close to extinction by a disturbance, then it will recover despite interspecific competition. The higher the per capita growth rate of all coexisting species when each is at low abundance, the more stable the coexistence (i.e. the shape of the cup in the classic ball and cup analogy, Fig. 2; see also Murdoch *et al.* 2003). Both stochastic and deterministic variability could generate such disturbances, pushing the community away from its expected density (i.e. the distance to the equilibrium point, Fig. 2). If such disturbances are sufficiently frequent relative to the time it requires to reach equilibrium, then a species will be pushed to extinction by successive negative events. There are thus two conditions for coexistence (Adler & Drake 2008): (1) the average per capita growth rates of all coexisting species must be positive when at low densities, and (2) these growth rates must be strong enough to overcome negative random events potentially pushing densities to extinction. Here we present two simple and general tools to understand the effect of variability on these two conditions. They are sufficiently general to account for stochastic and deterministic sources of variability.

### Long-term average growth rate

Assessing coexistence requires estimation of long-term average growth rates at low abundance (Chesson 1994). This poses the question of whether average environmental conditions and demographic traits can be used to assess coexistence. A problem arises in that the functional form of most ecological models describing growth is nonlinear. Jensen’s inequality is a well-established principle that states that the average of a nonlinear function is different from the function of the average (see Ruel & Ayres 1999 for a discussion of its application to ecology).

To illustrate this, consider fluctuations in the long-term growth rate of a geometrically growing population. The growth function for a given time step is linear, but the long-term growth function is nonlinear. The function giving the population size at time *t* in a discrete time model with annual growth rate is . After 2 years of growth, the population size is . If, for example the annual growth rate is constant and equals 1.1, then we have *N*(3) = 1.21*N*_{0}. Instead, with environmental variability driving the growth rate, but with the same average such as and , we obtain a negatively growing population *N*(3) = 0.96*N*_{0}. More generally, for a stochastic variable and a nonlinear concave function, the average of the function will be larger than the function of the average, while for a convex function, it will be smaller (further examples of the Jensen’s inequality are provided in Box 3). Because the per capita growth rate at low abundance often involves a nonlinear term (see Table 1), if a varying quantity affects this rate, then we must assess its long-term average with the nonlinear averaging technique.

Model of interspecific competition | Varying parameter | Sign of the second derivative of the expected per capita growth rate of the invader with respect to the varying parameter | |
---|---|---|---|

Invader | Resident | ||

The reported effect indicates that introducing variability in the parameter of interest either increases (+), decreases (−) or does not affect (0) the invader’s expected per capita growth rate relative to the deterministic situation. The effects are presented for cases where the invader and resident traits vary.
| |||

Lotka-Volterra | intrinsic rate of increase | 0 | 0 |

interspecific competition coefficient | 0 | 0 | |

carrying capacity | − | 0 | |

Lottery model | death rate | 0 | 0 |

per capita fecundity | 0 | + | |

Annual plant model | intrinsic rate of increase | 0 | + |

intraspecific competition coefficient | 0 | − | |

interspecific competition coefficient | 0 | + |

The second order Taylor expansion is a powerful technique to approximate the long-term average of a nonlinear response describing the per capita growth rate (see Chesson 1994 and Kuang & Chesson 2009 for application to temporal environmental variability and Chesson 2000b and Chesson *et al.* 2005 for spatial variability). The approximated average of a nonlinear function of a variable *x*, *F*(*x*), is:

And its variance is:

This approximation is adequate as long as the variance of the perturbations around the mean is not too large (coefficient of variation < 30%; Lande *et al.* 2003). The sign of the second derivative *F*′′(*x*), evaluated at equilibrium and for average conditions, determines whether the average of the function is larger or smaller than the function of the average (see Box 3). This is of central importance in assessing the impacts of any kind of variability on the expected per capita growth rate of an invader.

For situations with multiple sources of variability, we must also account for covariance between randomly fluctuating variables. More generally, the average of a function with multiple stochastic variables incorporates corrections for nonlinearities, plus a term related to the covariance between the *N* fluctuating variables *x*_{n}:

The first term is the function evaluated under average conditions (denoted by *). The double summation contains all second order cross-partial derivatives multiplied by the covariance between the variables *x*_{l} and *x*_{n}. Equation 3 thus shows that the covariance between two variables could also affect the invader’s growth rate. This exemplifies the need to better characterise multi-dimensional trade-offs (Clark *et al.* 2007).

To better illustrate this principle, take for example the lottery model of Chesson & Warner (1981) used to develop the storage effect theory. Consider two species, denoted 1 and 2, competing for vacant space following the random death of adults. The recruitment probability of a species is simply its relative abundance in the pool of seeds reaching the vacant site. The dynamics of species 1 are given by:

where *d*_{i} is the death rate and *f*_{i} is fecundity, which varies over time. The left-hand side term describes the mortality, while the right-hand side the lottery recruitment. For simplicity, assume the death rate is the same for both species and that species 2 is the superior competitor because of a larger average fecundity (). The per capita growth rate of species 1 when an invader is approximated as (Chesson & Warner 1981):

It is straightforward to calculate this growth rate and to find that species 1 cannot invade in the absence of any variability (the per capita growth rate is smaller than 1). The situation is, however, different if the fecundity of the resident species 2 fluctuates over time because of a deterministic response to environmental variability. The long-term average per capita growth rate would then be approximated using eqn 1 as:

The second derivative of eqn 5 with respect to *f*_{2} is positive (eqn 6), meaning that any variability in fecundity of species 2 will increase the growth rate of species 1 when at low abundance (the right-hand side of eqn 6). Note that the growth rate of a species is neither sensitive to temporal fluctuations in its own fecundity (i.e. the second derivative of eqn 5 with respect to f_{1} is zero), nor to the death rate. In other words, for species 1, only fluctuations in the fecundity of species 2 will affect the long-term average growth rate. This deterministic variability does not impede species 2’s growth rate when at low abundance and therefore it will promote coexistence if the variability is large enough (i.e. when the right term becomes larger than the central term).

### Population extinction risk

The invasibility analysis is not, in itself, sufficient to assess the effect of different types of variability on coexistence, because it ignores its effect as a perturbation to population density. Invaders are at low density by definition. When at low abundance, stochastic and deterministic variability might push a species to extinction despite a long-term average positive growth rate (Lewontin & Cohen 1969). Moreover, for communities that do not recover sufficiently quickly after a disturbance, variability could emerge as the main driver of persistence (Box 2).

The theory of stochastic population dynamics has been useful to conservation ecologists as a basis for population viability analysis (Boyce 1992). Early studies investigated the effects of demographic stochasticity and deterministic variability on mean growth rates, extinction risks and population distributions (e.g. Bartlett 1960; Lewontin & Cohen 1969; May 1973; Roughgarden 1976; Gabriel & Bürger 1992; Lande 1993; Ludwig 1996). For example, Lande (1993) showed that demographic stochasticity has almost no impact on population dynamics as population size becomes sufficiently large. The deterministic response to environmental variability may also affect persistence, and in particular if there is a temporal autocorrelation (Johst & Wissel 1997– but see Halley & Kunin 1999; Heino *et al.* 2000) or large fluctuations (Ovaskainen & Meerson 2010). These findings build on the relative importance of stabilising and destabilising aspects of stochastic population growth (May 1973; Lande 1993). May’s (1973) analysis indicates that as inter- and intraspecific competition coefficients become more similar, community stability decreases, and the influence of environmental variability on community dynamics dominates (see Box 2).

Analytical solutions for stochastic community models are often difficult to obtain. At low population densities, however, density-independent models approximate an invader’s dynamics and extinction is a possible outcome (Lewontin & Cohen 1969). Lewontin and Cohen showed that assessing extinction risk requires estimation of (1) the invader’s per capita average growth rate and variance and (2) the probability that an invader starting with *N*_{0} individuals at time *t* has a lower density after *T* years. After considerable algebra, they demonstrated that the extinction probability is simply given by the integral of the normal distribution *G*(*μ*_{x}, ) of growth rate *x* with mean *μ*_{G} and variance :

It is not necessary to solve this integral to understand the basics of how variability affects the extinction probability of an invader: for a given time interval and starting abundance *N*_{0}, the probability of extinction (1) increases as the average intrinsic per capita growth rate tends towards zero, and (2) is a saturating positive function of the variance in the per capita growth rate.

To interpret the consequences of variability on extinction probability, consider again the above lottery example. The mean growth rate *μ*_{G} in eqn 7 is given by eqn 6 and the variance is calculated using eqns 3 and 5. Extinction probability can then be found using eqn 7. Although conceptually simple, analytical solutions are too complex to be informative. A numerical example is shown instead in Fig. 3, illustrating the complex interplay between deterministic variability at the population level, the long-term average growth rate, and extinction risk. As shown above, the long-term average growth rate of the inferior competitor increases with temporal fluctuations in fecundity. Figure 3 also illustrates the emerging trade-off between the long-term average growth rate and the extinction probability when at low abundance: both increase with temporal fluctuations in fecundity. Even if such deterministic variability in fecundity has been shown to promote coexistence in large populations (Chesson & Warner 1981), it could promote random extinctions and thus impede coexistence in smaller populations (see Adler & Drake 2008 for an extensive study of the discrete lottery model). This example therefore shows that we could not predict the effect of different types of variability on coexistence, based exclusively on either the growth rate or the risk of extinction.

## Box 2 Variability and ecological drift

The expected per capita growth rate of a species when at low abundance (*r*_{i}) can be generally represented as function of the environment *E* and competition from residents *C*:

The first term determines the stability of coexistence, that is, the ability of a system to recover from perturbations. The second term represents random fluctuations. These could be caused by stochastic and deterministic variability. The first term could be calculated easily for simple models employing eqns 1–2 and then eqn 7 to assess extinction probability. According to Lewontin & Cohen (1969), the extinction probability of an invader decreases with and increases with *ɛ*. Thus, a population could go extinct despite a high positive growth rate, owing to a deterministic response to strong environmental fluctuations (May 1973– see also Fig. B2.1a).

Ecological drift is the deviation from expected dynamics; that is, when the right term in eqn B2.1 is large relative to the left term. An efficient metric for field ecologists to quantify drift is the variance in species composition between independent communities, exposed to the same average environmental conditions (Clark & McLachlan 2003). This variance is maximal when there is systematic fixation (extinction or dominance) and all species have equal chances of fixation. We illustrate this in Fig. B2.1 with simple simulations of the discrete Lotka-Volterra model of competition (see Table 1), with an additive source of environmental fluctuations (as in eqn B2.1). It shows that the variance between replicated runs increases over time as species become more similar (*α*_{ij} → *α*_{ii} = 1). The variance quickly stabilises for a highly stable community (*α*_{ij} = 0.5), while it increases constantly over time for a weakly stable one (*α*_{ij} = 0.95).

- (B2.1)

[ Ecological drift in a stable two species discrete version of the Lotka-Volterra model of competition. Environmental variability is introduced by independent normal random deviates to the carrying capacity of each species (mean of 0, standard deviation of 0.2). (a) Time series of species A’s relative abundance for weak (*α*_{ii} = 1 and *α*_{ij} = 0.5) and strong (*α*_{ii} = 1 and *α*_{ij} = 0.95) interactions. (b) Variability in species A’s relative abundance after 1000 time steps, based on 1000 replicated runs and as a function of the strength of interspecific competition. Maximal variability occurs when there is systematic fixation (one species dominates and the other goes extinct). Parameters: *r*_{1} = *r*_{2} = 1.25, *K*_{1} = *K*_{2} = 1. ]

## Variability across Levels of Organisation

So far our approach has been general and largely independent of the different types of variability and level of organisation. The tools we propose are general enough to encompass stochastic and deterministic variability at different levels of organisation. However, to apply them one must understand how variability translates across levels, from individuals to populations. More specifically, we need to predict the net combined effect of variability across levels of organisation on coexistence. Below, we provide several simple examples, building upon previous analyses of the lottery model, to illustrate how this could be achieved.

### Example 1. Demographic heterogeneity arising from phenotypic variability in fecundity

We should expect in nature that individuals differ in the number of offspring they produce and thus a simple way to introduce demographic heterogeneity into the lottery model is to consider that fecundity is a random variable, with no specific hypothesis as to the origin of its variability. We have seen above that some variability in the fecundity of a resident species will increase the per capita growth rate of an invading species (i.e. the ‘storage effect’, Chesson & Warner 1981). Understanding the source of this variance is thus a critical step in understanding coexistence.

So far, we have considered that fecundity could fluctuate over time, due for example to a species-specific deterministic response to temporal environmental variability. However, fecundity could also vary because of inter-individual differences (demographic heterogeneity). Consider the simple case where the fecundity of individual *k* of species *i* is the average for the species *μ*_{i} plus a random variable *ɛ*_{ik}. Thus, the average fecundity over all *N* individuals in the population is

and based on previous findings from stochastic population dynamics (Lande *et al.* 2003), the temporal variance of the fecundity is:

Here corresponds to variability arising from a deterministic response to environmental variation affecting the whole population and to the variability arising from demographic heterogeneity. This equation tells us that the effect of demographic heterogeneity on the variance of fecundity will precipitously decrease with population size. Because it does not influence much the variance in fecundity, demographic heterogeneity should thus not greatly influence community dynamics and coexistence (Chesson 1985). This result is, however, not general, as we will see with the next example.

### Example 2. Demographic heterogeneity arising from spatial environmental variation

Now suppose that we want to be more specific in the definition of demographic heterogeneity because field data tells us that it arises from spatial environmental variation. For this, consider the lottery model, examining the recruitment probability of species *i:*

Species *i* will grow if this quantity is larger than its relative abundance ().

The lottery model is implicitly spatial, but could be modified to encompass spatial variability in an environmental factor *E*, which, in turn, will influence spatial variation in recruitment probability. If the local environment modifies recruitment, for instance because soil pH impacts the seedling germination rate *g*_{ix}, then the local recruitment probability *P*_{ix} becomes:

For spatial variability in the environment, the recruitment probability should be treated as a random variable and its spatial average calculated using the above-described technique of nonlinear averaging. In the seedling germination example, the impact of spatial variability of recruitment probability will depend on the function relating germination rate to the environment. If we consider, for instance, a Gaussian-shaped function to describe the germination niche *g*_{i}(*E*_{x}), then it can be shown that the second partial derivative of eqn 10 relative to the spatially varying environmental factor *E* is positive (the solution is, however, too complex to be informative). This result echoes classic niche theory (MacArthur & Levins 1967) and similar lottery models with spatial variability in recruitment (Mouquet and Loreau, 2002; Snyder and Chesson, 2003; Gravel *et al.* 2006; Vellend 2006; Lichstein *et al.* 2007): for this specific model, demographic heterogeneity generated by a differential response to local environmental conditions promotes species coexistence.

### Example 3. Demographic heterogeneity arising from genotypic variability in the niche optimum

Assume now that demographic heterogeneity is the result of genotypic variation in the niche optimum (e.g. Vellend 2006; Lichstein *et al.* 2007). Here, the function *g*_{i}(*E*_{x}) would itself be a random variable and its expectation calculated using nonlinear averaging. The solution is again too complex to be informative, but an example is provided in panel c of Box 3. It shows that genotypic variability in the niche optimum reduces performance close to the optimum and increases it far from the optimum (see also Lichstein *et al.* 2007). The result will be increased niche overlap and thus a more limiting similarity (MacArthur & Levins 1967).

These examples indicate the need to understand how variability at one level of organisation impacts the average and variance at the next level of organisation. As such, demographic heterogeneity can have any one of the three principal effects on coexistence: no effect (example 1), promotion of niche differentiation (example 2) or reduction in niche differentiation (example 3). Despite the often conflicting and diverse literature on the subject (see above), these examples are useful because they suggest that the key element for understanding coexistence is how variability at the individual level affects both the average and the variance of the dynamics at the population level. More generally, the diverse outcomes of demographic heterogeneity we described suggest that ecologists not only need to understand the different types of variability at the individual level (i.e. stochastic and deterministic variability), but also how this variation manifests itself at the population level and affects dynamics.

## Connection to Current Theory on Variability and Coexistence

Most current theory on variability and coexistence is based on the concept of nonlinear averaging. Chesson (2000a) describes the population growth rate when at low abundance as a function of a fitness inequality term and a stabilising term related to niche differentiation (Chesson 2000a), akin to eqn 1 for nonlinear averaging. The fitness inequality term corresponds to the difference in competitive abilities at average conditions (the left term of eqn 1, ). When the growth function is nonlinear, stochastic and deterministic variability could promote stable coexistence through an effect on the right term in eqn 1. Hubbell’s (2001) neutral theory is a limiting case in which fitnesses are equal and there is no stabilising mechanism (see Adler *et al.* 2007 for discussion). Equivalent species do not invade resident equilibria (the long-term average growth rate when at low abundance is zero); as such, a neutral competitive community is not stable and is subject to ecological drift. Given sufficient time and variability and regardless if it is demographic or environmental, all but one species will go extinct.

Different types of variability can affect the long-term average ability of a species to exploit a resource (its R*–Tilman 1982) and thus increase or reduce the fitness inequality term. For instance, environmental fluctuations affecting the carrying capacity of the resident species in the Lotka-Volterra model of interspecific competition reduces the density of that species (May 1973), and environmental fluctuations affecting the carrying capacity of the invader reduces its growth rate (Table 1). Consequently, the net effect of environmental fluctuations could either be beneficial or detrimental to the invading species (Turelli 1978). If the carrying capacity of only one of the two species fluctuates, then that species will be at a disadvantage relative to its competitor, promoting the former’s competitive exclusion (Table 1). Another example is provided at Box 3, where genotypic variability in the optimum niche changes the shape of the niche. Nonlinear averaging shows that genotypic variability of the niche optimum reduces performance (akin to R*) at the species-level average optimum, while it increases performance away from it. In this case, for a given environmental condition, the fitness difference between two species will diminish with increasing genotypic variability.

It has been shown that deterministic variability may promote stable coexistence via two mechanisms (Chesson 2000a). First, it favours coexistence through a trade-off (called ‘relative nonlinearity’ by Chesson 1994) in long-term population growth rates in constant vs. varying competitive environments (Armstrong & McGehee 1980; Huisman & Weissing 1999). Depending on the function describing population dynamics, variability can have contrasting effects on competitiveness. For instance, the shape of the functional response for resource consumption in consumer-resource models (Type I vs. Type II) will affect species dynamics (see Box 3). Temporal fluctuations in resource availability combined with a Type II functional response reduces the long-term average fitness because it increases average resource availability when resident (i.e. an equalising mechanism – left side of eqn 1). Fluctuations in resource availability are also more detrimental to the long-term average growth rate of the species with the most nonlinear functional response (i.e. a stabilising mechanism – right side of eqn 1). Coexistence is then possible if the best competitor at average conditions also has a more nonlinear functional response. Second, environmental variation may promote storage effects (Chesson & Warner 1981), provided that species respond differently to the environment and that they experience proportionately stronger competition under favourable environments (nonlinearity in the growth function due to the positive second derivative – right side of eqn 1). These mechanisms have been formalised into a general framework for temporal (Chesson 1994) and spatial variability in the environment (Chesson 2000b). Levins (1979) proposed that fluctuations in the intensity of competition could be intrinsic (generated by consumer-resource dynamics) or extrinsic (from environmental fluctuations) and may allow several species to coexist on a single resource.

Clark *et al.* (2009) applied this theoretical framework to field studies and were the first to propose that multi-dimensional trade-offs could promote coexistence. They based their argument on the observation that, despite our failure to observe ecological trade-offs between pairs of species, elevated and structured variability in demographic rates can promote coexistence. Recently, Clark (2010) and Clark *et al.* (2010) provided compelling support to their proposition. Using temporal data for 33 tree species of the Southeastern United States, they estimated annual growth and reproduction rates for every individual. They found that interindividual correlations of demographic rates between species are smaller than those within species, which supports classical theoretical predictions for species coexistence. Such demographic heterogeneity could arise from various sources, including soil heterogeneity, pests, genotypic variability and historical contingencies. These results convincingly show that species-level approaches could mask much of the differentiation, because they place all variability in a single term (Box 1), masking finer differentiation that is found between individuals.

## Box 3 Examples of nonlinear averaging

We provide three examples of how variability can influence the average of well-studied nonlinear functional relationships. All expected responses are calculated using the technique of nonlinear averaging (eqn 1). Panel (a) illustrates a hypothetical relationship between the mortality rate of a population and the environment it experiences. The solid line represents the response to average environmental conditions. The dotted line corresponds to the average of this function with small variation in the environment. Because this function is concave (*M* = *e*^{−αE}) and consequently its second derivative with respect to the fluctuating environmental factor is negative, the average of the function is larger than the function of the average (Jensen’s inequality). Panel (b) shows a typical type II functional response, describing the nonlinear relationship between per capita population growth rate and resource availability. In this case, because the function is convex and its second derivative with respect to varying resource availability R is negative, the average of the function is lower than the function of the average. Panel (c) is a simplified representation of the niche using a Gaussian-shaped curve with an optimum *u* along an environmental gradient. This hypothetical niche does not result from specific assumptions about the underlying processes responsible for its shape. An individual will tolerate a certain range of conditions (i.e. individuals have a certain plasticity), but additional variability also comes from inter-individual differences. The second derivative of the function describing this niche is complex, switching from negative to positive values depending on average environmental conditions. We illustrate the average response for a population with genotypic variability in the niche optimum *u*. Elevated genotypic variance enlarges the species-level average niche and lowers its performance at the optimum.

- (B3.1)

[ Illustration of the Jensen’s inequality for commonly studied functional responses. ]

## The Emergence of Ecological Drift

In a homogeneous world, although a small amount of niche differentiation is sufficient to promote coexistence, community stability will be low if the growth rate of each species when nearing extinction is low (Murdoch *et al.* 2003). As we have seen above, traditional coexistence theory usually focuses on the single criterion of long-term average growth rate, overlooking the fact that stochastic and deterministic variability alter population densities and may promote drift. The integration of tools from coexistence theory (nonlinear averaging) and extinction risk theory (Adler & Drake 2008) shows that the strength of coexistence depends on the relative importance of stabilising mechanisms of community dynamics and the effects of variability on relative abundance (Box 2). Ecological drift will emerge despite stabilising mechanisms if stochastic/deterministic variability is high. Extinction by drift occurs when the amplitude and frequency of these perturbations are larger than the time needed to recover from them. Ecological drift could make species ecologically equivalent when their risks of extinction converge (Orrock & Fletcher 2005). Much higher niche differentiation is thus necessary to maintain coexistence in a variable world (Gravel *et al.* 2006).

We have provided tools to better understand how and when ecological drift should emerge from variable population dynamics. The equation for extinction probability (eqn 7) tells us that extinction risk will increase with stochastic and deterministic variability provided that it increases more the variance in growth rate at low abundance than its long-term average (from eqn 1). The first order partial derivative of the growth function with respect to the varying parameter informs us about the variance in the growth rate. If this term is positive, any variability will translate into population fluctuations (eqn 2). On the other hand, stochastic and deterministic variability will promote long-term average growth rate when the second order partial derivative is positive (eqn 1). The lottery model is again a good candidate to illustrate the emergence of ecological drift. There is no effect of variability in the adult death rate on the long-term average population growth rate in this model because the second order partial derivative with respect to this parameter is zero (Table 1). Variability in death rate, however, may considerably influence variance in growth rates because the first order partial derivative is positive. In other words, the variability in the death rate does not influence the long-term average growth rate at low abundance, but it increases its variance. Consequently, variability in the death rate, even if it originates from a deterministic and differentiated response to a fluctuating environment, will translate into strong fluctuations in community composition. If all species are subject to a similar amount of variability, then ecological equivalence will emerge because they will all have the same chance of persisting and there will be limited predictability in their abundance (Gravel 2007).

Past study has interpreted ecological equivalence mostly from a demographic point of view, as assumed in all neutral models. Our synthesis emphasises that it is crucial to distinguish ecological equivalence in terms of demography and dynamics. Demographic ecological equivalence occurs when all individuals of all species have the same per capita birth and death probabilities (Hubbell 2001). Ecological equivalence at the demographic level is still a matter of debate, especially in light of recent findings on niche multi-dimensionality (Clark 2009, 2010; Clark *et al.* 2010). But ecological equivalence could also occur at the level of population dynamics, when all species have the same probability of persisting in the community owing to ecological drift. Ecological equivalence of population dynamics is the emergent property of both demographic equivalence and variability in population dynamics. Because niche-differentiated species could still be prone to random fluctuations in abundance, we argue that a more general definition and assessment of neutrality should be based on the relative importance of stabilising mechanisms and ecological drift. That is, even if there are strong niche differences under certain conditions, at some times and in some places, traits might have little or no importance to coexistence.

The emergence of ecological drift with increasing stochastic and deterministic variability should be a fairly general phenomenon, because different types of variability may override deterministic forces structuring community dynamics (e.g. niche differentiation, Janzen-Connell effect and fitness inequalities). The relative strengths of stabilising mechanisms vs. disturbances are important for the relevance of ecological drift for coexistence (Box 2; May 1973). Stochastic and deterministic variability could affect all species similarly, which would promote ecological equivalence because all species would have the same extinction risk. Drift could, however, affect some species more than others, generating inequalities in extinction risk and variability in relative abundances (e.g. Vellend 2010). For instance, pronounced intrinsic differences between species could translate into differences in their expected relative abundances, with the rarer species being more likely to go randomly extinct. Some species could also respond more to a fluctuating environmental factor, and thereby be more likely to go extinct. The combination of drift and deterministic forces on population dynamics could therefore promote ecological equivalence (e.g. Gravel *et al.* 2006) or alternatively differentiate species (e.g. Lichstein *et al.* 2007).

## Future Directions

Complex variation in species responses to the environment may or may not signal differences in species traits, and if actual trait differences exist, these may or may not significantly influence population dynamics and species coexistence. Evidence of niche differentiation from field data is not evidence of stable coexistence, just as apparent ecological equivalence is not evidence of neutrality. We have emphasised that ecologists need to better understand the causes of observed variability and of how they scale up to affect population and community dynamics. We more specifically revealed two key aspects of species coexistence in a variable world that require more attention from empirical studies of coexistence: the emergence of ecological drift and the consequences of underlying causes of variability for long-term growth.

Our analysis has shed light on how we may improve empirical investigation on coexistence. Tests for species coexistence must first account for emergent ecological drift. Our analysis revealed that the strength of stabilising mechanisms relative to random disturbance events is also crucial for maintaining coexistence. We emphasised that despite average differences between species, emergent ecological drift could also prevent stable coexistence. Consequently, not only we need to test if all species have a positive growth rate at low abundance, we also need to test if the community dynamics are sufficiently stable to prevent emergent ecological drift. For instance, Adler *et al.* (2010) documented what they called an ‘embarrassment of niches’ in a grassland. They observed that stabilising mechanisms were much stronger than what was needed to offset any fitness inequality. This ‘excess’ of stabilising mechanisms might often be required to recover from regular variation in population density due to environmental and demographic stochasticity. Another approach is to explicitly quantify ecological drift. Clark & McLachlan (2003) for instance quantified the temporal dynamics of variability among replicated pollen records of temperate forest trees and found evidence for a strong stabilising mechanism. Similarly, others have studied the temporal turnover in community composition (Adler 2004; Chase 2007). Therefore, not only should future experimental designs aim at documenting both the average and variance of population growth rates at low abundance, but methods for time series analysis should aim to better quantify the error structure (e.g. covariation among species and intrapopulation variability) to more accurately assess emergent ecological drift.

Second, documenting differences in traits, trade-offs and correlations between species performance and the environment is only the first step towards an understanding of coexistence. Just as demonstrating ecological equivalence is not enough to argue for neutrality, finding ecological differentiation is not enough to argue for coexistence, because it might not affect long-term average population growth. Our synthesis emphasises that empirical tests of coexistence theory should be based on notions of stability and ecological drift (Siepielski & McPeek 2010). Good experimental examples (e.g. Adler *et al.* 2006, 2010; Levine & HilleRisLambers 2009; Siepielski *et al.* 2010) should inspire future study. Our analysis also reveals that population dynamic outcomes will depend the on the actual process affected by variability. For this, we need to understand if environmental and demographic stochasticity either increase or decrease average population growth rate and its variance. For instance, Clark (2010) documented that demographic heterogeneity decreases the interspecific competition among forest trees. The next step towards understanding coexistence in this system would require assessing how this variability at the individual level affects population and community dynamics. Doing this is not trivial. While much attention has been given to developing statistical methods to better quantify ecological trade-offs and their dimensionality, we believe that similar developments are required to better quantify the structure of variability in ecological time series.

In closing, we believe that ideas surrounding both neutrality and multi-dimensional trade-offs have merit because they force ecologists to think about how to incorporate variability in the dynamics of ecological communities. We emphasise that different sources of variability could promote ecological equivalence at the population level, despite niche differentiation. The neutral theory does not, however, provide an explanation for stable coexistence and niche differentiation remains the ultimate explanatory mechanism. We also note that variability does note promote coexistence in itself; it requires that the deterministic response to a variable environment is differentiated between species. Ecologists and statisticians must work together to develop accessible methods for assessing the structure of variability in specific systems and to quantify its contribution to coexistence.

## Acknowledgements

We thank C. Messier and C.D. Canham for stimulating discussions that led to this study and B. Beisner, V. Calcagno, J.S. Clark, A.R. Ives and two anonymous referees for helpful comments on previous versions of the manuscript. D.G. thanks the National Science and Engineering Research Council of Canada and the Canada Research Chair programme. F.G. wishes to acknowledge support from the James S. McDonnell Foundation through a ‘21st Century Science Initiative’ research award. M.E.H. thanks the Agence National de la Recherche of France for grants ‘EvolStress’ (ANR-09-BLAN-099-01) and ‘EvoRange’ (ANR-09-PEXT-01102).

## Glossary

**Demographic heterogeneity: ** Demographic variability arising from individual trait differences.

**Demographic stochasticity:** Demographic variability arising from the probabilistic nature of individual birth and death processes over a given time period.

**Demographic variance:** Demographic variability resulting from both demographic stochasticity and demographic heterogeneity.

**Ecological drift:** Population changes emerging from variable population dynamics. Can be measured as the variance between replicated time series of community dynamics.

**Environmental stochasticity:** Fluctuations in time or space of population level birth and death rates, due to deterministic responses to random environmental variation.

**Multi-dimensional trade-off:** An interspecific trade-off involving more than one niche axis.

**Neutrality:** Ecological equivalence among species. Ecological equivalence at the demographic level occurs when all individuals have equal birth, death and immigration rates. Ecological equivalence at the population level occurs when all species have the same probability of persisting in the community.

**Stability:** A system is stable when it returns to an equilibrium point or equilibrium trajectory following a perturbation.

**Stable coexistence:** Simultaneous occurrence of multiple species at a location, defined mathematically by a positive per capita growth rate of all species when at low abundance.

**Variability (total)**: The combination of stochastic and deterministic variability, both at the level of individuals and of populations.