The evolution of natural organisms is ultimately driven by the invasion and possible fixation of mutant alleles. The invasion process is highly stochastic, however, and the probability of success is generally low, even for advantageous alleles. Additionally, all organisms live in a stochastic environment, which may have a large influence on what alleles are favorable, but also contributes to the uncertainty of the invasion process. We calculate the invasion probability of a beneficial, mutant allele in a monomorphic, large population subject to stochastic environmental fluctuations, taking into account density- and frequency-dependent selection, stochastic population dynamics and temporal autocorrelation of the environment. We treat both discrete and continuous time population dynamics, and allow for overlapping generations in the continuous time case. The results can be generalized to diploid, sexually reproducing organisms embedded in communities of interacting species. We further use these results to derive an extended canonical equation of adaptive dynamics, predicting the rate of evolutionary change of a heritable trait on long evolutionary time scales.

Although the ecological importance and basic principles of adaptation to a variable environment have been long known, the corresponding genetic processes are not yet sufficiently understood. Ultimately, evolution is dependent on the fate of mutant alleles, and during the first generations after the appearance of a new variety its success is to a large extent dependent on chance events and the probability of extinction is high. A large body of theory (nicely reviewed by Patwa and Wahl 2008) treats the probability that an advantageous mutant survives the first crucial generations and becomes sufficiently abundant so that the risk of stochastic extinction can be ignored. This has in the literature been called the probability of “survival,” “establishment,” “fixation,” or “invasion,” depending on the context. We will here use the term “invasion.” In many cases invasion implies fixation, but not necessarily so if fitness is frequency dependent, such that a polymorphism is possible.

Starting with the simpler case of a constant environment, Haldane (1927) famously stated that the invasion probability of a mutant allele equals 2s, where s is the relative fitness advantage of the invading allele (Haldane assumed a constant, large population size, Poisson distribution of offspring and a small s). Later, Ewens (1969) and Eshel (1981) (see also Athreya 1992) generalized Haldane's result to arbitrary offspring distributions. They found the invasion probability to be approximately equal to 2s2, where σ2 is the variance in the number of offspring from a single individual, that is, a measure of the strength of genetic drift (or demographic stochasticity). For example, the Poisson distribution has a variance equal to its mean, which by assumption is equal to 1 + s here. Thus, Ewens’ and Eshel's approximation agrees with Haldane's result because s is assumed to be small.

Taking variable survival and/or reproduction rate into account is inherently difficult in the general case. The case of a variable fitness advantage s but constant population size N has been studied several times (e.g., Kimura 1954; Jensen 1973; Karlin and Levikson 1974; Takahata et al. 1975). Alternatively, a branching process approach can be used, which usually requires the assumption of an infinite resident population size. Smith and Wilkinson (1969) showed by this approach that an invading mutant will go extinct with certainty if &#55349;&#56636;(ln(mt)) < 0, where mt is the time-dependent average number of offspring per individual and &#55349;&#56636;(·) denotes the long-term, stationary, mean (Dempster 1955 fore-shadowed this result, see also Gillespie 1973). It is assumed that each mt is chosen independently from a fixed distribution—a so-called white noise environment. Later, Athreya and Karlin (1971) generalized this result to autocorrelated environments, and Karlin and Lieberman (1974) to diploid populations. Together, these results underline the importance of mean log growth rate for adaptations to variable environments, a fundamental result in bet-hedging theory (e.g., Cohen 1966; Seger and Brockman 1987). In a recent paper, Peischl and Kirkpatrick (2012) used novel analytical techniques to calculate the probability of invasion, given small fluctuations of s. They show that the invasion probability is proportional to a weighted time average of s, with more weight on points in time with low mutant abundance.

If the invading mutant has a fixed fitness advantage relative to the resident type, then the mutant growth rate will vary over time just like that of the resident population. This assumption has been used in a number of studies. Ewens (1967) showed that the probability of establishment in a cyclic population equals math formula (again assuming a Poisson distribution of offspring and a small s), where nH is the harmonic mean population size and n(0) is the resident population size at the time when the mutant first appears. This shows that the invasion of a mutant type is less likely if the amplitude of the population cycle is large (assuming a fixed arithmetic mean), since the harmonic mean is sensitive to variation, as opposed to the arithmetic mean. It can also be shown that invasion is more likely in a growing population than in a declining population (Ewens 1967; Kimura and Ohta 1974; Otto and Whitlock 1997). The results by Ewens (1967) and Otto and Whitlock (1997) for cyclic populations were later generalized to arbitrary offspring distributions by Pollak (2000), who among other things confirmed that the probability of invasion in a cyclic population is proportional to the harmonic mean population size divided by the population size at mutant introduction.

The more general case of both a variable strength of selection and a variable resident population size has been treated recently by Waxman (2011) and Uecker and Hermisson (2011) . In both studies, quite general expressions, but rather implicit, for the invasion probability are derived, Uecker and Hermisson (2011) further analyze simplifying special cases such as a deterministically growing population or a periodic (sinusoidal) environment.

Lastly, we would like to highlight a rarely cited result by Hill (1972) who, somewhat offhandedly, derived the expression

display math(1)

where P is the probability of mutant invasion, ne = nH is again the harmonic mean population size, math formula is the arithmetic mean selective advantage and q is the initial proportion of the mutant type. We will return to this result, and its assumptions, in later sections.

We here generalize several of the aforementioned results to the case of arbitrary ergodic population dynamics, subject to ergodic environmental fluctuations. We calculate the invasion probability of a mutant of small phenotypic effect in a large resident population. Mutant fitness, and in particular its selective advantage s, depends on the resident population size as well as the environmental fluctuations and may in some circumstances be negative as long as the long-term mean math formula is positive. Solutions are given for both discrete time and continuous time dynamics. The continuous time case allows for overlapping generations and is a particularly suitable model for unicellular organisms that reproduce through fission, such as bacteria or protozoa.

Model Description: Basic Assumptions

We consider the invasion of a mutant type in a monomorphic resident population of asexually reproducing individuals, under the assumptions that (1) all individuals are equivalent, that is, there is no age, stage, or spatial structure, (2) the resident population size is large enough that the growth of an invading mutant is independent of its own density, at least until the mutant abundance is large enough that the risk of stochastic extinction is negligible, and (3) the mutation is of small effect, such that the mutant type is ecologically close to the resident type, that is, it has in all possible environmental circumstances a per capita growth rate close to that of the resident.


Because we will move back and forth between the established conceptual frameworks of stochastic population dynamics, population genetics, and long-term evolution, a couple of concepts may have different meanings to readers with different background.

First, the “environment” of an invading mutant type consists of two basic components—the “external environment” and the “feedback environment.” We think of the external environment as a stochastic, ergodic process, which affects the survival and reproductive success of all individuals of the same type in the same way, such as stochastic weather fluctuations or a variable resource abundance. Ergodic means that irrespective of initial conditions, the environment will in the long term visit its full stationary distribution. The external environment is in itself not affected by the state of the focal population, in contrast to the feedback environment, which by definition depends on the current state of the focal population and possible interacting populations (Metz et al. 1992; Mylius and Diekmann 1995; Heino et al. 1998). In the simplest of cases, the feedback environment is population size and the external environment is a single parameter, such as temperature. Our analysis is staged in this simplified scenario but it is straightforward to generalize to the multidimensional case (see below).

Second, “fitness” can be understood either as long-term fitness, that is, the long-term average per capita growth rate of any given clone, or as the instantaneous per capita growth rate at any given moment. We use the qualifications “mean fitness” and “instantaneous fitness” to denote the two concepts, respectively (more precise definitions follow).

Finally, we use &#55349;&#56636;[z(t)], &#55349;&#56653;[z(t)], and ℂ[z(t),w(t)] to denote the mean, variance, and covariance, respectively, of the stochastic process(es) z(t) (and w(t)). If nothing else is specified, the stationary mean, variance and covariance, respectively, are intended. For brevity, we will sometimes use to denote the mean.


We start with the continuous time case—assuming individuals reproduce and die according to a time-inhomogeneous birth-and-death process. More formally, we assume that a resident-type individual has a birth rate, b(n(t), ɛ(t)), and death rate d(n(t), ɛ(t)), where n(t) is the resident population size and ɛ(t) is an environmental process. It is assumed that ɛ(t) is an ergodic, stochastic process continuous in time. The instantaneous fitness, that is, the per capita growth rate, f, is given by the difference between birth and death rate,

display math(2)

We denote the total dynamic environment determining the instantaneous fitness E(t). In the formalism here, E(t) = {n(t), ɛ(t)} and the growth, birth, and death rates can be written

display math(3)

We assume E(t) is ergodic, which should be a realistic assumption for many scenarios, albeit excluding long-term environmental trends or a steadily growing or declining population. Note that autocorrelation of the environmental process ɛ(t) is allowed, as long as it declines to zero at large time lags. More precisely, the total environment E(t) should explore its full stationary distribution much faster than the time scale of a mutant invasion (math formula, see below). It should also be noted that technically speaking the population process is not ergodic because n = 0 is an absorbing state. However, in the large population limit considered here, this is of minor importance.

Given the growth function above, it is straightforward to express the resulting dynamics of the resident population. Because we assume population size n to be large enough that demographic stochasticity can be ignored, the resident population dynamics are given by

display math(4)

We assume a single mutant individual appears in the population at t = 0. The mutant birth, death, and per capita growth rates are denoted by math formula, math formula, and math formula, respectively. The instantaneous mutant fitness advantage is written

display math(5)

Note that E(t) is still the environment given by the population dynamics of the resident population (and the external environment). A mutant type may have a fixed fitness advantage (s), but can also differ in its density dependence, its sensitivity to fluctuations of the external environment, or all of the above. s(E(t)) can in the general case change sign depending on the state of the environment E(t), but we assume its long-term (stationary) mean, math formula, is positive. In other words, the mutant type may be at a disadvantage for shorter periods of time, as long as it is advantageous on average.


For the discrete time case, we assume nonoverlapping generations. Each individual (independently) gives birth to a geometrically distributed number of offspring, with the mean number of offspring determined by the individual's instantaneous fitness. The probability of k offspring is

display math(6)

where p = 1/(1+λ) and λ is the mean number of offspring. The variance in offspring number is λ(λ + 1), which can be compared to the commonly used Poisson distribution, which has a variance equal to its mean, λ. A mechanistic motivation for the geometric distribution arises if an individual makes repeated reproduction attempts, each with the same probability of success, but stops at the first failure. From a more pragmatic point of view, however, there is clearly no natural population where individual reproductive success exactly follows a geometric or Poisson distribution. The geometric distribution is used here for mathematical convenience, in lack of a more general theory for all, or at least a family of distributions.

In discrete time, we define the instantaneous fitness function f as the natural logarithm of the per capita growth rate (λ), such that the mean number of surviving offspring of an individual of the resident type is given by math formula, where ɛ(t) here is a discrete time process, but with otherwise the same properties as in the continuous time case above. The dynamics of a large population of resident-type individuals is thus

display math(7)

Mutant Invasion

We here derive the main result—the probability of invasion of a mutant type, starting as a single individual at time t = 0. Invasion does not necessarily imply fixation. If coexistence of the mutant and resident types is possible, we assume the equilibrium mutant abundance is large, such that the invasion process can safely be analyzed under the assumption that mutant abundance has no effect on mutant fitness. More precisely, we assume there is a population size ni of the mutant type at which invasion can be considered certain but that at the same time ni << n, where n is the equilibrium resident population size. If the probability that a mutant population starting with a single individual invades is equal to P, then the probability that a population of ni mutants goes extinct is approximately given by math formula as long as P is small. A requirement is thus that math formula is close to zero, that is, that niP is large (niP > 5 gives an error less than 1%). If, as we will show, P is the size of math formula, we can express the necessary requirement that math formula for our analysis to hold.


As a starting point, we use a result by Kendall (1948), which states

display math(8a)

where PE is the ultimate survival probability of a time-dependent birth-and-death process and

display math(8b)

A heuristic interpretation of equation (8a)) is a weighted total death rate, with most weight on periods, usually at low t-values, with low numbers of mutants (the exponential factor can be interpreted as 1/(expected mutant population size at time t)). As mentioned in the Introduction, a similar weighting was found by Peischl and Kirkpatrick (2012).

The environment E(t) is in the general case stochastic and unpredictable. The necessary interpretation of PE (eq. 8a) is thus the “conditioned” survival probability (Waxman 2011), conditioned on the future environment E(t), t ≥ 0, which is the reason for the subscript E.

The unconditioned probability of invasion is given by the mean PE, and we here calculate the mean probability P0,

display math(9)

averaged across all possible future developments of environmental states, but still conditioned on initial conditions E(0). In particular, we seek the linear dependence of P0 on the mean fitness advantage math formula as math formula becomes small, that is, we seek the limit

display math(10)

In Appendix S1, we show that

display math(11)

for almost all possible future environments E(t), t ≥ 0 (the exceptions have probability zero). n(0) is the resident population size at the time of mutant arrival, but all other dependencies on initial conditions average out. Inserting equation (11) into equation (10) gives (see Appendix S1 for details)

display math(12)

and we can finally express the approximate invasion probability as

display math(13a)

where we define the effective population size ne as

display math(13b)

The identities math formula and math formula follow from the ergodicity assumption of n(t). More precisely, they follow from the assumptions that ln(n(t)) and 1/n(t) have a long-term mean growth rate of zero.

The definition of effective population size (eq. 13b) is somewhat arbitrary. Otto and Whitlock (1997) suggest defining ne such that math formula(the “fixation effective population size”), which in our case implies setting math formula. However, our proposed definition of effective population size (eq. 13b) has the appealing properties that (1) it is unitless—it does not depend on the chosen time unit, (2) it simplifies to ne = n/2 in cases when n is constant, (3) it can be interpreted as half the weighted harmonic mean population size, weighted by the total per capita event rate (b + d), and is thus congruent with the discrete time case below. A possible disadvantage with our definition is that the average fitness advantage, math formula, must be standardized with the mean birth rate, math formula. On the other hand, the unitless ratio math formula (eq. (13a)) can be interpreted as a standardized selection coefficient, measured on the time scale of the average generation time (in the deterministic case, with a constant population size, generation time equals 1/d = 1/b). Irrespective of the preferred definition of effective population size, equation (13a) is directly comparable to several previous results in discrete time (e.g., Ewens 1967; Otto and Whitlock 1997; Pollack 2000).

The approximation in equation (13a) is valid for small s, that is, not only is math formula small, but also its fluctuations. The mutant type can thus not be inherently different from the resident type—its instantaneous fitness must for all environmental states be close to that of the resident. The only realistic interpretation is a mutation of small phenotypic effect. We further investigate the applicability of this result in the Model Examples section below and in Appendix S3.


Using the assumption of geometrically distributed offspring, the ultimate survival probability of a mutant strategy appearing at t = 0 can be expressed exactly as (Haccou et al. 2005, Box 5.5):

display math(14a)


display math(14b)

The striking similarity between equations (14a), (14b) and the continuous time version equations (8a), (8b) makes it possible to carry out almost exactly the same derivation as above, only exchanging integrals with sums and setting the death rates d and math formula to 1. Due to the great similarity of the calculations we refrain from presenting the discrete time derivation here, and instead present the major results:

display math(15a)


display math(15b)

and, just like above, P0 is the probability of invasion conditioned on initial conditions E(0), n(0) is the resident population size at the time of mutant appearance, and nH is the harmonic mean population size. The requirement that the mutant phenotype is close to the resident is the same as above. This result agrees well with that of Ewens (1967), which gives the probability of fixation as math formula in a population with cyclic dynamics. Our result is generalized to a variable, density-dependent fitness advantage and arbitrary ergodic population dynamics. The difference by a factor two is due to different assumptions on the distribution of surviving offspring – the geometric distribution [used here], as opposed to the Poisson distribution [as used by Ewens].

The Diffusion Approximation

The diffusion approximation is very often utilized in population genetics and it can be used, with care, for the problem of mutant invasion in stochastic environments. Classically, the proportion p of the invading type is the dynamic state variable and under the assumption that p changes slowly (between generations) it is sufficient to calculate the mean and variance of the change Δp (Kimura 1962). In a stochastic setting, it is further necessary to assume that p changes slowly enough that the full stationary distribution of environmental states is experienced during a time-step Δt. Still, Δt has to be small enough that Δp is small. In other words, it is required that the invasion process is much slower than the stochastic environmental dynamics. Nonetheless, Hill (1972) derived the following expressions under the assumptions of discrete generations and Poisson distributed offspring:

display math(16a)
display math(16b)

where ne is the harmonic mean population size and math formula is the arithmetic mean selective advantage (math formula in our notation). Inserting equations (16a), (16b) into the standard equations of Kimura (1962) yields

display math(17)

expressing the invasion probability of a mutant appearing as a single individual at time 0 (p0 = 1/n(0)). Hill's result has as a first-order approximation (assuming nesA is large and discarding terms of order math formula and higher)

display math(18)

which coincides with our result (eq. 15a), apart from the difference in effective population size.

It is possible to derive equations similar to equations (16a), (16b) also for our models in discrete and continuous time (not shown). The resulting expressions, similar to equations (17) and (18), match our results above using the branching process approach (eqs. 13a, b and 15a, b). In short, it is possible to acquire much the same results using the diffusion approximation. This is not too surprising, because the necessary assumptions (large population size, slow invasion) are much the same. However, the conditions under which the diffusion approximation is valid, especially the averaging across the stationary distribution of environmental states in equations (16a), (16b), are somewhat unclear to us. For example, Hill's (1972) derivation misses the fact that in discrete time, fitness should be averaged on a logarithmic scale. We leave it to future studies to more thoroughly evaluate the conditions under which the diffusion approximation is appropriate. Here, we conclude that it is correct at least to the first order of s.

Model Examples and Tests of Accuracy


As a continuous time example of our main finding—the probability of mutant invasion—we choose a theta-logistic model with a birth rate, b, subject to environmental variation and a density-dependent death rate, d, according to

display math(19a)


display math(19b)

such that the instantaneous fitness becomes

display math(19c)

n(t) is the total population size, K is the carrying capacity, corresponding to the deterministic equilibrium population size, r is the per capita growth rate at low densities, and θ (together with r) controls the shape and strength of density dependence. ɛ(t) is a Gaussian process (more precisely an Ornstein–Uhlenbeck process [Stirzaker 2005]) with zero mean and an autocovariance function

display math(20)

where math formula is the stationary variance of the environmental fluctuations and the (auto-) correlation time TC dictates the environmental autocorrelation (the limit math formula corresponds to white noise, with no autocorrelation).

As a first example, we choose a resident population with strong density dependence (θ = 2) and study the invasion of a mutant with weaker density dependence (θ = 1.98), but the same equilibrium population size. In the deterministic case (math formula), the invasion fitness in this model depends only on the equilibrium population size of the resident, K, compared to that of the invading mutant, and it is a standard result that evolution will maximize K (Charlesworth 1971). However, in a variable environment selection will deviate from the deterministic prediction. The environmental fluctuations have no direct effect on mean fitness but the resulting fluctuations in population size in combination with a nonlinear density dependence create selection for weaker density dependence in this case. This is illustrated in Figure 1, where the density-dependent fitness of the resident (f, solid, grey line) and the invading mutant (math formula, dash-dotted line, mostly overlapping with f) are depicted together with the stationary distribution of resident population size (shaded histogram in background). The fitness difference (math formula, the thick dashed line is 100s) is negative for population sizes below K but positive above K. Mean population size is equal to K, but the strong curvature of s generates a positive average fitness advantage for the mutant (math formula).

Figure 1.

Instantaneous fitness of the resident type (solid, grey line) and a rare mutant (dash-dotted line) as functions of the resident population size in the continuous time theta-logistic model (eqs. 19a–c), disregarding environmental stochasticity (ɛ is set to 0 when plotting these functions). The dashed line shows the difference between mutant and resident fitness (×100). The background shading is a histogram (y-scale not shown) of the population sizes from a simulation of the stochastic resident population dynamics, where the environmental process is an Ornstein–Uhlenbeck process (eq. (20)). Parameter values: math formula r = 1, K = 106, θ(resident) = 2, θ(mutant) = 1.98, math formula = 0.7, TC = 1.

We tested the predicted probability of invasion by, first, generating a set of initial conditions from the stochastic dynamics of the resident population and, next, starting 105 separate invasion attempts from each initial condition, all initiated from a single mutant individual (simulation details are given in Appendix S2). Figure 2 shows the resulting estimated invasion probabilities plotted against initial population size n(0) (points with 95% confidence intervals). For the set of parameter values chosen here (see legend), the results follow our prediction very well (dashed line, math formula and ne are calculated from simulations of the population dynamics).

Figure 2.

Probability of mutant invasion (y-axis) as a function of the initial resident population size (x-axis) for the stochastic continuous time theta-logistic model (eqs. 19a–c, 20). The black dots (with 95% confidence intervals) indicate the estimated probability from 105 simulations, started with a single mutant individual. All invasion attempts for a given n(0) were started at the same initial condition. Initial conditions were generated by simulating the resident population for 100 time units and thereafter until the appropriate (equally spaced on the log x-axis) population size occurred. The dashed line is the prediction given by equation (13), where math formula and math formula were calculated from simulations. The background shading is a histogram of the resident population dynamics, with log-spaced bins. Parameter values are the same as in Figure 1.

We further investigate the robustness of our prediction in Appendix S3. To summarize, we find good agreement between our result and more exact numerical calculations (using eqs. 8a, b) as long as math formula is large and math formula is small. For this particular model, with these particular parameter values, our approximation has an average error less than 5% in the region math formula. At the lower limit, demographic stochasticity of the resident dynamics is too strong and, more importantly, the branching process approach is no longer valid because the resident population cannot be considered infinite from an invasion perspective. Above the higher limit (math formula), the variation in PE between alternative future environments is too large for our result to hold. In principle, the relationship P0 ∼ 1/n0 fails. It should here be noted that a diffusion approximation approach (sensu eq. 17) likewise fails at this limit—the difference between the two predictions is much smaller than the error. We also tested the sensitivity to strong environmental variation and autocorrelation, and found environmental autocorrelation to be more critical than variation per se, except close to the boundary where the risk of extinction of the resident population becomes substantial and the population undergoes frequent severe bottlenecks. See Appendix S3 for further details.

Technical note: In the derivation of equations (13a), (13b), we show that for each possible future environment, the probability of invasion converges to the limit as the mean fitness advantage math formula goes to zero. Numerical investigations (Appendix S3, Figs. S1 and S4) show that, at least for this model, the “mean” probability, averaged across all possible future environments, converges much faster than the invasion probabilities corresponding to single environmental realizations. This means that the value of math formula may not be as restricted to really small values as one might conclude from our derivation, and leaves room for future theoretical investigations on this topic.


The discrete time example is based on the classical logistic equation, with a fitness (log per capita growth rate) of the resident population given by

display math(21)

We introduce minute amounts of environmental variation here (&#55349;&#56653;[ɛ(t)] = 10−6), merely to avoid completely deterministic dynamics (and loss of ergodicity for some initial conditions) as we let population size (K) grow large.

Given stable population dynamics (r < 2) and no environmental fluctuations (&#55349;&#56653;[ɛ(t)] = 0), selection is neutral on the r parameter. If environmental variation is introduced through stochastic variation of K this model generates selection for decreasing r-values, basically because a low-r type has weaker density dependence (Turelli and Petry 1980). The mechanism is very similar to that described in the previous, continuous time example (Fig. 1). Here, we will instead consider the case of unstable dynamics, choosing a high r-value, which gives strong, overcompensating density dependence and chaotic dynamics (in the deterministic case) (May 1974). Selection is still for lower values of r. To illustrate several features of our results, we also introduce a trade-off between density dependence r and carrying capacity K, such that a high-r type is compensated with a higher K. More precisely, we study the two alternative types 1 and 2: {r1 = 2.8, K1 = 106} versus {r2 = 2.85, K2 = 1.0023 × 106}. Setting type 1 as the resident, type 2 has a fitness advantage (math formula = 0.0023) and can invade (Fig. 3A). On the other hand, if type 2 is dominating, type 1 has an advantage (math formula = 0.0034, Fig. 3B). The frequency dependence comes from the shift in population dynamics as one type or the other dominates the population. Type 2 has the higher r-value, which generates more variable population sizes (compare the distributions of the resident populations in Fig. 3A and B). The strong density fluctuations give type 1 an advantage because it has the lower r-value. However, once type 1 becomes more common, the population dynamics stabilize somewhat, such that the advantage is lost. The two types will thus both increase from low abundances and can coexist in the population. Figure 3C shows a successful invasion of type 2 (black dots) when type 1 (gray dots) is resident, and the subsequent coexistence.

Figure 3.

Mutual invasions of two types in the discrete time logistic model (eqs. 6, 7, 21). (A) Probability of type 2 invading type 1. (B) Probability of type 1 invading type 2. (A, B) Estimated invasion probability (black dots with 95% confidence intervals), based on 105 simulations starting at different initial resident population sizes. The dashed line indicates the prediction based on equations (15a), (15b), where math formula and nH were calculated from simulations. Background shading is a histogram of simulated resident population dynamics (y-scale not shown, but the same in (A) and (B). (C) A successful invasion of type 2 (black dots) into a resident population of type 1 (gray dots). The two types coexisted for at least 104 generations and showed no signs of one excluding the other (not shown). Parameters, type 1: r = 2.8, K = 106; type 2: r = 2.85, K = 1.0023 × 106.

Figures 3A and 3B show a good correspondence between the approximation in equations (15a), (15b) and simulation results. In Appendix S3, we investigate the sensitivity of our approximation to changes in the resident population size and the strength of selection (math formula). We find that the average error is within 5% in the region math formula. The upper limit here is about seven times higher than in the continuous time case, presumably at least partly due to the fast mixing of the wildly fluctuating dynamics—even rapidly invading mutants will during the invasion be exposed to a large, representative, portion of the stationary distribution of the resident type.

This example illustrates three things. First of all, that our results are valid for all types of ergodic dynamics of the resident type (chaos in this case). Second, that they are applicable to situations when invasion does not imply fixation. Third, that population dynamics may induce frequency dependence. In a constant environment with stable population dynamics, the feedback environment in the present model is one-dimensional—it is characterized by a single parameter, the equilibrium population size. If population sizes fluctuate, on the other hand, the environment in which a new mutant finds itself can no longer be described so easily—the full distribution of population sizes is necessary to determine its probability of invasion.

A few technical notes: This example is not as superficially constructed as it might appear at first sight. If an r-K trade-off is modeled as r = r0 + x and K = K0(1 + cx) (c > 0), one quite easily finds parameter values for which there exists an evolutionary branching point of the trait x (not shown). In other words, gradual evolution of x will converge to a parameter region in which coexistence of closely positioned types is possible (cf. Geritz et al. 1998). In conclusion, such parameter values are not totally unlikely—they will be provided by natural selection, given a suitable trade-off. Yet, the model as such is admittedly superficial and should not be taken too seriously. We choose it here for its simplicity and the possibility to demonstrate several features of our results with a single model. Also note that the resident dynamics are strictly speaking not chaotic—the state space is finite (there can only be a discrete number of individuals) and the dynamics are stochastic. However, the stochastic dynamics are very similar to the truly chaotic dynamics of a deterministic, continuous version of the same model.



The ergodic environment E can easily be generalized to a community context, or the case of several coexisting types in a population (or both). As long as the mutant represents a small phenotypic change of one of the interacting species or one of the coexisting types, the mean fitness advantage math formula is well defined and our results are readily applicable. Note that in the multitype case resident population size n has to be replaced with the number of individuals of the type from which the mutant descends.


It is likewise straightforward to consider the case of a diploid, randomly mating population. A new, invading mutant will initially only occur as a heterozygote and its growth is then equivalent to the asexual case. In continuous time, a “birth event” has to be interpreted as the event of coupling with a random individual and producing a single offspring. Each birth event produces a new heterozygote with probability 1/2, which means the birth rate b that goes into the equations is the rate of birth events each heterozygote is involved in divided by two. The assumption of random mating is crucial here because we cannot allow different mating success for males and females.

In the discrete time case, the reproductive success of each allele copy needs to follow a geometric distribution for our analysis to hold. This is, for example, the case if all individuals are mated and the number of offspring from each pair of mates has a geometric distribution (a binomial sampling, due to Mendelian segregation, of a geometrically distributed number yields a new geometric distribution).

In both the discrete and continuous time cases, it is the mean heterozygote fitness advantage that enters the equations as math formula. Completely recessive alleles, which only have an advantage as a homozygote, are thus not allowed. Further, it is not straightforward to generalize to the diploid, multitype case, because the multiple genotypes in which a mutant allele may then occur creates an extra source of “demographic stochasticity” not taken into account here.

Adaptive Dynamics

Given the probability of mutant invasion one can derive expressions for the rate at which new varieties will invade a population and the consequential rate of trait evolution. If new types appear as mutants of the resident type with a fixed mutation rate μ per individual, the rate of mutant appearance at any point in time is equal to the number of births times μ, which yields the average rate of successful invasions

display math(22a)


display math(22b)

in the continuous (eq. 22a) and discrete (eq. 22b) time case, respectively. Note, however, that μ has to be low enough such that only one mutant is invading at any one time. An immediate conclusion from equations (22a), (22b) is that evolution is generally slower in populations with highly variable population sizes, given the same arithmetic mean population size. This finding is certainly not new, but is here extended to more general conditions.

It is also possible to derive a generalized canonical equation of adaptive dynamics (Dieckmann and Law 1996), predicting the rate of evolutionary change over long evolutionary time. Considering the evolution of a continuous, heritable trait x we assume the instantaneous fitness of any individual in the population is given by f(xi, E(t)), where xi is the trait value of the individual and E(t) is the ergodic environment set by a resident type with trait value x. We can then write

display math

where math formula is the trait value of a mutant type,

display math(23)

is the instantaneous selection gradient and math formula is the phenotypic difference in x between the mutant and the resident type. Accordingly, we get

display math(24)

as long as Δx is small, which can be substituted into the expressions for P0 above. Following much, the same procedure as in Dieckmann and Law (1996) gives

display math(25)

for both the continuous and discrete time case. μ is the mutation rate per individual and math formula is the variance in mutational effects (on x). μ, math formula, and ne may all depend on the resident trait value x. The effective population size, ne, is in the continuous time case given by equation (13b). In discrete time, ne is equal to half the harmonic mean population size if the assumption of geometrically distributed number of offspring is used (eq. 15b). A Poisson distributed number of offspring instead yields an effective population size equal to the harmonic mean population size, using the diffusion approximation by Hill (eq. 18).

Equation (25) seemingly differs by a factor 1/2 from the original expression derived by Dieckmann and Law for the continuous time case. However, this difference is due to our definition of effective population size, which converges to n/2 in the deterministic, continuous time case. The expression given here has the advantage that it is the same for discrete and continuous time and that the effective population size in discrete time agrees with earlier definitions.

Equation (25) gives the expected long-term evolutionary change of a continuous trait x, given mutations are of small phenotypic effect and rare, such that consecutive invasions are separated in time. It constitutes a generalized canonical equation of adaptive dynamics, applicable to arbitrary ergodic environments and population dynamics.


We have here calculated the invasion probability of an advantageous mutant type under quite general conditions. We assume a large, unstructured, monomorphic population and a mutant of small effect, but put no restrictions on the type of population dynamics or the variability of the stochastic environment, other than the assumption of ergodicity. Environmental autocorrelation or slow population dynamics are allowed, as long as math formula is small enough that the invasion process is much slower than the population dynamics and environmental fluctuations. The mutant fitness advantage may depend on population density as well as environmental conditions. We have outlined how our results can be generalized to multitype, multispecies scenarios, as well as diploid, sexually reproducing organisms. We further use the derived invasion probability to calculate the rate of invasions of new types and to extend the canonical equation of adaptive dynamics, which shows how our results relate to the rate of adaptation in stochastic environments.

The branching process approach used here requires that the average fitness advantage math formula is small and that math formula is large. For theoretical purposes this may not be such a large problem, but it certainly restricts the number of natural or experimental populations to which our results can be readily applied. Single invasion experiments in the laboratory commonly involve selection coefficients larger than a percent or two, and experimental populations (except bacteria or protozoans) are commonly too small in numbers. In the laboratory or in the field, our predictions can nonetheless serve as benchmark values, in the lack of a more complete theory. We made some attempts to extend the theory using a diffusion approximation, but found the results largely conflicted with the same problems as our first derivation, especially when math formula is not small. There is still the possibility that the diffusion approximation does a better job in situations when math formula is small to intermediate—our numerical investigations were not suitable for that type of evaluation—but a more thorough investigation of the accuracy of the diffusion approximation for this problem is out of scope here. Moreover, the basis for the application of the diffusion approximation in this context is in our minds still somewhat shaky and needs further analysis.

Uecker and Hermisson (2011) used an analytical approach very similar to ours. (In fact, their eq. 16b is equivalent to our eq. A4) However, instead of considering the stochastic case and taking the limit math formula, Uecker and Hermisson studied a set of special cases where more complete solutions are attainable—letting the environment or the resident population change, but in a deterministic fashion. Despite the differences, many of their conclusions match ours. Among other things, Uecker and Hermisson demonstrate that in a periodic (sinusoidal) environment, the probability of invasion is independent of initial conditions if the frequency of environmental change is high enough (see also Otto and Whitlock 1997). In other words, if the environment changes much faster than the process of invasion, it is sufficient to take into account the averaged environmental conditions to calculate the probability of invasion (save for initial population size). Further, Uecker and Hermisson demonstrated that the strength of demographic stochasticity has a direct negative effect on the probability of invasion (our eq. 13a). It is also possible to show that several of the derived expressions by Uecker and Hermisson agree with ours if the limit math formula is taken. In our minds, the two studies complement each other nicely.

The importance of the geometric mean fitness, as emphasized in classical bet-hedging theory, is somewhat implicit in our presentation. In the discrete time case, we define instantaneous fitness f as the natural logarithm of per capita growth rate, which directly makes “mean fitness” correspond to the (logarithm of the) geometric mean growth rate. The classical trade-off between a high arithmetic mean and a low variance is thus not immediately apparent here, but is incorporated in our definition of “fitness.” Instead, the formalism here emphasizes nonlinearities of the density dependence, sensitivity to environmental fluctuations, and frequency dependence (see also a conceptual discussion in Ripa et al. 2010 on the definition of bet-hedging when fitness is frequency dependent).

In a broader perspective, our results and examples have highlighted several important but sometimes neglected aspects of trait evolution. Natural populations are subject to environmental stochasticity, fitness is density and frequency dependent, variable population sizes induce extra frequency dependence, and the strength or even direction of selection may differ depending on environmental circumstances. It is our hope that this study may inspire future work toward a more complete theory of trait evolution by natural selection.


H. Metz made several invaluable comments to an earlier draft of this article. We also thank N. Barton, J. Hermisson, and three anonymous reviewers for constructive comments on an earlier version of this article. JR thanks the Swedish Research Council for financial support. Some of the simulations were carried out at the LUNARC Centre for Scientific and Technical computing, Lund University.