Although the ecological importance and basic principles of adaptation to a variable environment have been long known, the corresponding genetic processes are not yet sufficiently understood. Ultimately, evolution is dependent on the fate of mutant alleles, and during the first generations after the appearance of a new variety its success is to a large extent dependent on chance events and the probability of extinction is high. A large body of theory (nicely reviewed by Patwa and Wahl 2008) treats the probability that an advantageous mutant survives the first crucial generations and becomes sufficiently abundant so that the risk of stochastic extinction can be ignored. This has in the literature been called the probability of “survival,” “establishment,” “fixation,” or “invasion,” depending on the context. We will here use the term “invasion.” In many cases invasion implies fixation, but not necessarily so if fitness is frequency dependent, such that a polymorphism is possible.

Starting with the simpler case of a constant environment, Haldane (1927) famously stated that the invasion probability of a mutant allele equals 2*s*, where *s* is the relative fitness advantage of the invading allele (Haldane assumed a constant, large population size, Poisson distribution of offspring and a small *s*). Later, Ewens (1969) and Eshel (1981) (see also Athreya 1992) generalized Haldane's result to arbitrary offspring distributions. They found the invasion probability to be approximately equal to 2*s*/σ^{2}, where σ^{2} is the variance in the number of offspring from a single individual, that is, a measure of the strength of genetic drift (or demographic stochasticity). For example, the Poisson distribution has a variance equal to its mean, which by assumption is equal to 1 + s here. Thus, Ewens’ and Eshel's approximation agrees with Haldane's result because *s* is assumed to be small.

Taking variable survival and/or reproduction rate into account is inherently difficult in the general case. The case of a variable fitness advantage *s* but constant population size *N* has been studied several times (e.g., Kimura 1954; Jensen 1973; Karlin and Levikson 1974; Takahata et al. 1975). Alternatively, a branching process approach can be used, which usually requires the assumption of an infinite resident population size. Smith and Wilkinson (1969) showed by this approach that an invading mutant will go extinct with certainty if ��(ln(*m _{t}*)) < 0, where

*m*is the time-dependent average number of offspring per individual and ��(·) denotes the long-term, stationary, mean (Dempster 1955 fore-shadowed this result, see also Gillespie 1973). It is assumed that each

_{t}*m*is chosen independently from a fixed distribution—a so-called white noise environment. Later, Athreya and Karlin (1971) generalized this result to autocorrelated environments, and Karlin and Lieberman (1974) to diploid populations. Together, these results underline the importance of mean log growth rate for adaptations to variable environments, a fundamental result in bet-hedging theory (e.g., Cohen 1966; Seger and Brockman 1987). In a recent paper, Peischl and Kirkpatrick (2012) used novel analytical techniques to calculate the probability of invasion, given small fluctuations of

_{t}*s*. They show that the invasion probability is proportional to a weighted time average of

*s*, with more weight on points in time with low mutant abundance.

If the invading mutant has a fixed fitness advantage relative to the resident type, then the mutant growth rate will vary over time just like that of the resident population. This assumption has been used in a number of studies. Ewens (1967) showed that the probability of establishment in a cyclic population equals (again assuming a Poisson distribution of offspring and a small *s*), where *n _{H}* is the harmonic mean population size and

*n*(0) is the resident population size at the time when the mutant first appears. This shows that the invasion of a mutant type is less likely if the amplitude of the population cycle is large (assuming a fixed arithmetic mean), since the harmonic mean is sensitive to variation, as opposed to the arithmetic mean. It can also be shown that invasion is more likely in a growing population than in a declining population (Ewens 1967; Kimura and Ohta 1974; Otto and Whitlock 1997). The results by Ewens (1967) and Otto and Whitlock (1997) for cyclic populations were later generalized to arbitrary offspring distributions by Pollak (2000), who among other things confirmed that the probability of invasion in a cyclic population is proportional to the harmonic mean population size divided by the population size at mutant introduction.

The more general case of both a variable strength of selection and a variable resident population size has been treated recently by Waxman (2011) and Uecker and Hermisson (2011) . In both studies, quite general expressions, but rather implicit, for the invasion probability are derived, Uecker and Hermisson (2011) further analyze simplifying special cases such as a deterministically growing population or a periodic (sinusoidal) environment.

Lastly, we would like to highlight a rarely cited result by Hill (1972) who, somewhat offhandedly, derived the expression

where *P* is the probability of mutant invasion, *n _{e}* =

*n*is again the harmonic mean population size, is the arithmetic mean selective advantage and

_{H}*q*is the initial proportion of the mutant type. We will return to this result, and its assumptions, in later sections.

We here generalize several of the aforementioned results to the case of arbitrary ergodic population dynamics, subject to ergodic environmental fluctuations. We calculate the invasion probability of a mutant of small phenotypic effect in a large resident population. Mutant fitness, and in particular its selective advantage *s*, depends on the resident population size as well as the environmental fluctuations and may in some circumstances be negative as long as the long-term mean is positive. Solutions are given for both discrete time and continuous time dynamics. The continuous time case allows for overlapping generations and is a particularly suitable model for unicellular organisms that reproduce through fission, such as bacteria or protozoa.