### Abstract

- Top of page
- Abstract
- Introduction
- Model
- Analysis
- Stationary distribution in terms of phenotypic selection gradient
- Applications
- Discussion
- Acknowledgments
- References
- Appendices

Many traits and/or strategies expressed by organisms are quantitative phenotypes. Because populations are of finite size and genomes are subject to mutations, these continuously varying phenotypes are under the joint pressure of mutation, natural selection and random genetic drift. This article derives the stationary distribution for such a phenotype under a mutation–selection–drift balance in a class-structured population allowing for demographically varying class sizes and/or changing environmental conditions. The salient feature of the stationary distribution is that it can be entirely characterized in terms of the average size of the gene pool and Hamilton's inclusive fitness effect. The exploration of the phenotypic space varies exponentially with the cumulative inclusive fitness effect over state space, which determines an adaptive landscape. The peaks of the landscapes are those phenotypes that are candidate evolutionary stable strategies and can be determined by standard phenotypic selection gradient methods (e.g. evolutionary game theory, kin selection theory, adaptive dynamics). The curvature of the stationary distribution provides a measure of the stability by convergence of candidate evolutionary stable strategies, and it is evaluated explicitly for two biological scenarios: first, a coordination game, which illustrates that, for a multipeaked adaptive landscape, stochastically stable strategies can be singled out by letting the size of the gene pool grow large; second, a sex-allocation game for diploids and haplo-diploids, which suggests that the equilibrium sex ratio follows a Beta distribution with parameters depending on the features of the genetic system.

### Introduction

- Top of page
- Abstract
- Introduction
- Model
- Analysis
- Stationary distribution in terms of phenotypic selection gradient
- Applications
- Discussion
- Acknowledgments
- References
- Appendices

Many phenotypes are quantitative and can be measured on a continuous scale. For instance, allocations of resources to growth, survival, defence, male and female function or offspring production are continuously varying strategies. Body shape and size, rates of transcription, enzymatic fluxes, intensities of desires, dates of first flowering or maximum flight speed are all phenotypes belonging to a continuum. Because of such a prevalence of continuous phenotypes in natural populations, it is relevant to try to understand their evolutionary dynamics and stationary distributions under the joint pressure of mutation, natural selection and random genetic drift. Nevertheless, few studies have analytically addressed the evolution of quantitative phenotypes under the action of these three evolutionary forces, and they often focus on situations of frequency-independent selection, where the recipient of the expression of the phenotype is the actor alone (e.g. Lande, 1976; Bürger *et al.*, 1989; Bürger & Lande, 1994).

Because resources come in finite supply, many phenotypic traits are actually subject to frequency-dependent selection at the intraspecific level, where the behaviour of one individual affects the fitness of others. These include resource competition efforts, mating and foraging tactics, sex ratio, optimal dispersal, parent–offspring conflict, anisogamy, storage effects, levels of social learning or waiting times in attrition fighting. The evolution of continuous phenotypes with frequency-dependent selection is more complicated to analyse than without, and simplifying assumptions are necessary in order to make the analysis tractable. Key assumptions include removing from the analysis one or several evolutionary forces, generally mutation and/or genetic drift, and focusing on a two-allele system coding for mutant and resident phenotypes, where the mutant deviates phenotypically only by small magnitude from the resident. Under these general assumptions, and at the risk of oversimplifying the presentation, one can identify three interrelated approaches for studying the evolution of continuous phenotypes.

The first could be labelled classical evolutionary game or kin selection theory for continuous phenotypes (e.g. Maynard Smith, 1982; Eshel, 1983; Taylor, 1989; Parker & Maynard Smith, 1990; Bulmer, 1994; Taylor & Frank, 1996; Frank, 1998; Pen, 2000; Ohtsuki & Iwasa, 2004; Vincent & Brown, 2005; Lion & Gandon, 2009). Here, the population is assumed to be of total infinite size. Genetic drift at the global scale is thus removed from the model and mutations are not explicitly considered in the formalization, as one is essentially interested in characterizing the end points of the evolutionary dynamics. These are the candidate evolutionary stable strategies (ESS). In practice, they are obtained from phenotypic selection gradients often through the form of the optimization of an individual fitness function (Maynard Smith, 1982; Parker & Maynard Smith, 1990; Vincent & Brown, 2005).

Because stable strategies are identified by comparing the fitness of pairs of strategies, namely, by focusing on the mutant-resident system, implicit in classical evolutionary game theory is an evolutionary dynamic that is assumed decomposable into two time scales (Eshel, 1996; Hammerstein, 1996; Eshel *et al.*, 1998): first, a fast time scale of short-term evolution. This is the time scale during which a novel mutation appears in a population monomorphic for a resident phenotype, and is either eliminated or selected to fixation before any other new mutation appears. The superposition of several of these trait-substitution events yields the second, slower time scale of steady long-term evolution of the phenotype. Evolution is thus regarded as a step-by-step transformation of the phenotype caused by the successive invasion of rare mutant alleles. The orbit of the phenotype in state space eventually converges towards a singular point, a cycle, or is altered forever in a strange attractor (Eshel, 1996; Hammerstein, 1996; Eshel *et al.*, 1998).

The second approach to the evolution of continuous phenotypes is adaptive dynamics. This broadens the first by focusing not only on phenotypic selection gradients but also on the time course of evolution (e.g. Dieckmann & Law, 1996; Geritz *et al.*, 1998; Ferrière *et al.*, 2002; Waxman & Gavrilets, 2005; Champagnat *et al.*, 2006; Dercole & Rinaldi, 2008; Leimar, 2009; Zu *et al.*, 2010). Here, evolution is also assumed to be decomposable into a two-time scale dynamics, but long-term evolution is made more explicit by the incorporation into the formalization of mutation rates and the evaluation of the time dynamics of the phenotype itself. In addition to characterizing candidate ESS and other singular points (phenotypic values at which the local selection gradient vanishes), the adaptive dynamics approach also allows one to explicitly track the changes in phenotype along the orbits in phenotype space towards singular points or through other attractors (Dercole & Rinaldi, 2008). But as under classical game theory, the stochastic effects introduced by genetic drift are often ignored in practice and candidate ESS are obtained from phenotypic selection gradients by way of the optimization of an individual fitness function (Geritz *et al.*, 1998; Dercole & Rinaldi, 2008).

The third approach to the evolution of continuous phenotypes under frequency-dependent selection may be called kin selection (or inclusive fitness) theory for finite populations (e.g. Rousset & Billiard, 2000; Leturque & Rousset, 2002; Roze & Rousset, 2003; Rousset, 2004; Rousset & Ronce, 2004; Taylor *et al.*, 2007a, b). Here, as under the two other approaches, a two-time scale evolutionary dynamic is assumed. As under classical evolutionary game theory, mutations to all possible phenotypes are not explicitly taken into account in the formalization. But, in contrast to the two other approaches, short-term phenotypic evolution is explicitly determined from changes (perturbations) of the fixation probability of a mutant allele introduced into a monomorphic population of residents. The fixation probability captures the effect of both natural selection and random genetic drift on the evolutionary dynamics, from the appearance of a mutant until its loss from or fixation in the population. Importantly, the fixation probability perturbations turn out to be proportional to phenotypic selection gradients for weak selection intensities, so that in practice candidate ESS are obtained from the optimization of an individual fitness function, as under the two other approaches (Leturque & Rousset, 2002; Rousset, 2004).

The identification of singular points of the evolutionary dynamics for continuous phenotype is thus obtained by broadly similar methods throughout evolutionary biology, and whether evolution occurs in finite populations (stochastic systems) or infinite populations (deterministic systems). But in the presence of several singular points, which may occur when the adaptive landscape is multipeaked, the long-term behaviour of a stochastic system may differ markedly from that of a deterministic system. Constant dynamic shocks introduced by the flow of mutations and the sampling effects occurring in finite population may accumulate and tip the balance from one singular point to the other. For a multipeaked fitness landscape, a higher peak may then eventually be singled out by the evolutionary dynamics even if the population can remain locked in a suboptimal peak for a very long time. This state space exploration process due to the interaction between mutation, selection and drift is ingrained in population genetics (Wright, 1931; Barton *et al.*, 2007; Hartl & Clark, 2007) and used as an equilibrium selection device in game theory (Foster & Young, 1990; Binmore *et al.*, 1995), but it has not been much explored in the context of the evolution of continuous phenotypes.

In this article, the substitution rate approach to the separation between short- and long-term evolution of population genetics (Gillespie, 1983, 1991; Orr, 1998; Sella & Hirsh, 2005) is used in order to derive a diffusion equation for the evolution of a continuous phenotype. Mutation, natural selection and random genetic drift are allowed to jointly affect the evolutionary dynamic when it takes place in a class-structured population with demographically varying class sizes and/or changes in environmental conditions. The approach highlights strong links between the adaptive dynamics framework and the direct fitness (or neighbour-modulated) method of kin selection theory. The article is organized as follows: Model introduces the biological assumptions of the model and specifies the separation of time scales hypothesis. Analysis derives a phenotypic substitution rate for class-structured populations (fast time scale) and a diffusion equation for long-term phenotypic evolution (slow time scale). Stationary Distribution in Terms of Phenotypic Selection Gradient connects the stationary distribution of the slow process to standard phenotypic selection gradients. Applications presents two applications of the stationary distribution, and Discussion discusses the results.

### Discussion

- Top of page
- Abstract
- Introduction
- Model
- Analysis
- Stationary distribution in terms of phenotypic selection gradient
- Applications
- Discussion
- Acknowledgments
- References
- Appendices

The stationary distribution of a one-locus continuous phenotype under a mutation–selection–drift balance in a class-structured population has been derived under the assumptions of weak selection intensities and a separation of time scales between short- and long-term evolution. If mutation rates are the same across classes and the mutation machinery is independent of the evolving phenotype, the stationary distribution can be entirely characterized in terms of the average size of the gene pool and Hamilton's (1964) inclusive fitness effect for demographically structured populations of finite size (Rousset, 2004; Rousset & Ronce, 2004; Taylor *et al.*, 2007b).

The stationary distribution shows that the exploration of the phenotypic space at steady state varies exponentially with the inclusive fitness effect cumulated over state space, which determines an adaptive landscape (eqn. 7, Figs 1 and 2). For a multipeaked fitness landscape, the various peaks of the landscape are those phenotypes that are candidate evolutionary stable strategies. The curvature of the stationary distribution at a candidate evolutionary stable strategy provides a natural measure of its stability by convergence (eqn. 8 and ineq. 9), which is consistent with those obtained in previous analyses (Eshel, 1983; Taylor, 1989; Geritz *et al.*, 1998; Rousset, 2004).

The results of this paper support Gillespie's (1991) enthusiasm that the separation of time between long- and short-term evolution makes tractable an apparently intractable model, which captures realistic aspects of natural populations, such as finite size, frequency-dependent selection, class structure, varying demography and mutation rates. Further, the stationary distribution of the phenotype can be expressed in terms of standard quantities; namely, phenotypic selection gradient obtained as derivatives of individual fitness functions weighted by relatedness coefficients, which are commonly used in evolutionary biology (Wenseleers *et al.*, 2010). These standard approaches thus allows one to obtain an approximate, but calculable estimate of the phenotypic distribution, the drift load or the variance in phenotype maintained in a population at a mutation–selection–drift balance, to which relaxing assumptions can be compared. In addition to identifying candidate evolutionary stable strategies, the stationary distribution also allows one to select among such alternatives and to identify stochastically stable strategies (Foster & Young, 1990, Binmore *et al.*, 1995) by letting the average size of the gene pool grow large.

The results of this paper also point to connections between the adaptive dynamics framework (Dieckmann & Law, 1996; Geritz *et al.*, 1998; Dercole & Rinaldi, 2008) and the direct fitness method of kin selection theory (Taylor & Frank, 1996; Rousset, 2004; Wenseleers *et al.*, 2010). The model developed here was inspired by these two approaches: in particular, the emphasis of adaptive dynamics on evaluating the time dynamics of evolving phenotypes and the emphasis of kin selection theory for finite populations on stochastic elements affecting the fate of mutant alleles. Here, as has already been suggested for branching points determination (Ajar, 2003), the direct fitness method of kin selection theory can be envisioned as adaptive dynamics at the intraspecific level with mutant–mutant interactions. Such interactions are difficult to avoid in small populations, as two interacting individuals are likely to descend from the same recent common ancestor.

One main limitation of the model from a theoretical point of view is the heuristic assumption of a separation of time scales between short- and long-term evolution (eqn. 1). Conditions on the mutation rate guaranteeing convergence to the separation of time scales would be interesting to document and could be addressed by more mathematically inclined research. Evaluating expressions for the average fixation probability of a mutant in the presence of varying mutation rates across classes should also be interesting, as this is relevant for the evolution in age-structured populations. From a more biological perspective, one main limitation of the model is its one-dimensional phenotypic nature. Addressing the co-evolution of multiple phenotypic traits and/or multispecies interactions opens avenues for future explorations.