Competitive speciation and costs of choosiness


  • M. KOPP,

    1. Section of Evolutionary Biology, Department Biology II, Ludwig-Maximilian University Munich, Planegg-Martinsried, Germany
    2. Mathematics and Biosciences Group, Max F. Perutz Laboratories and Faculty of Mathematics, University of Vienna, Vienna, Austria
    Search for more papers by this author

    1. Section of Evolutionary Biology, Department Biology II, Ludwig-Maximilian University Munich, Planegg-Martinsried, Germany
    2. Mathematics and Biosciences Group, Max F. Perutz Laboratories and Faculty of Mathematics, University of Vienna, Vienna, Austria
    Search for more papers by this author

Michael Kopp, Dr Bohrgasse 9, A-1030 Vienna, Austria.
Tel.: +43 1 79044 4583; fax: +43 1 79044 4551;


We investigate how costs of choosiness affect the evolution of assortative mating in a simple model of competitive speciation. The model allows for a comprehensive analysis by analytical and numerical techniques. We obtain results for two types of costs: mating costs, which restrict the number of males a choosy female can evaluate, and viability costs, which decrease a choosy female's survival. Mating costs significantly reduce the range of parameters for which speciation is possible, but only if the number of males a female can evaluate is small (less than 10). This type of costs can be eliminated if females are allowed to mate randomly at the end of the mating period. Although, in this case, it is not possible to achieve complete reproductive isolation, our results show partial isolation with strong phenotypic clustering. Viability costs counteract selection for assortative mating. As this selection is typically weak, speciation is possible only if viability costs, too, are weak. The above restrictions are less severe if extreme phenotypes have an intrinsically higher carrying capacity.


In the last decade, the study of sympatric speciation has gained much momentum (Coyne & Orr, 2004; Dieckmann et al., 2004; Gavrilets, 2004; Bolnick & Fitzpatrick, 2007). This has been in part due to empirical research (Schliewen et al., 1994; Gíslason et al., 1999; Barluega et al., 2006b; Savolainen et al., 2006a) and in part due to advances in theoretical modelling (e.g. Dieckmann & Doebeli, 1999; Kondrashov & Kondrashov, 1999; Kirkpatrick & Ravigné, 2002; Gavrilets, 2004). It is now widely accepted that sympatric speciation can – and does – occur (Bolnick & Fitzpatrick, 2007). However, it is still very much debated how common it is in nature. This controversy applies to both empirical data (Barluega et al., 2006a; Savolainen et al., 2006b; Schliewen et al., 2006; Stuessy, 2006) and models (Doebeli & Dieckmann, 2005; Doebeli et al., 2005; Gavrilets, 2005; Polechová & Barton, 2005; Waxman & Gavrilets, 2005a, b; Doebeli et al., 2007).

On the theoretical side, part of the debate has focused on costs of choosiness. One possible path to sympatric speciation is the evolution of nonrandom mating in populations subject to frequency-dependent disruptive selection. For example, in populations experiencing intraspecific competition, fitness may be reduced for individuals with an intermediate phenotype. Females can prevent the production of low-fitness offspring by choosing males whose phenotype is close to their own. In other words, there is selection for positive assortative mating (Rosenzweig, 1978; Doebeli, 1996; Dieckmann & Doebeli, 1999; Doebeli & Dieckmann, 2000; Matessi et al., 2001; Bolnick, 2004; Doebeli, 2005; Bolnick, 2006; de Cara et al., 2008; Otto et al., in press; Pennings et al., 2008; Ripa, in press), in a process similar to reinforcement (Servedio & Noor, 2003). If assortative mating gets strong enough, the population can split into two reproductively isolated clusters (Gourbiere, 2004; Bürger & Schneider, 2006; Bürger et al., 2006; Schneider & Bürger, 2006; Doebeli et al., 2007). However, speciation may fail if choosy females experience fitness costs.

Whether or not costs of choosiness represent a serious obstacle for sympatric speciation has been a matter for much recent debate (Matessi et al., 2001; Bolnick, 2004; Gourbiere, 2004; Kirkpatrick & Nuismer, 2004; Doebeli, 2005; Doebeli & Dieckmann, 2005; Gourbiere & Mallet, 2005; Waxman & Gavrilets, 2005a, b; Bürger & Schneider, 2006; Bürger et al., 2006; Schneider & Bürger, 2006; de Cara et al., 2008; Otto et al., in press). For example, Gavrilets repeatedly claims that ‘costs of being choosy…can easily prevent speciation’ (Gavrilets, 2004, p. 398) or even that ‘absence of costs’ is among the ‘conditions for sympatric speciation’ (Gavrilets, 2005). Others have rejected this conclusion and maintain that ‘the evolution of assortative mating remains possible even when such costs are substantial’ (Doebeli & Dieckmann, 2005, see also Doebeli, 2005; Doebeli et al., 2007).

There are several reasons for this disagreement. To some degree, what constitutes ‘substantial’ costs or ‘possible’ speciation is a matter of personal interpretation. Different authors have used different criteria for successful speciation (e.g. evolution of strong assortative mating vs. formation of phenotypic clusters), and have focused on different aspects of the speciation process (strength of selection for assortative mating, time to speciation, maintenance of polymorphism and cluster formation). Matters are further complicated by the fact that results are usually derived from individual-based simulations, which cover only a limited range of parameters and make generalizations difficult. Only very recently, analytical approaches have been suggested by three groups of authors (de Cara et al., 2008; Otto et al., in press; Pennings et al., 2008; building on preliminary work by Matessi et al., 2001), which allow for a comprehensive understanding of the evolutionary dynamics under simple but nontrivial conditions.

Here, we use the model by Pennings et al. (2008) to analyse costs of choosiness. The key simplification of this model relative to previous studies (e.g. Dieckmann & Doebeli, 1999; Bolnick, 2004) is that the genetic basis of the ecological trait (which underlies competition) is reduced to a single biallelic locus. The benefit is that the model can be analysed in great detail and in the full parameter space by using a combination of analytical and numerical techniques. In Pennings et al. (2008), choosiness was cost free. Here, we introduce two types of costs, which we refer to as mating costs and viability costs. Mating costs arise if choosy females run the risk of not finding an acceptable male before the end of the mating period (Gavrilets & Boake, 1998; Arnegard & Kondrashov, 2004; Bolnick, 2004; Bürger & Schneider, 2006; Bürger et al., 2006; Schneider & Bürger, 2006). Viability costs directly decrease the survival of choosy females (Doebeli et al., 2007; Otto et al., in press), for example, due to an increased predation risk. Our main interest is in how costs of choosiness alter the strength and direction of selection on assortative mating, and, in particular, whether costs prevent the evolution of phenotypic isolation.

Model description

Below, we give a short, but self-contained description of the model; for additional details, the reader is referred to Pennings et al. (2008). We assume that natural selection acts on a single, diploid locus (termed the ecological locus) with two alleles, ‘+’ and ‘−’. The three ecological genotypes (+/+), (+/−) and (−/−) may be viewed as coding for a quantitative trait (e.g. with value x, 0 and −x), but we will not need to make explicit reference to the phenotype in most of the following. The total population size is inline image, where inline image and inline image denote the number of individuals in the three genotype classes, and ‘hom’ and ‘het’ stand for homozygotes and heterozygotes. In symmetric cases, we will write inline image. The relative genotype frequencies will be denoted by inline image and Phet = Nhet/N. Unless otherwise stated, the terms ‘homozygotes’ and ‘heterozygotes’ will always refer to the ecological genotype. Time is continuous, and generations are overlapping. Individuals are assumed to be either simultaneous hermaphrodites (no selfing) or males and females with a sex ratio of 1 : 1.

Viability selection

Viability selection is modelled using a Lotka–Volterra approach (Roughgarden, 1972). The carrying capacity for heterozygotes is Khet, whereas the carrying capacity for both homozygotes is Khom = Khet(1−k), with k ≤ 1. For k > 0, Khet > Khom, that is, the carrying capacity function (K as a function of phenotype) is unimodal, whereas for k < 0, Khet < Khom and the carrying capacity function is bimodal. The strength of competition is 1 between identical genotypes, 1 − c between heterozygotes and either of the homozygotes, and 1 − c ′ between the two different homozygotes (c ′ ≥ c). Typically, we will assume c ′ = 1 − (1 − c)4, which corresponds to a Gaussian relationship at the phenotypic level. The total amount of competition experienced by the various genotypes is

image( (1a))
image( (1b))

These can be viewed as ‘ecologically effective population sizes’, with 1 − c and 1 − c ′ acting as competition coefficients. In the absence of viability costs (see below), the genotype-specific death rates are given by


An extension of the model to cases with an asymmetric carrying capacity or competition function is possible (see Pennings et al., 2008), but did not lead to qualitative changes in the results. We therefore limit our analysis to the symmetric case here and only briefly comment on the asymmetric case in the Results section.

Assortative mating

Assortative mating is modelled as female choice and is based on similarity with respect to the ecological locus. Females are characterized by evolvable mating traits m and m′, which indicate their readiness to reject males of a different genotype. When a female encounters a male of her own genotype mating occurs with probability 1. When a heterozygous female encounters a homozygous male (or vice versa), mating occurs with probability 1 − m. Similarly, when a (+/+) female meets a (−/−) male (or vice versa), mating occurs with probability 1 − m′. In the following, we will always assume that m and m′ are determined by the same set of genes and that m′ is an increasing function of m. For numerical calculations (e.g. in the figures), we choose m′ = 1 − (1 − m)4, making mating probability a Gaussian function of phenotypic distance. We consider two types of costs: mating costs and viability costs.

Mating costs

Mating costs arise if a choosy female runs the risk of not finding a suitable mate by the end of the mating period. Here, we follow an approach introduced by Gavrilets & Boake (1998) and subsequently used by a number of authors (Matessi et al., 2001; Arnegard & Kondrashov, 2004; Bolnick, 2004; Doebeli, 2005; Waxman & Gavrilets, 2005b; Bürger & Schneider, 2006; Bürger et al., 2006; Schneider & Bürger, 2006). Denote the probability that an encounter of a female with a random male leads to mating by

image( (3a))
image( (3b))

Furthermore, assume that there is a fixed maximum number M of encounters or mating trials before the mating period ends. Then, the mating rates of homozygous and heterozygous females are

image( (4a))
image( (4b))

which depend on m and m′. For M → ∞, mating costs are absent and we recover model 2 of Pennings et al. (2008). In the terminology of Kirkpatrick & Nuismer (2004), this is the ‘animal model’, whereas the opposite case M = 1 is the ‘plant model’.

 The rates of matings between females of type i and males of type j, Qi,j, are given by

image( (5a))
image( (5b))
image( (5c))
image( (5d))
image( (5e))

To understand these equations, note that φfemale gives the probability that a female mates at all, whereas the fractional term gives the probability that she mates with a male of a particular genotype.

Flexible mating strategy

Females can avoid costs of choosiness by accepting a random male at the end of the mating period (Doebeli & Dieckmann, 2005). Therefore, we also consider a variant of the above model, in which females that have been unsuccessful in M trials at assortative mating go on to mate randomly at their next encounter. In this case, inline image and φfemale,het are the rates of assortative mating only, and the total pairwise mating rates Qi,j become

image( (6a))
image( (6b))
image( (6c))
image( (6d))
image( (6e))

Viability costs

Alternatively, or in addition to effects on female mating rate, costs of choosiness could also affect viability (see Doebeli, 2005; Doebeli & Dieckmann, 2005; Doebeli et al., 2007; Otto et al., in press). There can be a number of reasons for this, such as direct costs of the choosiness mechanism, increased predation risk due to mate choice, or a disadvantage in competition for the best feeding or nesting sites.

We consider two subtypes of viability costs: absolute and relative ones (using the terminology of Otto et al., in press). Absolute viability costs depend directly on the parameters m and m′, independent of the availability of mating partners. For example, absolute costs could arise if the choosiness mechanism itself is costly. For mathematical reasons, we assume a soft selection scheme, where costs depend on the difference between a female's own choosiness and the mean choosiness in the population:


with a cost function f that is nondecreasing in both of its arguments. An example is the simple linear cost function


where δ,δ ≥ 0 measure costs linked to m and m′ respectively. Because of soft selection, these costs do not affect the dynamics of a population that is monomorphic with respect to choosiness. Absolute viability costs can be introduced in various alternative ways, but models seem to behave very similar as long as costs are not too strong. For example, we also numerically analysed a hard selection model with a multiplicative cost term, d· = (C·/K·)(1 + δm + δm′), and obtained results comparable to the ones discussed below (not shown).

 In contrast to absolute costs, relative costs depend on the number of males a female rejects before mating. Mating costs, as introduced above, are necessarily relative. However, also viability costs can easily be relative, for example if predation risk is proportional to the number of unsuccessful mating trials. In particular, we assume


The coefficient γk measures the cost for the kth unsuccessful mating trial. If costs are equal for each trial, γk ≡ γ, eqn 9 simplifies to


where (1/π·−1) is the average number of rejected males per female of a given ecological type.

 We only consider viability costs that act equally on males and females. For hermaphrodites, this assumption holds automatically. For separate sexes, it is a necessary condition to maintain an 1 : 1 sex ratio.

Although, in principle, different types of costs can act simultaneously, in this paper, we will only analyse them separately. Thus, a population subject to mating costs will not also experience viability costs and vice versa.

Population dynamics

Using the above definitions and the laws of Mendelian inheritance, the birth rates of the various ecological genotypes are given by

image( (11a))
image( (11b))

Finally, the population dynamics are given by

image( (12a))
image( (12b))

Equations 11a, 11b, 12a and 12b assume that all females in the population follow the same mating strategy (i.e. their m and m values are identical). In the next section and in the Appendix, we show how the model can be extended to incorporate mutant mating strategies.

Invasion analysis

Our main aim is to understand the evolution of the choosiness parameters m and m′, which we will summarize in the vector m = (m,m′). For this purpose, we assume a population that is monomorphic with respect to m, and we ask under what conditions a rare mutant with a slightly different value inline image can invade. (Recall that m′ is an increasing function of m; so, a single mutation will increase or decrease both parameters.) Without costs of choosiness, considerable analytical progress is possible. In particular, it can be shown that mutants with increased choosiness can invade if and only if the total fitness of homozygotes is greater than that of heterozygotes (where fitness is determined by both natural and sexual selection, see below and Pennings et al., 2008). The reason is that choosy females tend to have more homozygous offspring. With costs of choosiness, no such simple criterion exists, because females now face a trade-off between their own fitness and that of their offspring. Thus, invasion depends not only on the sign but also on the magnitude of the fitness difference between homozygotes and heterozygotes. Choosiness can increase only if the benefit of having more homozygous offspring more than offsets the disadvantage of having a reduced mating rate or viability. To determine the strength and direction of selection on m, it is therefore necessary to calculate the invasion fitness gradient (Geritz et al., 1998).

We assume that the population dynamics are fast relative to the evolutionary dynamics (Hammerstein, 1996). This means that invasion of new modifier alleles for choosiness always occurs at the population dynamical equilibrium of a ‘resident population’ with a monomorphic mating strategy. For speciation to be possible, the resident population must be polymorphic at the ecological locus. Due to the symmetries of the model, a symmetric equilibrium with inline image always exists. For viability costs with soft selection, the ratio n = Nhet/Nhom at this equilibrium can be derived following Pennings et al. (2008). In all other cases, it needs to be calculated numerically for 0 < m < 1. The stability of the symmetric equilibrium is not guaranteed under nonrandom mating (Matessi et al., 2001; Kirkpatrick & Nuismer, 2004; Bürger & Schneider, 2006; Bürger et al., 2006; Pennings et al., 2008). It can be determined by numerically calculating the leading eigenvalue of system 12a and 12b.

Let νhet be the number of heterozygote individuals (with genotype (+/−) at the ecological locus) that carry the mutant choosiness allele. Similarly, νhom is the total number of mutant homozygotes [i.e. the sum of the number of mutants with ecological genotypes (+/+) and (−/−)]. The dynamics of the mutant subpopulation is described by


with the elements of the matrix inline image being

image( (14a))
image( (14b)))
image( (14c))
image( (14d))

where terms marked with a tilde refer to the mutants. The leading eigenvalue inline image of inline image is the invasion fitness (Metz et al., 1992) for the inline image mutant. It can be visualized using pairwise invasibility plots (Geritz et al., 1998; see Appendix 1 for some examples and discussion of peculiarities). The direction of selection for choosiness is determined by the invasion fitness gradient (Geritz et al., 1998)


D describes how, in the vicinity of the resident mating strategy, a rare mutant's strategy influences its (invasion) fitness. Choosiness (m) increases if D is positive and decreases if D is negative. A convergence-stable intermediate equilibrium is achieved if D = 0 and ∂D/∂m < 0. In most cases, a convergence-stable equilibrium for m also is an evolutionarily stable strategy (ESS). However, in Appendix 1, we also present an example where it is a branching point, which might give rise to a polymorphic mating strategy in the population (Fig. A1f).

Figure A1.

 Pairwise invasibility plots for female choosiness m in the presence of costs of choosiness. M is the maximal number of mating trials per female in the model with mating costs. δ determines the magnitude of absolute viability costs linked directly to m. For each resident choosiness value m, it is shown whether a mutant with choosiness inline image can (white) or cannot (grey) invade the resident population. Intersections of boundary lines mark evolutionarily singular points (Geritz et al., 1998). (a) No costs. For the chosen ecological parameters (k = 0.12, c = 0.4), there are two such points, with the lower one being an evolutionary attractor and the higher one an evolutionary repellor (corresponding to the P/C regime). (b, c) Mating costs (δ = δ′ = 0). The evolutionary attractor is uninvasible, i.e. it is an evolutionarily stable strategy (ESS). (d–f) Absolute viability costs linked to m (M = δ′ = 0). The insets show close-ups of the neighbourhood of the evolutionary attractor (the left-most singular point). In (d) and (e), the attractor is an ESS. In (f), the attractor is invasible and, hence, an evolutionary branching point.

In Appendix 2, we derive the invasion fitness gradient for mating costs as well as absolute and relative viability costs. We also determine the sign of the invasion fitness gradient at random mating (m = m′ = 0) and at complete isolation (m = m′ = 1), which determines whether these two states can be evolutionary endpoints. For intermediate m, we must resort to numerical analysis of eqn 15. The model with a flexible mating strategy is analysed in Appendix 3. We show that, in this model, invasion again depends only on the sign of the fitness difference between homozygotes and heterozygotes.


Basic model without costs

The basic model without mating or viability costs (with M → ∞ and f = 0) has been analysed in Pennings et al. (2008). Here, we briefly summarize its results. In the model, all females are assured to mate, but rare males are less likely to be chosen by a female. Thus, female choosiness induces sexual selection against rare males (Kirkpatrick & Nuismer, 2004). (This effect has been termed a ‘cost to rarity’ by Doebeli & Dieckmann (2003), but it is not a cost that is paid by the choosy females themselves.) Because females do not pay for being choosy, choosiness increases if and only if homozygotes have higher fitness than heterozygotes. Depending on the ecological parameters k and c, there are five regimes for the evolution of female choosiness (Fig. 1a). Three regimes are monostable and lead to a unique outcome: random mating (m = 0, R), partial isolation (0 < m < 1, P) or complete isolation (m = 1, C). In addition, there are two bistable regimes, in which the outcome depends on initial conditions: random mating or complete isolation (R/C), and partial isolation or complete isolation (P/C). Furthermore, for low values of c, the polymorphic equilibrium at the ecological locus (which is a prerequisite for speciation) is unstable for at least some (intermediate) values of m (see Bürger & Schneider, 2006; Bürger et al., 2006), which prevents the evolution of complete isolation in small steps.

Figure 1.

 Evolutionary regimes for female choosiness m in the model with mating costs, as a function of the ecological parameters k and c. k is the strength of stabilizing (positive values) or disruptive (negative values) selection arising from phenotype-specific carrying capacities. c is the strength of frequency-dependent selection induced by competition. Mating costs are quantified by the number of mating trials per female, M. M → ∞ corresponds to the model without costs (model 2 of Pennings et al., 2008). The five regimes are named according to the outcome of evolution in small steps: R = random mating (m = 0); P = partial isolation (0 < m < 1); C = complete isolation (m = 1); R/C = random mating or complete isolation; P/C = partial or complete isolation. In the R/C and P/C regimes, low initial values of m lead to random mating or partial isolation, respectively, whereas high initial values of m lead to complete isolation. Only points where complete isolation can be reached from an initial value of m ≤ 0.999 (m ≤ 0.823) are assigned to these bistable regimes (see main text). In (a), the dotted line shows the boundary of the bistable regimes if the domain of attraction of the complete isolation equilibrium can be arbitrarily small (critical m → 1). In (b–f) (i.e. for M < ∞), the complete isolation equilibrium is always locally stable, but often with a vanishingly small domain of attraction. In the hatched area, the polymorphic equilibrium at the ecological locus is unstable for at least some values of m. In (d) and (e), the C regime is present for values of k smaller than those shown in the plots. In (f), only the R regime exists, and the polymorphic equilibrium is always unstable.

The pattern of the invasion fitness gradient in each regime is shown in Fig. 2, along with the fitness difference between homozygotes and heterozygotes (ΔW). Note that, although both functions have the same roots, they have different shapes. In particular, in the C, P/C and R/C regimes, D tends to zero for m → 1, because, in this parameter region, heterozygotes are almost absent and a change in m has little influence on the offspring phenotype distribution. This effect is not reflected in the fitness difference ΔW. Therefore, it is always the invasion fitness gradient that determines not only the direction but also the speed of evolution (according to the ‘canonical equation’ of adaptive dynamics; Dieckmann & Law, 1996).

Figure 2.

 The invasion fitness gradient D (solid line, left axis) and the fitness difference between homozygotes and heterozygotes ΔW (dashed line, right axis) as a function of female choosiness m, in the absence of costs and for each of the five evolutionary regimes (cf. Fig. 1a). Choosiness increases whenever D and ΔW are positive and decreases when they are negative. Parameters: (C) k = 0.05,c = 0.3, (P) k = 0.05,c = 0.8, (R) k = 0.6,c = 0.95, (PC) k = 0.12,c=0.4, (RC) k = 0.2,c = 0.2.

In Pennings et al. (2008), we explain the above results by the interplay of natural and sexual selection. Briefly, natural selection (due to the carrying capacity and competition functions) is negatively frequency dependent and reflects the structure of ecological niches. It favours a specific proportion of heterozygotes (which may be zero). Sexual selection (due to female choosiness) is positively frequency dependent and favours common genotypes. It is sexual selection that creates the bistability in the R/C and P/C regimes and the potential instability of the ecological polymorphism. Both natural and sexual selection can prevent the evolution of full assortative mating and, instead, lead to a stable intermediate value of choosiness.

We note two differences in Fig. 1a when compared with the analogous Fig. 1c in Pennings et al. (2008): first, the present figure extends to negative k values, showing (not surprisingly) that speciation is easiest if the carrying capacity function is bimodal (provided competition creates enough negative frequency dependence to maintain the ecological polymorphism). Second, we demand that the domain of attraction of complete isolation in the P/C and R/C regimes must not be too small. In Fig. 1, parameter combinations are included into these regimes only if complete isolation evolves from a starting population with m′ ≤ 0.999 (meaning that the probability of a (+/+) female to accept a (−/−) male, and vice versa, is ≥ 0.001). The original boundary for stability of complete isolation is included as a dotted line in Fig. 1a, which shows that there is a large area in parameter space where complete isolation is mathematically stable, but has a vanishingly small domain of attraction.

Mating costs

Mating costs do not change the basic structure with five evolutionary regimes. However, mating costs induce sexual selection against rare and choosy females (in addition to sexual selection against rare males). Therefore, for given values of k and c, mating costs decrease the likelihood that complete isolation can evolve from random mating in small steps. This results in a shift of the complete isolation regime towards lower (and negative) k values (Fig. 1b–f).

Yet, marked deviations from the no-costs case appear only for M ≲ 10. A special case is given for M = 1, where no amount of choosiness can evolve at all. This is easy to understand, as a female that has only a single mating opportunity should never reject it. The following discussion refers only to the case M ≥ 2.

Figure 3a illustrates how mating costs affect the invasion fitness gradient D(m). Costs decrease D for intermediate values of choosiness, but not for very small or very large m. In the limit m → 0, D remains unchanged. This means that local stability of the random mating regime is unaffected by mating costs (Appendix 2). The reason is that any benefit of being choosy scales linearly with m for m→0, whereas the costs scale like mM (the probability to reject a mating partner is proportional to m, and females go unmated if they reject M mating partners). Consequently, the line separating the C, P and P/C regimes from the R and R/C regimes is identical in all subplots of Fig. 1 with M ≥ 2.

Figure 3.

 The invasion fitness gradient D(m) for female choosiness m in the presence of costs of choosiness. For values of m to the right of the crosses, the proportion of heterozygotes is less than 1/3 (bimodal phenotype distribution). To the right of the dots, the proportion of heterozygotes is less than 0.01 (strong phenotypic clustering). The ecological parameters are k = 0.05, c = 0.3 (cf. Fig. 2, first panel). (a) Mating costs with a limited number of mating trials, M, per female. From top to bottom M = ∞ (thick line), 10, 8, 6, 5, 4, 3, 2, 1. Decreasing M leads to a switch from the C regime (M = 10, 8, 6 to the P/C regime (M = 5, 4, 3) and further on to the P regime (M = 2) and R regime (M = 1), cf. Fig. 1. Note the break in the y-axis and the different scales in its lower and upper parts. (b) Absolute viability costs linked directly to m (i.e. eqn 8 with δ = 0), with (from top to bottom) δ = 0, 0.01, 0.02, 0.03, 0.04. (c) Absolute viability costs linked directly to m (i.e. δ = 0), with (from top to bottom) δ = 0, 0.01, 0.02, 0.03, 0.04. (d) Relative viability costs linked to the average number of males rejected by a female before mating (according to eqn 10), with (from top to bottom), γk ≡ γ = 0, 0.01, 0.02, 0.03, 0.04.

For m → 1, mating costs induce strong sexual selection against heterozygotes (both males and females), as long as these are very rare. As a consequence, complete isolation is always locally stable, at least in the mathematically strict sense (Appendix 2). This means that mating costs actually increase the invasion fitness gradient in the P and R regimes, where complete isolation is unstable without costs (note that Fig. 3a shows only the C regime). However, this effect is limited to values of m very close to 1 and is thus of no real biological significance. Indeed, the domain of attraction of the m = 1 equilibrium is exceedingly small for parameter values outside those marked as the C, P/C and R/C regimes in Fig. 1 (see above). Once we require a minimal domain of attraction for complete isolation (see above), mating costs lead to a slight reduction in these regimes. Where complete isolation is stable in the absence of costs, the effect of mating costs on the invasion fitness gradient vanishes for m → 1.

Furthermore, strong mating costs increase the area where the polymorphic equilibrium at the ecological locus becomes unstable (for some values of m in [0,1]). The reason is that mating costs reduce the fitness of rare phenotypes (due to increased sexual selection). This may lead to a failure of speciation, which requires a polymorphic population. Unlike in the no-costs case (Pennings et al., 2008), the monomorphic equilibria at the ecological locus are always locally stable for sufficiently large m (Appendix 2), but this should be of little significance as long as the polymorphic equilibrium is locally stable, too.

Finally, we also performed some limited numerical analysis of an asymmetric model with inline image and inline image (results not shown; see Appendix E and Fig. E1 in Pennings et al., 2008 for the evolutionary regimes of the asymmetric model without costs). We find that consequences of mating costs are similar to that in the symmetric case. In particular, there is no significant effect for M ≳ 10.

Flexible mating strategy

One way to avoid costs of choosiness is for females to become less choosy towards the end of the mating period. In the simplest case, we might assume that a female behaves choosily during the first M encounters with males, but mates randomly afterwards. In this model, reproductive isolation can never be complete, because some females will always reach the random mating step. For the sake of continuity, we nevertheless retain nomenclature such as ‘C regime’ for the evolution of m = 1, even though this no longer refers to complete isolation in the strict sense. As each female is guaranteed to mate, costs of choosiness are absent in this model. We therefore recover the invasion criterion from the model with M → ∞, namely that choosiness increases (the invasion fitness gradient is positive) if and only if homozygotes have higher fitness than heterozygotes (Appendix 3).

Figure 4 shows the distribution of evolutionary regimes in the k vs. c plane for the model with a flexible mating strategy, together with the maximal possible degree of reproductive isolation (measured as the equilibrium proportion of heterozygotes for m = 1). For small values of M, the domain of the C regime increases in size, whereas the domains of the P regime and of the bistable regimes decrease. These results can be explained as follows: the P regime tends to be replaced by the C regime because, for small M, even the C regime leads only to partial isolation (see above). The bistable regimes become less prominent because a small M (high readiness of females to mate randomly) reduces sexual selection on males, which is the driving force behind bistability due to its positive frequency dependence. Unlike in the previous model (Fig. 1), the case M = 1 is not qualitatively different from the other cases. Obviously, the maximal possible degree of reproductive isolation (i.e. the proportion of heterozygotes at m = 1) decreases with decreasing M. We observe ‘almost complete’ isolation (less than 1% heterozygotes) if and only if the effects of mating costs in the corresponding nonflexible model are weak (M ≳ 10). For stronger mating costs, a flexible strategy leads to only partial isolation even in parameter regions where complete isolation evolves in the absence of flexibility. In these cases, we find a large parameter region where evolution of assortative mating leads to pronounced phenotypic clustering (for M ≥ 2, the phenotype distribution at a stable equilibrium with m = 1 is always at least bimodal).

Figure 4.

 Evolutionary regimes for female choosiness m in the alternative model with a flexible mating strategy. Females are choosy in their first M encounters with males but mate randomly afterwards. Plots in the left-hand column are analogous to those in Fig. 1 (see Fig. 1 for further details). Plots in the right-hand column show the proportion of heterozygotes at the maximal choosiness value m = 1; black: proportion of heterozygotes less than 0.001, dark grey: ≤0.01, intermediate grey: ≤ 0.1, light grey ≤(1/3) (bimodal), white: > 1/3. The dotted line is identical to the line separating the C, P/C and R/C regimes from the P and R regimes in the left-hand plots, with m = 1 being locally evolutionarily stable to the left of this line.

Viability costs

Absolute viability costs

The effect of absolute viability costs with soft selection on the invasion fitness gradient D(m) can be calculated analytically (Appendix 2). For a given cost function f, D(m) is decreased by the derivative


(see eqn A3). For the linear cost function (eqn 8), it follows that D(m) is reduced by a constant, δ, if costs are directly linked to m (Fig. 3b). By contrast, the effect of costs linked to m′ is proportional to (1 − m)3, meaning that it is strongest for low m and vanishes for m→1 (Fig. 3c). (Note that costs linked to m′ reduce the invasion fitness gradient by a constant if the gradient is derived with respect to m′ instead of m. In general, one should use a scale, proportional to m or m′ or any combination thereof, where choosiness modifiers have a constant average effect, independent of the resident choosiness.) Because of soft selection, there is no change in the stability of the ecological polymorphism in the resident population.

It is easy to see that arbitrary shapes of the cost function f can lead to an arbitrary number of additional evolutionary regimes (other than those described in Fig. 2), because they may create additional (stable or unstable) equilibria. For example, in Fig. 3b, for δ = 0.02, there is a regime with two stable intermediate equilibria for m. More generally, if costs are proportional to m, complete isolation (m = 1) can never be stable, as the invasion fitness gradient tends to 0 for m→1 even in the absence of costs. This is because m′ is already close to 1 for large m (under the assumption of the Gaussian shape m′ = 1 − (1 − m)4) and selection in homozygotes against mating with heterozygotes vanishes once the heterozygotes are nearly extinct. Note that there is no such effect if costs are linked to m′ rather than m.

Instead of focusing on isolation in the strict sense, it may be more relevant biologically to discuss whether assortative mating can evolve to a point where two clear clusters emerge in phenotype space. A minimal condition for cluster formation is the evolution of a bimodal phenotype distribution (Nhet ≤ 1/3N), where heterozygotes are less frequent than either type of homozygotes. More stringently, one might demand that heterozygotes should be almost absent from the population (e.g. Nhet = 0.01N). The values of m corresponding to both these criteria are highlighted in Fig. 3 (crosses and dots respectively). Whether or not a given degree of clustering can evolve depends on the shape of the invasion fitness gradient (in the absence of costs), on the shape of the costs function and on the mutational step size at the choosiness locus (i.e. on the intermediate values of m that are actually realized). Viability costs will always impede cluster formation if the minimum in the derivative of the cost function df exceeds the maximum in the invasion fitness gradient D(m) (without costs) for sufficiently small m. By contrast, cluster formation is always possible if the maximum in df is smaller than the minimum in D(m). The minimal and maximal values of D acting in the population before either bimodality or strong clustering is reached are shown in Fig. 5. For the range of parameters analysed, D is in the order of 10−1–10−2. We note that if sexual selection on males is absent (as in model 1 of Pennings et al., 2008), D is higher, but still within the same order of magnitude (results not shown).

Figure 5.

 The invasion fitness gradient D(m) in the absence of costs of choosiness. The graphs are contour plots in the c vs. k plane (see Fig. 1) for the minimal (a,c) and maximal (b,d) value of D before the population reaches either a bimodal phenotype distribution (a,b) or strong phenotypic clustering with 1% heterozygotes (c,d). The values shown correspond to the minimum and maximum of the top most functions in Fig. 3 to the left of the crosses and dots respectively. From right to left, the thick lines are isoclines for D = 0, 0.05 and 0.1. The thin lines are isoclines in steps of 0.01. Only isoclines for positive values of D are shown.

Relative viability costs

The effect of relative viability costs is intermediate between those of absolute viability costs and of (relative) mating costs (Fig. 3d). Like for mating costs, the effect of relative viability costs is maximal for intermediate m (although this maximum is much less pronounced for costs that are proportional to the number of rejected males, eqn 10) and vanishes for m → 1. Complete isolation is always locally stable (although the domain of attraction may be vanishingly small, see above) as long as costs are not extremely high (γ < 1/4; see Appendix 2). Relative viability costs also decrease the stability of the ecological polymorphism, in a pattern similar to that shown in Fig. 1. In contrast to mating costs, however, relative viability costs are linear in m for m → 0. This is because a part of the costs is already paid after the first potential partner is rejected (given that γ1 > 0 in eqn 9). Close to random mating, therefore, relative viability costs have an effect similar to that of absolute costs with a linear cost function.


We have investigated how costs of choosiness affect the evolution of assortative mating in a simple model of competitive speciation. Our analysis focuses on two types of costs: mating costs, which restrict the number of mating trials per female, and viability costs, which reduce survival. We show that mating costs have a significant effect only if the number of mating trials per female is less than about 10. Viability costs must be weak for assortative mating to evolve, because selection for assortative mating is weak in itself.

The strength of selection for assortative mating

Sufficiently strong costs can inhibit the evolution of any trait. But how strong is too strong for the evolution of assortative mating driven by resource competition? Clearly, assortative mating can evolve only if positive selection due to competition (and, potentially, sexual selection) is stronger than negative selection resulting from costs of choosiness. Therefore, the first question to be asked is how strong positive selection for assortative mating can be.

As shown in Fig. 5, the strength of selection for assortative mating depends strongly on the shape of the carrying capacity function (which is determined by the parameter k). For positive k, that is for unimodal carrying capacity functions, the invasion fitness gradient in the absence of costs is typically in the order of 10−2. Therefore, the selection coefficient s for a new mutation that increases choosiness by 10% is in the order of 10−3. This is weak, but not very weak selection and well within the range of selection coefficients inferred from molecular data (e.g. Andolfatto, 2007). It is worth noting that selection on mating modifiers is stronger by at least two orders of magnitude than selection on dominance modifiers in Fisher's classical theory for the evolution of dominance at mutation–selection balance (Mayo & Bürger, 1997; note, however, that this is not true under frequency-dependent disruptive selection, where the strength of selection on dominance is comparable with that on assortative mating in our model, e.g. Peischl & Bürger, 2008). Thus, speciation should still be possible in the presence of weak costs, but it is likely to be prevented if costs are moderate or high. Furthermore, the fate of mutations with a given selection coefficient depends strongly on the effective population size Ne. In particular, selection dominates genetic drift if Ne >> 1. Irrespective of costs, this suggests that competitive speciation is more likely in species with large effective population sizes.

Selection for assortative mating is significantly stronger if the carrying capacity function is bimodal (k < 0). For example, for k = −0.5, the invasion fitness gradient for m can exceed 0.1. This is hardly surprising, as a bimodal carrying capacity directly favours extreme phenotypes (i.e. homozygotes). Under these conditions, speciation should be possible even in the presence of moderate costs. Indeed, the prime obstacle to speciation in this case seems to be the potential instability of the ecological polymorphism for low and intermediate values of c (as opposed to the existence of stable low or intermediate evolutionary equilibria for m). A bimodal carrying capacity function can result, for example, if the population uses two discrete resources. Given the large effort invested in models with unimodal resource distributions (following Dieckmann & Doebeli, 1999), we suggest that more attention should be paid to models with bimodal or discrete resource distributions (e.g. Doebeli, 1996; Kisdi & Geritz, 1999; Ito & Shimada, 2007; Ripa, in press). It seems possible that competitive speciation then appears more likely than inferred from previous models.

The strength of costs

The next question is how strong costs of choosiness are likely to be relative to the strength of selection for assortative mating. This is an empirical question, but unfortunately, too few data exist for a general answer (Kokko et al., 2006; Bolnick & Fitzpatrick, 2007). Absolute viability costs, which are independent of a female's actual search effort, are particularly hard to measure, although it has been suggested that costs for female preferences (which may be comparable with absolute costs of choosiness) might be very low (Kokko et al., 2006). Some studies show that relative viability costs (such as search costs) can be quite high (e.g. Byers et al., 2005). This conclusion is corroborated by indirect evidence from studies showing that females behave less choosily under conditions of resource limitation (Byers et al., 2006), predation risk (Godin & Briggs, 1996) or increased energy expenditure (Milinski & Bakker, 1992). In other species, however, viability costs seem to be very low (Gibson & Bachman, 1992).

With regard to mating costs, Bolnick & Fitzpatrick (2007) recently reviewed data about the number of males females visit before mating. These estimates should be viewed as a lower bound for the parameter M in our model (the maximal number of males a female can evaluate during the mating period), because it is not clear whether females in the study populations accepted a male because the benefit of further search was low (i.e. the chosen male satisfied the female's preferences) or because the costs (risk of remaining unmated) were high. In some cases, females evaluate up to 100 males before mating, but often, they mate after looking at fewer than 10. As, in our model, we find a significant effect for M ≲ 10, this suggests that mating costs can indeed be an important factor opposing competitive speciation.

Furthermore, there is some evidence that mating costs might play a larger role if the ecological trait has a more complex genetic architecture. Although our estimate that mating costs are relevant for M ≲ 10 is in accordance with the results by Schneider & Bürger (2006) and Bürger et al. (2006), it is in slight contrast to Bolnick (2004) and Waxman & Gavrilets (2005b), who find a significant effect already for M ≲ 30. These differences can be explained by differences in the effective number of ecological phenotypes. The more different types there are, the less likely it is that a choosy female finds a male of exactly her own type. Bolnick (2004) and Waxman & Gavrilets (2005b) study the time to speciation in models with four to 10 ecological loci. As pointed out by Bolnick (2004), this time is largely determined by the time it takes to sort genetic variation in an initially highly polymorphic population that is close to random mating. Assuming, for simplicity, that all phenotypes are equally frequent, the probability of a female encountering a male of the same phenotype within M = 10 trials is 0.983 for one locus (as in our model), 0.692 for four loci and 0.386 for 10 loci. After 30 trials, these probabilities increase to 0.971 and 0.769 for four and 10 loci respectively. Schneider & Bürger (2006) and Bürger et al. (2006) also allow for more ecological phenotypes than we do, but their focus is on the number of phenotypic clusters formed at the ecological equilibrium for a given level of choosiness, and this number is typically quite low. Nevertheless, both Bolnick (2004) and Bürger et al. (2006) find evidence that costs become more important as the number of ecological loci increases.

Several authors have focused on the extreme case M = 1, mostly because it is analytically tractable (Drossel & McKane, 2000; Gourbiere, 2004; Kirkpatrick & Nuismer, 2004; de Cara et al., 2008). Not surprisingly, these studies find that speciation is impossible in the presence of costs (cf. Fig. 1f). Indeed, this is obvious even without formal analysis, as a female should never reject the only male she can expect to meet. However, the case M = 1 is clearly not representative of the general model, as even for M = 2, the results are qualitatively very different. Therefore, claims regarding the general importance of costs should not be based on this extreme case.

An argument by Bolnick & Fitzpatrick (2007) suggests an interesting difference between the expected strength of mating costs and viability costs. Mating costs should be most severe in low-density populations, where encounters between potential mates are rare. However, this is not likely to be the case in populations experiencing strong intraspecific competition, such as the ones considered in our model. An opposite argument can be put forward for viability costs: these are indeed most likely to be strong in resource-limited, high-competition environments, and this might create a ‘catch 22’ for speciation (Bolnick & Fitzpatrick, 2007).

The pattern of effects

Different types of costs not only differ in their strength but also in the precise pattern of change they inflict on the invasion fitness gradient (Fig. 3). This, in turn, determines their exact biological consequences. Consider a situation where speciation is possible in the absence of costs (C regime). In principal, costs that have a strong effect at low degrees of choosiness (low m) tend to prevent the onset of speciation, whereas costs that are strong at high degrees of choosiness tend to prevent the completion of speciation. Viewed differently, costs whose effect is weak at low m but gets stronger with increasing m tend to halt speciation at partial isolation, whereas costs whose effect decreases near complete isolation (i.e. for m → 1) favour bistability. They can prevent speciation from random mating, but not the completion of speciation, for example after secondary contact. Thus, there is a potential difference in the effect of costs on sympatric speciation vs. reinforcement (i.e. completion of speciation after secondary contact).

It is worth pointing out that both types of relative costs (mating costs and relative viability costs) belong to this latter class (Fig. 3a, d). Relative costs have no effect at or near complete isolation, because the only way for a mutant (homozygote) female to increase her mating rate is to mate with a male from the opposite homozygote class. However, the (heterozygote) offspring resulting from this mating will have a very low mating rate themselves, due to the strong sexual selection against rare heterozygotes. Both mating costs and relative viability costs have their largest effect at intermediate m. However, the effect of mating costs vanishes also towards random mating (Fig. 3a), whereas relative viability costs have a strong effect already at low degrees of choosiness (Fig. 3d). In consequence, mating costs favour bistability between partial isolation and complete isolation (P/C regime; see Fig. 1), whereas relative viability costs favour bistability between random mating and complete isolation (R/C regime; not shown).

Absolute viability costs can exhibit any number of patterns. For example, linear costs linked to m have a constant effect on the invasion fitness gradient (Fig. 3b), whereas linear costs linked to m′ have their maximal effect in populations close to random mating (Fig. 3c). It is also conceivable that costs are low for low m, but increase for large m (not shown). Such costs would most probably lead to partial isolation.

All costs that do not tend to zero for m → 1 will prevent the evolution of strictly complete isolation (Fig. 3b). However, even in this case, we often find that isolation can still become ‘almost complete’, and that the resulting population shows a strongly bimodal distribution with almost no intermediate phenotypes present (see also Doebeli et al., 2007). If gene flow is severely restricted, the two incipient species might conceivably evolve complete reproductive isolation by additional mechanisms (e.g. divergent evolution along other trait axis, temporal and/or spatial isolation, gradual build-up of incompatibilities).

Avoidance of costs

In many situations, females should be able to avoid costs of choosiness by flexibly adjusting their mate choice behaviour. In this paper, we have investigated a model in which females avoid mating costs by abandoning choosiness after M unsuccessful trials at assortative mating (Fig. 4). More generally, such a strategy is always possible if the cost of rejecting a male increases over time. In the examples used in the figures, this is true for mating costs (where all costs are paid only after the Mth unsuccessful mating trial), but not for viability costs: absolute viability costs are paid before the first male is encountered and, thus, can never be avoided. Relative viability costs with a linear cost function (eqn 10) are identical in every round, so that the cost/benefit ratio is the same for each new trial.

As female mating behaviour is generally highly plastic (Bolnick & Fitzpatrick, 2007), cost-saving mechanisms should be common in nature. This is particularly true for mechanisms avoiding mating costs, because these entail sexual selection against females that is effectively created by the females’ own behaviour. In other words, refusing to mate if the likelihood of future encounters with males is low does not seem to be a viable adaptive strategy. We stress that a flexible mating strategy is favoured for any level of choosiness in the population, even at m = 1. It is, therefore, always the flexible strategy that is the evolutionarily stable endpoint, and models for costs of choosiness should account for this fact.

A flexible strategy will never allow the evolution of complete reproductive isolation. Our results show that ‘almost complete’ isolation results whenever costs are weak in the nonflexible case. Otherwise (M ≲ 10 in our model), cost avoidance leads to partial isolation as the evolutionary outcome in a large parameter region. In this case, we find strong phenotypic clustering (about 10% heterozygotes), but gene flow between the clusters is still considerable, and there is no obvious mechanism leading to complete isolation.


Previous authors have voiced largely different opinions about costs of choosiness and their importance for competitive speciation (see Introduction). For our model, as may often be the case, the truth is found in the middle. In the light of our results, occasional claims that ‘absence of costs’ is a prerequisite for speciation (e.g. Gavrilets, 2005) are exaggerated: evolution of assortative mating is not structurally unstable and does not break down immediately once biologically meaningful costs are allowed for. On the other hand, even moderate costs will clearly inhibit speciation in many biological scenarios. Therefore, any model of competitive speciation is incomplete if it does not account for this point.


We thank M. Durinx, E. Kisdi, P. Pennings, A. Rettelbach, C. Rueffler, S. Schindler, K. Schneider and an anonymous reviewer for helpful comments on the manuscript. MK and JH are members of the Mathematics and Biosciences Group at the University of Vienna, which is funded by the Vienna Science and Technology Fund (WWTF). We also acknowledge support through an Emmy-Noether grant of the Deutsche Forschungsgemeinschaft (DFG) to JH.


Appendix 1: pairwise invasibility plots for female choosiness

Figure A1 shows pairwise invasibility plots (Geritz et al., 1998) for female choosiness m in the models with mating costs and with absolute viability costs. In the absence of costs, the nondiagonal boundary lines are vertical (Fig. A1a; see Pennings et al., 2008). This is also true in the flexible model, which is cost-free, too (not shown). As a consequence of vertical boundary lines, modifiers with small effect can invade if and only if modifiers with large effect can invade (see also Otto et al., in press).

Pairwise invasibility plots with vertical boundary lines represent a nongeneric case in the classification of Geritz et al. (1998), and it has been argued that they arise if the model assumptions are overly simplistic (Dieckmann & Metz, 2006). As seen in Fig. A1b–e, the boundary lines are no longer vertical if choosiness is costly. From the point of view of Dieckmann & Metz (2006), therefore, costs of choosiness contribute to the added ecological realism that is needed to break up the degeneracy of the simple model.

With mating costs (Fig. A1b, c), the nondiagonal boundary line for the singular strategy corresponding to the stable intermediate equilibrium of m is always decreasing. This equilibrium is therefore always an ESS (i.e. uninvasible), and in the neighbourhood of such an equilibrium, only modifiers with small effect can invade. With absolute viability costs that are proportional to m, the boundary line at this singular point is decreasing for small δ, but can be increasing for high δ. In the latter case, the singular point is a so-called evolutionary branching point (Geritz et al., 1998), i.e. invasion will lead to a protected polymorphism of coexisting mating strategies.

Appendix 2: invasion analysis with costs: analytical results

In this section, we derive the invasion fitness gradient for the model with mating costs and for the models with absolute and relative viability costs. In addition, we derive the conditions for stability of random mating and complete isolation in each of these models. In particular, we show that stability of random mating is unaffected by mating costs (for M ≥ 2), and that complete isolation is always (locally) stable in the presence of mating costs, but never stable in the presence of absolute viability costs directly linked to m. We also study the stability of the monomorphic equilibria in the model with mating costs. We start with the model with absolute viability costs, where the assumption of soft selection makes a full analytical treatment possible.

Absolute viability costs and soft selection

The invasion fitness gradient  As absolute viability costs with soft selection do not affect the population dynamics, we can calculate the invasion fitness gradient analytically. Denote the matrix of the mutant invader dynamics in the model without costs by A(0). The full invasion matrix with costs (see eqn 13) is then given by

image( (A1))

As the contribution of costs is proportional to the identity matrix, the leading eigenvalue in the model with costs relates to the eigenvalue without costs as

image( (A2))

and the invasion fitness gradient is

image( (A3))

Here, v = (v1,v2) and w = (w1,w2) are the leading left and right eigenvectors of the resident matrix Am,m, and ΔQ is defined as in eqn A29.

As the entries of w are proportional to the equilibrium distribution of the residents, we have w1/w2 =n/2, and hence w2/(vw) = 1/(v2 + v1n/2). From eqn A30 in the limit M → ∞, we obtain inline image with

image( (A4))

and thus

image( (A5))

The ratio n = Nhet/Nhom depends only on the resident population with monomorphic mating type. Due to soft selection, it keeps the same value as in the model without costs and can be derived from a fourth-order equation (see Pennings et al., 2008). The value of v1/v2 follows from the eigenvalue equation. For a linear cost function (eqn 8) and a Gaussian mating function, we further have

image( (A6))

Stability of random mating  In particular, for random mating (m = m′ = 0), we find v1/v2 = n/2 and

image( (A7))

and from eqns 12a and 12b,

image( (A8))

From the condition D(m = 0) = 0, we get n as the solution of a cubic equation. Using this solution in eqn A8, we then obtain k as a function of c and c ′, defining the boundary of the area where random mating is stable.

Stability of complete isolation  At complete isolation (m,m′ → 1 and Nhet → 0), we find v1/v2 →|3 + 4(dhet − dhom)|−1 and dhom → 1, and thus

image( (A9a))

As ∂m′/∂m → 0 for m,m′ → 1, we always have D(m = m′ = 1) < 0 if δ > 0, i.e. complete isolation is unstable. For δ = 0, the stability condition for complete isolation (D > 0) is

image( (A9b))
Relative viability costs and hard selection

Any hard selection scheme, as used for the relative viability costs (as well as for the mating costs, see below), will affect the population dynamics. We thus cannot use the results from the cost-free model for n = Nhet/Nhom, and generally need to rely on numerical methods. However, for invasion at random mating or at complete isolation a full analytical treatment is still possible.

The invasion fitness gradient  With relative viability costs, the invasion matrix (eqn 10) is given by

image( (A10))

and the invasion fitness gradient by

image( (A11))

where v and w are the left and right leading eigenvectors of inline image. The above equation can be evaluated to

image( (A12))

Stability of random mating  For random mating (m =m′ = 0), where v1/v2 = n/2, and with a Gaussian mating function, the invasion fitness gradient reduces to

image( (A13))

The condition D(0) = 0 leads to n = inline image and we obtain the boundary lines of the random mating regimes from eqn A8, as for the absolute viability costs (results not shown).

Stability of complete isolation  At complete isolation,

image( (A14))

As in the case of absolute viability costs, we have v1/v2 = |3 + 4(dhet − dhom)|−1 with dhom = 1. As


for n = 0, the heterozygous death rate diverges if ∑kγk →∞. We then have v1/v2 → 0 and conclude that complete isolation is always stable as long as ∑k(kγk/2k−1) < 1 (i.e. γ < 1/4 for γk ≡ γ).

Mating costs and hard selection

The invasion fitness gradient  For mating costs, the invasion matrix reads

image( (A15))

where inline image is again the invasion matrix without costs.

 We then obtain the invasion fitness gradient as

image( (A16))

with v and w again denoting the left and right leading eigenvectors of inline image. The above equation can be evaluated to

image( (A17))

where inline image is the gradient without costs and

image( (A18))
image( (A19))

is the derivative of the average female mating rate with respect to choosiness.

Stability of random mating  As inline image, we see that mating costs do not affect the invasion fitness gradient at random mating (m = m′ = 0) for M ≥ 2 (which is given by eqn 13 of Pennings et al., 2008).

Stability of complete isolation  For a resident population at complete isolation (Nhet = 0 and m = m′ = 1), the invasion matrix reads as follows:

image( (A20))

 From the equilibrium condition for the dynamics of the resident population (eqns 12a and 12b), we find dhom = 1 − (1/2)M. Using this and eqn 2 gives dhet =2(1 − c)(1 − k)[1 − (1/2)M]/(2 − c ′). We then see, in particular, that the matrix element a22 is 0 for M = 1 and always negative for M ≥ 2 and inline image. For inline imageinline image sufficiently close to 1, we further see (by considering leading order terms) that the trace of inline image is always negative, whereas the determinant is negative for M = 1, but positive for M ≥ 2. This corresponds to a positive leading eigenvalue for M = 1, but a negative leading eigenvalue for M ≥ 2. We conclude that mutants of a very small effect for weaker choosiness can invade for M = 1, but never for M ≥ 2. Strict complete isolation is therefore always locally stable for M ≥ 2. Note, however, that this result is of limited relevance as it is often possible for mutants with a slightly larger effect to invade (depending on the parameter values for k, c and c′).

Stability of monomorphic equilibria  Local stability of the monomorphic equilibrium (say, with the + allele) is given if rare mutants with the − allele cannot invade. As all these mutants will appear as heterozygotes, we can focus on the heterozygote fitness. The monomorphic equilibrium is stable if


Here, φmale, het is the mating rate of heterozygous males. From eqns 12a and 12b, we get dhom = 1 and from eqn 2dhet = (1 − k)(1 − c) follows. The mating rate of a female heterozygote is φfemale, het = 1 − mM. A heterozygote male must be chosen by a homozygote female in her first attempt as otherwise she will certainly choose a male of her own type. Each of N females will meet the rare heterozygote male first with probability 1/N and take him with probability 1 − m. We thus have φmale, het = 1 − m and

image( (A21))

For c < 1 and k < 1, this is always negative for sufficiently large m → 1.

Appendix 3: Invasion criterion for the model with flexible mating strategy

In this section, we prove the invasion criterion for the model where costs of choosiness are avoided due to a flexible female mating strategy, as defined in eqns 6a–6e. The criterion claims that invasion of mutants for increased choosiness is possible if and only if the fitness of homozygotes Whom is larger than the fitness of heterozygotes Whet in the population dynamical equilibrium of the residents.

Note first that the total female mating rates are all equal to 1 for this model as every female is guaranteed to mate. Explicitly, using eqns 6a–6e,

image( (A22))
image( (A23))

We can thus write the elements of the invasion matrix for rare mutants inline image (eqn 13) as follows,

image( (A24))
image( (A25))
image( (A26))
image( (A27))

Define the so-called resident matrix Am,m where the mutant variables inline image and inline image are set equal to the resident values m and m′. As this matrix reproduces the population dynamics of the resident population, it must have leading eigenvalue 0 in the population dynamic equilibrium. Consider now the difference of the mutant and the resident matrix,

image( (A28))


image( (A29))


image( (A30))

We can prove that

image( (A31))

and thus ΔQ ≷ 0 for inline image. To see this, note that

image( (A32))

and ∂πhom/∂m = −Nhet/N < 0 and

image( (A33))



The derivation for m′ is analogous.

Next, we denote by w the right leading eigenvector of the mutant matrix inline image and by v the left leading eigenvector of the resident matrix Am,m (with corresponding eigenvalue 0). We can then express the leading eigenvalue inline image of inline image as

image( (A34))

Because of the Perron–Frobenius theorem, all elements of the vectors v and w must be positive. We conclude that inline image for inline image, and hence mutants for stronger choosiness can invade, if and only if v2 > v1. For the last step note that the column sums of the resident matrix reproduce the fitness values,

image( (A35))
image( (A36))

Because of the equilibrium condition, we have

image( (A37))

and thus Whet and Whom have opposite signs. From v1a11 + v2a21 = 0, we get

image( (A38))

and thus v2 > v1 if and only if Whet < 0 < Whom.