Is self-fertilization an evolutionary dead end?


  • Boris Igic,

    Corresponding author
    • Department of Biological Sciences, University of Illinois at Chicago, Chicago, IL, USA
    Search for more papers by this author
    • These authors contributed equally to this work.
  • Jeremiah W. Busch

    Corresponding author
    • School of Biological Sciences, Washington State University, Pullman, WA, USA
    Search for more papers by this author
    • These authors contributed equally to this work.

Authors for correspondence:

Boris Igic

Tel: +1 312 996 6072


Jeremiah W. Busch

Tel: +1 509 335 1246




A compound hypothesis positing that self-fertilization is an evolutionary dead end conflates two distinct claims: the transition from outcrossing to selfing is unidirectional; and the diversification rate, or the balance of the speciation and extinction rate, is negative for selfing species. Both claims have enjoyed widespread informal support for decades, but have recently come under suspicion. Sources of data that apparently contradict strongly asymmetric mating system transitions often rely on statistical phylogenetic tests plagued by profound flaws. Although recently developed models mend preceding approaches, they have been employed sparingly, and many problems remain. Theoretical investigations, genetic data and applications of new phylogenetic methods provide indirect support for an association of selfing with negative diversification rates. We lack direct tests of reversals from selfing to outcrossing, and require data concerning the genetic basis and complexity of independently evolved outcrossing adaptations. The identification of the mechanisms that limit the longevity of selfing lineages has been difficult. Limitations may include brief and variable durations of selfing lineages, as well as ongoing difficulties in relating additive genetic and nucleotide variation. Furthermore, a common line of evidence for the stability of mixed mating – based simply on its frequent occurrence – is misleading. We make specific suggestions for research programs that aim to provide a richer understanding of mating system evolution and seriously challenge Stebbins’ venerable hypothesis.

‘The prevalence of outcrossing in flowering plants may result from selection for selfing within gene pools often being countered by selection for outcrossing between gene pools. Exclusively outcrossed plants may thus be favored by clade selection over those that rely partly on selfing.’

G. C. Williams (1992, p. 36)

I. Introduction

The vast majority of flowering plants are hermaphrodites, bearing both ovules and pollen enclosed in the same flower. They display an impressive diversity in form and function, with substantial variation within and across species in the degree to which self-fertilization is avoided or embraced (Barrett, 2002). Given the important effects of mating systems on the distribution of genetic diversity (Charlesworth, 2003), evolutionary biologists have long sought to establish their empirical distribution in nature and to explain the ultimate sources of this profound variability (Mather, 1943; Baker, 1959; Jain, 1976). Over the last 50 yr, an impressive body of theoretical and experimental work has demonstrated that the evolution of selfing is driven by multifarious agents of selection, potentially dominated by its transmission advantage and ability to provide reproductive assurance (Schoen et al., 1996). Selfing, however, is not the predominant mode of reproduction in angiosperms (Stebbins, 1974; Barrett & Eckert, 1990; Goodwillie et al., 2005; Igic & Kohn, 2006), and multiple mechanisms may potentially effect intermediate levels of self-fertlization (Holsinger, 1991; Stephenson et al., 2000; Kalisz et al., 2004; Goodwillie et al., 2005). Many recent models have proposed factors that potentially stabilize mixed-mating systems, a blend of selfing and outcrossing observed within individuals and populations (reviewed by Goodwillie et al., 2005; Johnston et al., 2010).

Given the widespread occurrence, yet apparently short-lived persistence, of selfing as a strategy, an oft-invoked hypothesis has been that species selection discourages this mating system (Dobzhansky, 1950; Stebbins, 1957; Lewontin, 1970; Lande & Schemske, 1985; Williams, 1992; Holsinger, 2000; Takebayashi & Morrell, 2001). Such higher order selection, manifested through different birth and death rates of lineages associated with outcrossing and selfing populations and species, could ensure that selfing lineages fail, presumably because of their limited capacity to adapt to changing environments, or their susceptibility to the accumulation of deleterious mutations (Takebayashi & Morrell, 2001). As a result, the repeated evolution of selfing is thought to be an evolutionary ‘blind alley’, or ‘dead end’ (selfing as an evolutionary dead end or SEDE hypothesis; Dobzhansky, 1950; Stebbins, 1957, 1974; Lande & Schemske, 1985). Takebayashi & Morrell (2001) offered a compelling reformulation of this compound hypothesis, which is implicitly composed of two parts, positing that: (1) the transition rate from selfing to outcrossing is zero, and (2) the net diversification rate – the difference between speciation and extinction rates – is negative for selfing species. At least in principle, the SEDE hypothesis can be invalidated if either of these component hypotheses is rejected.

Whether or not it strictly holds true, few believe that transitions from selfing to outcrossing are common. However, the complete irreversibility of selfing remains a contentious claim (Takebayashi & Morrell, 2001; Goodwillie et al., 2005). Regardless of whether or not reversals to outcrossing occur, even rarely, the unidirectional evolution of selfing lies at the philosophical heart of Stebbins’ characterization of selfing lineages as evolutionary ‘blind alleys’, contextually interpreted as inescapable entrapments that prevent the elaboration of radically new adaptive devices (Stebbins, 1957; Takebayashi & Morrell, 2001). Despite its importance, the second part of the hypothesis, concerning unequal diversification rates, has been largely ignored in the literature. It is simply astonishing that the SEDE hypothesis is over 50 yr old and directly invokes species selection, yet no genetic model aimed at the examination of the evolution of mating systems includes extinction or speciation as a possible outcome, despite being both explicitly posited by SEDE and acknowledged as crucial processes (Lande & Schemske, 1985; Williams, 1992; Schultz & Lynch, 1997; Holsinger, 2000; Igic & Kohn, 2006; Igic et al., 2008; Schoen & Busch, 2008; Glémin & Ronfort, 2013). One of the most important obstacles for the accumulation of data directly addressing the SEDE hypothesis has been the failure of phylogenetic models to account for the simultaneous action of natural selection at multiple levels of evolutionary hierarchy, especially above the level of individuals (Goldberg et al., 2010).

The strict-sense use of the term ‘species selection’ is often reserved for processes affecting traits emergent at the level of species, such as geographic range (reviewed in Jablonski, 2008; cf. Gould, 1982). Traits such as outcrossing rate and stature are defined at the level of individual organisms, and the outcome of the differential survival of lineages associated with their aggregate values is commonly termed ‘effect-macroevolution’ (Jablonski, 2008). A broader and commonly used definition of species selection holds that selection can also operate on such traits expressed at the individual level. For example, species selection may be expressed through differences in speciation and extinction rates of lineages, whose values are functions of aggregate trait distributions (Lewontin, 1970; Van Valen, 1973; Stanley, 1976; Gould, 2002; Okasha, 2006). Here, we imply species selection in this broad sense. We also conflate selection at the level of populations and species, although this practice is generally undesirable. The significance and magnitude of species selection remain unknown, but it is taken for granted that selection operates above the level of individual plants, and particularly on traits that strongly influence the amount of heritable variation (Lewontin, 1970; Gould, 1982; Williams, 1992; Jablonski, 2008; Rabosky & McCune, 2010). At least in principle, species selection could therefore affect the distribution and stability of mating systems among extant species, requiring any coherent theory on the subject to consider selection above the gene or individual level (Williams, 1992).

Here, we review progress in the evaluation of SEDE, with an emphasis on inferential limitations associated with the widespread failure to consider multilevel selection. We first discuss the ultimate reasons why reversals from selfing to outcrossing may be unlikely, and provide guidance for the identification of systems in which reversals are most likely to be driven by natural selection. We also interpret the lack of evidence for the expected differences in population-genetic parameters between selfers and outcrossers, given the macroevolutionary expectation that selfing species have short durations. We then critique phylogenetic methods for the inference of the directionality of mating system evolution, as well as the widely employed reasoning used to support the evolutionary stability of mixed-mating systems, which both fail to consider that variation in lineage diversification rates may be associated with selfing and outcrossing. We end with suggestions on the evidence necessary to reject the two component hypotheses of SEDE, and we further discuss the value of asking the eponymous question, suggesting alternative paths for future inquiry.

II. Microevolutionary perspectives on SEDE

1. Constraints on reversals from selfing to outcrossing

According to the SEDE hypothesis, transition rates from outcrossing to self-fertilization are unidirectional, with reversals to outcrossing untenable as a result of both genetic and environmental barriers. Generally, evolutionary reversals are considered to occur when populations revert exactly to an ancestral character state (Bull & Charnov, 1985). The most narrowly cast perspective on reversals would require that recently derived selfing populations exactly and homologously retrace evolutionary shifts, at the same pathways and sites, which recover the outcrossing mating system (Bull & Charnov, 1985; Goldberg & Igic, 2008). We believe that Stebbins’ formulation of SEDE and most other interpretations take a much looser definition of reversal, which holds that any reversal from selfing to outcrossing is strongly discouraged, whether or not it is exactly homologous (Stebbins, 1957; Lande & Schemske, 1985; Takebayashi & Morrell, 2001). Re-evolution of outcrossing, following loss, requires the introduction of mutations that increase outcrossing in an otherwise selfing species (Nasrallah et al., 2004). The environment faced by these initial mutants with a newly regained propensity for outcrossing, however, has outsized importance. We focus on the population-genetic and ecological mechanisms that constrain the spread of such mutants, and highlight conditions that enhance the probability of their establishment – or full reversal – in response to natural selection. This framing makes the occurrence and detection of reversals plausible, and permits the identification of study systems in which the irreversibility component of the SEDE hypothesis can be best challenged by data.

Selfing can evolve in response to an extremely diverse array of selective agents (Jain, 1976; Goodwillie et al., 2005), but its intrinsic 3 : 2 transmission advantage is ever present (Fisher, 1941; up to 50%). This specific advantage arises because a selfing individual, in a stable population, transmits a total of three copies of alleles: two through self-fertilized seeds, and a single copy as an outcrossed pollen parent. By contrast, outcrossing individuals donate, on average, one copy as a seed parent (pollen for that seed having arrived from an outcrossing event with another individual) and another copy as a pollen parent (Fisher, 1941; Lloyd, 1979, 1992). Inbreeding depression is usually invoked as the principal counterweight to this advantage. Because elevated homozygosity accompanying the evolution of selfing may effectively purge partially recessive, harmful mutations, inbreeding depression should correspondingly decline in response to increased selfing (Lande & Schemske, 1985; Byers & Waller, 1999; Takebayashi & Morrell, 2001; Crnokrak & Barrett, 2002; Winn et al., 2011). The conditions for reversal to outcrossing are the opposite of those for the evolution of selfing (Lloyd, 1979, 1992; Busch & Delph, 2012). Massive regain of inbreeding depression, particularly following hybridization events, may be a necessary prerequisite for reversals to partial or complete outcrossing.

Although reversals to outcrossing are generally viewed as unlikely because of the concurrent reduction in outcrossing rate and inbreeding depression, it is widely underappreciated that the ecological context may amplify the costs of outcrossing. If selfing evolves because of reproductive assurance, the conditions for a reversal are even more restrictive than if selfing evolves principally because of its transmission advantage, assuming that all else is equal (Busch & Delph, 2012). Outcrossing mutations would have to overcome both the transmission advantage of selfing and the greater seed production of plants capable of self-fertilization. These are potentially large disadvantages at the gene and individual levels (Cheptou & Schoen, 2007). Reversals to outcrossing are also unlikely when selfing evolves in pollinator-limited environments (Inoue et al., 1996), because pollination vectors are necessary to produce outcrossed seed. In the absence of pollinators, reversals would be impossible, with the sole exception of a simultaneous shift to an abiotic mode of cross-pollination. Similarly, the mode and timing of self-pollination also influence the probability of reversals to outcrossing, as derived phenotypes that self-fertilize early in a flower's life (i.e. cause seed discounting) or limit outcrossed paternity (i.e. cause pollen discounting) are more amenable to reversal, as these discounts favor outcrossing (Lloyd & Schoen, 1992), as long as not all ovules in a population are selfed before outcrossing is possible. Reversals are therefore most likely when pollinators are common or have re-appeared, when selfing does not increase individual seed production, and when selfing entails high rates of seed and pollen discounting.

The genetic basis and complexity of the outcrossing mechanism are important because they control directly the likelihood that an outcrossing phenotype can be reassembled in a selfing population. At one end of the spectrum of complexity, self-incompatibility systems involve closely co-evolving pistil- and pollen-expressed genes. The derivation of selfing in ancestrally self-incompatible lineages causes the loss of tightly co-evolved allelic diversity in the genes embedded at the S-locus. The loss of this diversity can happen very rapidly if S-alleles bearing loss-of-function mutations cause selfing, or less quickly (c. 4Ne generations) if mutations outside this region cause selfing (Igic et al., 2008). Several analyses of recently derived self-pollinating taxa have shown that the first mutations permitting selfing are loss-of-function mutations at the S-locus (Bernacchi & Tanksley, 1997; Nasrallah et al., 2004; Busch et al., 2011; Tsuchimatsu et al., 2012). If such mutations commonly trigger the evolution of selfing, transitions may be rapid, leaving little time before an irreversible collapse of S-locus diversity, as a minimum number of S-alleles is required to permit outcrossing (Igic et al., 2008). Reversals in these systems would entail one of several unlikely events: back-mutations or hybridization with a closely related self-incompatible lineage that restores enough S-allele diversity to permit outcrossing, or the evolution of novel self-incompatibility mechanisms. Such events should be rare, regardless of whether the strength of natural selection is sufficient for outcrossing phenotypes to overcome the transmission advantage of selfing and associated selective factors (Goodwillie et al., 2005).

Morphological adaptations that prevent self-pollination (e.g. flower size, dichogamy or herkogamy) may involve a much smaller loss of complexity (Baldwin et al., 2011). The suite of morphological adaptations that accompanies shifts to self-fertilization, and comprises the ‘selfing syndrome’ (Sicard & Lenhard, 2011), is dominated by quantitative reductions in the size of floral organs. Examples abound, and are not limited to Arenaria, Capsella, Clarkia, Collinsia, Eichhornia, Leptosiphon, Mimulus and Solanum (Lin & Ritland, 1997; Runions & Geber, 2000; Armbruster et al., 2002; Fishman et al., 2002; Georgiady et al., 2002; Fishman & Stratton, 2004; Goodwillie et al., 2006; Vallejo-Marin & Barrett, 2009; Slotte et al., 2012). The selfing syndrome is frequently associated with multiple quantitative trait loci (QTLs) (for a review, see Sicard & Lenhard, 2011). An understanding of the constraints on reversals to outcrossing requires an understanding of the genetic bases of specific functional traits that control the selfing rate (Herlihy & Eckert, 2007; Vallejo-Marin & Barrett, 2009; Kalisz et al., 2012), in addition to the potential pathways of trait evolution that may reduce selfing rates. Novel functional constraints on floral traits in selfing flowers (Anderson & Busch, 2006), coupled with the loss of variation at multiple QTLs influencing the selfing rate, represent potentially strong limits on the ability of natural selection to drive reversals from selfing to outcrossing.

Although there are manifold reasons why reversals to outcrossing appear to be theoretically unlikely, few direct tests have been conducted (Robacker & Ascher, 1978; Bixby & Levin, 1996). Perhaps one reason for the dearth of work is that many collectively view such evolutionary events as exceedingly rare. Indeed, the independent origins of self-incompatibility systems and morphological adaptations for outcrossing have obviously occurred during the evolutionary history of angiosperms (Stebbins, 1957; Steinbachs & Holsinger, 2002; Igic et al., 2008). The spirit of the SEDE hypothesis, however, requires that reversals from selfing to outcrossing, driven by natural selection, are impossible, given the characterization of selfing as a blind alley. Direct tests of the irreversibility of mating system evolution are needed, particularly in taxa whose present-day selfing rates are high, but in which the costs of outcrossing are expected to be low. Precisely this set of conditions is observed in Aquilegia canadensis, where increased selfing rates are associated with nearly complete seed discounting and extremely high inbreeding depression. Herkogamy is a major trait influencing the selfing rate and selection may easily drive reversals (Herlihy & Eckert, 2002, 2007), unless mutations increasing the selfing rate have positive pleiotropic effects on viability (Jordan & Otto, 2012). Experimental studies of reversals require, at a minimum, an understanding of the same parameters as necessary to understand the evolution of selfing: the mode of self-pollination, its associated discounts and the functional genetic basis of outcrossing mechanisms (Lloyd, 1992; Barrett & Harder, 1996; Herlihy & Eckert, 2004). Experimental evolution of the selfing rate can occur extremely rapidly and may even be most likely in highly selfing populations (Roels & Kelly, 2011), so tests of this hypothesis are likely to result in robust tests of the unidirectionality component of the SEDE hypothesis.

2. Which mechanisms cause the negative diversification rate of selfing lineages?

Stebbins (1957) argued that selfing lineages are ‘blind alleys’, evolutionary paths with little opportunity to adapt to changing environments. In this section, we examine the genetic evidence supporting and detracting from the widespread expectation that extinction rates of selfing lineages exceed their speciation rates.

Selfing may constrain adaptive evolution by reducing the levels of polymorphism and, particularly, additive genetic variation (Darlington, 1939; Mather, 1943; Stebbins, 1957). There is considerable support for the notion that selfing lineages harbor less nucleotide polymorphism, although it is unclear how genome-wide nucleotide polymorphism is linked to potential adaptive phenotypic evolution (Pollak, 1987; Charlesworth & Charlesworth, 1995; Nordborg, 2000; Charlesworth & Wright, 2001; Glémin et al., 2006). In theory, selfing should cause a reduction in genetic diversity equal to Ne/(1 + F) (where the inbreeding coefficient F is a function of the selfing rate: s/2 − s). Far greater losses are expected, however, if selfing reduces the efficacy of recombination (Ross-Ibarra, 2004; Morrell et al., 2005; Glémin et al., 2006; Glémin & Ronfort, 2013), fixes during population bottlenecks (or facilitates bottlenecks; Schoen & Brown, 1991; Schoen et al., 1996; Goldberg & Igic, 2012), or reduces migration among populations (see Ingvarsson, 2002, however, for potential increases in Ne at the species level). Given these correlates of selfing, it is not surprising that it is associated with much larger reductions in Ne than expected (Charlesworth, 2003). What, then, are the consequences of potentially small Ne in selfing species on the process of adaptive evolution?

Potentially large reductions in Ne in selfing species should cause previously deleterious mutations with sufficiently small effects to become effectively neutral (Charlesworth & Wright, 2001), permitting a greater fraction of the harmful mutational load to fix by drift (Heller & Maynard Smith, 1979; Charlesworth et al., 1993; Glémin, 2003). Comparisons of the ratio of nonsynonymous to synonymous substitutions between outcrossing and selfing populations have largely failed to find such a pattern (Wright et al., 2002; Cutter et al., 2008; Haudry et al., 2008; Escobar et al., 2010). Although a meta-analysis found that selfing taxa had higher values for the neutrality index (Glémin et al., 2006) – reflecting segregating nonsynonymous polymorphism – there is a surprising lack of support for weakened purifying selection in selfers (Bustamante et al., 2002; Wright et al., 2008). An alternative is that selfing causes extinction because it robs lineages of potentially beneficial genetic variation that is positively selected. Glémin & Ronfort (2013) compared the probability and rate at which beneficial mutations fix in outcrossing and selfing species, with the possibility of lineage extinction. Extinction is less likely in outcrossing species when beneficial mutations are dominant or codominant, and when standing genetic variation plays a substantial role in adaptation (Glémin & Ronfort, 2013). Adaptation in highly selfing lineages is independent of the dominance of mutations and is often faster, but escape from extinction is unlikely when selfing greatly reduces Ne. New methods for the estimation of the fraction of adaptive substitutions may help to evaluate whether selfing retards positive selection, as may be expected if many mutations are, in fact, beneficial, at least in some environments (Eyre-Walker & Keightley, 2009; Gossmann et al., 2010; Slotte et al., 2010; Carneiro et al., 2012).

Selfing may limit purifying or positive selection, but these lineages may not persist long enough to leave footprints of attenuated adaptation in the genome. Phylogenetic studies have shown that the origins of selfing are often rather recent (Cutter et al., 2008; Foxe et al., 2009; Escobar et al., 2010; Ness et al., 2010; Busch et al., 2011; Pettengill & Moeller, 2012). Indeed, the rate of extinction (μ) for self-compatible (SC) species may be very high, and the average lineage durations (1) are therefore expected to be very short, possibly on the order of 200 000 yr (Goldberg et al., 2010). This problem is particularly worrisome for studies that compare rates of substitution between outcrossing and selfing species, as a shift to a higher selfing rate will require, at a minimum, 4Ne generations before effectively neutral mutations stochastically fix within lineages. Although shifts in patterns of polymorphism will occur more rapidly, there is the lingering problem of how to quantify the adaptive value of genetic variation maintained at the species level. Quantitative genetic approaches that explicitly evaluate additive genetic variation in the face of changing environments may be best positioned to test this hypothesis (Charlesworth & Charlesworth, 1995; Bartkowska & Johnston, 2009). More generally, reductions in Ne should limit both positive and purifying selection (Bachtrog & Charlesworth, 2002; Wright & Andolfatto, 2008), and these simultaneous handicaps may be required to cause extinction over time scales overlapping the low lineage durations estimated for selfing lineages.

III. Macroevolutionary perspectives on SEDE

With over five decades in their wake, modern studies of plant mating system evolution have yielded data on selfing and outcrossing rates for approximately 500 species and numerous analyses of the history of mating systems, employing a variety of formal models and theoretical and empirical arguments. The two components of the SEDE hypothesis – irreversibility of transition to selfing and the negative diversification rate of selfing lineages – may be evaluated by reconstructing evolutionary histories (Takebayashi & Morrell, 2001). In this section, we defend one principal argument: although studies conducted to date contain valuable data shaping our expectations about the process of mating system evolution, reliance on flawed statistical phylogenetic models (Maddison, 2006; Goldberg & Igic, 2008) and a widespread lack of appropriate sample sizes have so far prevented rigorous tests of dead-end hypotheses. Because they are essential to a comprehensive understanding of SEDE, we critique the methods employed over the past decade, examine the implications for the dominant paradigm arising from the results, and discuss the value of existing or prospective solutions.

1. Models that violate the premise: state-independent character evolution

Given a character with a finite number of states (k; for binary mating system characters of SEDE, k = 2) from an arbitrary number of species, we can evaluate a continuous-time Markov model of character evolution to infer the rates and history of changes in this character on a phylogeny (Pagel, 1994; Schluter et al., 1997; Lewis, 2001). Model parameters are the transition rates between character states. Many software packages readily evaluate the likelihoods of parameters under a Markov (M) model with k number of states (this family of models is commonly termed ‘MK’) of character evolution, to find maximum likelihood or Bayesian estimates of transition rates and ancestral states, and accommodate sources of phylogenetic and parameter estimation uncertainty (Pagel, 1999; Pagel & Meade, 2006; FitzJohn et al., 2009; Maddison & Maddison, 2010). This family of models, however, has one strong weakness in the context of SEDE: it cannot be used to infer the differences in diversification rates, and its use will therefore disable accurate inference of character evolution.

Regardless of the methods employed for the parameter estimation procedure, incorporation of uncertainty, subsequent model choice or statistical tests, these models can only be used to estimate parameter values under the explicit assumption that character evolution proceeds within the confines of a phylogenetic tree. Maddison (2006) provided what may be the clearest explanation for why state-independent evolution models, such as Mk, may mis-specify the underlying process of character evolution:

‘[They treat] the tree as if it existed prior to the evolution of the character, like a series of branching paths along which the character was constrained to follow in its evolution. This may be a valid assumption if the character of interest is neutral. […] If a character affects fitness and population sizes, fidelity to habitat, reproductive isolation, or migration, then the character may not have been a mere passenger along the phylogeny's branches.’

The character states themselves are presumed not to affect the evolution of species and, as a result, the rate of speciation or extinction for each state is therefore constrained to be equal – independent of the state in which a given lineage finds itself – whether such a lineage is selfing or outcrossing. Consequently, this simple model may often be underparameterized and highly problematic for tests of SEDE, because the key component of SEDE is that the character states have different net diversification rates. Inference of character evolution should minimally be able to evaluate competing models containing parameters proposed by the SEDE hypothesis. These parameters must include speciation and extinction rates associated with selfing and outcrossing, as well as rates of forward and reverse transitions between these two states, a task that Mk2 models cannot accomplish. If the evolutionary process resembles that proposed by SEDE, Mk2 inference of transition rates may be hopelessly confounded by diversification parameters, which are not accounted for (Igic et al., 2006; Maddison, 2006).

A more technical concern with Mk models that is potentially disastrous to the accuracy of character evolution analyses involves root state calculations (Goldberg & Igic, 2008). In particular, weights must be assigned to the two character states at the tree's root to compute the likelihood of the data given the model, but the common practices of setting these weights as equal or of using the stationary frequencies are not consistent with a model of irreversible transitions, and yield poor results (Goldberg & Igic, 2008). FitzJohn et al. (2009, appendix 1) solved this problem by weighting the root states in proportion to their likelihoods of yielding the observed data, but this solution is not presently implemented in most software packages.

2. Binary State Speciation and Extinction (BiSSE) models: allowing state-dependent character evolution

In view of the many shortcomings of the existing approaches, there is little doubt that Maddison et al. (2007) developed the most exciting and far-reaching advances for phylogenetic studies of SEDE in over a decade. They proposed the Binary State Speciation and Extinction (BiSSE) model of character evolution, at least theoretically able to accommodate the action of species selection and to recover the process of evolution proposed by SEDE. Likelihood calculation under BiSSE uses a model with six parameters, instead of two used by Mk2, to estimate the parameters illustrated in Fig. 1(c,e). These parameters allow outcrossing and selfing lineages to take on distinct speciation (λO, λS) and extinction (μO, μS) rates, as well as bidirectional transitions (qOS, qSO). For model testing purposes, constraints can be added to evaluate models of evolution. Parameters can be constrained (λO = λS, qSO = 0, etc.), and their fit compared using model selection procedures.

Figure 1.

A single simulated clade containing selfing (S; shown in red) and outcrossing (O; shown in black) species, evolving under selfing as an evolutionary dead end (SEDE; speciation rates: λO = 0.3, λS = 0.4; extinction rates: μO = 0.1, μS = 0.5; transition rates: qOS = 0.15, qSO = 0). In each case, branches of the phylogeny are colored to indicate the known character state at a particular time. Likewise, tip label circles denote extant character states (known), and node label pies indicate the inferred ancestral state, using one of two models. (a) The entire history of the clade is shown, including both extant and extinct lineages. (b) Reconstruction of ancestral states with Mk2 or Binary State Speciation and Extinction (BiSSE). The following panels were constructed from data obtained after a continued run of the same simulation until it reached 400 species. (c) The inferred Mk2 model parameters. (d) Parameter values inferred under Mk2. Vertical dashed lines show the true parameter values (e) Model parameters for BiSSE. (f) Parameter values inferred under BiSSE. Vertical dashed lines show the true parameter values. Some error is due to the simulation procedure, but BiSSE recovers transition rates very close to those used in simulations (qOS = 0.15, qSO = 0; both within the 95% credible set), unlike Mk2.

In the context of SEDE, the most interesting use of BiSSE may involve the examination of the comparative evidence for state-dependent net diversification. If a model choice procedure justifies the employment of a parameter-rich model, we may garner support for λS < μS (negative net diversification rate of selfing lineages) and qSO = 0 (irreversibility). The software package diversitree implements state-of-the-art models, simplifies parameter modification and allows easy model choice procedures in the R programming language (FitzJohn, 2012; R Development Core Team, 2012). We perform sample simulations in diversitree in order to illustrate the profound possible confounding effects of diversification and character state evolution, and to further highlight its implications for the phylogenetic inference of processes that produce the observed patterns of mating system evolution.

3. State-dependent vs state-independent models: an example

We use a single realization of a constant-rate stochastic process to illustrate the most relevant aspects of the models of character evolution, as well as some difficulties with inference (Fig. 1). The simulated clade is allowed to grow to 30 extant species. The entire history of simulated lineages is saved (Fig. 1a). We use an arbitrary set of model parameters, which conform to the assumptions of SEDE (λO = 0.3, λS = 0.4, μO = 0.1, μS = 0.5, qOS = 0.15, qSO = 0). The extant species are used to infer Mk2 (state-independent) and BiSSE (state-dependent) model parameters of character evolution, and ancestral states are inferred at each node, shown as the proportional likelihoods of selfing or outcrossing. This simple approach to the simulation of a fixed number of species is flawed in a number of important ways (Hartmann et al., 2010), as is the use of a single simulation run and scant justification for a particular model choice. We merely intend to conduct an illustration of important concepts widely neglected in the plant mating system literature.

As first pointed out by Takebayashi & Morrell (2001), the Mk models tend to produce transition probability estimates that mirror the proportion of tips in each state, so that the proportion of selfers is approximated by qOS/(qSO + qOS). We now know that this is, in large part, caused by the compound effects of several model violations (Nosil & Mooers, 2005; Goldberg & Igic, 2008). Indeed, the transition estimates with Mk2 fail in the predicted manner (Fig. 1d; the posterior parameter estimates do not overlap the true values, although the small-sample likelihood ratio test (LRT) fails to reject qSO = 0 or qSO qOS). Under Mk2, differences in speciation and extinction rates cannot explain the distribution of states at the tips. Therefore, assuming that the process is at equilibrium, the maximum likelihood estimates of reverse parameters tend to take on nonzero values, such that models with a low reversal rate (qSO = 0) could not explain the observed data well. These results include the correct rooting strategy (FitzJohn et al., 2009), which improves model performance.

Inference with BiSSE is not entirely accurate, and is associated with large uncertainty in parameter estimation, in part caused by the simulation procedure and small clade size. Nevertheless, it recovers transition rates reasonably close to those used in simulations (Fig. 1f; qOS = 0.15, qSO = 0, both within the 95% credible set), and appropriately fails to overestimate confidence in character transition rate estimates, unlike Mk2. As an approximate indicator of power, in this particular instance, the use of parameter-rich BiSSE is justified when the simulated clade reaches c. 100 extant taxa, and the illustrated precision (Fig. 1f) is achieved at c. 300–400 species. Of course, the variance in this stochastic process can be considerable.

The need for a relatively large number of species and sampling effort may seem discouraging. By comparison, the six studies reviewed by Takebayashi & Morrell (2001) averaged 33.5 species (range 10–60), often representing poor sampling from their respective clades. Although we argue for great care in analyses, there are also many reasons for guarded optimism. Methods that enable analyses with partial random sampling or unresolved trees are available and perform surprisingly well (FitzJohn et al., 2009). Although we cannot overlook a long list of other problems, a generally improved set of methods that enable good practices in character evolution is available. Promising advances based on the state-dependent evolution framework of BiSSE will no doubt bring us estimates of the pervasiveness and magnitude of species selection, as foreseen by Williams (1992). Yet, although inevitable increases in the scale of phylogenetic analyses will make powerful analyses relatively commonplace, we strongly advocate cautious interpretation of results that rely on the inference of character evolution, especially those derived exclusively from extant species character states and phylogenies.

4. Further difficulties with phylogenetic tests

Rigorous tests of the SEDE hypothesis are likely to remain very difficult when we only possess character state data (outcrossing rates) and phylogenies. The robustness of phylogenetic approaches to myriad possible model violations is presently unknown. Although it may be reasonable to look for help in the fossil record, that too may prove to be of limited value for the inference of mating system evolution. Fossilized taxa with measurable traits that strongly predict the outcrossing rate are rare and, even if they were more common, the generating process may yield confusing and deceiving data (Fig. 1a).

Both direct and indirect evidence hints that extinction rates can be relatively high (Niklas et al., 1983; Goldberg et al., 2010). When coupled with a high overall extinction rate (ε μ/λ ≈ 0.5), unequal diversification rates associated with selfing and outcrossing (λS − μS < λO − μO), as proposed by the SEDE hypothesis, should yield an evolutionary history disproportionately littered with remains of extinct selfing lineages. The exact fraction of selfing plants in the fossil record is expected to vary over time, partly depending on the magnitude of extinction and speciation rates (Igic & Kohn, 2006), with the expected equilibrium frequency qOS/(rO − rS). It is therefore possible that most fossil strata may yield a preponderance of flowers with phenotypes consistent with selfing (e.g. cleistogamous or small flowers and inflorescences), even if the ancestor of an extant clade were outcrossing (see Fig. 1a). This argument hinges on equiprobable recovery of selfing and outcrossing lineages, as well as our ability to unambiguously quantify mating systems using fossilized traits. Paleontological studies of traits indicative of selfing and outcrossing may be affected by similar problems.

The incorporation of several additional realistic factors into a BiSSE-like model could increase variance in parameter estimates. Many model mis-specifications and sources of error yield test results biased away from irreversibility (i.e. tests that fail to reject SEDE), which proposes a boundary parameter value. For example, it is unclear whether we may ever have sufficient power to confidently separate, even in large-scale phylogenetic analyses, an estimated reversal rate of 10−6 from one that is exactly zero. Similarly, it will be difficult to examine the accuracy and power of tests employing ancestral state reconstructions, given the unknown processes that may be unaccounted for. Further, phylogenetic tests can only examine the evolutionary history of a subset of lineages – those that yielded the extant taxa. Biases in species descriptions, with respect to character states (for example, clumping of outcrossers and splitting of selfers), may, in turn, bias the inference of speciation and extinction rates. In addition, if reversals happen rarely enough, our best methods may have insufficient power to detect them. The strongest evidence supporting or contradicting any dead-end hypothesis will therefore probably contain a multi-pronged approach, with demographic, ecological and genomic evidence buttressing statistical phylogenetic analyses, and together yielding a thoughtful and generally convincing argument for instances of regain.

IV. SEDE and the distribution of outcrossing rates in angiosperms

Motivated by its relatively common occurrence in angiosperms, the search for conditions under which mixed mating is an optimal reproductive strategy has dominated recent developments in the evolution of mating systems. The most promising new approaches account for correlated fertility components (Johnston et al., 2010) or pleiotropy among selfing rate, pollen export and viability (Jordan & Otto, 2012) to show that mixed-mating (and selfing) mutants can invade and persist in populations under a wide range of considered evolutionary parameters, including high inbreeding depression. These results are a boon for empiricists studying the population and quantitative genetics of plant mating systems, because they are key to both explaining the apparent paradoxical data – high selfing rates with high inbreeding depression (Eckert & Herlihy, 2004; Yang & Hodges, 2010) – and spurring further work on the search for the realistic ranges of parameters relevant to mating system evolution in natural populations, which remain virtually unknown.

Nevertheless, the results of these and many other models are frequently mistaken as having a primary bearing on the observed distribution of mating systems. This would only be true if mating systems generally had no effect on diversification rates (Lande & Schemske, 1985; Igic & Kohn, 2006). On the contrary, mating systems may have a universally high impact on diversification rates (Williams, 1992; Holsinger, 2000; Takebayashi & Morrell, 2001; Igic & Kohn, 2006; Schoen & Busch, 2008; Goldberg et al., 2010), and the distribution of outcrossing rates cannot be solely determined by the frequency of appropriate population-genetic conditions for the maintenance of genotypes with a particular propensity to outcross or self within a lineage. Progress towards an understanding of the processes shaping the empirical distribution of outcrossing rates in flowering plants will therefore benefit from the inclusion of more information on state-dependent diversification into population-genetic models of mating system evolution.

Although we still lack a formal set of models to disentangle selection at different levels of evolutionary hierarchy, the framework of state-dependent diversification offers a new perspective on the shape of the empirical distribution of mating systems, its interpretations and possible causes, topics that have generated considerable interest over the past several decades. Analyses of the shape of this distribution are saddled with uncertainties, but it appears increasingly clear that the distribution is approximately bimodal (Schemske & Lande, 1985). The modes of a discretized distribution with equal-size bins are roughly at zero and one, with more outcrossers than selfers, and a significant proportion of species displaying intermediate values of outcrossing rates (Fig. 2; Goodwillie et al., 2005; Igic & Kohn, 2006). The presence of species with intermediate outcrossing rates is commonly interpreted as evidence suggesting their long-term stability and existence as an adaptive strategy. Here, we briefly demonstrate how species selection could generate the observed pattern, with mixed-mating or selfing species present, even if these states are evolutionary dead ends (Schemske & Lande, 1985; Igic & Kohn, 2006).

Figure 2.

The inferred distribution of outcrossing rates in angiosperms (after Igic & Kohn, 2006). Although a significant proportion of mixed-mating species exists (0.2 < < 0.8), this does not imply the stability of mixed mating. Instead, unidirectional transition from the outcrossing state (black) to mixed mating (crimson) to selfing (red), together with a wide set of state-specific speciation and extinction rates (per lineage per Myr), can yield this distribution. Note that slightly altered parameter combinations can accommodate a variety of changes in the shape of the distribution, which could possibly explain differences between wind- and animal-pollinated species, or woody and herbaceous species.

Consider, for example, the distribution of outcrossing rates presented in Fig. 2. Because species with intermediate outcrossing rates are fairly common, binary state encoding of mating systems may not be able to capture the underlying dynamic (Goodwillie et al., 2005). We expand the two-state SEDE model into a similar three-state model of evolution, which allows for mixed mating (M) in addition to selfing (S) and outcrossing (O), and lets each state take on separate net diversification rates (rO, rM and rS), together with unidirectional ‘stepping stone’ transitions from outcrossers to mixed-mating species to selfers (qOM, qMS). We have shown previously using a deterministic exponential growth model that, if rO − qOM is larger than both rM − qMS and rS, the entire clade increases with the exponential rate rO − qOM and these three states attain the equilibrium ratios (Igic et al., 2008; Schoen & Busch, 2008):

display math

A heuristic set of parameter values, rO = 0.2, rM = −0.1, rS = −0.1, qOM = 0.1, qMS = 0.1 (each in units per lineage per million years), is therefore sufficient to approximate the inferred distribution of outcrossing rates (65% outcrossers, 25% mixed, 10% selfers), despite the fact that both mixed-mating and selfing species are treated as evolutionary dead ends. This arbitrary assignment of the model parameters is anchored by a net diversification rate (rO − qOM = 0.1), which is close to the rate averaged across angiosperms (Magallon & Sanderson, 2001), and transition rates that approximate those observed for the breakdown of self-incompatibility in Solanaceae (Igic et al., 2006). It is rather straightforward to expand this model to an arbitrary number of states (FitzJohn, 2012), although unconstrained models will suffer for rapidly increasing numbers of parameters.

It is clear, therefore, that the mere occurrence of any given proportion of mixed-mating species does not preclude the possibility that they, too, are evolutionary dead ends. Although mixed-mating mutants can fix within populations and be maintained at a population-genetic equilibrium, a key parameter in considering their evolutionary stability is the direction and magnitude of net diversification associated with selfing, mixed-mating and outcrossing species. Under a stochastic model of such a diversification process, we expect to observe some large (and old) lineages entirely composed of either mixed-mating or selfing species. These probabilities depend on speciation, extinction and transition rates, as well as the total elapsed time (Maddison et al., 2007). The simple exponential null process outlined here is generally associated with a very high stochastic variance, so that the ‘blind alleys’ envisioned by Stebbins may greatly differ in their lengths, such that some selfing lineages may be irreversibly headed for extinction, but take a long time to get there (Stebbins, 1957). It is important to note that the qualitative nature of the described effects does not depend on absolute irreversibility and negative diversification rates in taxa with elevated selfing rates, as posited by SEDE. Strong transition asymmetry and matching differences in net diversification rates suffice. The existence of a certain proportion of mixed-mating species may thus be expected, not paradoxical. Alternatively, mixed-mating species may well persist over evolutionary timescales, but any claims regarding evolutionary stability should be substantiated by a demonstrated association with significantly positive net diversification rates. At the time of this review, no study has found that an ancestrally self-compatible (i.e. selfing or mixed-mating) group persisted significantly longer than is expected by a null birth–death process with rM > 0.

V. Redirection of study to challenge Stebbins’ claims

Throughout this article, we have drawn attention to areas of research needed to evaluate the two interdependent components of Stebbins’ SEDE hypothesis in flowering plants: the transition rate from selfing to outcrossing is zero; and self-fertilization is associated with a negative diversification rate. A diverse array of analyses in myriad families has the potential to inform us about the history of mating systems, lead to increased accuracy of macroevolutionary parameter estimates, and may suggest strong generalities about the genetic and ecological mechanisms involved. Mechanistic insights in future studies are likely to rely on detailed knowledge of the genetic basis and complexity of morphologically based outcrossing mechanisms. Although there has been little success in clearly elucidating the population-genetic mechanisms that limit the longevity of selfing plants, major advances would benefit from connecting the magnitude of standing molecular variation to additive genetic variation and adaptive evolution. Unfortunately, these renewed foci may have limited utility in revealing the specific species-level costs of selfing, given the implied rapid turnover of such taxa in the angiosperm record.

Tests of the hypothesis of strict irreversibility are plagued by a diverse set of problems, some of which may be remedied with careful experimental work. Collectively, we have long understood that the evolution of selfing is a strongly asymmetric transition associated with a negative diversification rate. The demonstration of an association between selfing and a positive net diversification is equally, if not more, challenging. The inference of transition and diversification rate parameters is certainly closely intertwined in the case of mating system evolution, and estimates of both will suffer from common sources of error and bias, such as errors in both the estimation of mating system and inference of phylogeny (Takebayashi & Morrell, 2001), heterogeneous rates of evolution (Rabosky, 2010), nonrandom sampling of species, inconsistent or biased species definitions (e.g. state-dependent clumping and splitting; see Goodwillie, 1999), variable times of the speciation process (Etienne & Rosindell, 2012), punctuated trait changes (Goldberg & Igic, 2012) and biases in branch length estimates (Revell et al., 2007). A simple statistical phylogenetic rejection of irreversibility, especially with badly mis-specified models, does not advance our understanding of mating system evolution or shed light on why a reversal may have occurred in a particular lineage. Ideally, empirical support for a reversal from selfing to outcrossing would require two necessary attributes: ancestral condition of selfing within a lineage, and derived condition of higher outcrossing rates or a novel outcrossing mechanism from such an ancestral state. Although we are unaware of a multipart demonstration for such a reversal to outcrossing at this time, future work should endeavor to differentiate whether such reversals are essentially biologically prohibited, as implied by SEDE, or are simply sufficiently rare that they have escaped our collective detection thus far. The inference of significantly positive net diversification rates is within methodological reach, but is going to be almost always associated with very large uncertainties expected from an exponential process. It is presently unclear whether we will generally have sufficient statistical power to reliably detect, for example, a very slightly positive average net diversification rate, given the long list of errors and biases.

The best hope for broad investigations of the SEDE hypothesis in flowering plants rests with the myriad nonhomologous contrivances that facilitate or enforce outcrossing and selfing. A particularly interesting set of questions may therefore challenge our understanding of mating system transitions and their connection with diversification processes. For example, are particular floral traits most conducive to uni- or bidirectional shifts in the mating system? What is the genetic architecture of the transition from obligate outcrossing to selfing? Does either the mode of selfing or the order of mutational substitutions that generate selfing phenotypes influence the probability of reversal or differentially alter diversification rates? Are there threshhold rates of selfing above which a lineage is committed to a negative diversification rate? And, for a given incremental move towards complete selfing, are a variety of outcrossing mechanisms more or less easily regained? Substantial progress is likely to emerge from studies aimed precisely at answering such questions, particularly when they provide detailed information into the underlying genetic and ecological mechanisms.


Comments by two reviewers and by M. Rausher improved the focus and clarity of the manuscript. The authors thank E. Goldberg, C. Herlihy, S. Kalisz and J. Whittall for stimulating discussions, and their laboratories for providing critiques of the initial manuscript. They are also indebted to E. Goldberg for originating and sharing many key ideas presented here.