The origin and maintenance of separate sexes (dioecy) is an enduring evolutionary puzzle. Although both hermaphroditism and dioecy occur in many diverse clades, we know little about the long-term evolutionary consequences of changing sexual system. Here we find evidence for at least 133 transitions between sexual systems in mosses, representing an almost unparalleled lability in the evolution of their sexual systems. Furthermore, in contrast to predictions, the transition rate from hermaphroditism to dioecy was approximately twice as high as the reverse transition. Our results also suggest that hermaphrodites may have higher rates of diversification than dioecious mosses. These results illustrate the utility of mosses for understanding the genomic and macroevolutionary consequences of hermaphroditism and dioecy.

The microevolutionary forces governing the frequency of separate sexes (dioecy) are well studied in natural populations, but the macroevolutionary outcomes of these processes remain uncertain (Charnov et al. 1976; Bawa 1980; Freeman et al. 1997). It is often assumed that selection for outcrossing must be strong for an organism to give up the reproductive assurance and economy of hermaphroditism (Darwin 1878). Population genetic theory shows that male sterility can evolve under a broad array of conditions, particularly when inbreeding is deleterious (Charlesworth and Charlesworth 1978). However, the evolution of complete dioecy requires mutations causing both male and female sterility. Female-specific sterility mutations can only invade a hermaphroditic population provided that they have compensatory effects on male fitness (Charnov et al. 1976). Moreover, the persistence of these mutations is likely only if the male and female-sterility loci are physically linked (Charlesworth and Charlesworth 1978), as the independent inheritance of these loci would create genotypes that contain both sterility alleles. In contrast, a broader set of mutational events, including polyploidy and hybridization, can lead to hermaphroditism (Robertson et al. 2011). In addition, mutations that produce hermaphrodites can be transmitted through selfing or outcrossing (Maynard Smith 1971), provide reproductive assurance (Busch and Delph 2012), and avoid many of the costs associated with separate sexes (e.g., the costs of recombination or sexual conflict, Otto and Lenormand 2002). As a result, mutations that produce self-fertility may be adaptive under a broad range of conditions. In spite of the strength of this theory, we have little empirical knowledge of the rates (or probabilities) of transition between these states.

The lack of information regarding transitions between sexual systems stems from the fact that in most lineages of sexual eukaryotes, one or the other sexual system predominates. For example, although hermaphroditism has evolved numerous times in the noninsect invertebrates, the majority of animals are dioecious (Jarne and Auld 2006). On the other hand, although dioecy has evolved in most major lineages of angiosperms, the vast majority of flowering plants are hermaphroditic (Renner and Ricklefs 1995). Thus, while both hermaphroditism and dioecy are found in diverse lineages, current data suggest that the conditions (ecological, demographic, or genetic) favoring the evolution of separate sexes are idiosyncratic (Platt and Brooks 1997; Miller and Venable 2000; Ashman 2002; Case and Barrett 2004), and that the general evolutionary consequences of changes in sexual system remain inconclusive (Barraclough et al. 1995; Heilbuth 2000; Heilbuth et al. 2001; Butler et al. 2007; Eppley and Jesson 2008; FitzJohn et al. 2009).

Mosses are unusual among eukaryotes in that approximately 60% of the species are dioecious and 40% are hermaphroditic (Wyatt and Anderson 1984). In dioecious species, sex determination happens at the haploid (gametophyte) stage in the life cycle. The diploid part of the life cycle (the sporophyte) is always heterozygous at the sex-determining locus, and at meiosis the female and male determining factors (U and V, following Bachtrog et al. 2011) segregate to haploid female and male spores, respectively. These spores ultimately develop into multicellular haploid female and male gametophytes, the dominant part of the moss life cycle. Sporophytes are only produced when a U-bearing egg on a female gametophyte is fertilized by a V-bearing sperm from a male gametophyte; sporophytes homozygous at the sex-determining locus are never produced in dioecious mosses. Because both males and female possess a sex-limited genomic region (sometimes a large portion of a nonrecombining chromosome, Allen 1945), the evolution of sexual dimorphism in bryophyte may be accelerated relative to organisms in which only one sex has a sex-limited chromosome (i.e., XY or ZW systems; Bachtrog et al. 2011). Indeed, bryophytes exhibit sex-specific patterns of parental care, morphology, and life-history trade-offs (Stark et al. 2001; McDaniel 2005; McDaniel et al. 2007; Horsley et al. 2011). However, because fertilization in bryophytes requires sperm to swim through the terrestrial environment, sperm limitation may be common (Bisang et al. 2004), so the advantages of self-fertility are likely to be considerable. Discerning the patterns of evolution in hermaphroditism and dioecy in mosses will provide insights into the relative importance of key population parameters in governing sexual systems in general.

To study the evolution of sexuality in the mosses, we examined the patterns of evolution of dioecy and hermaphroditism using a generic-level phylogeny. Our reconstruction shows that sexual system is highly labile in this group. We also found that transitions to dioecy occurred at twice the frequency as transitions to hermaphroditism, a finding that was robust to both phylogenetic uncertainty and the potentially confounding effects of character-associated diversification rates. While not definitive, results also suggest that hermaphroditism may be associated with higher net diversification rates. Collectively these results suggest that mutations with antagonistic effects on male and female fitness may arise frequently in mosses, and that the population genetic conditions favoring their fixation may be more common than previously appreciated.

Materials and Methods

The analyses of sexual system transitions and diversification used the genus-level phylogenetic data from Shaw et al. (2005) that included 286 representatives of completely dioecious genera, 138 representatives of completely hermaphroditic genera, and 69 representatives of genera that include both dioecious and hermaphroditic taxa (mixed genera). The sexuality data for all genera were assembled from taxonomic monographs or descriptive floras (Supplementary Table 1), and the species numbers for all genera were obtained from Crosby et al. (2000). All data are deposited in the Dryad database ( under the accession number doi:10.5061/dryad.101tb.


We first obtained the optimal genus-level tree of mosses inferred by Shaw et al. (2005). To account for uncertainty in phylogenetic relationships and branch lengths, we also performed 100 nonparametric bootstrap replicates on the Shaw et al. (2005) four-locus concatenated alignment (Felsenstein 1985). Each bootstrap replicate consisted of a maximum likelihood (ML) search of a bootstrapped dataset using RAxML-VI-HPC version 7.0.4 (Stamatakis 2006). The ML analyses used the GTRCAT nucleotide substitution model. All resulting trees were rooted using Sphagnum as the outgroup, following Shaw et al. (2005).

Prior to the character evolution and diversification analyses, we transformed the molecular branch lengths of the estimated optimal ML tree and all the bootstrap trees to ultrametric branch lengths. To do this, we used penalized likelihood (Sanderson 2002) implemented in r8s, version 1.71 (Sanderson 2003). In the r8s analysis, we arbitrarily set the root age to 1 and used a smoothing parameter value of 100.


First, to get a minimum estimate of the number of transitions in sexual state, we reconstructed the ancestral sexual system using maximum parsimony, implemented in Mesquite (Maddison and Maddison 2010), on the ML tree. This analysis does not account for the branch lengths and allows only one transition per branch.

Next, to test the null hypothesis of equal transition rates from dioecy to hermaphroditism, we performed an asymmetry likelihood ratio test (LRT) using the ML and bootstrap trees. We transformed branches with a length of 0 (from the r8s analysis) to 1.0 × 10-7 in the trees with such branches. This LRT, performed with Mesquite (Maddison and Maddison 2010), compares the likelihood of a single parameter model of equal transition rates between mating systems with a two-parameter, asymmetric model, which allows separate rates of transitions to hermaphroditism and to dioecy (Pagel 1999). Significance of the LRT was evaluated based on a χ2 distribution with 1 df. Still, a biased, character-associated diversification rate could produce a similar result, even in the absence of different transition rates between character states (e.g., Maddison 2006).

We next examined the relationship between sexual system and net diversification rate with sister clade diversification tests (Mitter et al. 1988; Vamosi and Vamosi 2005). For the optimal tree and all bootstrap trees, we identified all sister clade comparisons in which one clade consisted only of taxa that were dioecious and the other clade consisted of only hermaphroditic taxa. Clades that included genera with polymorphic or unknown sexual systems were not included in the analysis. We first compared the number of sister clade pairs with more hermaphroditic taxa and dioecious taxa using a sign test (Mitter et al. 1988). However, this test does not include information about the magnitude of difference in diversification. Therefore, we also performed Wilcoxon signed rank tests comparing the number of hermaphroditic and dioecious taxa in each sister clade comparison (Wiegmann et al. 1993) and the overall proportion of hermaphroditic and dioecious taxa in each sister clade comparison (Barraclough et al. 1995). The sister clade comparisons were done by examining the trees in Mesquite (Maddison and Maddison 2010).

The distribution of binary character states among species in a phylogeny depends on both the transition rates between character states and the diversification rate associated with each character state (Maddison 2006). The BiSSE modeling approach can estimate jointly transition rates among character states and character-associated diversification rates, and thus parse their potentially confounding effects (Maddison et al. 2007; FitzJohn et al. 2009). Our dataset, which includes only a single representative taxon per genus, is not ideal for these approaches. The large size of several of the genera makes the BiSSE model analysis that allows terminally unresolved clades computationally infeasible (FitzJohn et al. 2009). Instead, we took an approach that assumes that we have randomly sampled a fraction of the possible taxa. Although sampling a single representative per genus likely does not represent random sampling and we have only ∼5% of the total extant species in our tree, this analyses provides another way of exploring the possibly inter-related effects of transitions among sexual states and diversification. These BiSSE model analyses were implemented using the diversitree R package ( We were unable to obtain ML estimates for all bootstrap trees, but we present results from the first 50 bootstrap trees to run to completion.

Results and Discussion

The maximum parsimony reconstructions identified a minimum of 64 transitions between hermaphroditism and dioecy in the genus level phylogeny (Fig. 1). If we assume only a single additional transition in each mixed genus, there are 133 transitions among sexual systems in mosses. This result is consistent with smaller scale studies in other bryophytes (Devos et al. 2011; Jesson et al. 2011) and illustrates a remarkable lability in sexual system.

Figure 1.

A maximum parsimony reconstruction of hermaphroditism (light gray) and dioecy (dark gray) in mosses, showing a minimum of 64 transitions between sexual systems.

The asymmetric LRT also found evidence of a significantly higher transition rate from hermaphroditism to dioecy then from dioecy to hermaphroditism using the ML tree and all 100 bootstrap trees (in ML tree LR = 10.2; 1 df; P < 0.001). The transition bias was remarkably robust to phylogenetic uncertainty, with the evolution to dioecy occurring at twice the rate of the evolution to hermaphroditism in all bootstrap replicates (mean transition bias 2.01, SD among bootstrap replicates = 0.047). However, the estimates of transition rate varied more among bootstrap replicates (q01: mean = 39.3, SD = 27.9; q10: mean = 80.5, SD = 58.0; note that the root to tip distance on the tree is 1).

It is possible that the estimates of the transition rates between sexual systems may be biased if the sexual systems are associated with different rates of diversification (Maddison 2006). For example, the estimated transition bias toward dioecy in mosses could be caused by an elevated net diversification in lineages with separate sexes and not an elevated rate of transitions to dioecy. We first performed a sister clade analysis to test the null hypothesis of no difference in net diversification associated with sexual system, and found no evidence to link character state and net diversification. In 17 of 31 comparisons (with one tie) in the ML tree, the hermaphroditic clade included more species than the dioecious clade (P= 0.36). This pattern of more species in the hermaphroditic sister clades was also found in 94 of the 100 bootstrap trees, although the difference was only significant in six replicates. Similarly, hermaphroditic clades had an average of 6.2 more species than their dioecious sister clades (SD = 82; Wilcoxon sign rank test P= 0.179). The Wilcoxon signed rank test showed significantly more hermaphroditic species in 27 of the 100 bootstrap replicates. Still, the sister clade analysis excludes any genera with multiple sexual systems or genera that are not part of a sister clade comparison. Thus, it may have low power to detect differences in diversification rates.

The BiSSE analyses, which simultaneously estimate transition rates and character-associated diversification, often appeared to have some trouble optimizing the likelihood function. The parameter estimates varied tremendously among some of the runs, which could indicate low power to estimate the parameters or a failure of the optimization. Still, overall, these analyses agree with the previous analyses (Fig. 2). The median transition bias from hermaphroditism to dioecy across 50 bootstrap trees was 1.68 (q01: median = 82.3; q10: median = 158.6). The net diversification rate was higher for hermaphrodites than dioecious species in 38 of the 50 bootstrap replicates (with a median difference of 52.7). Interestingly, dioecious species had a higher speciation rate in 45 of 50 replicates but also a higher extinction rate in 44 of the 50 replicates. The transition bias and net diversification bias must be in opposition to maintain both sexual systems at relatively high frequencies on the bryophyte phylogeny—indeed, relatively few BiSSE runs fell in B and D quadrants of Figure 2. These data also mirror similar analyses from angiosperms where hermaphrodites had a higher diversification rate than dioecious lineages (Heilbuth 2000).

Figure 2.

A plot of transition bias (the larger of q01 and q10 divided by the smaller) and net diversification ([λ0 - μ0] - [λ1 - μ1], in both cases with negative values indicating a bias toward hermaphrodites.

Recurrent transitions to dioecy are not unprecedented (Renner and Ricklefs 1995), but a significant transition bias is unexpected for three main reasons. First, the evolution of hermaphroditism may occur by multiple mutational mechanisms, including hybridization and whole genome duplications, both of which occur frequently in mosses (Natcheva and Cronberg 2004; Crawford et al. 2009). In contrast, the evolution of dioecy involves a minimum of two tightly linked mutations with opposing effects on male and female fertility (Charlesworth and Charlesworth 1978). Second, mutations conferring the ability to self-fertilize may be passed on through selfing or outcrossing and therefore have an immediate transmission advantage (Maynard Smith 1971). Moreover, mutations to hermaphroditism give their carriers reproductive assurance (Baker 1955; Ghiselin 1969), which would seem to be important given that fertilization in mosses requires that motile sperm swim through the terrestrial environment. Third, inbreeding avoidance, which is a major factor promoting the evolution of self-incompatibility in angiosperms (Busch and Delph 2012), may be less likely to contribute to the evolution of dioecy in mosses. Because mosses mate as haploids, self-fertilization results in a diploid that is completely homozygous (Klekowski 1969). Thus, all recessive deleterious alleles are unmasked in each selfing event (barring complementation among homologous copies in recent polyploids), so that they should be purged more rapidly than in flowering plants where such alleles may persist for many generations. Consistent with this inference, self-fertilization appears to predominate in hermaphroditic mosses (Eppley et al. 2006; Perroud et al. 2011) and no phenotypic differences have been found between inbred and outbred offspring in hermaphroditic mosses (Taylor et al. 2007; Perroud et al. 2011), suggesting little or no cost of inbreeding for hermaphrodites.

Our results suggest that the recurrent evolution of dioecy is most likely to be driven by sexual specialization (Charnov et al. 1976; Bawa 1980). Dioecious mosses are often sexually dimorphic, indicating that the optimal phenotype for males, which broadcast sperm, is different from that for females, which nurture the relatively large embryo until it matures, produces spores, and senesces. Theory shows that the abolishment of one sexual function in a hermaphrodite requires mutations that also have strong compensatory effects on the fertility of the other sexual function (Charlesworth and Charlesworth 1978). Although alleles with sexually antagonistic effects segregate in species with separate sexes (Cox and Calsbeek 2009), it is unknown how often alleles that cause this pattern of sexual reallocation arise in hermaphroditic species (Abbott 2011). The numerous transitions to dioecy that we found strongly suggest that such alleles frequently occur in hermaphroditic moss species. In addition, the increase in frequency of sexually antagonistic alleles may be facilitated by the presence of both female and male-limited genomic regions that arise as a consequence of genetic sex determination in bryophytes (Bachtrog et al. 2011).

Collectively these findings demonstrate that the moss sexual system is evolutionarily highly dynamic. The observation that dioecy evolves frequently in organisms that depend upon restricted conditions for fertilization and apparently have limited inbreeding depression strongly suggests an adaptive value of separate sexes beyond the benefits of outbreeding alone (Freeman et al. 1997). These results additionally highlight the utility of mosses for studying the genetic architecture of sexual antagonism and the evolution of sexual systems in general.

Associate Editor: M. Johnston


J. Shaw and C. Cox kindly provided the phylogenetic tree and species number data used in our analyses. D. Charlesworth, S. Otto, D. Promislow, and M. Wayne, and two anonymous reviewers made helpful comments on an earlier version of this manuscript.