Self-fertilization is expected to reduce genetic diversity within populations and consequently to limit adaptability to changing environments. Little is known, however, about the way the evolution of self-fertilization changes the amount or pattern of the components of genetic variation in natural populations. In this study, a reciprocal North Carolina II design and maximum-likelihood methods were implemented to investigate the genetic basis of variation for 15 floral and vegetative traits in four populations of the annual plant Amsinckia spectabilis (Boraginaceae) differing in mating system. Six variance components were estimated according to Cockerham and Weir's “bio” model c. Compared to the three partially selfing populations, we found significantly lower levels of nuclear variance for several traits in the nearly completely self-fertilizing population. Furthermore, for 11 of 15 traits we did not detect nuclear variation to be significantly greater than zero. We also found high maternal variance in one of the partially selfing populations for several traits, and little dominance variance in any population. These results are in agreement with the evolutionary dead-end hypothesis for highly self-fertilizing taxa.

One of the most common evolutionary patterns among flowering plants is the transition from an outcrossing to a self-fertilizing mode of reproduction (Stebbins 1950, 1957; Grant 1981; reviewed in Takebayashi and Morrell 2001). Several ecological and genetic theories have been put forth to explain this, including reproductive assurance when pollen receipt is unreliable or scarce (Baker 1955; Stebbins 1957; Kalisz et al. 2004; Moeller and Geber 2005); greater propensity for local adaptation due to decreased recombination rates (Stebbins 1957; Lloyd 1979); Fisher's (1941) automatic transmission advantage, where a gene conferring complete self-fertilization while still allowing pollen dispersal quickly spreads in a population of outcrossers; and the lower cost of producing selfed offspring (Schoen and Lloyd 1984).

Despite these apparent advantages, selfing must have associated inhibitions to its evolution or persistence, as only 10–15% of seed-plant taxa have a selfing rate greater than 80% (Schemske and Lande 1985; Barrett and Eckert 1990; Goodwillie et al. 2005). The main genetic force opposing the automatic transmission advantage is inbreeding depression, where the fitness of selfed progeny is lower than that of outcrossed offspring. Inbreeding depression is a ubiquitous feature of populations (Charlesworth and Charlesworth 1987), and can inhibit the transition to a predominantly selfing mode of reproduction. All else being equal, increased selfing is expected to evolve only when selfed progeny are more than one-half as fit as outcrossed progeny (Lloyd 1979; Lande and Schemske 1985). Although the transition to selfing is common, there appear to be no ancient highly selfing lineages, prompting Stebbins (1957) to contend that extreme selfing is an evolutionary dead end (see also Grant 1981; Wyatt 1988; Takebayashi and Morrell 2001).

Stebbins’ (1957) original proposal that highly selfing lineages are “evolutionary dead-ends” is based on the expected loss of genetic variation associated with selfing, which consequently limits the potential for adaptation and speciation, and hence increases the probability of extinction in selfing lineages. An assumption of this hypothesis is that selfing lineages cannot revert to outcrossing. Stebbins (1957) cited several kinds of evidence that suggested selfing lineages are most likely derived from outcrossing ancestors. More recently, in a review of phylogenetic studies, Takebayashi and Morrell (2001) found no study to provide unequivocal support from the transition of highly selfing to outcrossing.

There are several reasons to expect high levels of selfing to result in decreased genetic variation. Self-fertilization has direct impacts on population structure, which may in turn affect the capacity of populations to maintain neutral genetic variation. The increased homozygosity within populations resulting from inbreeding acts to reduce the effective population size (Pollak 1987; see also Ingvarsson 2002 for a review of this topic). At least for neutral molecular variation, a heavily inbred population behaves more like a population of half its census size. The effective population size will also be influenced by extreme bottlenecks resulting from recurrent extinction and recolonization events that are common in selfing populations (Baker 1955; Awadalla and Ritland 1997; Liu et al. 1998; Ingvarsson 2002; Charlesworth 2003). In addition to ecological and demographic considerations, self-fertilization is expected to enhance the effects of genetic hitchhiking accompanying selective sweeps (Maynard Smith and Haigh 1974; Hedrick 1980) and background selection against deleterious mutations (Charlesworth et al. 1993; Charlesworth and Charlesworth 1995; Glemin et al. 2006) both of which reduce genetic variation. These effects in highly inbreeding populations result from the reduced effective recombination rate in homozygous lines; crossing-over events at homologous loci will have little or no effect, because the alleles at those loci are identical (Charlesworth et al. 1993; Nordborg 2000). Evidence for these processes has been found in studies of Leavenworthia (Liu et al. 1998, 1999), Lycopersicon (Baudry et al. 2001), Brassica (Purugganan et al. 2000), Arabidopsis (Bergelson et al. 1998; Savolainen et al. 2000; Wright et al. 2002), Mimulus (Fenster and Ritland 1992; Charlesworth 2003) and in Amsinckia (Pérusse and Schoen 2004). It is important to emphasize that recombination within intermediate selfing populations appears to be sufficient to result in similar patterns of molecular diversity as that of random mating (Charlesworth et al 1993; Glemin et al 2006). For brevity we will use “selfing” to refer to very highly selfing populations or species and “partially selfing” for intermediate selfing rates.

Molecular marker-based estimates of genetic diversity, however, cannot directly address to what extent selfing reduces the genetic variation important for adaptive change, namely quantitative genetic variation. Theory suggests that selfing should reduce such a variation. The magnitude, however, is dependent on the mechanism maintaining genetic variation in natural populations (reviewed in Charlesworth and Charlesworth 1995). Mutation–selection balance will be less effective in maintaining variants in a selfing population. New mutations are more quickly exposed to selection in selfing populations, reducing their frequency as compared to an outcrossing population. Overdominance may be effective in maintaining genetic variation in outcrossing populations. It is, however, unlikely to do so in highly inbred populations unless selection coefficients against the homozygous genotypes are nearly symmetrical, which seems biologically implausible. Under some assumptions, however, the reduction of genetic diversity in selfers will be limited and as such selfing may not have a strong influence on adaptability. In a survey of the level of quantitative genetic variation in plants, Charlesworth and Charlesworth (1995) found that selfing is associated with a reduction in the genetic coefficient of variation (see figs. 3 and 4 in Charlesworth and Charlesworth 1995). The generality of this conclusion remains open to question as it was based on only 12 studies. Little empirical evidence is available to determine which of the proposed mechanisms is most relevant to natural populations and to what extent selfing reduces quantitative genetic variation. Thus it is premature to assess whether genetic variation is sufficiently reduced to eliminate adaptive potential and drive selfing populations to extinction.

Here we describe an analysis of quantitative genetic variation for 15 traits in four populations of Amsinckia spectabilis (Boraginaceae) differing in rate of self-fertilization. Three populations had intermediate selfing rates and one population was highly selfing. We therefore were able to compare quantitative genetic variation among partially selfing populations differing in selfing rate as well as between partially selfing populations and a highly selfing population.

Materials and Methods


Amsinckia (Boraginaceae) is a genus of annual plants centered in western North America (Ray and Chisaki 1957a,b). The genus is characterized by small groups of close relatives—either species or populations—that differ widely in rate of self-fertilization and associated floral morphology (Schoen et al. 1997; Li and Johnston 2001). Large-flowered, distylous taxa are more highly outcrossing than smaller-flowered, homostylous relatives (Johnston and Schoen 1996; Schoen et al. 1997). Distylous taxa exhibit two floral morphs (pins and thrums) that differ reciprocally in style and stamen lengths. In pins, the stigma is high, and the anthers are low at the base of the corolla. Conversely, thrums have a short stigma presented near the nectary at the base of the corolla and have longer anthers. Distyly is unusual in Amsinckia in that it is not accompanied by marked self- and intramorph incompatibility (Ray and Chisaki 1957a). Small genetic differences separate highly selfing and more outcrossing taxa suggesting that self-fertilization is recently derived (Schoen et al. 1997; M. O. Johnston and W. J. Hahn, unpubl. data).

We studied four California populations of A. spectabilis reflecting the wide range of mating systems within this genus (Fig. 1, Table 1). Selfing rates were estimated directly for one population, and by inference from closely similar populations for the other three. Nipomo (91003) and Santa Maria (91004) are large-flowered distylous populations. Distylous populations require pollinators for seed set (M. O. Johnston, pers. obs.; Pérusse and Schoen 2004). Previous analysis using allozyme variation in the Nipomo population found a selfing rate of 0.55 with a 95% confidence interval (CI) of 0.35–0.70 (Johnston and Schoen 1996). A nearby distylous population (Lompoc, not included here), had a selfing rate of 0.33 with 95% CI of 0.19–0.45 (Johnston and Schoen 1996). We thus consider the selfing rate of Santa Maria (91004) to be approximately 0.3 to 0.6. Montana de Oro (95002) is a large-flowered “mixed” seaside population displaying a continuous range of stigma–anther separation as well as pins and thrums. We consider its selfing rate to be approximately 0.73, as found for another such mixed population (Lompoc 17 or La Purisima; 95% CI 0.54–0.84; Johnston and Schoen 1996). Mixed populations set abundant seed in the absence of pollinators (M. O. Johnston, pers. obs.). Zmudowski State Beach (91011) is a small-flowered homostylous population that can be considered nearly completely selfing, for the following reasons. First, the inbreeding-depression method found the selfing rate to be functionally 100% (Johnston and Schoen 1995, 1996). This method compares the mean fitness of offspring produced naturally to those from completely selfed and outcrossed matings. Second, no allozyme polymorphisms were found for this population (91011). Third, direct estimate using allozymes in a nearby small-flowered homostylous population (Alisal Slough) found a selfing rate of 0.998 (95% CI 0.78–1.0; Johnston and Schoen 1995, 1996). Fourth, full seed set occurs in the absence of pollinators in all small-flowered homostylous A. spectabilis populations studied (Schoen et al. 1997; M. O. Johnston, pers. obs.).

Figure 1.

Dissected Amsinckia spectabilis flowers, showing the floral morphs that exist within the studied populations. Nipomo (91003) and Santa Maria (91004) are distylous, consisting of pins and thrums. Montana de Oro (95002) is a mixed population, consisting of morphs that vary from pin or pin-like through homostylous to thrum or thrum-like. The Zmudowski State Beach (91011) population is small-flowered homostylous.

Table 1. Amsinckia spectabilis populations: floral characteristics, collection location in California, and selfing rate.
Floral typeLocation (number)Selfing rate
  1. 1Johnston and Schoen 1996.

  2. 2Not directly measured in this population; selfing rate from similar populations; see text.

Large flower heterostylousNipomo (91003)0.551
Santa Maria (91004)0.33–0.552
Large flower, mixed (homostylous and heterostylous)Montana de Oro State Park (95002)0.732
Small flower homostylousZmudowski State Beach (91011)∼12


Seeds were collected from the Santa Maria population in 2000 and from the other three populations in 1995, and were stored in dry conditions at 4°C. The parental individuals were planted in December 2004 and grown throughout the winter of 2004–2005 in a Conviron growth chamber at Dalhousie University. Individuals from different populations were interspersed haphazardly among one another to avoid confounding differences in parental effects among populations with the unknown positional effects of the growth chamber. The plants were grown under 10 h day length with day-time temperature of 24°C and night-time temperature of 15°C.

Flowering occurred throughout the period of January–March 2005. Eight individuals from each population were crossed in a reciprocal North Carolina II design excluding self-pollinations (herein referred to as a “set,”Fig. 2). Five types of relatives result from this design: full-siblings (FS), maternal half-siblings (MHS), paternal half-siblings (PHS), reciprocal full-siblings (RFS), and reciprocal-half siblings (RHS). RHS share one parent that is the father of one and the mother of the other. With RFS, the father of one individual is the mother of the other, and vice versa. Crosses were performed within “sets” comprising eight individuals. Each pollination set was conducted three times for Santa Maria (91004) and Montana de Oro (95002), and twice for Zmudowski (91011) and Nipomo (91003) using different individuals to create each crossing set (Fig. 2). The number of maternal–paternal combinations (cross) within each set was constrained due to the reproductive biology of A. spectabilis. For instance, a plant in the growth chamber can reliably produce approximately 20 flowers (M. P. Bartkowska, pers. obs.), and each flower can produce a maximum of four seeds. By repeating the crossing design for each population in the manner described above, we ensured that a sufficient number of offspring resulted from each maternal–paternal combination and at the same time maximized the number of parental individuals surveyed from each population. Only two crossing sets were conducted for the Zmudowski State Beach population (91011) and the Nipomo population (91003) due to the limited availability of parental plants flowering concurrently during the pollination period (January–March of 2005). As such, the total number of parental individuals sampled from the natural population was 24 for Santa Maria (91004) and Montana de Oro (95002), and 16 for Zmudowski (91011) and Nipomo (91003).

Figure 2.

Breeding design. Eight individuals from a population were crossed in a reciprocal North Carolina II design to form a set. There were three such sets for Santa Maria (91004) and Montana de Oro (95002), and two for Zmudowski (91011) and Nipomo (91003).

Several pollinations per maternal–paternal combination were performed throughout the flowering period. The order of the pollen donor was haphazardly assigned during pollinations. To avoid potential resource limitations on the developing fruit, one unpollinated flower was left between two pollinated flowers. All flowers acting as pollen recipients were emasculated one day prior to pollination to prevent autonomous self-fertilization. Hand pollinations were performed by removing an anther from the pollen donor and gently rubbing it over the stigma of the pollen recipient. Only anthers from fully opened flowers with no visible signs of senescence were used to minimize variation introduced by differences in pollen quality due to pollen age. Fruits matured approximately two weeks following pollination, at which time ripe seeds were collected and stored for future use.


In April 2005, five seeds per maternal–paternal combination were germinated on moistened filter paper in petri plates at approximately 5°C. Most seeds had germinated nine days after being moistened. Batches of seedlings were then individually potted into 4 cm × 4 cm square pots over the following four days. Four offspring were planted for the majority of maternal–paternal combinations; however, for some combinations, no seeds germinated. In some cases, an extra seedling was planted (extra seedling was chosen haphazardly from available seedlings). A total of 1296 individual seedlings were planted.

A seedling from each maternal–paternal combination was assigned a random position within one of four blocks in a photoperiod-controlled greenhouse room. Following the initial placement, individually potted plants were rotated periodically between locations within blocks to randomize the environmental variation. Day length was limited to 10 h and natural light was supplemented with 400 W high-pressure sodium lamps throughout the day. The length of the longest cotyledon, length and width of the longest north-facing leaf, and the number of rosette leaves greater than 3 cm were measured 14, 37, and 42 days, respectively, following planting. Individual plants were moved to a larger greenhouse room 40 days following planting (beginning 13 May 2005), where they were exposed to natural day lengths supplemented with sodium lamps for 14 h throughout the day.

Of the 1296 individuals planted, 1260 survived to flower. The number of days from planting to the opening of the first flower was recorded for each individual. Several flowers from each individual were collected and fixed in formalin acetic acid (FAA) ethanol and stored for further study of floral characters. The second open flower was chosen to avoid confounding floral measurements with floral position. Flower size differs among open flowers on a fiddlehead, with the newly opened flowers being the smallest and flowers beginning to senesce being the largest (M. P. Bartkowska, pers. obs.; see also Li and Johnston 2001). The total number of flowers per plant was counted following plant death.


From the preserved flowers, two were chosen haphazardly for detailed study of floral morphology. These were dissected under an Olympus SZH10 stereo microscope (Olympus America Inc., Melville, NY) connected to a video imaging system and computer. Measurements of floral traits performed were on images of dissected floral parts using the public domain NIH image program (ver. 1.36b developed at the U.S. National Institutes of Health and available on the internet at http://rsb.info.nih.gov/nih-image/). Traits were defined and measured as in Li and Johnston (2001). For each flower, the following measurements were obtained: length of each sepal (KSL); length and width of a haphazardly chosen corolla lobe (CPTL and CLBW, respectively); pistil height from base of ovary to the top of the stigma (PISL); mid height of the shortest and longest stamen (SSIL = (shortest SSIL + longest SSIL) ÷ 2); stamen height range (SSIL_RANGE = longest SSIL – shortest SSIL); surface area of each anther (SANSA) calculated from the anther length (SANL) and width (SANW) as SANL × SANW; stigma surface area (PSTA) calculated from the stigma height (PSTH), width (PSTW), and length (PSTL); and functional anther–stigma separation distance (ASD). The stigma surface area (PSTA) was estimated as a five-sided box and was calculated as


The functional anther–stigma distance, ASD, was the minimum distance separating the top of the stigma from the anthers. If the stigma top was below the bottom of the tallest anther and above the anther top of the shortest stamen, then ASD was 0. Otherwise, ASD was positive and was calculated as follows. If the stigma was below the shortest anther bottom, then ASD was the difference between the base of the shortest stamen and the pistil length. If the stigma was above the tallest anther top (PISL> SSIL), then ASD was the difference between the pistil height (PISL) and the insertion length of the tallest stamen (SSILL). The final trait value assigned to an individual for each trait was taken as an average of measurements obtained from the two flowers. Approximately 2000 flowers were photographed. Several flowers from the intermediate selfing population, Montana de Oro, were excluded from this analysis. Due to time limitations, we were unable to collect flowers from every individual in this population that met our criteria for flower choice. In total, over 48,000 measurements were taken.


Differences among populations in the measured traits were explored using multivariate analysis of variance (MANOVA) with the GLM procedure in SAS version 9.1 (SAS Institute Inc., Cary, NC). A one-way MANOVA with all populations was conducted, as well as all pairwise population contrasts. We obtained exact calculations of the Wilk's lambda test statistic using the MSTAT = EXACT option. Separate univariate ANOVAs were also conducted.

The “bio” model c of Cockerham and Weir (1977), designed for reciprocal breeding designs, was used to partition the total phenotypic variance, Vp, into six components,


where Vn is variance due to additive nuclear effects; Vt is variance due to nonadditive nuclear effects; Vmat is variance due to maternal effects (includes maternal environmental effects and variance in genes specific to maternal function); Vpat is variance due to paternal effects (includes environmental effects affecting pollen donation, and variance among genes specific to paternal function); Vk is variance caused by interactions between parental contributions to progeny phenotype apart from the interaction of nuclear genetic effects, also referred to as the specific reciprocal combining ability variance (includes interaction between nuclear genes of one parent and cytoplasmic genes of the other or between cytoplasmic genes contributed from each parent); and Vres is residual environmental variance (within-family effects). In this manner, the relative importance of all major components excluding epistasis and gene-by-environment interactions can be determined.

We used the MIXED procedure in SAS to generate restricted maximum likelihood (REML) estimates of these variance components. This program evaluates the covariances between different types of relatives, which can be expressed as linear combinations of the variance components in the following manner:


The TYPE = LIN(5) option in the MIXED procedure provided direct estimates of the variance components. Negative estimates were constrained to zero. Code for this model was adapted from Fry (2004).

Estimates of additive (Va) and dominance variance (Vt) can be directly obtained from the observational components of the “bio” model c (Lynch and Walsh 1997). The relationship between the observational and causal components of variance, however, depends on the degree of inbreeding (F). For a randomly mating base population the observational components of the “bio” model c are related to causal components of variation in the following manner:


In a completely inbred base population the relationship is as follows:


It is worth noting that the causal components of the residual variance (Vres) also differ with the degree of inbreeding present in the base population. For a randomly mating population Vres includes additive and dominance variance in addition to environmental variation (Ve):


Conversely, in a completely inbred population, Vres is entirely due to environmental variation. This is because siblings within families will have the same genotype, thus within family variation will be due to environmental variation. For the populations that outcross to some extent we cannot accurately convert the observational components of variation into the causal components because we lack direct estimates of the level of inbreeding. Nonetheless, for these populations we can be confident that the relationship between the observational and causal components will lie somewhere between the expectation for random mating and complete inbreeding for the three populations that outcross to some extent.

Likelihood-ratio tests were performed on all non-negative estimates to test whether the estimates of the variance components were significantly different from zero. This was done by successively constraining each nonzero variance component to zero, and subtracting the likelihood of this model, L0, from the maximum likelihood for the data using all nonzero variance components, Lmax. Twice the difference in these likelihoods follows a chi-square (χ2) distribution, with one degree of freedom (the difference in the number of parameters between the two models; Shaw 1987). It should be noted, however, that this test is not strictly distributed as chi square when negative estimates of variance components are prohibited, as in the model used here (reviewed in Shaw and Geyer 1997). These constraints can make the usual log-likelihood test overly conservative. One approach in reducing this problem is to treat the log-likelihood test as a one-sided test by using critical values that correspond to two times alpha. Although this approach alters the P-values reported in this study, it does not change which variance components are reported to be significantly different from zero. Thus, we opted to use the standard test for likelihood testing.

Maximum-likelihood methods used for parameter estimation do not require the usual assumptions of least squares methods. For significance testing, however, these assumptions must be met. Frequency distributions of the residuals, plots of the residuals versus the trait and of the residuals versus the predicted values for each trait within a population were evaluated. These showed little deviation from normality, or from independence and homogeneity of variance for any of the traits for all populations. The deviations from the assumptions were generated by a few individuals within each population that had particularly low and high trait values. No transformation greatly improved the normality of the data and therefore all tests for variance components measured in single population models were conducted on raw data.

To compare the genetic architecture of traits among populations, we first standardized the trait values to zero mean and unit variance for each population.

Standardizing the population variance allowed us to compare variance components independently of total phenotypic variance. We also collapsed the models used to compare populations to three variance components: Vn, Vmat, and Vres. The other variance components contributed to very few traits (Supporting Appendix S1). Restricting the number of variance components to three before contrasting populations should yield greater power to detect differences in Vn among populations. This approach should also avoid the problem of underestimating the difference in log-likelihoods between competing models, caused by an artificially good fit for the null hypothesis model due to a large number of variance components. For these analyses we also pooled the two large-flowered heterostylous populations, Nipomo and Santa Maria. These populations were not found to differ significantly with regards to levels of nuclear genetic variation for any trait (results not shown). Only for days to first flower were these populations found to differ in the total amount of genetic variation, most likely stemming from larger amounts of maternal variance for this trait in Santa Maria (results not shown).

We compared genetic structure using likelihood-ratio tests conducted between all possible pairs of the three groups (Nipomo plus Santa Maria, Montana de Oro, Zmudowski). We tested for differences at two levels. First, we assessed whether the total variance structure (i.e., the total genetic variation) differed between population pairs by comparing a model in which variance components were free to vary between the populations (full model with six parameters estimated, giving Lmax) to one in which the parameters were constrained to be the same in both populations (constrained model defined by the null hypothesis with three parameters, giving Lo). In the constrained model, the grand mean and the mean of each population were fit as fixed effects and families were fit as random effects. Under this model, the two populations being compared were free to differ in mean but the model was optimized for only one overall estimate of each variance component. The log-likelihood for the full model, Lmax, was obtained by taking the sum of the log-likelihoods from the mixed models run on each individual population (Lmax  pop1+Lmax  pop2). We can obtain the likelihood of the data from two populations given that variance components are free to converge to different optima for the two populations by recognizing that the likelihood of a set of independent observations is the product of the likelihoods of the individual observations (Hilborn and Mangel 1997). Because logarithms are additive, the negative log-likelihoods obtained from the single population models add to yield the log-likelihood given that variance components are free to converge to different optima for the two populations. This is equivalent to allowing the MIXED procedure in SAS to optimize for different values of the variance components for the two populations being evaluated. For this test there are three degrees of freedom (six variance components estimated in the full model minus three estimated in the constrained model).

Second, we tested whether the contribution of nuclear additive variance, Vn, differed between population pairs. Here the null hypothesis of no difference was modeled by constraining the nuclear variance in each population to be the same. This value was obtained from the constrained model in the total variance test above. For this log-likelihood test there is one degree of freedom (six variance components estimated for Lmax minus five variance components fit for L0; see Shaw 1987 for greater detail concerning likelihood tests for variance components). We also compared maternal variance among populations using the same approach.



The multivariate test of differences among populations was statistically significant (Wilks’ lambda = 0.016; P < 0.0001). All the multivariate pairwise population contrasts were also significant (all P < 0.0001). Univariate ANOVAs were statistically significant for all traits (all P < 0.0001). Trait means and pairwise tests of differences between populations are shown in Table 2. For the majority of traits there was not an evident trend in mean associated with mating system. A trend did exist for five floral traits (corolla lobe length, corolla lobe width, stigma surface area, anther surface area, and functional anther–stigma distance), which were all largest in the heterostylous populations (Nipomo, Santa Maria), intermediate in the mixed population (Montana de Oro), and smallest in the homostylous population (Zmudowski; Table 2).

Table 2.  Mean (standard deviation and number of observations) for the 15 traits examined in four A. spectabilis populations. Trait means that share a common superscript are not significantly different as determined by t-tests with bootstrapped P-values adjusted for multiple comparisons.
TraitNipomo Mean (SD, N)Santa Maria Mean (SD, N)Montana de Oro Mean (SD, N)Zmudowski Mean (SD, N)
Cotyledon length (mm)12.9a16.0b14.4c10.7d
(1.9, 250)(2.2, 395)(2.4, 350)(1.7, 268)
Leaf length (mm)123.0a116.6b115.2b103.0c
 (13.3, 250)(16.7, 395)(13.0, 350)(15.6, 269)
Leaf width (mm)10.6a11.7 b13.5c6.7d
 (1.5, 250)(1.8b, 395)(2.2, 350)(1.6, 269)
Leaf number (mm)34.2a32.9b29.2c41.1d
 (5.2, 250)(4.8, 395)(4.2, 353)(6.1, 269)
Days to first flower57.6a56.5b51.3c62.9d
 (4.6, 247)(4.1, 392)(3.3, 351)(4.1, 253)
Total number of flowers430.5a387.7b262.2c347.7d
 (108.5, 250)(89.7, 394)(77.0, 353)(100.1, 259)
Sepal length (mm)9.7a10.0b9.8a,b7.5c
 (1.0, 239)(1.1, 397)(1.1, 244)(0.6, 246)
Corolla lobe length (mm)28.8a29.8b24.5c19.3d
 (3.2, 236)(3.4, 345)(2.7, 244)(1.4, 246)
Corolla lobe width (mm)10.4a10.1a8.4b5.2c
 (1.4, 239)(1.5, 364)(1.41, 244)(0.7, 246)
Stamen insertion length (mm)13.9a16.4b12.8c12.3c
 (4.4, 239)(5.0, 367)(2.5, 244)(0.8, 246)
Stamen insertion length range (mm)0.7a0.8a0.9a1.1b
 (0.6, 239)(1.1, 367)(0.7, 244)(0.5, 246)
Pistil insertion length (mm)14.4a13.4b11.2c11.8c
 (4.5, 239)(4.9, 367)(2.2, 244)(0.8, 246)
Stigma surface area (mm2)3.5a3.4a2.5b1.3c
 (0.7, 239)(0.8, 366)(0.6, 244)(0.2, 246)
Anther surface area (mm2)2.8a3.2b2.3c1.5d
 (0.7, 239)(0.8, 367)(0.5, 244)(0.2, 246)
Functional anther-stigma separation (mm)7.4a8.4b2.9c0.3d
 (2.3, 239)(2.5, 367)(2.6, 244)(0.7, 246)


The selfing population, Zmudowski State Beach, had the fewest traits for which significant nuclear variance (Vn) was detected (Fig. 3; Supporting Appendix S1). These included stigma surface area, days to first flower, sepal length, and corolla lobe width. The latter two traits were found to have a significant contribution of nuclear variance in all other populations. Excluding the selfing population, all other populations also had significant contributions of nuclear variance to the following traits: leaf length, leaf width, leaf number, stamen insertion length, pistil insertion length, and functional anther stigma separation (Supporting Appendix S1). Significant nuclear variance was also detected in Nipomo for anther surface area, Santa Maria for days to first flower, and both Santa Maria and Montana de Oro for corolla lobe length and stigma surface area (Supporting Appendix S1).

Figure 3.

Figure 3.

Proportion of total phenotypic variance for six components of variance in Cockerham and Weir's (1977)“bio” model c. Results from log-likelihood tests comparing total variance structure, Vgen, and nuclear variance, Vn, among populations are reported in rows above the individual population estimates. Populations that do not differ share a common letter. Populations have been shortened as follows: Nip. + S. M. represents the pooled heterostylous populations, Nipomo and Santa Maria; Mont. denotes Montana de Oro; Zmud. is Zmudowksi State Beach.

Figure 3.

Figure 3.

Proportion of total phenotypic variance for six components of variance in Cockerham and Weir's (1977)“bio” model c. Results from log-likelihood tests comparing total variance structure, Vgen, and nuclear variance, Vn, among populations are reported in rows above the individual population estimates. Populations that do not differ share a common letter. Populations have been shortened as follows: Nip. + S. M. represents the pooled heterostylous populations, Nipomo and Santa Maria; Mont. denotes Montana de Oro; Zmud. is Zmudowksi State Beach.

Maternal effects contributed to all of the traits expressed up to and including days to first flower in the mixed population Montana de Oro, and also to cotyledon length, leaf length, and days to first flower in Santa Maria (Fig. 3; Supporting Appendix S1). Maternal effects extended to floral traits including days to first flower, stamen insertion length, for both populations, and also for stamen insertion length in Santa Maria and corolla lobe length in Montana de Oro (Fig. 3; Supporting Appendix S1).

The other genetic components contributed to very few traits. There was no dominance variance, Vt, detected in any population (Supporting Appendix S1). Paternal effects, Vpat, were detected only for days to first flower in Montana de Oro. Variance in specific reciprocal combining ability, Vk, was detected for cotyledon length in both Nipomo and Montana de Oro and for average anther insertion length in Montana de Oro (Supporting Appendix S1).


Log-likelihoods obtained for the models in which components are free to vary among populations as well as those for models in which components are constrained to be the same in the populations being compared are presented in Supporting Appendix S2. We did not detect a difference either in nuclear variance or total genetic variance among any of the populations for five traits: sepal length, corolla lobe width, range of stamen insertion length, stigma surface area, and anther surface area. These traits also had the lowest variance in all the populations.

The two large-flowered heterostylous populations Nipomo and Santa Maria, did not differ in nuclear variance for any trait, and differed in total genetic variance only for days to first flower (results not shown). We therefore pooled these two populations for further comparisons of genetic variance structure (see Methods). The pooled heterostylous populations and the mixed population Montana de Oro had similar levels of nuclear variance for most traits (Vn was significantly different only for corolla lobe length; Supporting Appendix S2; Fig. 3). These groups, however, differed in the total level of genetic variation, Vgen for seven traits: cotyledon length, leaf number, days to first flower, total flower number, corolla lobe length, stamen insertion length, and functional anther–stigma distance (Fig. 3). The difference in genetic variance in the first five of these traits can be attributed to significantly higher maternal effects in Montana de Oro (Supporting Appendix S2).

Without regard to statistical significance, the highly selfing population Zmudowski State Beach had the lowest nuclear variance estimate for 11 traits, was second lowest for three traits (sepal length, corolla lobe length, and anther surface area), and was tied for the lowest estimate with Nipomo for corolla lobe width (Supporting Appendix S1). Using log-likelihood tests, we detected significantly less total genetic variance and nuclear variance in the highly selfing population compared to the pooled heterostylous populations and also to Montana de Oro for several traits (Fig. 3; Supporting Appendix S2). Zmudowski had significantly less total genetic variance for leaf width, leaf number, stamen insertion length, pistil length, and anther–stigma distance. For five of these traits (leaf length, leaf width, leaf number, stamen insertion length, and anther–stigma distance) this difference could be attributed to significantly lower nuclear variance. Significantly less nuclear variance was also detected for leaf length. There was only one trait, days to first flower, for which a difference in maternal variance was detected between the heterostylous populations and the highly selfing one (Vmat lower in Zmudowski; Supporting Appendix S2; Fig. 3). In comparison to Montana de Oro, Zmudowski harbored less total genetic variance for nine traits: cotyledon length, leaf width, leaf number, days to first flower, total flower number, corolla lobe length, stamen insertion length, pistil length and anther–stigma distance. For cotyledon length, days to first flower, total flower number, corolla lobe length, and anther–stigma distance, this difference is at least partially due to higher Vmat in Montana de Oro (Supporting Appendix S2). Differences in leaf width, leaf number, total flower number, stamen insertion length, and anther–stigma distance can either be entirely or partially attributed to lower Vn in Zmudowski (Supporting Appendix S2; Fig. 3). There was no comparison for which the highly selfing population had significantly more total genetic variance or nuclear variance.


We found evidence that quantitative genetic variance is reduced with very high rates of self-fertilization. The majority of traits studied in the three populations that outcross to some extent harbored the potential to respond to selection. In contrast, the highly selfing population had only four traits for which heritable variation was found to be significantly greater than zero. This population had the lowest amount of nuclear variance in all traits for which significantly different levels of nuclear variance were detected among populations. These results suggest that the highly selfing population of A. spectabilis may be an evolutionary dead-end as defined by Stebbins (1957). From the finding of high maternal variance in Montana de Oro, this study also highlights the potential problem of incorrectly inferring a correlation between the amount of total genetic variation and heritable genetic variation in a population.


Among all traits and populations, only four estimates of Vt and Vk were found to be significantly greater than zero. We therefore reduced the “bio” model c to include only Vn, Vmat, and Vres to compare variances among populations. Reducing the number of parameters estimated for the models used in the log-likelihood tests eliminates the problem of over-parameterizing and should increase the power to detect differences in the main components of interest. We also pooled the two heterostylous populations, Nipomo and Santa Maria, to decrease the number of pairwise comparisons and to increase the power for detecting differences in variance components among populations. These populations were similar with regards to both the amount and type of variance detected. These populations did not differ in nuclear variance for any trait, and it was only for days to first flower that differed significantly in total genetic variance, which was most likely the result of greater maternal effects in Santa Maria.

The effect of mating system on the level of nuclear variance appears to occur only with extreme selfing. Despite an expected difference in the selfing rate between Montana de Oro and the heterostylous populations, Nipomo and Santa Maria, these two groups generally harbored similar levels of nuclear variation (Supporting Appendices S1 and S2; Fig. 3). These groups, however, differed in total genetic variation, Vgen for seven traits: cotyledon length, leaf number, days to first flower, total flower number, corolla lobe length, stamen insertion length, and functional anther–stigma separation distance. The difference in genetic variance in the first five of these traits can be attributed to significantly higher maternal effects in Montana de Oro (Supporting Appendix S2).

In contrast, the highly selfing population, Zmudowski State Beach, had the fewest traits for which significant contributions of nuclear genetic variance were detected (Supporting Appendix S1) and generally tended to have the lowest estimates of genetic variance (Fig. 3; Supporting Appendix S1). All but one of the significant differences in Vn detected among populations involved comparisons to the highly selfing population; in all instances Vn was lower in the highly selfing population (Supporting Appendix S2; Fig. 3). There were also differences in total genetic variation between this population and the more outcrossing ones. These tended to be the result of either higher maternal effects in the more outcrossing populations, or lower Vn in the highly selfing population (Supporting Appendix S2; Fig. 3).

Comparing the nuclear variance between the highly selfing and more outcrossing populations underestimates the difference in the amount of heritable variation available to respond to selection, that is additive variance. Nuclear variance is not equal to additive variance. Estimates of nuclear variance can be converted to additive variance if the inbreeding coefficient is known (Table 2). For a randomly mating population Va= 4 Vn and for a completely selfing population Va= 2 Vn (Falconer and Mackay 1996; Fry 2004). Because we do not know the inbreeding coefficient for the partially selfing populations we expect additive variance to be intermediate to these values. This difference in conversion factor causes difficulty in interpreting the results from log-likelihood ratio tests comparing variance components between populations. Because the inbreeding coefficient is unknown in these populations, we cannot assess the exact amount by which these populations differ in additive variance. We can be certain, however, that the difference in additive variation between the selfing and more outcrossing populations is underestimated by as much as twofold. It should also be noted that the response to selection is not exactly proportional to additive variance under partial selfing (Kelly 1999).


It is intriguing that we observed large differences in maternal-effects variance among populations for several characters. For example, Vmat for days to first flower was 2% in Nipomo, 22% in Santa Maria and Montana de Oro, and 0% in Zmudowski State Beach (Supporting Appendix S1; Fig 3). Variation in phenotypes of offspring due to effects of the maternal parent can be attributable to several factors including maternal environmental effects; cytoplasmic effects that are often maternally inherited; endosperm dosage effects; and an additive, heritable component resulting from variation caused by nuclear genes specific to maternal function (see Mousseau and Fox 1998 for a review of this topic). The underlying cause of maternal variance cannot be evaluated here as multigenerational crossing designs are required to evaluate the heritable component of maternal variance. Irrespective of the source, the high level of maternal variance in several traits in this population highlights the problem of relying on total genetic variance to explore differences in heritable variation among populations. If only broad-sense heritability was considered, we would erroneously conclude that Montana de Oro harbors much greater ability to respond to selection as compared to the heterostylous populations.


Evidence that mating system affects evolutionary potential in natural populations has been slow to accumulate. Loss of molecular variation associated with the evolution of selfing has been documented in studies of allozyme polymorphism (Hamrick and Godt 1997), in nucleotide variability in species of the genera Leavenworthia (Liu et al. 1998, 1999), Lycopersicon (Baudry et al. 2001) Arabidopsis (Bergelson et al. 1998; Savolainen et al. 2000; see review in Wright et al. 2002 and also in Charlesworth 2003), and in work on the intron-length variation in Amsinckia (Pérusse and Schoen 2004). The relationship between molecular variation and evolutionary potential, however, may be weak (Latta 1998; McKay and Latta 2002; Latta and McKay 2002). The conclusions regarding the effect of mating system from previous studies that directly compare levels of standing quantitative genetics for populations or sister species with different rates of self-fertilization are inconsistent. A study of two Mimulus species, one largely outbreeding and the other highly inbreeding, by Carr and Fenster (1994) is one of the few studies that directly compared levels of quantitative genetic variation in selfing populations to more outcrossing populations. They found that on average the genetic coefficient of variation in the more inbreeding species was about one-third of that in the more outbreeding species. In a later study investigating the genetics of sex-allocation among species that differ in mating system, Fenster and Carr (1997) also observed lower heritabilities in the selfing M. micranthus versus the partially selfing M. guttatus for floral traits. Another study employing Cockerham and Weir's (1977)“bio” model, Lyons (1996) found no difference in additive variance between populations of Leavenworthia crassa with selfing rates of 77% and 97%. In a survey of the level of quantitative genetic variation in plants, Charlesworth and Charlesworth (1995) found that selfing is associated with a highly significant reduction in the genetic coefficient of variation (see Figs. 3 and 4 in Charlesworth and Charlesworth 1995). This effect, however, was based on 12 studies using paternal plant components of variance or regressions of progeny values on paternal plant values, and as such the reliability of this conclusion remains open to question. More recently, in a review of heritability estimates for functional traits in plants, Geber and Griffen (2003) found heritability estimates to be larger in species with outcrossing or partially selfing systems (h2= 0.29 for both mating systems) compared with inbreeding species (h2= 0.15). For floral traits Ashman and Majetic (2006) found limited evidence that self-compatible species had lower heritabilities than self-incompatible species but this effect did not remain statistically significant following Bonferroni adjustment.

We observed marked differences in the amount of genetic variation maintained between the selfing population and the three populations that outcross. Although we cannot directly convert the estimates of nuclear variance into heritability estimates in this study (discussed above) we can estimate a range for heritability. Averaged across all traits, heritability is expected to lie between 0.15 and 0.35 (0.15 if taken to be completely inbred to 0.3 if assumed random mating) in the heterostylous populations Nipomo and Santa Maria, between 0.2 and 0.45 in the mixed population Montana de Oro, and approximately 0.06 for the highly selfing population Zmudowski State Beach. Our estimates are comparable to those previously reported. Our finding that the selfing population tends to maintain less heritable variation than the partially selfing populations contributes to the growing body of evidence that the evolution of selfing is associated with a loss of adaptive potential.

The inability of selfing populations to revert to outcrossing is a corollary of the dead-end hypothesis. If reversion were possible, then the theories predicting extinction of selfers, particularly due to loss of genetic variation or mutational meltdown (reviewed in Takebayashi and Morrell 2001) would no longer apply. Within the genus Amsinckia there is evidence that outcrossing does not evolve from selfing. Specifically, a reconstruction of the molecular phylogeny within the group found short phylogenetic branch lengths separating selfers and their nearest outcrossing relatives, suggesting that there are no ancient self-fertilizing taxa in Amsinckia (Schoen et al. 1997; M. O. Johnston and Hahn unpublished data). Our results suggest that the selfing population has little or no ability to respond to selection on traits associated with mating system, such as corolla lobe length, pistil length, stamen length, and functional stigma–anther separation. Although the lack of standing genetic variation for traits affecting mating system may hinder reversion to outcrossing, the greatest inhibition is likely to be the low opportunity to fertilize the ovules of other selfing individuals.


We found large differences among traits in the genetic components of variation and in residual variance, Vres. There was no clear pattern for how variance components differed among the traits we investigated (Supporting Appendix S1). Floral and vegetative traits did not have consistently different proportions of nuclear variance (related to heritability). Traits that had lower proportions of nuclear variance did not consistently have either low nuclear variance or high residual variance. Other studies investigating heritability have also found differences among traits (see review in Merila and Sheldon 2000); although those studies generally examined differences among suites of traits (e.g., morphological vs. life-history traits). Generally, traits closely related to fitness and hence under strong selection have been found to have low heritability estimates in animals (Gustafsson 1986; Mousseau and Roff 1987; Roff and Mousseau 1987; see also review in Merila and Sheldon 2000) and also in plants (Stratton 1992; Campbell 1997; Geber and Griffen 2003; Ashman and Majetic 2006). Initially these results were framed under the prediction that traits closely related to fitness should have lower levels of additive variance because alleles conferring highest fitness will be driven quickly to fixation. More recently greater attention has been given to the alternative explanation that low heritability of fitness-related traits is caused by high Vres (Houle 1992; Merila and Sheldon 2000). Fitness traits are expected to have higher Vres because of the compounding of mutational input, nonadditive gene action, or environmental variance across loci (Price and Schluter 1991; Houle 1992). The few studies that have examined the source of low heritability in fitness traits have found it to be due to high environmental variance (in animals Houle 1992; Merila and Sheldon 2000; for plants see Stratton 1992; Campbell 1997). Knowledge of which traits are more closely related to fitness would allow us to investigate how different selection regimes shape the components underlying phenotypic variation within populations. To resolve the issue of whether low Va or high Vres is the cause of difference in heritability among traits, more studies that report the components of variation along with measure of selection are required.


Our study is one of the few to demonstrate that the ability to respond to selection is greatly reduced by selfing. For most traits the selfing population had less nuclear genetic variation. Interestingly, we detected no genetic variation in this population for 11 of 15 traits including those related to mating system. Furthermore, the similar levels of nuclear genetic variance among the three partially selfing populations suggest that some degree of outcrossing—perhaps small—maintains additive genetic variance within populations.

Associate Editor: J. Kohn


We are grateful to J. D. Fry for help in modifying SAS code. We thank J. K. Kelly, J. R. Kohn, and R. G. Shaw for comments on an earlier version of this work, and S. Good-Avila, J. K. Kelly, R. G. Latta, and D. Roff for helpful discussion. For assistance with plant propagation and data collection we thank A. Bland, J. Corbin, T. Darwish, C. Kozela, E. Lapalme, P. Li, K. Olson, P. Singh, and L. Weir. This work was supported by NSERC grants to MOJ