Effects of population size and isolation on heterosis, mean fitness, and inbreeding depression in a perennial plant


  • Christopher G. Oakley,

    1. Department of Biological Science, Florida State University, Tallahassee, FL 32306-4295, USA
    2. Present address: Department of Plant Biology, Michigan State University, East Lansing, MI 48824-1312, USA
    Search for more papers by this author
  • Alice A. Winn

    1. Department of Biological Science, Florida State University, Tallahassee, FL 32306-4295, USA
    Search for more papers by this author

Author for correspondence:
Christopher G. Oakley
Tel: +1 517 432 5285
Email: coakley@msu.edu


  • In small isolated populations, genetic drift is expected to increase chance fixation of partly recessive, mildly deleterious mutations, reducing mean fitness and inbreeding depression within populations and increasing heterosis in outcrosses between populations.
  • We estimated relative effective sizes and migration among populations and compared mean fitness, heterosis, and inbreeding depression for eight large and eight small populations of a perennial plant on the basis of fitness of progeny produced by hand pollinations within and between populations.
  • Migration was limited, and, consistent with expectations for drift, mean fitness was 68% lower in small populations; heterosis was significantly greater for small (mean = 70%, SE = 14) than for large populations (mean = 7%, SE = 27); and inbreeding depression was lower, although not significantly so, in small (mean = −0.29%, SE = 28) than in large (mean = 0.28%, SE = 23) populations.
  • Genetic drift promotes fixation of deleterious mutations in small populations, which could threaten their persistence. Limited migration will exacerbate drift, but data on migration and effective population sizes in natural populations are scarce. Theory incorporating realistic variation in population size and patterns of migration could better predict genetic threats to small population persistence.


Many populations are subdivided into locally breeding subpopulations connected by migration (Wright, 1931; Hanski & Gaggiotti, 2004). Population subdivision can increase the role of genetic drift in evolution in a structured population because subpopulations are smaller than the population they comprise (Whitlock et al., 2000; Whitlock, 2002; Glémin et al., 2003; Roze & Rousset, 2004). Drift in small populations is predicted to affect the frequency and distribution of recessive or nearly recessive deleterious mutations. These mutations are, in turn, expected to affect the dynamics of adaptive evolution and to act synergistically with demography to increase the extinction risk of small populations or metapopulations (Lynch et al., 1995; Higgins & Lynch, 2001). The expected effects of genetic drift on the distribution of recessive or nearly recessive deleterious mutations can be evaluated empirically from analysis of multiple populations that differ in effective size as a result of natural variation in census size and degree of isolation.

Small effective population size can affect the frequency and distribution of partly recessive deleterious mutations, because genetic drift in small populations increases both homozygosity within populations and chance differentiation among them. Although dominant mutations are effectively removed by selection because they are always expressed, recessive or nearly recessive mutations can be protected from selection because they are at most weakly expressed in heterozygotes. In small subpopulations, rare, strongly deleterious mutations may be lost because of chance fluctuations in allele frequencies (Glémin, 2003). Mildly deleterious alleles may become fixed because of the lower efficacy of selection in removing them (Kimura et al., 1963; Bataillon & Kirkpatrick, 2000; Whitlock et al., 2000).

The effects of drift on the frequency and distribution of deleterious, largely recessive mutations can be evaluated by estimating heterosis between populations, and mean fitness and inbreeding depression within populations. We define heterosis as the increase in fitness of crosses between populations relative to crosses within populations (Crow, 1948; Whitlock et al., 2000), and inbreeding depression as the decrease in fitness of selfed progeny relative to those produced by outcrossing within a population (Charlesworth & Charlesworth, 1987; Charlesworth & Willis, 2009). Heterosis and mean fitness depend on the degree of fixation of partly recessive deleterious mutations, whereas inbreeding depression depends on deleterious, partly recessive mutations that are segregating within populations. Because genetic drift increases homozygosity, we expect lower mean fitness in small relative to large populations, as well as greater heterosis in crosses between populations. We also expect lower inbreeding depression within small relative to large populations because the loss of rare, strongly deleterious alleles by chance, the fixation of mildly deleterious alleles, and the greater coefficient of relatedness among individuals in small populations (Byers & Waller, 1999; Keller & Waller, 2002) are all expected to reduce genetic differences between selfed and outcrossed progeny. Although the reduced genetic difference between selfed and outcrossed progeny can lower estimates of within-population inbreeding depression, the increased frequency of fixed deleterious mutations underlying this effect can pose a risk to long-term population persistence (Lynch et al., 1995; Higgins & Lynch, 2001).

The effects of drift depend on both subpopulation size and the pattern and magnitude of migration. Consequently, accurate predictions for effects of drift on mean fitness, inbreeding depression, and heterosis in natural systems require descriptions of both effective sizes and the amount and direction of migration among subpopulations. Inbreeding depression (reviewed by Husband & Schemske, 1996; Armbruster & Reed, 2005, Winn et al., 2011) and heterosis in crosses among plant populations (Fenster, 1991; van Treuren et al., 1993; Ouborg & Van Treuren, 1994; Willi & Fischer, 2005; Busch, 2006) have been reported for numerous plant species, but relating these measures to drift requires estimates of size and rates of migration for multiple conspecific populations, which remain rare. Some of the expected effects of drift on the distribution of deleterious, partly recessive mutations are supported by comparisons of widely separated populations of different size (Paland & Schmid, 2003; Willi & Fischer, 2005) or of populations separated by different distances (Fenster 1991; Fenster & Galloway, 2000; Bailey & McCauley, 2006), but mean fitness, heterosis, and inbreeding depression have not been investigated in systems for which both population size and migration have been estimated.

Recently developed coalescent methods of genetic data analysis permit simultaneous estimation of relative effective population sizes and rigorous statistical comparisons of different models of migration among populations (Beerli & Felsenstein, 1999, 2001; Beerli & Palczewski, 2010). We used a coalescent analysis of neutral genetic markers to describe relative effective sizes and patterns of migration among 15 subpopulations of a short-lived perennial plant. We combine these estimates with results from comparisons of the fitness of progeny from hand pollinations within and between eight small and eight large subpopulations to determine if smaller populations have lower mean fitness and express greater heterosis and less inbreeding depression than larger populations of this species. The results have implications for the management of small populations in patchy habitats.

Materials and Methods

Study system

Hypericum cumulicola (Small) P. Adams (Hypericaceae) is a federally endangered, short-lived perennial plant endemic to the rosemary scrub of the southern Lake Wales ridge in Florida. Rosemary scrub occurs in discrete patches of white sand on paleodunes that are at least several million yr old (Christman & Judd, 1990). Within a patch of rosemary scrub, H. cumulicola is a specialist of open sandy gaps that support few to many hundreds of individuals. The gaps are maintained by periodic fire (Quintana-Ascencio & Morales-Hernandez, 1997; Quintana-Ascencio & Menges, 2000; Quintana-Ascencio et al., 2003), which may kill adults of this species, but gaps can be quickly recolonized from a persistent seed bank (P. F. Quintana-Ascencio, unpublished). The entire species has been described as a metapopulation on the basis of its distribution in discrete patches and the presence of suitable but unoccupied habitat (Quintana-Ascencio & Menges, 1996). H. cumulicola is not clonal, and its flowers are visited primarily by small solitary bees (Evans et al., 2003). Limited seed dispersal and pollinator movements suggest that dispersal among patches of scrub is rare (Dolan et al., 2008). A species-wide allozyme survey (Dolan et al., 1999) revealed high homozygosity (FIS = 0.76) and low genetic diversity within populations and strong differentiation among populations (FST = 0.76); estimates for finer spatial scales are not available. Flowers of H. cumulicola are self-compatible, but pollinator exclusion indicates limited (7%) autonomous selfing (Evans et al., 2003), suggesting that the high estimate of FIS is the result of pollinator-mediated selfing and/or biparental inbreeding rather than autogamous selfing.

In June 2007, we selected 16 subpopulations of H. cumulicola separated by between 0.25 and 12.0 km in southern Highlands County, Florida, to serve as focal subpopulations for hand pollinations (Fig. 1). We counted all individuals in each subpopulation in 2007 and categorized subpopulations containing 11–25 individuals as small, and those with 124 to > 1000 individuals as large. We counted all individuals again in 2011, and to estimate maximum possible subpopulation size, we measured the area of suitable habitat available to each subpopulation from geographic information system maps of vegetation type. The means for each estimate of population size were compared for the two size categories by randomization tests based on 10 000 randomizations, written in R (R Development Core Team, 2011). We used one-tailed tests because of our a priori expectation that the large populations would have greater values for each measure of population size.

Figure 1.

Spatial arrangement of the 16 focal Hypericum cumulicola populations as distance (in m) from centers of columns. Population size is indicated by column height, and column labels are population identifiers. Arrows indicate the pattern of migration between populations, and weights of the arrows are proportional on a log10 scale.

Estimating subpopulation differentiation, effective size, and migration

We collected tissue from the tip of a leafy vegetative stem from each of up to 30 plants in each of our 16 focal subpopulations (a total of 311 individuals). We did not carry out genetic analysis for one subpopulation (subpopulation 13) because it was located between two others (subpopulations 15 and 8; Fig. 1) that showed low differentiation in preliminary analyses (Supporting Information, Table S1). Samples were stored in silica gel until DNA was extracted by means of a CTAB procedure, modified from Doyle & Doyle (1987). Samples were screened for 10 unlinked microsatellite loci described by Edwards et al. (2007). Two small populations were scored for only seven loci because they were completely monomorphic for all seven. Because three of these loci were the most variable, this is unlikely to bias our results. For PCR conditions and genotyping protocols, see Methods S1. Tests of genotypic disequilibria were performed and population-level FIS values were calculated using GENEPOP v4.1 (Raymond & Rousset, 1995). Population-level FIS values were subsequently adjusted to account for loci that were monomorphic within a particular population (i.e. for globally variable markers, monomorphism within a population was interpreted as FIS of one for that locus). We chose not to attempt indirect estimates of the selfing rates (e.g. RMES, robust multilocus estimate of selfing, David et al., 2007) because it is not possible to distinguish selfing from biparental inbreeding owing to small population size, which is suggested by the low rate of autogamy in H. cumulicola (Evans et al., 2003). We calculated the number of alleles per locus, Nei’s gene diversity within subpopulations, hierarchical F statistics, and pairwise FST with SPAGeDi (Hardy & Vekemans, 2002). SPAGeDi was also used to test isolation by distance using a permutation test of the regression of pairwise FST and the log of geographic distance between populations.

To evaluate the extent and pattern of migration among subpopulations and to estimate relative inbreeding effective subpopulation sizes accounting for migration, we used the coalescent-based program Migrate-n v. 3.2.6 (Beerli & Felsenstein, 1999, 2001). This program calculates the posterior probability of the parameters Θ, the mutation-scaled effective size (4Neμ for a diploid organism, where μ is the mutation rate and Ne is the effective population size) and M, the mutation-scaled migration rate (m/μ, where m is the migration rate). Although absolute Ne and m cannot be estimated without knowledge of the mutation rates of the markers, relative estimates can be obtained on the assumption that subpopulations do not differ in mutation rate.

Migrate-n offers several advantages over estimates of migration based on FST. The latter assume an island model (but see, Rousset, 1997; Hardy & Vekemans, 1999) of migration and cannot accommodate asymmetric migration (e.g. between sources and sinks). Both a nonlinear relationship between FST and the number of migrants per generation and sampling error in estimates of allele frequencies can make estimates based on FST imprecise (Whitlock & McCauley, 1999). Each method require assumptions, but Migrate-n allows realistic patterns of migration, explicitly models the mutation process, provides a robust approximation to the stepwise mutation model appropriate for the microsatellite markers that we used (Beerli, 2007), and is more powerful and therefore less prone to sampling error in small population samples (Beerli & Felsenstein, 1999). Coalescent methods that explicitly account for migration also provide more accurate estimates of relative effective population sizes.

Because genetic variation was limited in our data (Table 1), we evaluated migration in a pairwise fashion. We used a Bayesian framework to compare marginal likelihoods (Beerli, 2006; Beerli & Palczewski, 2010) for several models of migration for all 66 pairwise combinations of the 12 subpopulations that were polymorphic for at least one locus. For each pair, we compared models of migration representing panmixia, bidirectional migration (two different Θ and M), both possible source–sink models (different Θ with unidirectional M), and isolation. A more detailed description of these methods, including parameter values for prior distributions, is provided in Methods S2 and Oakley (2011).

Table 1.   Census sizes, habitat patch areas, measures of genetic diversity, FIS, and mutation-scaled effective population sizes for 16 subpopulations of Hypericum cumulicola
SubpopulationCensus size (2007)Census size (2011)Harmonic mean census sizeHabitat area (m2)Mean number of alleles per locus F IS Nei’s gene diversity4Neμ (95% credible interval)
  1. Subpopulations 4 and 10 could not be censused again in 2011 because of a recent fire. No genetic data were collected from subpopulation 13.

  2. μ, mutation rate; Ne, effective population size.

 71123.420121100.001 (0–0.252)
 3153020.016551.20.90.0140.261 (0.024–0.508)
 417n/an/a11 5511.310.1460.393 (0.124–0.692)
 11869.055011.70.630.2040.549 (0.264–0.868)
 61846.576861100.001 (0–0.284)
 8181214.445422.10.590.2360.121 (0–0.336)
 52235.366661.20.880.0650.001 (0–0.232)
 2252223.410 5441100.001 (0–0.280)
 161246383.613 1111.90.590.2370.313 (0.014–0.834)
 11159122138.175 7441.80.710.1830.637 (0.332–0.988)
 10174n/an/a42781.30.970.1020.287 (0.044–0.546)
 14192534282.413 8431.10.90.0050.065 (0–0.326)
 121964775.833121.60.770.2060.471 (0.194–0.784)
 92855693.627 4241.40.930.1510.379 (0.116–0.666)
 13497536515.825 771n/an/an/an/a
 151001301462.820 6912.70.330.2371.277 (0.884–1.750)

Hand pollinations and progeny fitness

Hand pollinations within and between the 16 focal subpopulations were conducted in the field in June–August of 2007. Six plants in each subpopulation were selected at random to serve as both maternal and paternal parents for outcrosses within and between subpopulations. Each plant received and donated four types of pollen: self pollen from a different flower on the same plant; outcross pollen from a different individual within the subpopulation; and outcross pollen from each of two different subpopulations, one large and one small. All outcrosses were reciprocal. We emasculated all flowers before pollination to prevent unintentional self-fertilization and caged whole plants to prevent pollinator access.

Plants seldom produced more than three open flowers per day, so pollinations were conducted over multiple days; the order of pollination treatments was random to the extent allowed by flower availability. Pollinations were repeated when flowering stems were broken or when fruits were damaged by seed predators. For half of the subpopulations, outcross pollinations within and between populations were repeated three times for all maternal plants to generate seeds for another study. The data for these crosses are included in the analyses reported here.

Fruit capsules typically matured in 5–6 wk, and a small drop of Elmer’s white glue was applied to each after 4 wk to prevent dehiscence. Fruits were collected between late July and early October. We recorded whether each pollination produced a fruit and counted the seeds produced by each fruit. In total, we conducted 727 hand pollinations, which generated over 3000 seeds.

In December 2007, all seeds from successful fruits were placed in a germinator under 11 h days, with 22 : 10°C, day : night temperatures. Seeds from each maternal family × pollination type combination were placed in a Petri dish with moist fine sand. Upon emergence of the cotyledons, seedlings were transplanted to flats filled with a 50 : 50 mixture of field-collected soil and fine sand. Up to 10 seedlings from each maternal family × pollination type combination were transplanted, for a total of 1577. Seedlings were reared in the glasshouse, watered with distilled water, and fertilized with 10% strength 20-20-20 fertilizer monthly. After 3 months, seedlings were transplanted to 20-cm-deep, 7.5 × 7.5 cm pots filled with pure field soil collected from a scrub site at Archbold Biological Station with soil suitable for H. cumulicola but not currently supporting a population. We recorded whether plants survived and reproduced for 2 yr. For plants that reproduced, fecundity was estimated as the total number of flowers produced over 2 yr, corresponding to the average longevity of plants in their natural habitat (Quintana-Ascencio et al., 2007). For each subpopulation, mean fitness for each pollination type was estimated as the product of population means (calculated from maternal family means) for proportion fruit set, seed number per fruit, proportion germinating, combined proportion surviving and reproducing, and fecundity.

Analysis of mean fitness, heterosis, and inbreeding depression

Our data exhibited characteristics common to estimates of fitness; mortality at early stages caused missing data for some families at later stages and the distribution of cumulative fitness is a mixture of discrete and skewed continuous values that is resistant to transformation to normality. Aster models (Geyer et al., 2007; Shaw et al., 2008), which can accommodate mixture distributions, were not suitable for our data because they cannot be implemented for complex mixed models, or to incorporate the effects of fruit and seed set on cumulative fitness. We proceed with an ANOVA analysis based on raw population means (derived from family means), noting that although it is conservative, it gave results similar to those from analysis based on family means, and to an analysis of rank-transformed population means (not shown). The effects of pollination type, maternal-subpopulation size class, their interaction, and maternal population nested within size class were analyzed in a mixed-model ANOVA (Proc MIXED, SAS Institute, 2000) on mean fitness for each pollination type for each subpopulation. The effects of paternal-subpopulation size class and the interaction between maternal- and paternal-subpopulation size classes were also included in the ANOVA. We could include the interaction between pollination type and either maternal- or paternal-subpopulation size class, but not both. The choice is irrelevant for selfs and outcrosses within subpopulations because the maternal and paternal subpopulations are the same. We chose to include the interaction with maternal-subpopulation size class because fruits and seeds from between-population crosses develop in the maternal environment. Maternal subpopulation nested within maternal-subpopulation size class was treated as a random effect, and all other terms were treated as fixed.

Our primary interests were in the effects of pollination type and the interaction of maternal-population size class and pollination type, the former indicating the potential for overall inbreeding depression and heterosis, and the latter indicating that small and large maternal subpopulations differ in one or both of these quantities. We further examined the relationship between subpopulation size and pollination type by comparing least-square means of pollination types for the large- and small-subpopulation size classes.

Population mean fitness, heterosis, and inbreeding depression were calculated as follows, where W is an estimate of fitness and the subscripts s, ow, and ob refer to self, outcross within, and outcross between subpopulations, respectively. Population mean fitness was estimated as Wow because this species is not autogamously selfing (Evans et al., 2003). Heterosis was calculated as (Wob − Wow)/Wob, and inbreeding depression for each subpopulation was calculated as (Wow − Ws)/Wow. Heterosis depends on the fitness of outcrosses within subpopulations relative to that of outcrosses between subpopulations, and inbreeding depression depends on the fitness of selfs relative to that of outcrosses within subpopulations. When Ws > Wow or Wow > Wob, we report measures of relative performance, which is calculated with the greater of the two values as the denominator (Ågren & Schemske, 1993). Relative performance is bounded by −1 and 1, a range that gives equal weight to negative and positive values and prevents bias in averages as a result of the extreme values that can occur when fitness is zero or near zero for one cross type (Paland & Schmid, 2003; Picóet al., 2004; Bailey & McCauley, 2006; Oakley & Winn, 2008).

We tested explicitly for differences in heterosis and inbreeding depression for small and large subpopulations on the basis of 10 000 randomizations. Tests were one-tailed because of our a priori expectations of lower mean fitness and inbreeding depression and greater heterosis in small subpopulations than in large ones. The comparison of heterosis for subpopulations of different size class is complicated, because it could be influenced by both maternal and paternal subpopulation sizes, but because we found no significant effects of paternal-subpopulation size class on fitness, we based the comparison on maternal-subpopulation size class only.


Field measures of population size

The mean 2007 census sizes of subpopulations designated as small were significantly smaller than the means for those designated as large (Table 1; randomization test, Pone-tailed = 0.009). Subpopulations classified as large on the basis of census size in 2007 also had significantly larger census size in 2011 (randomization test, Pone-tailed = 0.007), greater harmonic mean census size over 2 yr (randomization test, Pone-tailed = 0.003), and occupied larger habitat patches (randomization test, Pone-tailed = 0.031).

Genetic variation and differentiation

Genetic variation within subpopulations was low; the average number of alleles per locus was 1.5 (range 2–7) and the average Nei’s gene diversity was 0.12 (Table 1). Three of the 15 subpopulations (all small) for which genetic data were collected were monomorphic for all microsatellite loci. Tests of genotypic disequilibria (not shown) were hampered by high homozygosity and low diversity, but only four cases of all possible pairs of loci within populations showed significant disequilibria and subsequent results are qualitatively robust to randomly removing one locus per pair, so we proceeded using the full dataset. Hierarchical F statistics indicate high homozygosity and differentiation (FIS = 0.60, FST = 0.68, FIT = 0.84). Population-level estimates of FIS were, with few exceptions, very high (Table 1), and negatively correlated with census population size (= −0.63, P = 0.01), although this result is driven mostly by the much lower FIS in the largest population. Differentiation was also pervasive among all subpopulations; 95% of pairwise FST estimates exceeded 0.2, and most exceeded 0.5 (Table S1). There was no pattern of isolation by distance (P = 0.75), but this is perhaps not surprising given the generally high values of FST even at distances of a few hundred meters.

Migration and population relative effective size

Migration is uncommon in this system, even over spatial scales of just a few hundred meters. In 94% of our pairwise comparisons, a model of isolation was more likely than any of the other migration models tested (Fig. 1; Notes S1; Oakley, 2011). We will therefore hereafter refer to subpopulations as populations. Our estimates of the mutation-scaled effective size (Θ) spanned three orders of magnitude (Table 1). Although credible intervals for individual estimates were wide, large and small population categories based on census size in 2007 differed significantly in their mode Θ (randomization test, Pone-tailed = 0.026), and Nei’s gene diversity tended to be larger for larger populations (randomization test, Pone-tailed = 0.053).

Our data may not meet some assumptions of the Migrate-n program, such as stable population sizes and homogenous mutation rates among markers, but the qualitative agreement of our coalescent-based estimates of effective population size with estimates of population sizes from other empirical measures, and the agreement of our conclusions about migration with pairwise FST values, suggest that violations of assumptions were not severe.

Mean fitness, heterosis, and inbreeding depression

Cross type by population size class combinations differed by more than fourfold in mean fitness (Fig. 2). We found no significant effects of maternal- or paternal-population size class or their interaction on fitness (Table 2). The main effects of maternal population nested within population size class and of pollination type were also not significant, although P-values were < 0.10 for both. The interaction between pollination type and maternal-population size class was significant (Table 2), indicating possible differences between small and large populations in the magnitude of mean fitness, heterosis, and/or inbreeding depression (Tables S2, S3).

Figure 2.

Means of population means for fitness for each pollination type for small and large population size classes of Hypericum cumulicola. Error bars indicate SE.

Table 2.   Mixed-model ANOVA for fitness fit with the restricted maximum likelihood (REML) method, in which random effects for balanced designs are tested with a Z-statistic
ANOVA effects Z F df P
  1. Denominator degrees of freedom for fixed effects are approximations based on the containment method (SAS Institute, 2000).

Fixed effects
 Maternal-population size class 1.101,140.31
 Paternal-population size class 2.621,420.11
 Pollination type 2.972,420.06
 Maternal size class × paternal size class 0.011,420.93
 Maternal size class × pollination type 3.672,420.03
Random effect
 Maternal population (size class)1.39  0.08

We found a significantly greater mean fitness of outcrosses within populations for large than for small populations (Table 3). On average, fitness of outcrosses within populations was 68% lower in small populations (Fig. 2). Comparison of least-square means also supported significantly greater fitness for outcrosses between populations than for outcrosses within populations for small populations but not for large populations, indicating significant heterosis for small populations only (Table 3). The magnitude of heterosis in small populations was 70% (SE = 14) but was only 7% (SE = 27) in large populations, and this difference was significant (randomization test, Pone-tailed = 0.024). We found no relationship between heterosis and either geographic or genetic distance (results not shown), but this is perhaps not surprising given the consistently strong isolation we report.

Table 3.   Results of a priori comparisons of least-square means for fitness of progeny from different cross types for small and large populations of Hypericum cumulicola
Comparisons of least-square means t P
  1. P-values are for one-tailed tests.

Large, outcross within vs small, outcross within2.200.016
Small, outcross within vs small, outcross between−3.400.001
Large, outcross within vs large, outcross between0.410.34
Small, outcross within vs small, self−1.120.13
Large, outcross within vs large, self1.120.13

We found no significant differences in fitness of selfs and within-population outcrosses for either population size category (Table 3), and therefore no significant inbreeding depression. Mean inbreeding depression was −29% (SE = 28) for small populations and 28% (SE = 23) for large populations; this difference was not statistically significant (randomization test, Pone-tailed = 0.059), but the large magnitude of the difference and the small P-value could indicate a biologically relevant difference.


Genetic drift is expected to play a role in the evolutionary dynamics and extinction risk of small and/or spatially structured populations. We found that populations of H. cumulicola are nearly completely isolated from each other and that smaller populations show a stronger signature of genetic drift than larger populations. We found significantly lower mean individual fitness and significantly greater heterosis for small populations, as well as a trend toward lower inbreeding depression, consistent with a role for drift in fixing partly recessive deleterious mutations in small populations. The combination of strong isolation among populations, chance fixation of deleterious mutations, and previously reported low rates of population growth (Quintana-Ascencio et al., 2003) could synergistically increase the risk of extinction in this endangered plant.

Migration and effective size

Our coalescent analysis suggested extreme spatial structuring of our focal populations of H. cumulicola. The absence of migration, even among populations separated by only a few hundred meters (Fig. 1; Notes S1), suggests little potential for migration to provide either demographic or genetic rescue to small populations. This species has a persistent seed bank, and dormant seeds could carry a signal of effective size and/or migration that is not reflected in our sample of above-ground individuals, but it is unlikely that the seed bank would be sufficiently more genetically variable than the above-ground population to alter our conclusions about migration and relative effective population sizes (Dolan et al., 2008).

Agreement among several sources of evidence suggests that current census sizes reflect consistent long-term differences in the relative sizes of populations. Coalescent-based estimates of relative population effective sizes confirmed differences between small and large population assignments based on census size, despite limited precision in the coalescent-based estimates (Table 1). Populations designated small on the basis of census size also had smaller harmonic mean census size and lower average Nei’s gene diversity and occupied smaller habitat patches (Table 1). Differences in habitat patch area support consistent differences in long-term relative sizes because populations are quickly re-established from the seed bank after fire, and census size shortly after fire is positively correlated with habitat area (P. F. Quintana-Ascencio, unpublished).

Mean fitness, heterosis, and inbreeding depression

We found evidence, consistent with expectations, for a greater effect of genetic drift on the frequency of partly recessive unconditionally deleterious mutations in small than in large populations. On average, mean individual fitness was much lower in small populations, heterosis was much greater for small than for large populations, and inbreeding depression was, on average, also lower in smaller populations. All of these effects were measured in a relatively benign glasshouse environment. Because the expression of deleterious mutations can be greater in harsher field environments (reviewed by Armbruster & Reed, 2005), the effects we measured may underestimate the magnitude of the consequences of partly recessive deleterious mutations in small populations.

The 68% lower mean individual fitness of within-population outcrosses in small vs large populations that we found is striking and consistent with the fixation of deleterious alleles by drift. Lower individual mean fitness in small populations has been reported for other species (Paland & Schmid, 2003; Willi & Fischer, 2005; Willi et al., 2005), and although methodological differences complicate direct comparisons, the magnitude of the effect we report appears to be greater than in previous studies (c. 10% in Paland & Schmid, 2003 and c. 20% in Willi & Fischer, 2005), possibly because of smaller population sizes (Paland & Schmid, 2003) and lower genetic diversity (Willi & Fischer, 2005) in our system.

The strong heterosis we observed and its relationship to population size are also striking, particularly given that the distances between populations that we crossed ranged only from 1 to 9 km. Strong heterosis has been observed in crosses between widely isolated natural populations of other taxa (Ebert et al., 2002; Busch, 2006), but comparisons of the magnitude of heterosis for populations of different sizes are rare, and results are inconsistent. Paland & Schmid (2003) reported 16.5% greater fitness of between- than of within-population crosses for six small (n < 200) populations of the self-compatible biennial plant Gentianella germanica and no evidence for heterosis for six populations larger than 200 individuals. For 13 populations of the self-incompatible perennial Ranunculus reptans, Willi & Fischer (2005) found up to 35% greater fitness in crosses between than in crosses within populations, and that heterosis was greater in populations inferred to be smaller on the basis of lower genetic diversity. Because neither of these results for R. reptans was supported when data from self-incompatible crosses were eliminated from the analyses, they may have been driven by self-incompatibility in within-population crosses. Studies of other species revealed no relationship between population size and either heterosis or inbreeding depression (van Treuren et al., 1993; Hauser & Loeschcke, 1994; Ouborg & Van Treuren, 1994), but these may lack the replication at the population level needed to detect effects of population size.

We found no significant inbreeding depression within populations for either large or small populations of H. cumulicola, although, on average, inbreeding depression was lower in small populations. Low inbreeding depression might be expected for this species on the basis of previously reported high homozygosity, possibly because of genetic drift, which would promote both purging of rare strongly deleterious mutations (Glémin, 2003; Pujol et al., 2009) and the fixation of more mildly deleterious alleles (Whitlock et al., 2000). Although purging caused by inbreeding and subsequent selection against homozygous progeny is also expected to reduce inbreeding depression (Lande & Schemske, 1985; Husband & Schemske, 1996; Byers & Waller, 1999; Crnokrak & Barrett, 2002), this process should only be effective in the largest of our populations. Negative inbreeding depression (i.e. outbreeding depression within populations) for small populations is puzzling, especially in light of strong heterosis in crosses between populations, because outbreeding depression is predicted to increase with geographic distance (Frankham et al., 2011). Negative estimates of inbreeding depression, calculated as relative performance, have been reported for several highly selfing taxa (reviewed in Winn et al., 2011), and Paland & Schmid (2003) also found negative inbreeding depression only in small populations. No biological explanation for negative inbreeding depression has been offered, and it remains enigmatic.

The lower inbreeding depression we report for smaller than for large populations was not statistically significant but may reflect a true difference, because the same partly recessive deleterious alleles thought to underlie the significant differences in heterosis and mean fitness we found are also expected to contribute to inbreeding depression. Because we conducted twice as many outcross pollinations as self-pollinations, we had greater statistical power to detect differences in heterosis and the fitness of progeny from within-population outcrossing than to detect inbreeding depression. The P-value for our comparison of inbreeding depression in small, and large populations (0.06) approached significance, so a conclusion of no biologically important difference may be premature. The one other study that has compared inbreeding depression for more than three large and small populations of the same species did find significantly greater mean inbreeding depression for large (13.1%) than for small (− 15.4%) populations (Paland & Schmid, 2003).

Consequences of genetic drift

The strong heterosis we found for small populations of H. cumulicola suggests that genetic drift can fix deleterious, largely recessive mutations or drive them to high frequency. The resulting genetic load could act synergistically with demographic and environmental stochasticity to threaten population extinction by mutational meltdown (Lynch et al., 1995; Higgins & Lynch, 2001). Although lower mean individual fitness does not necessarily reduce population absolute fitness (Wallace, 1975, 1991; Agrawal, 2010), many populations of H. cumulicola have finite rates of increase near or below one (Quintana-Ascencio et al., 2003), so suppression of vital rates by fixed deleterious mutations could realistically increase the risk of extinction. The absence of inbreeding depression in small populations could favor the evolution of even greater self-fertilization (but see, Pujol et al., 2009; Barringer et al., 2012). Increased selfing could further decrease effective population size and the potential for already rare gene flow between populations, further accelerating the fixation of mildly deleterious alleles and the risk of extinction.

Existing theory and empirical evidence support effects of population size and structure on evolutionary dynamics and population persistence (Lynch et al., 1995; Newman & Pilson, 1997; Saccheri et al., 1998; Whitlock et al., 2000; Higgins & Lynch, 2001; Whitlock, 2002; Paland & Schmid, 2003), but current theory makes unrealistic assumptions about homogeneity of population size and migration. We found variation in population size, and although migration was rare overall, it was variable in both magnitude and direction. Treating this set of populations as a single panmictic unit or even as a metapopulation might greatly overestimate the potential for long-term persistence. Although the degree of structure we report for H. cumulicola may be extreme, it highlights the existence of heterogeneity in population sizes and the magnitude and pattern of migration in nature. Theory will need to accommodate the realistic variation in population size and isolation found in nature to make useful predictions about the effects of population structure on evolution and extinction.

Conservation implications

Demographic and environmental stochasticity pose the most immediate threats to species of conservation concern (Lande, 1988; Schemske et al., 1994), but even species with protected habitat may be threatened by the consequences of fixed deleterious mutations. We echo the suggestion (see for example, Fox et al., 2008) that intentional purging of captive endangered species would be bad practice because it would promote fixation of some deleterious alleles. On the other hand, crosses between nearby populations might alleviate the negative consequences of fixed deleterious mutations (Frankham et al., 2011). Data obtained from other endangered species using experimental approaches such as ours could be used to parameterize models to predict the consequences of fixed deleterious mutations.


We are deeply grateful to E. S. Menges, P. F. Quintana-Ascencio, and C. W. Weekley for advice and logistical support and to members of the Soltis lab for hosting C.G.O. and for training in molecular techniques. We thank Archbold Biological Station and the Florida Fish and Wildlife Conservation Commission for permission to work on their properties and R. Pickert and N. Ranalli for providing vegetation shape files. We thank P. Beerli for help with Migrate-n and the High Performance Computing Center at FSU for computational resources. We also thank K.J. Jacobsen, J. Mola, J. Monge, M.G. Shuster, R. Walsh, and K. Wray for assistance with the work, and K. Womble for preparing the figure. We thank three anonymous reviewers for comments that improved the manuscript. M.C. Whitlock and members of D.W. Schemske’s laboratory provided useful comments on an earlier version of the manuscript. This project was funded by a Florida State University CRC grant to A.A.W. and an NSF Doctoral Dissertation Improvement Grant (DEB-0808435) to A.A.W. and C.G.O.