Deleterious Mutations can Reduce Differentiation in Small, Subdivided Populations

We study the effects of multilocus selection on genetic differentiation at linked neutral loci by using computer simulations. Two types of selection are examined, purifying selection against deleterious mutations and stabilizing selection at a quantitative, polygenic trait. Deleterious recessive mutations cause background selection that is predicted to reduce the heterozygosity at linked loci and to increase population differentiation (measured as F_ST or G_ST). Our simulations show that background selection in small subdivided populations can lead to an immigrant advantage that homogenizes the populations and reduces differentiation. Such a reduction is not observed in a case of high selfing rate (90% selfing), as reduced variation within populations cancels out the effects on absolute differentiation caused by background selection to give no net change in G_ST. In general, inbreeding leads to increased differentiation both in a completely neutral case and under background selection. Strong stabilizing selection at a quantitative trait with the same optimal phenotype in each population slightly increases the level of differentiation over that observed under a neutral model.