The majority of plant species and many animals are hermaphrodites, with individuals expressing both female and male function. Although hermaphrodites can potentially reproduce by self-fertilization, they have a high prevalence of outcrossing. The genetic advantages of outcrossing are described by two hypotheses: avoidance of inbreeding depression because selfing leads to immediate expression of recessive deleterious mutations, and release from drift load because self-fertilization leads to long-term accumulation of deleterious mutations due to genetic drift and, eventually, to extinction. I tested both hypotheses by experimentally crossing Arabidopsis lyrata plants (self-pollinated, cross-pollinated within the population, or cross-pollinated between populations) and measuring offspring performance over 3 years. There were 18 source populations, each of which was either predominantly outcrossing, mixed mating, or predominantly selfing. Contrary to predictions, outcrossing populations had low inbreeding depression, which equaled that of selfing populations, challenging the central role of inbreeding depression in mating system shifts. However, plants from selfing populations showed the greatest increase in fitness when crossed with plants from other populations, reflecting higher drift load. The results support the hypothesis that extinction by mutational meltdown is why selfing hermaphroditic taxa are rare, despite their frequent appearance over evolutionary time.