• Mixed mating;
  • pollen discounting;
  • prior selfing;
  • selfing rate;
  • viability

Mutations that alter the morphology of floral displays (e.g., flower size) or plant development can change multiple functions simultaneously, such as pollen export and selfing rate. Given the effect of these various traits on fitness, pleiotropy may alter the evolution of both mating systems and floral displays, two characters with high diversity among angiosperms. The influence of viability selection on mating system evolution has not been studied theoretically. We model plant mating system evolution when a single locus simultaneously affects the selfing rate, pollen export, and viability. We assume frequency-independent mating, so our model characterizes prior selfing. Pleiotropy between increased viability and selfing rate reduces opportunities for the evolution of pure outcrossing, can favor complete selfing despite high inbreeding depression, and notably, can cause the evolution of mixed mating despite very high inbreeding depression. These results highlight the importance of pleiotropy for mating system evolution and suggest that selection by nonpollinating agents may help explain mixed mating, particularly in species with very high inbreeding depression.