Fine scale spatial structure (FSSS) of cytoplasmic genes in plants is thought to be generated via founder events and can be amplified when seeds germinate close to their mother. In gynodioecious species these processes are expected to generate FSSS in sex ratio because maternally inherited cytoplasmic male sterility genes partially influence sex expression. Here we document a striking example of FSSS in both mitochondrial genetic markers and sex in roadside populations of Silene vulgaris. We show that in one population FSSS of sexes influences relative fruit production of females compared to hermaphrodites. Furthermore, FSSS in sex ratio is expected to persist into future generations because offspring sex ratios from females are female-biased whereas offspring sex ratios from hermaphrodites are hermaphrodite-biased. Earlier studies indicated that pollen limitation is the most likely mechanism underlying negative frequency dependent fitness of females. Our results support the theoretical predictions that FSSS in sex ratio can reduce female fitness by decreasing the frequency at which females experience hermaphrodites. We argue that the influence of FSSS on female fitness is complementary to the influence of larger scale population structure on female fitness, and that population structure at both scales will act to decrease female frequencies in gynodioecious species. Better comprehension of the spatial structure of genders and genes controlling sex expression at a local scale is required for future progress toward understanding sex ratio evolution in gynodioecious plants.