Costs related to pollen competition have rarely been considered, but are expected in the case of sexual conflict where male and female sexual functions have opposing evolutionary interests. In Collinsia heterophylla, delayed stigma receptivity is beneficial as it enhances pollen competition. A sexual conflict over timing of stigma receptivity has been proposed in this species as early pollination, following one-time pollinations, is advantageous to pollen donors at a cost of reduced maternal seed set (measured as seed number). In this study, we explored whether the maternal cost was still present following an additional pollination. We hypothesized that the cost is caused either by harm related to early pollen presence or by factors unrelated to harm. We performed pollinations at different stages of floral development, either one or two pollinations (24-h time lag), and varied the size of the first pollen load in the latter category. Early pollination reduced seed biomass also after two-time pollinations, suggesting a persistent maternal cost of early pollen presence. Further, pollen load size modified seed production, possibly indicating that dose-dependent harm influences the maternal cost of early fertilization. Our results strongly suggest negative effects of pollen competition on maternal fitness following early pollination, which is consistent with the existence of a sexual conflict over timing of stigma receptivity. In conclusion, we propose that much could be gained if more plant studies considered the potential for fitness costs in relation to sexual conflict, particularly those investigating pollen–pistil interactions.