Fleshy fruits, like drupes and berries, have evolved many times through angiosperm history. Two hypotheses suggest that fleshy fruit evolution is related to changes in the seed mass fitness landscape. The reduced dispersal capability following from an increase in seed mass may be counterbalanced by evolution of traits mediating seed dispersal by animals, such as fleshy fruits. Alternatively, increasing availability and capabilities of frugivores promote evolution of fleshy fruits and allow an increase in seed size. Both these hypotheses predict an association between evolution of fleshy fruits and increasing seed size. We investigated patterns of fruit and seed evolution by contrasting seed mass between fleshy and non-fleshy fruited sister clades. We found a consistent association between possession of fleshy fruits and heavier seeds. The direction of fruit type change did not alter this pattern; seed mass was higher in clades where fleshy fruits evolved and lower in clades where non-fleshy fruits evolved, as compared to their sister clades. These patterns are congruent with the predictions from the two hypotheses, but other evidence is needed to distinguish between them. We emphasize the need to integrate studies of seed disperser effectiveness, seed morphology, and plant recruitment success to better understand the frugivores’ role in fleshy fruit evolution.