Tropical forests are seriously threatened by fragmentation and habitat loss. The impact of fragment size and forest configuration on the composition of seed rain is insufficiently studied. For the present study, seed rain composition of small and large forest fragments (8–388 ha) was assessed in order to identify variations in seed abundance, species richness, seed size and dispersal mode. Seed rain was documented during a 1-year period in three large and four small Atlantic Forest fragments that are isolated by a sugarcane matrix. Total seed rain included 20,518 seeds of 149 species of trees, shrubs, palms, lianas and herbs. Most species and seeds were animal-dispersed. A significant difference in the proportion of seeds and species within different categories of seed size was found between small and large fragments. Small fragments received significantly more very small-sized seeds (<0.3 cm) and less large-seeded species (>1.5 cm) that were generally very rare, with only one species in small and eight in large fragments. We found a negative correlation between the inflow of small-sized seeds and the percentage of forest cover. Species richness was lower in small than in large fragments, but the difference was not very pronounced. Given our results, we propose changing plant species pools through logging, tree mortality and a high inflow of pioneer species and lianas, especially in small forest fragments and areas with low forest cover. Connecting forest fragments through corridors and reforestation with local large-seeded tree species may facilitate the maintenance of species diversity.