Habitat fragmentation and disturbance are known to impact animals and plants in different ways, depending on species’ characteristics and the type and scale of habitat modification involved. In contrast, direct or indirect ramifications on mutualistic relationships between plants and animals are less clear, possibly because general patterns are confounded by the diffuse nature of many of these interactions. Here, we examine how fragment size and/or severe disturbance of a Kenyan mountain cloud forest affects the frugivore community and seed removal of a large-seeded, bird-dispersed tree of the forest interior, for three consecutive years. Forest deterioration reduced avian visitation and seed removal rates independent of fragment size, consistently so despite strong temporal variation in fruit production over the three-year study. In disturbed forest fragments, seed removal rates were on average 3.5 times lower than in more intact ones. Strong differences in both visitation and seed removal rates were largely attributable to shifts in frugivore assemblages, characterized by loss or reduced abundance of the most effective seed dispersers, most of which were forest specialists. Although some disturbed fragments benefited from visits of non-forest dependent seed dispersers, such ‘resilience’ was not predictable or reliable in time or space. We conclude that disruption of seed disperser-seed interactions in highly fragmented and disturbed tropical forests may be persistent in time when resiliency is inadequate, possibly posing long-term effects on tree communities.