Although non-native herbivores are well known to increase the prevalence of exotic species in plant communities, herbivore removal is only sometimes sufficient for native plant recovery. Better predicting the influence of grazer removal, and identifying effective restoration approaches, requires a mechanistic understanding of the processes limiting native plant recovery in post-grazing landscapes. We evaluated the factors limiting the recovery of native shrubs in exotic annual grasslands after a century of sheep grazing on Santa Cruz Island, CA. To evaluate the life stage transitions most limiting the recovery of the two dominant native shrubs (Eriogonum arborescens and Artemisia californica), we quantified seed rain and censused establishment and growth rates over 5 years that varied widely in rainfall. In addition, we conducted competitor removal experiments to evaluate the influence of exotic annual grasses on shrub establishment. We found that seed rain is unlikely to limit native recovery; shrub recovery was most limited at the seedling establishment stage. For Eriogonum, the wet El Niño year increased establishment by almost an order of magnitude over average rainfall years, while grass competition reduced growth and survival by two thirds. Artemisia failed to recruit in any year of the study. Our results suggest that management efforts seeking to increase shrub recovery need to overcome barriers at the establishment life stage, but once such limitations are overcome, adult shrubs will persist with exotic grass competitors. Our work shows how understanding the factors limiting native recovery can inform the restoration of post-grazing landscapes.