While it is generally assumed that specialist species are more vulnerable to disturbance compared with generalist counterparts, this has rarely been tested in coastal marine ecosystems, which are increasingly subject to a wide range of natural and anthropogenic disturbances. Habitat specialists are expected to be more vulnerable to habitat loss because habitat availability exerts a greater limitation on population size, but it is also possible that specialist species may escape effects of disturbance if they use habitats that are generally resilient to disturbance. This study quantified specificity in use of different coral species by six coral-dwelling damselfishes (Chromis viridis, C. atripectoralis, Dascyllus aruanus, D. reticulatus, Pomacentrus moluccensis, and P. amboinensis) and related habitat specialization to proportional declines in their abundance following habitat degradation caused by outbreaks of the coral eating starfish, Acanthaster planci. The coral species preferred by most coral-dwelling damselfishes (e.g., Pocillopora damicornis) were frequently consumed by coral eating crown-of-thorns starfish, such that highly specialized damselfishes were disproportionately affected by coral depletion, despite using a narrower range of different coral species. Vulnerability of damselfishes to this disturbance was strongly correlated with both their reliance on corals and their degree of habitat specialization. Ongoing disturbances to coral reef ecosystems are expected, therefore, to lead to fundamental shifts in the community structure of fish communities where generalists are favored over highly specialist species.