There have been considerable advances in our understanding of the tolerance of species interaction networks to sequential extinctions of plants and animals. However, communities of species exist in a mosaic of habitats, and the vulnerability of habitats to anthropogenic change varies. Here, we model the cascading effects of habitat loss, driven by plant extinctions, on the robustness of multiple animal groups. Our network is constructed from empirical observations of 11 animal groups in 12 habitats on farmland. We simulated sequential habitat removal scenarios: randomly; according to prior information; and with a genetic algorithm to identify best- and worst-case permutations of habitat loss. We identified two semi-natural habitats (waste ground and hedgerows together comprising < 5% of the total area of the farm) as disproportionately important to the integrity of the overall network. Our approach provides a new tool for network ecologists and for directing the management and restoration of multiple-habitat sites.