Expansion of agriculture is a principal driver of biodiversity losses in the tropics, prompting suggestions that plantations should be made more hospitable to wildlife. Such “wildlife-friendly” practices contrast with the alternative “land sparing” strategy, which promotes separation of agricultural and conservation areas. Focusing on the wildlife-friendly strategy of retaining fragments of forest within the agricultural matrix, here we report the abundance and diversity of birds within oil palm plantations, fragments, and contiguous forest. Abundances of imperiled bird species were 60 times lower in fragments and 200 times lower in oil palm than in contiguous forest. Forest fragments also did not increase bird abundances in adjacent oil palm, had lower species richness than contiguous forest, and had an avifaunal composition that was more similar to oil palm than to contiguous forest. Therefore, from a conservation perspective, any investment in the retention of fragments would be better directed toward the protection of contiguous forest.