In this article, I examine systems of care aimed at improving citizens and ruined colonial buildings in Salvador, Bahia, Brazil's Pelourinho World Heritage Site. Such UNESCO-sanctioned trusteeship, or the identification of buildings, bodies, and practices in need of a tutelage that would recuperate them as signs of a common humanity, maintains and exacerbates segmentations of knowledge essential to imperial control. I thus work to reconsider the Pelourinho, and cultural heritage, as imperial formations in light of UNESCO's system of producing world heritage through the specification of “exceptional universal value” in which the exceptional object obfuscates not simply as an emergency, but through its monumentalization as an ostensibly shared property. This attempt to gain a clearer understanding of empires' real effects is catalyzed by a number of residents of the Pelourinho who, in their subjection to decades of state-directed surveillance intended to make them into living human ancestors, have come to reject sentimental attachments to buildings or the purveyors of philanthropy. Yet the ways they do so are revelatory of new approaches to exceptions: Residents reject victimhood as a state of being injured and instead weave accounts of the structures that engender, and continue to reproduce, such violence. I follow in the path of this quite iconoclastic version of “historical reconstruction” as a way to draw out an ambivalently postcolonial Brazil whose own claims to exceptionality may be understood, like those put together by the woman I call Topa, as forced by entwined historical processes, rather than isolated emergencies or remainders beyond the political. My aim is to show how empires can be linked across space and time, without relying on empirical mapping of their constitutive parts and ideological props—recognizing, instead, the specificity of empire's effects.