In recent years, there has been a drive to strengthen existing public accountability arrangements and to design new ones. This prompts the question whether accountability arrangements actually work. In the existing literature, both accountability ‘deficits’ and ‘overloads’ are alleged to exist. However, owing to the lack of a cogent yardstick, the debate tends to be impressionistic and event-driven. In this article we develop an instrument for systematically assessing public accountability arrangements, drawing on three different normative perspectives. In the democratic perspective, accountability arrangements should effectively link government actions to the ‘democratic chain of delegation’. In the constitutional perspective, it is essential that accountability arrangements prevent or uncover abuses of public authority. In the learning perspective, accountability is a tool to make governments effective in delivering on their promises. We demonstrate the use of our multicriteria assessment tool in an analysis of a new accountability arrangement: the boards of oversight of agencies.