Management effectiveness evaluations are an increasingly common approach to measuring conservation outcomes within protected areas. While these evaluations have the potential to provide valuable data to guide management, the accuracy of evaluation data is not reported. We investigated how evaluation data are collected and used, the criticisms made of evaluation methods, and the processes employed to address criticisms or ensure accuracy. We found that most evaluation tools use qualitative indicators of management success and rely heavily on the knowledge of protected area managers. Criticisms of the evaluation methods have led to improvements to the data collection process, but the precision and accuracy of these data are almost never measured. We believe that measuring the accuracy of evaluation data will provide important opportunities to improve the rigor of future evaluations and build confidence in the use of these data for adapting protected area management.