The aim of this paper is to describe a problem for calibrationism: a view about higher order evidence according to which one's credences should be calibrated to one's expected degree of reliability. Calibrationism is attractive, in part, because it explains our intuitive judgments, and provides a strong motivation for certain theories about higher order evidence and peer disagreement. However, I will argue that calibrationism faces a dilemma: There are two versions of the view one might adopt. The first version, I argue, has the implausible consequence that, in a wide range of cases, calibrationism is the only constraint on rational belief. The second version, in addition to having some puzzling consequences, is unmotivated. At the end of the paper I sketch a possible solution.