In external beam radiation therapy, digitally reconstructed radiographs (DRRs) and portal images are used to verify patient setup based either on a visual comparison or, less frequently, with automated registration algorithms. A registration algorithm can be trapped in local optima due to irregularity of patient anatomy, image noise and artifacts, and/ or out-of-plane shifts, resulting in an incorrect solution. Thus, human observation, which is subjective, is still required to check the registration result. We propose to use a novel image registration quality evaluator (RQE) to automatically identify misregistrations as part of an algorithm-based decision-making process for verification of patient positioning. A RQE, based on an adaptive pattern classifier, is generated from a pair of reference and target images to determine the acceptability of a registration solution given an optimization process. Here we applied our RQE to patient positioning for cranial radiation therapy. We constructed two RQEs—one for the evaluation of intramodal registrations (i.e., portal-portal); the other for intermodal registrations (i.e., portal-DRR). Mutual information, because of its high discriminatory ability compared with other measures (i.e., correlation coefficient and partitioned intensity uniformity), was chosen as the test function for both RQEs. We adopted translation and 1° rotation as the maximal acceptable registration errors, reflecting desirable clinical setup tolerances for cranial radiation therapy. Receiver operating characteristic analysis was used to evaluate the performance of the RQE, including computations of sensitivity and specificity. The RQEs showed very good performance for both intramodal and intermodal registrations using simulated and phantom data. The sensitivity and the specificity were 0.973 and 0.936, respectively, for the intramodal RQE using phantom data. Whereas the sensitivity and the specificity were 0.961 and 0.758, respectively, for the intermodal RQE using phantom data. Phantom experiments also indicated our RQEs detected out-of-plane deviations exceeding and 2.5°. A preliminary retrospective clinical study of the RQE on cranial portal imaging also yielded good sensitivity and specificity . Clinical implementation of a RQE could potentially reduce the involvement of the human observer for routine patient positioning verification, while increasing setup accuracy and reducing setup verification time.