We thank the reviewers for their helpful comments.
INTERRATER CORRELATIONS DO NOT ESTIMATE THE RELIABILITY OF JOB PERFORMANCE RATINGS
Article first published online: 7 DEC 2006
Volume 53, Issue 4, pages 873–900, December 2000
How to Cite
MURPHY, K. R. and DESHON, R. (2000), INTERRATER CORRELATIONS DO NOT ESTIMATE THE RELIABILITY OF JOB PERFORMANCE RATINGS. Personnel Psychology, 53: 873–900. doi: 10.1111/j.1744-6570.2000.tb02421.x
- Issue published online: 7 DEC 2006
- Article first published online: 7 DEC 2006
Interrater correlations are widely interpreted as estimates of the reliability of supervisory performance ratings, and are frequently used to correct the correlations between ratings and other measures (e.g., test scores) for attenuation. These interrater correlations do provide some useful information, but they are not reliability coefficients. There is clear evidence of systematic rater effects in performance appraisal, and variance associated with raters is not a source of random measurement error. We use generalizability theory to show why rater variance is not properly interpreted as measurement error, and show how such systematic rater effects can influence both reliability estimates and validity coefficients. We show conditions under which interrater correlations can either overestimate or underestimate reliability coefficients, and discuss reasons other than random measurement error for low interrater correlations.