This research was supported by the Foundation of Educational Research (SVO) in the Netherlands (project 4075). The authors wish to thank Gideon J. Mellenbergh and Jannemieke van de Gein for their advice and comments and Wendy Yen for her editorial comments.
A Method of Estimating Rater Reliability
Version of Record online: 12 SEP 2005
Journal of Educational Measurement
Volume 26, Issue 1, pages 29–40, March 1989
How to Cite
van den Bergh, H. and Eiting, M. H. (1989), A Method of Estimating Rater Reliability. Journal of Educational Measurement, 26: 29–40. doi: 10.1111/j.1745-3984.1989.tb00316.x
- Issue online: 12 SEP 2005
- Version of Record online: 12 SEP 2005
A method for assessing rater reliability by means of a design of overlapping rater teams is presented. The products to be rated are split randomly into m disjoint subsamples, m equaling the number of raters. Each rater rates at least two subsamples according to a prefixed design. The covariances or correlations of the ratings can be analyzed with LISREL models, resulting in estimates of the rater reliabilities. Models in which the rater reliabilities are congeneric, tauequivalent, or parallel can be tested. We address problems concerning the identification and the degrees of freedom of the models and present two examples based on essay ratings.