Get access

Probability interpretations of intraclass reliabilities

Authors

  • Jules L. Ellis

    Corresponding author
    1. Ellis Statistical Consultations, Grotestraat 63, 6511 VB Nijmegen, the Netherlands
    2. School of Psychology and Artificial Intelligence, Radboud University Nijmegen, 6500 HE Nijmegen, the Netherlands
    • Correspondence to: Jules L. Ellis, Ellis Statistical Consultations, Grotestraat 63, 6511 VB Nijmegen, the Netherlands.

      E-mail: julesellis@xs4all.nl

    Search for more papers by this author

Abstract

Research where many organizations are rated by different samples of individuals such as clients, patients, or employees frequently uses reliabilities computed from intraclass correlations. Consumers of statistical information, such as patients and policy makers, may not have sufficient background for deciding which levels of reliability are acceptable. It is shown that the reliability is related to various probabilities that may be easier to understand, for example, the proportion of organizations that will be classed significantly above (or below) the mean and the probability that an organization is classed correctly given that it is classed significantly above (or below) the mean. One can view these probabilities as the amount of information of the classification and the correctness of the classification. These probabilities have an inverse relationship: given a reliability, one can ‘buy’ correctness at the cost of informativeness and conversely. This article discusses how this can be used to make judgments about the required level of reliabilities. Copyright © 2013 John Wiley & Sons, Ltd.

Ancillary