Get access

A crossed random effects modeling approach for estimating diagnostic accuracy from ordinal ratings without a gold standard

Authors

  • Yunlong Xie,

    1. Biostatistics and Bioinformatics Branch, Division of Epidemiology Statistics and Prevention Research, Eunice Kennedy Shriver National Institute of Child Health and Human Development, National Institutes of Health, Bethesda, MD 20892, U.S.A.
    Search for more papers by this author
  • Zhen Chen,

    Corresponding author
    • Biostatistics and Bioinformatics Branch, Division of Epidemiology Statistics and Prevention Research, Eunice Kennedy Shriver National Institute of Child Health and Human Development, National Institutes of Health, Bethesda, MD 20892, U.S.A.
    Search for more papers by this author
  • Paul S. Albert

    1. Biostatistics and Bioinformatics Branch, Division of Epidemiology Statistics and Prevention Research, Eunice Kennedy Shriver National Institute of Child Health and Human Development, National Institutes of Health, Bethesda, MD 20892, U.S.A.
    Search for more papers by this author

Correspondence to: Zhen Chen, Biostatistics and Bioinformatics Branch, Division of Epidemiology Statistics and Prevention Research, Eunice Kennedy Shriver National Institute of Child Health and Human Development, National Institutes of Health, Bethesda, MD 20892, U.S.A.

E-mail: chenzhe@mail.nih.gov

Abstract

In diagnostic studies without a gold standard, the assumption on the dependence structure of the multiple tests or raters plays an important role in model performance. In case of binary disease status, both conditional independence and crossed random effects structure have been proposed and their performance investigated. Less attention has been paid to the situation where the true disease status is ordinal. In this paper, we propose crossed subject-specific and rater-specific random effects to account for the dependence structure and assess the robustness of the proposed model to misspecification in the random effects distributions. We applied the models to data from the Physician Reliability Study, which focuses on assessing the diagnostic accuracy in a population of raters for the staging of endometriosis, a gynecological disorder in women. Using this new methodology, we estimate the probability of a correct classification and show that regional experts can more easily classify the intermediate stage than resident physicians. Copyright © 2013 John Wiley & Sons, Ltd.

Ancillary