Estimating Diagnostic Accuracy of Raters Without a Gold Standard by Exploiting a Group of Experts
Article first published online: 24 SEP 2012
© 2012, The International Biometric Society
Volume 68, Issue 4, pages 1294–1302, December 2012
How to Cite
Zhang, B., Chen, Z. and Albert, P. S. (2012), Estimating Diagnostic Accuracy of Raters Without a Gold Standard by Exploiting a Group of Experts. Biometrics, 68: 1294–1302. doi: 10.1111/j.1541-0420.2012.01789.x
- Issue published online: 21 DEC 2012
- Article first published online: 24 SEP 2012
- Received September 2011. Revised March 2012. Accepted May 2012.
- Diagnostic error;
- Imperfect tests;
- Model selection;
Summary In diagnostic medicine, estimating the diagnostic accuracy of a group of raters or medical tests relative to the gold standard is often the primary goal. When a gold standard is absent, latent class models where the unknown gold standard test is treated as a latent variable are often used. However, these models have been criticized in the literature from both a conceptual and a robustness perspective. As an alternative, we propose an approach where we exploit an imperfect reference standard with unknown diagnostic accuracy and conduct sensitivity analysis by varying this accuracy over scientifically reasonable ranges. In this article, a latent class model with crossed random effects is proposed for estimating the diagnostic accuracy of regional obstetrics and gynaecological (OB/GYN) physicians in diagnosing endometriosis. To avoid the pitfalls of models without a gold standard, we exploit the diagnostic results of a group of OB/GYN physicians with an international reputation for the diagnosis of endometriosis. We construct an ordinal reference standard based on the discordance among these international experts and propose a mechanism for conducting sensitivity analysis relative to the unknown diagnostic accuracy among them. A Monte Carlo EM algorithm is proposed for parameter estimation and a BIC-type model selection procedure is presented. Through simulations and data analysis we show that this new approach provides a useful alternative to traditional latent class modeling approaches used in this setting.