Using latent variable modeling and multiple imputation to calibrate rater bias in diagnosis assessment
Article first published online: 5 NOV 2010
Copyright © 2010 John Wiley & Sons, Ltd.
Statistics in Medicine
Volume 30, Issue 2, pages 160–174, 30 January 2011
How to Cite
Siddique, J., Crespi, C. M., Gibbons, R. D. and Green, B. L. (2011), Using latent variable modeling and multiple imputation to calibrate rater bias in diagnosis assessment. Statist. Med., 30: 160–174. doi: 10.1002/sim.4109
- Issue published online: 3 JAN 2011
- Article first published online: 5 NOV 2010
- Manuscript Accepted: 13 SEP 2010
- Manuscript Received: 15 JAN 2010
- Markov chain Monte Carlo;
- ordinal probit;
- leniency error
We present an approach that uses latent variable modeling and multiple imputation to correct rater bias when one group of raters tends to be more lenient in assigning a diagnosis than another. Our method assumes that there exists an unobserved moderate category of patient who is assigned a positive diagnosis by one type of rater and a negative diagnosis by the other type. We present a Bayesian random effects censored ordinal probit model that allows us to calibrate the diagnoses across rater types by identifying and multiply imputing ‘case’ or ‘non-case’ status for patients in the moderate category. A Markov chain Monte Carlo algorithm is presented to estimate the posterior distribution of the model parameters and generate multiple imputations. Our method enables the calibrated diagnosis variable to be used in subsequent analyses while also preserving uncertainty in true diagnosis. We apply our model to diagnoses of posttraumatic stress disorder (PTSD) from a depression study where nurse practitioners were twice as likely as clinical psychologists to diagnose PTSD despite the fact that participants were randomly assigned to either a nurse or a psychologist. Our model appears to balance PTSD rates across raters, provides a good fit to the data, and preserves between-rater variability. After calibrating the diagnoses of PTSD across rater types, we perform an analysis looking at the effects of comorbid PTSD on changes in depression scores over time. Results are compared with an analysis that uses the original diagnoses and show that calibrating the PTSD diagnoses can yield different inferences. Copyright © 2010 John Wiley & Sons, Ltd.