What Is the Best Way to Estimate Hospital Quality Outcomes? A Simulation Approach
Article first published online: 21 FEB 2012
© Health Research and Educational Trust
Health Services Research
Volume 47, Issue 4, pages 1699–1718, August 2012
How to Cite
Ryan, A., Burgess, J., Strawderman, R. and Dimick, J. (2012), What Is the Best Way to Estimate Hospital Quality Outcomes? A Simulation Approach. Health Services Research, 47: 1699–1718. doi: 10.1111/j.1475-6773.2012.01382.x
- Issue published online: 5 JUL 2012
- Article first published online: 21 FEB 2012
- Agency for Healthcare Research and Quality
- Clinical and Translational Science Center
- Weill Cornell Medical College
- Biostatistical methods;
- incentives in health care;
- patient outcomes/functional status/ADLs/IADLs;
- quality of care/patient safety (measurement)
To test the accuracy of alternative estimators of hospital mortality quality using a Monte Carlo simulation experiment.
Data are simulated to create an admission-level analytic dataset. The simulated data are validated by comparing distributional parameters (e.g., mean and standard deviation of 30-day mortality rate, hospital sample size) with the same parameters observed in Medicare data for acute myocardial infarction (AMI) inpatient admissions.
We perform a Monte Carlo simulation experiment in which true quality is known to test the accuracy of the Observed-over-Expected estimator, the Risk Standardized Mortality Rate (RSMR), the Dimick and Staiger (DS) estimator, the Hierarchical Poisson estimator, and the Moving Average estimator using hospital 30-day mortality for AMI as the outcome. Estimator accuracy is evaluated for all hospitals and for small, medium, and large hospitals.
Data Extraction Methods
Data are simulated.
Significant and substantial variation is observed in the accuracy of the tested outcome estimators. The DS estimator is the most accurate for all hospitals and for small hospitals using both accuracy criteria (root mean squared error and proportion of hospitals correctly classified into quintiles).
The mortality estimator currently in use by Medicare for public quality reporting, the RSMR, has been shown to be less accurate than the DS estimator, although the magnitude of the difference is not large. Pending testing and validation of our findings using current hospital data, CMS should reconsider the decision to publicly report mortality rates using the RSMR.