Prediction Models Assessing Transplant Center Performance: Can a Little Knowledge be a Dangerous Thing?
Article first published online: 19 JAN 2006
American Journal of Transplantation
Volume 6, Issue 2, pages 245–246, February 2006
How to Cite
Schold, J. D. and Howard, R. J. (2006), Prediction Models Assessing Transplant Center Performance: Can a Little Knowledge be a Dangerous Thing?. American Journal of Transplantation, 6: 245–246. doi: 10.1111/j.1600-6143.2005.01216.x
- Issue published online: 19 JAN 2006
- Article first published online: 19 JAN 2006
- Received 21 October 2005, revised 15 November 2005 and accepted for publication 16 November 2005
The demand for and availability of quality indicators for healthcare providers has grown dramatically over the past decade. Government and private oversight of healthcare quality exists through offices such as the Agency for Healthcare Research and Quality and the Joint commission on Accreditation of Healthcare Organizations (1,2). The Center for Medicare and Medicare Services (CMS) has plans to emphasize quality of care and payment for performance in the future. Studies have reported improved outcomes associated with quality monitoring in the context of cardiac bypass procedures (3,4). In this commentary, we explore the benefits and potential pitfalls of applying these indicators in the field of transplantation and review a new approach of internal quality control for transplant centers to monitor outcomes on a continuous basis.
The manuscript by Axelrod et al. presents a quality control model (CUSUM) for kidney and liver transplant centers to assess irregularities in performance markers based on outcomes adjusted for patient and donor characteristics (5). The novelty of the model includes real-time surveillance and the integration of temporal factors for detecting performance levels, such that frequent recent failures may be more indicative of quality variants than aggregated over a fixed period of time. The manuscript further compares their approach and results to the standard transplant center evaluations currently used by the Organ Procurement and Transplantation Network (OPTN) (6). While the authors clearly state that their model is to be exclusively utilized for internal quality control, it is noteworthy that their performance indicators were quite similar to the OPTN annual assessments. Based on these results and other diagnostics given in this paper, several questions remain regarding the overall utility of these quality indicators, the common or proper interpretation of the results and whether the indicators are appropriate in the field of transplantation.
The field of transplantation has the luxury of extensive data collection from which a vast array of important research has translated into significant improvements in patient care. However, despite the prolific amount of available data, our ability to predict individual patient outcomes remains limited (7). This lack of predictive ability is also indicated in the current manuscript in which the areas under the Receiver Operating Characteristic curves for the models adjusted for all available explanatory factors were approximately 65% (with 50% being a completely worthless diagnostic tool) which is considered poor in traditional academic point systems. Contributing to this relatively low predictive level is that despite the vast amount of information that is utilized in these assessments, there are many additional factors that these models do not or cannot account for. Factors such as the resources available to patients, degree of compliance, patient mental capacity and emotional state, environmental risk factors, patient support networks, changes in insurance status and many additional clinical diagnostics are a sample of variables that may strongly influence outcomes. In regard to center evaluations, these caveats are particularly critical as regions may be highly diverse with respect to these noncodified factors. In other words, beyond the significant random variation associated with assessing patient outcomes, a systematic bias may exist for centers as a function of their population characteristics and case-mix.
One of the dangers of objective measures in this context is that centers that are primarily focused on improving their assessment criteria may be more inclined to accept ‘better’ patients that they believe are more likely to follow protocols or generally have superior prognoses. In a similar fashion, centers may choose to use greater scrutiny accepting donor organs. While some of these factors may be adjusted for in models, as described previously, there are a number of exogenous factors that may influence outcomes and cannot be readily codified. Conversely, given the strong and robust survival advantage associated with transplantation, the potential competing emphasis of performing as many transplants as possible may be compromised. Other dangers may exist as centers may be strongly motivated to maximize early survival rates during which time they are evaluated while potentially sacrificing long-term outcomes. Additionally, the ramifications of ‘false positives’ (flagging poor performance inappropriately) may have a significant impact on dedicating resources for investigating ‘inferior’ outcomes that may often be innocuous circumstances. For the CUSUM or OPTN models, centers with small transplant volumes are unlikely to ever be flagged for performance (good or bad) given insufficient statistical power, while large volume centers may often be given a significant grade with debatable clinical relevance. Moreover, for smaller centers the influence of individual factors may be exaggerated with the voluminous adjustment factors for a relatively small number of patients.
Given these caveats, the question arises whether agencies that report center performance should explicitly state these limitations and explain the reliability of the evaluation tools in a manner the general population may appropriately interpret. Insurance companies, patients and transplant professionals may often assume that such assessments are completely reliable and interpret them accordingly. At the same time, patients and centers have great incentive and perhaps an inherent right to be empowered with quality assessments that may have life-altering consequences. While it is difficult to argue with the concept of evaluating performance of transplant centers, the evaluation tool should be reliable and accurate. Transplant center's survival may be predicted on their evaluations, especially if outcomes are rated as significantly lower. Government agencies may not renew center certification (relevant with the proposed conditions of participation currently being considered by CMS), insurance companies may not contract with such centers, and patients may select centers with better outcomes (even if those centers with higher outcomes may be below their true predicted outcomes).
In summary, while there is a clear demand for quality indicators in all areas of healthcare, the complex nature of transplantation renders quality assessments problematic. Given the significant ramifications of quality indicators, the responsibility of disseminating the limitations of evaluations in a transparent fashion is with the agency conducting the assessment. As a community, we must be cognizant that the goal of high quality care to all patients is not compromised by the assessment tools used to evaluate it.
- 1Agency for Healthcare Research and Quality. AHRQ Quality Indicators. http://www.qualityindicators.ahrq.gov/ (accessed 12 Oct 2005).
- 2Joint Commission on Accreditation of Healthcare Organizations. http://www.jcaho.org/ (accessed 12 Oct 2005).
- 6US Transplant—Scientific Registry of Transplant Recipients. 2004 OPTN/SRTR Annual Report. 2004. Ann Arbor , MI .