- Top of page
Report cards evaluating transplant center performance have received significant attention in recent years corresponding with the Centers for Medicare and Medicaid Services issue of the 2007 Conditions of Participation. Our primary aim was to evaluate the association of report card evaluations with transplant center volume. We utilized data from the Scientific Registry of Transplant Recipients (SRTR) along with six consecutive program-specific reports from January 2007 to July 2009 for adult kidney transplant centers. Among 203 centers, 46 (23%) were low performing (LP) with statistically significantly lower than expected 1-year graft or patient survival at least once during the study period. Among LP centers, there was a mean decline in transplant volume of 22.4 cases compared to a mean increase of 7.8 transplants among other centers (p = 0.001). Changes in volume between LP and other centers were significant for living, standard and expanded criteria deceased donor (ECD) transplants. LPs had a reduction in use of donors with extended cold ischemia time (p = 0.04) and private pay recipients (p = 0.03). Centers without low performance evaluations were more likely to increase the proportion of overall transplants that were ECDs relative to other centers (p = 0.04). Findings indicate a significant association between reduced kidney transplant volume and low performance report card evaluations.
- Top of page
Report cards for transplant centers have been generated on a biannual basis for over a decade by the Scientific Registry of Transplant Recipients (SRTR). These report cards are publicly available and include descriptive information of the waitlist population, donor and recipient characteristics as well as risk-adjusted outcomes of patients transplanted at each center in the United States. In March of 2007, the Center for Medicaid and Medicare Services (CMS) issued a Conditions of Participation (COP) for solid-organ transplant centers . Among many other standards for center compliance to receive public support (i.e. Medicare funding), the COP describes an explicit association between risk adjusted posttransplant outcomes and certification of centers by CMS. In particular, the document indicates that centers that fail to meet performance metrics will be subject to quality review, may be required to enter a Systems Improvement Agreement and potentially a loss of funding. The primary outcomes utilized to evaluate the performance of transplant centers are the risk-adjusted 1-year graft and patient survival using very similar metrics as currently reported by the SRTR .
Quality assurance programs and regulatory oversight have many laudable purposes in any healthcare context. This oversight may provide incentives for quality monitoring by individual programs as well as provide the capacity to evaluate practice patterns of high- and low-performing centers. Quality assurance may serve to protect patients from harm and provide transparent healthcare information to facilitate informed decision making. In fact, current SRTR metrics and CMS oversight based on the COP has led to numerous transplant center quality reviews that identified center deficiencies and led to improvement efforts [3, 4]. The HRSA Breakthrough Collaborative also provided a framework of best practices characteristic of centers with superior outcomes .
Despite important potential benefits of report cards, there may also be unintended consequences of public reporting and quality oversight. One of these potential consequences is that report cards unintentionally motivate providers to restrict access to care to patients considered to constitute a risk to their performance metrics . This type of patient selection has been illustrated in other healthcare contexts associated with implementation of report cards [7-9]. In a similar fashion, there may be legitimate concerns that transplant centers become more conservative with respect to selecting donor organs as a consequence of public reporting of performance . In this sense, report cards may unintentionally lead to greater rates of organ discard, longer cold ischemia times or disincentivize procurement of hard to place organs. In addition, centers may be reluctant to enter patients in research studies and potentially innovative interventions in which standard adjustment may not fully capture the risk level of a given transplant . Finally, despite the stated purpose of report cards to provide transparent information, empirical evidence indicates that they have minimal effect on transplant center selection and thus may not benefit patients to the degree that was intended .
The primary aim of this study was to evaluate the association of kidney transplant center volume and the incidence of low performance evaluations. We hypothesized that centers that receive low performance evaluations may alter protocols and are more likely to reduce transplant volume. We further aimed to evaluate whether there were changes in the composition of patient characteristics transplanted at centers that have received low performance evaluations.
- Top of page
There were two primary sources of data for this study. First, we evaluated data directly from archived SRTR performance reports from January, 2007 through July, 2009 (six consecutive biannual reports). We restricted our analysis to adult kidney centers to focus the study in a single context representing the most common form of solid-organ transplantation. For each adult kidney transplant center in the United States, we documented the observed and expected 1-year graft and patient survival as well as results of the risk adjusted models indicating whether centers were statistically significantly lower than their expected survival (p < 0.05 based on the reported two-sided test in the SRTR reports) from each biannual cohort. For the purpose of this study we utilized indications of ‘significantly lower than expected survival’ to classify centers with low performance. Of note, this is different than the standards that the SRTR currently utilizes which includes two additional criteria for flagging rules based on the difference and ratio of observed and expected events. We also utilized SRTR data to evaluate patient characteristics associated with centers with and without low performance evaluations and changes in the distribution of patient characteristics over time. As depicted in Figure 1, the study population included adult solitary kidney transplant recipients qualifying for the January, 2007 Program-Specific Report (PSR), transplants occurring from January, 2004 to June, 2006 (Baseline Cohort), and transplants qualifying for the January, 2010 report from January, 2007 to June, 2009 (End Cohort). Patients transplanted at centers that did not perform at least 10 transplants applicable for the January, 2007 report were excluded from the study. This exclusion was based on unique considerations for performance evaluations of centers with low volume and a reduced ability to assess changes with sufficient statistical power at smaller centers. Programs that did not perform transplants that qualified for the January, 2010 report were also excluded from the primary analysis based on the inability to evaluate differences in the composition of the cohorts at these centers over time.
We compared changes in transplant volume between 2007 and 2010 cohorts between centers that did and did not have a low performance evaluation within this period. We utilized unpaired t-tests to compare the overall change in volume as well as by donor type (defined as living, standard criteria and expanded criteria deceased donor kidneys). We also utilized a multivariable general linear model evaluating center-level factors associated with changes in volume which included baseline volume and the proportion of patients and transplant characteristics at each center. We compared the proportion of patient, donor and transplant characteristics between centers with and without low performance evaluations during the study period. We also evaluated changes in the proportion of these characteristics over the study period among centers with and without low performance evaluations. Finally, we generated hierarchical mixed models to assess whether patient-level characteristics were independently associated with transplantation at centers with low performance accounting for clustering within centers. All analyses were conducted in SAS v. 9.3 (Cary, NC, USA).
- Top of page
There were a total of 236 kidney transplant centers in the initial cohort; among these 211 were considered eligible for this study defined as performing at least 10 transplants applicable for the initial January, 2007 SRTR report. Of these centers, eight did not conduct transplants eligible for the January, 2010 report and were not included in the primary analyses. The mean proportion of recipient, donor and transplant characteristics at centers with and without low performance evaluations over the study period is displayed in Table 1. As indicated, the distribution of characteristics was relatively similar between centers with and without low performance evaluations; the only statistically significant difference was a higher proportion of retransplant recipients at centers with low performance evaluations. These comparisons were consistent in the multivariable mixed model, with the exception that after multivariable adjustment, retransplantation was not significantly associated with center groups and centers with a higher proportion of patients with extended pretransplant dialysis time (greater than 3 years) were associated with greater likelihood to be low performing (adjusted odds ratio = 1.31, 95% C.I. 1.02–1.67, p = 0.03).
Table 1. Center-level characteristics by performance evaluations
| ||Centers with low performance evaluation between 2007 and 2009|| |
|Transplant characteristics||No (n = 157)a||Yes (n = 46)a||p-Value|
|Recipient age > = 65 (%)||14%||13%||0.22|
|Donor age > = 60 (%)||7%||7%||0.54|
|Living donor recipient (%)||36%||33%||0.17|
|SCD transplant (%)||53%||56%||0.08|
|ECD transplant (%)||11%||10%||0.48|
|Caucasian recipient (%)||54%||55%||0.78|
|African American recipient (%)||24%||25%||0.73|
|Other race recipient (%)||22%||20%||0.57|
|Diabetes as primary diagnosis (%)||24%||23%||0.23|
|Retransplant recipient (%)||11%||13%||0.01|
|PRA > = 10% (%)||32%||31%||0.74|
|CIT > = 24 h (%)||21%||23%||0.46|
|Recipient pretransplant dialysis time > = 36 months (%)||37%||35%||0.36|
|Recipient BMI > = 30 (%)||30%||30%||0.96|
|Recipient college education or more (%)||22%||21%||0.24|
|Donor race African American (%)||12%||13%||0.51|
|Recipient private primary insurance (%)||37%||35%||0.47|
|Recipient Medicare as primary insurance||55%||60%||0.05|
|Recipient Medicaid as primary insurance||6%||5%||0.41|
|Female recipient (%)||39%||39%||0.88|
Forty-six (23%) centers had at least one occurrence of a lower than expected survival for 1-year graft or patient survival within the six biannual reports. The proportion was relatively consistent by the size of the center (Table 2). Ten percent of centers received a lower than expected survival evaluation for graft survival alone, 3% for patient survival alone and 9% of centers received low evaluations for both outcomes. The distribution of the number of overall indications of lower than expected survival by center over the study period is displayed in Figure 2. The figure indicates the number of periods that centers received a low performance evaluation for either graft or patient survival out of the six possible biannual periods. As displayed, 12% of centers had between one and two low performance periods and 11% of centers had three or more low performance evaluation periods.
Table 2. Distribution of lower than expected survival reports (2007–2009)
| ||Number of centers with lower than expected 1-year survival during the study period|| |
|Center volume for Jan, 2007 program-specific report (transplants performed January 2004–June 2006)||None||Graft survival only||Patient survival only||Both graft and patient survival||Either graft or patient survival||All centers|
| ||(74%)||(9%)||(6%)||(115)||(26%)|| |
| ||(71%)||(16%)||(0%)||(12%)||(29%)|| |
| ||(91%)||(9%)||(2%)||(0%)||(11%)|| |
| ||(74%)||(6%)||(6%)||(15%)||(26%)|| |
|All centers n (%)||157||20||7||19||46||203|
| ||(77%)||(10%)||(3%)||(9%)||(23%)|| |
Figure 2. Distribution of the number of lower than expected survival reports per kidney transplant center (2007–2009).
Download figure to PowerPoint
Overall, 106 (52%) of centers in the study reduced their kidney transplant volume between the June, 2007 and January, 2010 PSR. This proportion was relatively consistent based on the initial volume of the center: 45% of centers with less than 80 transplants, 55% of centers with 80–139 transplants, 55% of centers with 140–226 transplants and 54% of centers with at least 227 transplants had a decline in kidney transplant volume. Among the 46 centers that received a low performance evaluation over the study period, 72% had a decline in kidney transplant volume compared to 47% of centers without a low performance evaluation (p = 0.003). Among low performing centers the proportion of centers that had a decline in volume was relatively consistent by increasing volume. Specifically, 58%, 79%, 81% and 73% of low performing centers had a decline in volume respectively by first to fourth quartile of baseline center volume. There were no center-level factors associated with change in transplant volume in the multivariable general linear model other than an occurrence of a lower than expected survival evaluation (p = 0.002). In this model, incidence of a low performance evaluation was associated with 30.0 fewer transplants between periods after adjustment for the baseline center-level proportion of living and deceased donor transplants and demographic characteristics of both recipients and donors.
The average change in transplant volume over the study period was significantly different between centers with and without a low performance evaluation over the study period. As indicated in Figure 3, 39 and 26 centers received at least one lower than expected survival for 1-year graft and patient survival respectively during the study period. Of the 23% of centers that received a lower than expected graft or patient survival report during this period there was a mean decline of 22.4 transplants. This was significantly different than the mean increase of 7.8 transplants observed at all other centers (p = 0.001). The decline in volume was greater among centers with three or more poor reporting periods (mean = 24 fewer transplants) compared to centers with 1–3 poor reporting periods (mean = 17 fewer transplants). There was a statistically significant negative association between the number of low reporting periods and change in transplant volume (ρ = −0.30, p < 0.001, based on the Spearman correlation test). Differences in the change in transplant volume between low performing and all other centers were evident among each of living, standard criteria and expanded criteria donor transplants (Figure 4). Both low performers and other centers had a reduction of living donor transplantation, but the reduction was significantly more pronounced among low performing centers (p = 0.01). Centers without low performance evaluations had an increase in the number of both standard criteria and expanded criteria donor transplants, while centers with low performance evaluations had a reduction in both types of deceased donor transplants. Among the eight centers that were excluded from the study due to lack of transplants in the final evaluation cohort, one center had a significantly lower than expected survival period and all centers had a significant reduction in transplant volume until their last reported period (mean = 42 fewer transplants).
Figure 4. Mean change in transplant volume by donor type associated with center performance. *p-value reflects test for change in volume between centers with and without low performance evaluations.
Download figure to PowerPoint
There were several statistically significant differences in the change in the distribution of patient and transplant characteristics from the baseline to the end study cohort between centers that did and did not have low performance evaluations (Table 3). Centers without low performance evaluations were significantly more likely to transplant additional expanded criteria donor kidneys as a proportion of all transplants over time relative to other centers (p = 0.04). Low performing centers reduced use of donors with extended cold ischemia time (p = 0.04) and also reduced transplantation of patients with primary private payer insurance (p = 0.03) relative to other centers.
Table 3. Change in center-level characteristics with and without low performance evaluation
|Transplant characteristics||Centers without low performance evaluations between 2007 and 2009 (n = 157)a||Centers with low performance evaluations between 2007 and 2009 (n = 46)a||p-Valueb|
|Recipient age > = 65 (%)||+3%||+3%||0.99|
|Donor age > = 60 (%)||+2%||0%||0.07|
|Living donor recipient (%)||−3%||0%||0.06|
|SCD recipient (%)||+2%||0%||0.48|
|ECD recipient (%)||+2%||0%||0.04|
|Caucasian recipient (%)||−1%||−3%||0.15|
|African American recipient (%)||+1%||0%||0.42|
|Other race recipient (%)||+1%||+3%||0.01|
|Diabetes as primary diagnosis (%)||+1%||+2%||0.70|
|Retransplant recipient (%)||−1%||0%||0.44|
|PRA > = 10% (%)||+4%||+1%||0.19|
|CIT > = 24 h (%)||−2%||−6%||0.04|
|Recipient pretransplant dialysis time > = 36 months (%)||+2%||+4%||0.31|
|Recipient BMI > = 30 (%)||+3%||+2%||0.42|
|Recipient college education or more (%)||0%||+1%||0.30|
|Donor race African American (%)||−1%||+1%||0.08|
|Recipient private primary insurance (%)||0%||−4%||0.03|
|Recipient Medicare as primary insurance||+1%||+4%||0.13|
|Recipient Medicaid as primary insurance||0%||0%||0.59|
|Female recipient (%)||0%||−2%||0.32|
The baseline (i.e. June, 2007 cohort) observed 1-year overall graft survival was significantly lower among centers that subsequently received low performance evaluations relative to other centers (87.8% vs. 92.9% respectively, p < 0.001). However, the mean expected overall graft survival for the same cohort was not significantly different between these groups (91.9% vs. 92.3%, p = 0.10). Both low performing centers and other centers had an increase in observed overall graft survival (1.9% vs. 1.1%) but changes were not statistically significantly different between the groups. Similarly, both groups of centers had an increase in expected overall graft survival between the 2007 and 2009 cohorts but the changes were not statistically significantly different between low performers and other centers (1.6% vs. 1.2%, respectively, p = 0.08).
- Top of page
The primary findings of our study indicate that there is a significant association between outcomes of performance evaluations of US kidney transplant centers and kidney transplant volume. In particular, centers that receive low performance evaluations were more likely to exhibit a reduction in transplant volume over time compared to other centers. Declines in transplant volume were apparent for living, standard criteria and expanded criteria donor transplants among low performing centers and significantly different than other centers. The use of donations with extended cold ischemia time and recipients with private insurance were also relatively reduced among low performing centers compared to other centers. Finally, centers without low performance evaluations had an incremental expansion of the proportion of transplants that were expanded criteria donors which was not evident among low performing centers.
There are several potential explanations for the primary findings of this study. First, centers that have been identified with poor outcomes may be more likely to have staff turnover which may lead to declines in transplant volume. Centers that have been evaluated with lower performance may generally become more conservative in overall acceptance rates of candidates and donor organs . This may be due to a desire to consolidate activities and reduce staffing workload or reduce certain aspects of transplant services that are more difficult to manage. Centers under Systems Improvement Agreements may also place patients on hold or transfer to other centers’ waiting lists which could diminish transplant activity. Alternatively, centers that do not have low performance evaluations may be more likely to expand use of innovative procedures and transplant services .
A primary concern of the study findings is that performance oversight may unintentionally lead to stagnation of the growth of transplantation and ultimately limit access to care for patients waiting for a donor organ. Compared to many other healthcare contexts, the substantial ‘demand’ for care relative to the supply of organs in transplantation may facilitate this ability to be selective and there is wide variability in the composition and characteristics of transplant candidates between US transplant centers [14-16]. In fact, results of a survey given to transplant personnel indicated that respondents at centers that reported low performance evaluations indicated a greater likelihood to implement more restrictive selection criteria for patients and donor organs . Conversely, the substantial survival and quality of life improvement associated with kidney transplantation, even among higher risk organs, implies that any factor limiting transplant volume leads to poorer outcomes for the broader transplant population [17, 18]. This is an important consideration in recent years as national transplant volume has been stagnant since 2006 after a period of two decades in which there was consistent growth of kidney transplantation.
An additional concern of report cards is that assessment of the quality of providers is imprecise and may fail to accurately account for the acuity of patient populations or discriminate between providers with good and poor quality of care . This may lead to changes in practice, including alterations in candidate or donor selection criteria, which are not necessarily based on true differences in quality of care. One of the challenges understanding whether report cards accurately assess true low (or high) performers is the lack of a true gold standard. That is, there is relatively little evidence as to whether centers that do receive low performance evaluations truly have deficiencies in care standards or alternatively whether centers that not identified as outliers truly have adequate quality of care. This may impede our understanding about what level of measured quality of care is actionable and should necessitate regulatory involvement and which cases may be a function of random variation in performance. Lastly, contraction of services may make is more difficult to enroll sufficient numbers of patients into clinical trials and promote innovative scientific interventions.
Another question that stems from these findings is whether reduction in transplant volume is even rational from the center perspective related to performance evaluations. That is, regardless of the impact of limiting access to care to transplant candidates, does reduction in volume per se lead to better assessed performance for centers? There may be some evidence to suggest that changes in practice could enhance performance evaluations independent of actual changes in quality of care. There are data indicating that models utilized for risk adjustment for the program-specific reports have limited predictive capacity suggesting that factors not included in the models affect outcomes and could potentially vary between centers [19, 20]. As such, limiting transplants based on risk factors not captured in the models may improve measured performance of centers. However, it is important to note that, this would only pertain to unmeasured factors and alteration in factors that are included in the risk adjustment would have no bearing on measured performance. In addition, other than a modest difference between low performers and other centers with regard to changes in the proportion of ECD transplants, ischemia time and insurance status, findings from this study did not identify marked changes in known characteristics of recipients and donors. Rather, the findings appear to indicate that alterations in transplant volume for low performing centers are not explained by a comprehensive shift toward risk aversion. One potential intervention that may ameliorate declines in volume among low performing centers is to improve understanding of risk adjustment methodology and a critical evaluation of the sources of risks among centers. For example, an analysis of characteristics of transplants that is most strongly associated with poor outcomes for low performing centers could lead to targeted improvements in care for those groups of patients and less reluctance to reduce transplants among groups that were not associated with poor outcomes.
During the study period, observed graft survival rates did improve among centers with low performance evaluations. However, expected survival rates (after risk adjustment) also increased and as such changes in standardized mortality ratios used for report cards were relatively unaffected. Furthermore, there is evidence that kidney donor selection has little, if any impact, on SRTR report cards . In other words, there is limited evidence that aggressive centers with regard to donor kidney selection are at greater risk for low performance evaluations [21, 22]. Cumulatively, evidence seems to suggest that the strategy of limiting transplants from selected donors is relatively ineffectual with respect to improving performance evaluations while potentially denying critical treatment to transplant candidates.
In general, transplant characteristics between low performing centers and other centers are relatively similar. While there may be underlying characteristics of patients that can affect unbiased measurement of patient acuity, these data suggest that at an aggregate center-level, baseline characteristics of known variables of the patient populations that receive low performance evaluations are not markedly different from populations at other centers.
An interesting finding of the study was that low performing centers were more likely to reduce utilization of deceased donor kidneys with extensive (>24 h) cold ischemia time relative to other centers. Although it should be acknowledged that this effect was relatively minor, differences between centers based on performance may suggest a reluctance to accept nonlocal donor offers which are generally perceived to convey additional risk among low performing centers . However, there are also recent studies to suggest that donor kidneys with extended cold ischemia time are at risk for delayed graft function but not for long-term outcomes including graft loss and death that would affect performance evaluations [24, 25]. These results may suggest that donor kidneys that are otherwise viable and may not be associated with diminished outcomes applicable for performance evaluations are turned down by low performing centers more frequently but without significant potential gain on report cards.
An additional finding of the study was that low performing centers have a significant but relatively modest, reduction of privately insured patients compared to other centers. The influence of report cards on contracting by insurance companies has been illustrated in a previous study indicating that there is a decline in candidate registrations that were privately insured associated with changes in graft survival rates . In fact, there may be evidence that, particularly for younger patients and those with logistic means to travel, patients do select centers that have better reported outcomes [12, 27]. In addition, results likely reflect the influence of private insurance agencies propensity to contract (or terminate contracts) with centers that meet (or fail to meet) certain performance criteria. These results also highlight the potential economic ramifications of low performance evaluations to transplant centers.
There are several limitations of this study that are important to consider for interpretation of the findings. First, results of the study are observational and retrospective and as such it is critically important not to infer direct causality of the associations observed in this study. Particularly for this topic, there may be a host of factors that are likely related to performance evaluation of centers which are not captured in these data and could potentially affect the study findings. Another limitation of the study is an inability to specifically address the longitudinal course of low performance evaluations and changes in volume. It is difficult to determine when center changes may have occurred relative to performance evaluations and how to best reconcile changes in volume that may be related to prior evaluations versus prospective reports. In this study we evaluated changes over a 3-year period with the assumption that some centers may alter protocols and volume at different time periods within this period. However, there may be further specific information about alterations in volume in the specific intervals that occur directly before or after reports are publicized. We also limited the definition of low performance to those centers that were statistically lower than expected graft and patient survival, as such we did not determine whether results may be different based on the most current CMS or SRTR flagging rules. We also did not have information regarding acceptance rates or organ offers to centers that were eventually discarded. These data would be helpful to describe the potential impact on selectivity of centers following performance evaluations. Finally, we did not investigate whether changes in this study were temporary or whether centers that received low performance evaluations but then improved in measured performance, increased subsequent transplant volume.
In summary, findings of this study indicate a significant association between low performance evaluations of kidney transplant centers and a reduction in transplant volume. Results highlight a potential unintended impact of regulatory oversight on center practice that may deleteriously impact access to care for transplant candidates. An unintended consequence of transplant center report cards may be to stimulate more conservative behavior among centers that receive low performance assessments. Further investigation about the specific changes in practice that are a result of report cards and whether further education or interventions may ameliorate these effects are needed. Continued growth of national kidney transplant volume is of critical importance to the end-stage renal disease population and ongoing efforts to identify barriers to this growth should be a priority of the transplant community.