American Society of Transplant Surgeons Transplant Center Outcomes Requirements—A Threat to Innovation

Authors


* Corresponding author: Michael Abecassis, mabecass@nmh.org

Abstract

The transplant center regulations recently published by the Centers for Medicare and Medicaid (CMS) mandate that observed program-specific survival outcomes to fall within expected risk-adjusted outcomes. Meeting these outcomes is essential to continued participation in the Medicare program. Both donor and recipient variables not considered in current risk adjustment models can result in inferior outcomes and therefore may cause an overestimation of transplant center expected performance, precluding participation in the federally funded Medicare program. We reviewed the most recent four reporting periods published by the Scientific Registry for Transplant Recipients on their public website. We identified kidney, liver and heart transplant programs that were flagged for having outcomes statistically lower than expected as well as those that failed to meet CMS criteria. We also analyzed whether center volumes correlated with outcomes in these centers. We highlight the need for mitigating factors that could justify inferior outcomes under specific circumstances. Failure to reach consensus on such a mechanism for appeal may result in risk-averse behavior by transplant centers with respect to innovation and therefore hamper the ability to advance the field of transplantation. We propose a methodology that may address this emerging dilemma.

Abbreviations 
ASTS

American Society of Transplant Surgeons

CAD

coronary artery disease

CMS

Centers for Medicare and Medicaid

HCC

hepatocellular carcinoma

HCV

hepatitis C virus

OPTN

Organ Procurement and Transplantation Network

PRA

Panel Reactive Antibodies

SRTR

Scientific Registry for Transplant Recipients

UNOS

United Network for Organ Sharing

Introduction

Data on graft and patient outcomes are collected by United States transplant centers, analyzed and published twice a year. The observed results are subjected to risk adjustment by the Scientific Registry of Transplant Recipients (SRTR) to the extent that risk predictor data are available. The United Network for Organ Sharing (UNOS), functioning as a contractor separate from the SRTR for the Organ Procurement and Transplantation Network (OPTN), then reviews the center-specific program data (analyzed by the SRTR) in order to determine whether programs are performing at acceptable levels. Also, the SRTR publishes these data on a public website. In these reports, programs’ outcomes are designated as statistically higher, similar or lower to risk-adjusted expected outcomes.

On March 30, 2007, the Centers for Medicare and Medicaid (CMS) issued updated and vastly modified specific conditions of participation in the Medicare program for transplant centers, asserting its authority for oversight of patient safety (1). CMS cited specific outcomes requirements for center certification, maintenance of certification and recertification.

The designation of statistically different outcomes from expected that appears on the SRTR website is based on a 2-tailed t-test whereas the outcomes requirements for both the OPTN and CMS are both based on a 1-tailed t-test (see Methods). Moreover, it is critical to differentiate between the uses of similar outcomes data by OPTN and CMS, given that the implications differ greatly. Whereas OPTN utilizes outcomes data for the purpose of flagging programs whose outcomes deviate from expected outcomes and then initiates a peer-review process, CMS uses these outcomes standards for the more punitive purpose of determining whether a program should be certified for, or terminated from participation in the Medicare program. If a center is found to have observed patient deaths or graft failures that exceed expected (risk-adjusted) outcomes criteria, and a similar situation exists in any program-specific report in the previous 2 years, CMS issues a Condition-Level deficiency that may result in Medicare decertification. Thus, the implication of the CMS regulations is that any deficiencies in compliance may result in the inability of a transplant center to receive payment for transplant procedures for Medicare beneficiaries. This can also have implications for center of excellence designation by commercial payers and may result in impaired access to transplantation for patients who are not Medicare beneficiaries.

A center faced with a Condition-Level deficiency may request a review from CMS regarding ‘mitigating factors’. From a technical perspective, if a transplant center has a Condition-Level deficiency due to the failure to meet the outcomes requirements for a particular program, the transplant center will be given up to 210 days to come into compliance, or alternatively the program will not receive certification, recertification, or it may in fact become decertified. Because of the lag time in data cohorts reported to the OPTN and CMS by the SRTR, it is likely that a center that fails to meet outcomes standards during one reporting period will likewise fail to meet the standards in the following reporting period.

In a previous communication, the American Society of Transplant Surgeons (ASTS) Executive Committee, the governing body elected by the ASTS to represent its membership, highlighted a number of issues raised by these regulations and chronicled the dialogue between CMS and ASTS regarding the transplant center regulations (2). The ASTS is fully in support of instituting criteria designed to ensure that transplantation in the US is safe and effective. Notwithstanding this common objective and the collegial interface between CMS and ASTS underscored by an accompanying manuscript from CMS (3), the ASTS continues to question the appropriateness of the risk adjustment used in the CMS certification process. There are two problematic issues with the current risk adjustment methodology. First, the risk adjustment does not take into account important factors associated with outcomes that have not been previously required to be reported by the transplant centers. Second, without some provision for potentially inferior outcomes as new technologies undergo evaluation, innovation may be hampered and the field of transplantation may stagnate.

The focus of this communication is to assess the potential impact of the current CMS outcome requirements for participation in the Medicare program on the assessment and determination of transplant center performance. In addition, we assess the relationship of outcomes measures with center experience and volume. We also highlight specific mitigating factors that are known to affect outcomes, but which are not currently considered in the risk adjustment that defines the program's ‘expected’ outcomes. Finally, we propose a methodology that would allow for the analysis of predefined high-risk transplants separate from standard transplants.

Methods

We reviewed the publicly available center-specific SRTR reports (http://www.ustransplant.org) for kidney, liver and heart programs to identify programs that did not meet expected risk-adjusted standards. We reviewed the most recent report (Jan 1, 2005–Jun 30, 2007), specifically identifying programs that were reported to have ‘lower than expected’ performance outcomes for either graft or patient 1-year survival (observed survival minus expected >3, observed to expected ratio >1.5 and 2-tailed test p-value ≤0.05, from a multivariable Cox proportional hazards regression model). In addition, we identified programs that failed to meet both OPTN and CMS criteria (observed survival minus expected >3, observed to expected ratio >1.5 and 1-tailed test p-value ≤ 0.05). Accordingly, 1-tailed test p-values were determined by dividing the 2-tailed test p-values on the public SRTR website by two. Since a p-value of 0.05 to 0.10 is not flagged on the public SRTR website as ‘significantly lower than expected’, yet it is flagged by CMS (and OPTN), there are programs that fail to meet CMS outcomes requirements (by virtue of a two-sided test p-value between 0.05 and 0.10 versus a one-sided test p-value of ≤ 0.05) in addition to those flagged as ‘significantly different than expected’ on the public SRTR website.

We also reviewed the program experience (volume) over the same reporting periods in an attempt to correlate deficiencies in outcomes with center experience.

Results

In the most recent report (Jan 1, 2005–Jun 30, 2007), out of 256, 12 (5%) kidney programs were flagged by the SRTR public website as having 1-year patient survival outcomes statistically lower than expected. Nine additional kidney programs that were not significantly different than expected using the 2-tailed test p-value nevertheless failed to meet the patient survival criteria required by the CMS conditions of participation (Table 1). Among these 21 total programs (8%), although only 7 had been flagged by the SRTR public website as demonstrating 1-year patient survival outcomes that were statistically lower than expected in the preceding 2 years, 15 (6%) had at least one additional report in the preceding 2 years where the 1-year patient survival also failed to meet CMS criteria and thus would not qualify for CMS certification under the current regulations. For 1-year graft survival, 19 (7%) kidney programs were flagged by the SRTR public website and an additional six programs failed to meet CMS criteria. Of these 25 total programs (10%), 10 had been flagged in the SRTR public website in the preceding 2 years and 13 (5%) had failed to meet the CMS requirement at least once in the previous 2-year period and therefore would not qualify for CMS certification under the current conditions of participation (Table 2).

Table 1.  Number of kidney programs that fell below expected 1-year patient survival outcomes
SRTR PSR periodBased on SRTR public website (two-sided test)Additional CMS one-sided test p-value < 0.05Total CMS
  1. *Denotes failure to meet criteria in the most recent period as well as at least one additional reporting period in the previous 2 years. Programs that failed to meet one-sided test p-value < 0.05 would not qualify for CMS certification under the current regulations.

01/01/05–06/30/07*12 921
07/01/04–12/31/0612 719
01/01/04–06/30/06121022
07/01/03–12/31/0521 627
*Centers with ≥2 deficient periods 7 815
Table 2.  Number of kidney programs that fell below expected 1-year graft survival outcomes
SRTR PSR periodBased on SRTR public website (two-sided test)Additional CMS one-sided test p-value < 0.05Total CMS
  1. *Denotes failure to meet criteria in the most recent period as well as at least one additional reporting period in the previous 2 years. Programs that failed to meet one-sided test p-value < 0.05 would not qualify for CMS certification under the current regulations.

01/01/05–06/30/07*19625
07/01/04–12/31/0614418
01/01/04–06/30/0615419
07/01/03–12/31/0517320
*Centers with ≥2 deficient periods10313

Among 128 liver transplant programs, 9 (7%) did not meet the CMS 1-year patient survival criteria in the most recent report (7 or 5% were flagged in the SRTR public website), 6 (5%) had a similar finding (4 flagged in SRTR website) in the preceding 2 years and therefore would not qualify for CMS certification (Table 3). Among 9 liver programs that did not meet the 1-year graft CMS survival criteria (SRTR–7), 7 (5%) also failed to meet CMS graft survival criteria (SRTR–6) in the preceding 2 years and therefore would not qualify for CMS certification (Table 4).

Table 3.  Number of liver programs that fell below expected 1-year patient survival outcomes
SRTR PSR periodBased on SRTR public website (two-sided test)Additional CMS one-sided test p-value < 0.05Total CMS
  1. *Denotes failure to meet criteria in the most recent period as well as at least one additional reporting period in the previous 2 years. Programs that failed to meet one-sided test p-value < 0.05 would not qualify for CMS certification under the current regulations.

01/01/05–06/30/07*729
07/01/04–12/31/06718
01/01/04–06/30/06606
07/01/03–12/31/05628
*Centers with ≥2 deficient periods426
Table 4.  Number of liver programs that fell below expected 1-year graft survival outcomes
SRTR PSR periodBased on SRTR public website (two-sided test)Additional CMS one-sided test p-value < 0.05Total CMS
  1. *Denotes failure to meet criteria in the most recent period as well as at least one additional reporting period in the previous 2 years. Programs that failed to meet one-sided test p-value < 0.05 would not qualify for CMS certification under the current regulations.

01/01/05–06/30/07*729
07/01/04–12/31/06718
01/01/04–06/30/06516
07/01/03–12/31/05617
*Centers with ≥2 deficient periods617

Out of 137 heart transplant programs, 8 (6%) did not meet CMS 1-year patient survival criteria (SRTR–6), 4 (3%) had also failed CMS criteria (SRTR–3) in the preceding 2 years (Table 5) and of 9 (7%) programs that failed to meet CMS 1-year graft survival criteria (SRTR–7), 4 (3%) also failed CMS graft survival criteria (SRTR–3) in the preceding 2 years and would not qualify for CMS certification (Table 6). Of note, the four heart transplant programs that failed to meet patient outcomes criteria more than once in the 2-year period were the same programs that failed to meet graft survival criteria.

Table 5.  Number of heart programs that fell below expected 1-year patient survival outcomes
SRTR PSR periodBased on SRTR public website (two-sided test)Additional CMS one-sided test p-value < 0.05Total CMS
  1. *Denotes failure to meet criteria in the most recent period as well as at least one additional reporting period in the previous 2 years. Programs that failed to meet one-sided test p-value < 0.05 would not qualify for CMS certification under the current regulations.

07/01/04–12/31/06*628
01/01/04–06/30/06437
07/01/03–12/31/05538
01/01/03–06/30/05516
*Centers with ≥2 deficient periods314
Table 6.  Number of heart programs that fell below expected 1-year graft survival outcomes
SRTR PSR periodBased on SRTR public website (two-sided test)Additional CMS one-sided test p-value < 0.05Total CMS
  1. *Denotes failure to meet criteria in the most recent period as well as at least one additional reporting period in the previous 2 years. Programs that failed to meet one-sided test p-value < 0.05 would not qualify for CMS certification under the current regulations.

07/01/04–12/31/06*729
01/01/04–06/30/06347
07/01/03–12/31/05729
01/01/03–06/30/05718
*Centers with ≥2 deficient periods314

Of interest, 29 out of a total 267 (11%) transplant centers had at least one program that did not meet CMS criteria. Of the 21 centers that failed to meet CMS criteria for kidney, 4 also failed to meet liver criteria (2%). Of the 8 centers that failed to meet CMS criteria for liver, 4 (3%) also failed to meet criteria for kidney. None of the centers that failed to meet CMS criteria for heart also failed to meet criteria for either kidney or liver.

In order to determine whether kidney program experience (volume) correlated with outcome deficiencies, we analyzed kidney transplant volumes in centers that failed CMS criteria for either patient or graft survival compared to those that met the performance evaluation criteria (Figure 1). Volumes in centers that failed to meet CMS criteria ranged from 35 to 491 transplants over the 2.5-year period, demonstrating that small and large centers fell into deficient categories and both are affected by the CMS criteria.

Figure 1.

Correlation between kidney program deficiency and program volume.

A similar center volume analysis was performed for liver transplant programs (Figure 2). The volumes in these centers ranged from 46 to 387 over the 2.5-year period, again demonstrating a lack of correlation between center outcome deficiency and volume.

Figure 2.

Correlation between liver program deficiency and program volume.

We also performed a similar analysis for heart transplant programs (Figure 3). The transplant volumes in deficient centers ranged from 11 to 84 over the 2.5-year period. Again, no correlation was demonstrated between center volume and the finding of outcomes deficiencies.

Figure 3.

Correlation between heart program deficiency and program volume.

Discussion

Based on our findings, up to 10% of US kidney, liver and heart transplant programs, almost twice as many as currently flagged in the SRTR public website, will fail to meet CMS outcomes for either patient or graft survival, potentially resulting in the issuance of a Condition-Level center deficiency by CMS. Our data also show that more than half of centers that failed to meet criteria in the most recent reporting period had at least one other deficient report in the preceding 2 years. Once a program receives a Condition-Level deficiency, it is unlikely that the outcomes will be corrected in 210 days, given the time lag caused by the overlap in cohorts. Assuming a constant transplant rate, approximately 80% of the patients in the current period will remain in the following cohort, 60% in the subsequent cohort and so forth. Each period encompasses 6 months and therefore since 210 days is barely longer than a 6-month reporting period, only one subsequent reporting period would be included. Based upon our analyses, it is expected that up to 10% of US transplant centers will be at risk of losing Medicare funding for transplants.

Our findings raise the following questions: what are the factors that lead to worse outcomes in these centers? Are these related to donor selection, recipient selection or deficiencies in the transplant programs? Clearly, the latter would warrant appropriate action that might include decertification. But what if the observed outcomes differ from expected outcomes as a result of inadequate risk adjustment? Although data currently analyzed and risk adjusted by SRTR are fairly complete for donor criteria, the same may not hold true for certain recipient variables. There may be a substantial amount of center variability that is not accounted for in the current risk adjustment models. Therefore, if a center performs transplants on patients with a disproportionate prevalence of these unadjusted risk predictors, the ‘expected’ outcomes to which the observed outcomes are compared may not be accurate, leading to unjustified determinations of outcomes deficiencies.

Mitigating factors

CMS has made it clear that centers with deficiencies can request a review based on mitigating factors allowing CMS, in limited circumstances, to extend approval to a transplant program that demonstrates that there are exceptional factors present. The notion that a process should exist for transplant centers to implicate mitigating factors as a cause for inferior outcomes has evolved as a result of extensive deliberations between ASTS and CMS, including ASTS responses to the transplant center regulations (2). On July 17, 2008, CMS issued a request to the transplant community, including ASTS, eliciting feedback about CMS guidance to transplant programs regarding the mitigating factors provision (3). The proposal by CMS on which comments were requested included a process for the request of approval based on mitigating factors.

The ASTS response to this request was consistent with our previous efforts to foresee potential problems associated with an overly strict approach to the use of SRTR-generated program-specific graft and patient outcomes as a sole determinant for CMS certification. Specifically, in addressing the section on ‘risk-adjusted anomalies’ in the CMS document, the ASTS pointed out a few representative examples of the fundamental flaw in using SRTR risk-adjusted data as a sole determinant for certification. These included patients undergoing transplantation following desensitization protocols for ABO and cross-match incompatibility, patients undergoing liver transplantation for hepatocellular carcinoma (HCC) and other malignancies such as cholangiocarcinoma, hepatitis C virus (HCV) positive recipients and recipients with significant comorbidities such as advanced coronary artery disease (CAD).

Without exception, these individual circumstances have been shown to have a negative impact on both patient and graft outcomes although the results following transplantation may be significantly better than without transplantation (4). But in these as well as other situations, there is no appropriate risk adjustment included in the SRTR ‘risk-adjusted’ expected survival rates that form the benchmark for the transplant program's performance assessment. In the case of desensitized recipients (known to have inferior outcomes to nonsensitized or ABO compatible patients), only the panel reactive antibody (PRA), indicating the level of sensitization, is used for risk adjustment. But PRA may have no relevance to the impact of desensitization protocols for positive cross-match (5) and has absolutely no relevance to ABO compatibility. With respect to patients with HCC, although the presence or absence of HCC is considered in the current methodology, no variables relating to the biological behavior and the risk of recurrence (6) are entered into the SRTR risk adjustment. For cholangiocarcinoma, despite the fact that survival results in selected cases and in specialized centers are excellent (7), the overall outcomes are dismal given the high rate of recurrence (8). Yet, there is no risk adjustment for this diagnosis. Also, patients with end-stage liver disease secondary to HCV have worse outcomes than those with other etiologies of liver disease following liver transplantation due to recurrent disease (9–11). Moreover, donor age has been shown to have a significant impact on the graft and patient survival of HCV positive recipients especially with older donors. This interaction has only recently been added to the SRTR risk adjustment. Finally, recipients may have significant comorbidities that result in inferior outcomes. For example, the presence of CAD is associated with inferior outcome after kidney and liver transplantation (12–14). Currently, a patient with minimal coronary disease, e.g. 30% stenosis in one coronary artery, would receive the identical risk adjustment to a patient who has undergone a coronary bypass, has had recurrent myocardial infarction and subsequent angioplasty and stenting. The presence of congestive heart failure, the left ventricular function and many other prognostically significant indicators for survival are not included at all in SRTR risk adjustment.

In our letter to CMS, the ASTS used these examples to illustrate the point that current SRTR risk adjustment models should not be used in isolation for certification and decertification decisions. ASTS also suggested that CMS provide an expert panel review of mitigating circumstances, in the form of an advisory group that included not only CMS, but transplant professional societies that might have a better understanding of the clinical issues surrounding these mitigating factors.

In September 2008, CMS issued a document entitled ‘process for requesting consideration of mitigating factors in CMS’ determination of medical approval of organ transplant centers’. The document greatly resembled the ‘draft’ that had been provided to ASTS in the July 17, 2008 communication and on which ASTS had based its comments. In this document, CMS reiterated that ‘a transplant program may request that CMS consider mitigating factors in the initial approval and reapproval of a transplant program that does not meet one or more Conditions of Participation’. The regulation described three general areas to be reviewed by CMS in determining whether approval could occur based on mitigating factors including: (1) outcomes measures, (2) availability of other Medicare-approved transplant programs/centers in the area and (3) extenuating circumstances that might have a temporary effect on meeting the Conditions of Participation.

The section on ‘risk-adjusted anomalies’ outlined in the CMS document allows for consideration that performance has been adversely affected by transplant risks not captured in the SRTR risk-adjustment methodology. Included in this category of mitigating factors is the extent to which the program uses experimental protocols and the effect on outcomes if the pertinent cases are removed from the database. However, CMS provided no methodology regarding how patients whose transplants are considered ‘experimental’ should be analyzed.

ASTS proposal

In view of the preceding discussion, the ASTS proposes the following for consideration by CMS. First, it is clear those patients who undergo transplantation as part of experimental protocols that have passed the rigors of institutional review boards may have outcomes that are inferior to those undergoing transplantation under ‘standard of care’ conditions, yet are still included in the review of a center's outcome. Therefore, although we have not demonstrated that inferior outcomes in these programs are related to the undertaking of experimental protocols, the current paradigm threatens scientific innovation and advancement, given that the consequences of inferior outcomes may result in Medicare decertification, loss of center of excellence designation and poor public perception. Unquestionably, we agree that there needs to be quality control so that underperforming programs can be flagged for review. However, we do not believe that the intent of the regulations was to stifle the conduct of scientific research.

Thus, we propose that the outcomes of patients undergoing transplantation under approved experimental protocols should be analyzed separately. Research trial patients would be excluded from risk-adjusted SRTR program reports. Under this proposal, transplant centers would register these patients prospectively with the OPTN (at the time of transplantation) as patients undergoing transplantation under experimental protocols. To address the potential for over use of this exemption by transplant centers, the percent of transplant recipients undergoing transplantation under an experimental protocol could be ‘capped’ for each program. Also, approval of experimental protocols would need to occur a priori and would require approval from the center's institutional review board. In addition, transplant programs could request that specific exemptions be considered and an expert panel would review and approve categories of ‘exempted’ transplants on an annual basis. In order to meet reporting requirements and maintain transparency, the risk-adjusted outcomes of transplants performed under this designation would be published in the SRTR report but no statistical comparison would be made or used for determinations of certification.

Second, it is clear that better predictive models are sorely needed in order to improve the validity of currently used risk-adjustment methodologies. However, such validated models may take years to develop, since the necessary data may not have been collected. Therefore, it is essential to develop a strategy whereby transplant centers are not penalized for performing what might be considered reasonable but ‘high-risk’ transplants on the basis of either donor or recipient characteristics, especially when these are not considered experimental. One such approach would be to calculate ‘expected’ risk-adjusted outcomes only for transplants with the most standard donor and recipient characteristics and exclude defined high risk transplants from these calculations as they relate to program performance. Alternatively, CMS could use current SRTR risk-adjusted data in a manner more akin to their use by UNOS/OPTN as a way to improve program performance, rather than the more punitive action of termination of Medicare certification. The latter carries with it numerous unintended consequences, including but not limited to, access to transplantation.

Conclusions

Throughout these deliberations, ASTS has stated its unwavering position that centers with consistently poor outcomes that cannot be explained by mitigating circumstances and who do not undertake effective remediation should not be permitted to participate in the Medicare program. However, ASTS has repeatedly advised CMS that a rigid application of the outcomes condition of participation could result in centers refusing to accept high-risk patients and that the certification policy, if applied rigidly, could result in the termination of centers that generally provide high quality care. The ASTS is concerned that a serious and surely unintended consequence of these compliance metrics may be that centers will tend to avoid novel therapies that would normally advance the field of transplantation for fear of losing Medicare accreditation. Thus, we feel strongly that given the imperfect risk-adjustment methodologies currently in place and the fact that validated predictive models will require significant data collection and time (and thus are unlikely to occur in the near future), SRTR reports should never be the sole basis for such punitive an action as termination unless other aggravating circumstances are also present to indicate substandard care and a risk to patient safety.

Transplant centers facing the Medicare certification process are entering uncharted waters. We propose a process by which patients undergoing transplantation under defined experimental protocols that may result in inferior outcomes would be analyzed separately from patients undergoing ‘standard of care’ transplantation. Clearly, safeguards would need to be put in place in order to prevent the overuse of this designation. This proposal will minimize both the negative implications of introducing innovative therapies, as well as safeguard against the injudicious use of organs by transplant centers. In addition, in the absence of adequate risk adjustment, we believe that CMS needs to reconsider their decision not to incorporate certain recognized mitigating factors in their final analysis.

Ancillary