## Introduction

Continuous quality improvement programs have emerged as key components of high-performing health care organizations. The collection and timely analysis of clinically relevant data is crucial in accomplishing quality improvement initiatives and ensuring the highest quality care for patients. Organ transplantation is unique among medical specialties in the quantity and quality of data collected on a national basis. Through data collected from the Organ Procurement and Transplantation Network (OPTN), accurate and clinically relevant data are available that can provide information to improve transplantation outcomes. Currently, however, these data are analyzed on an episodic basis and provided to centers semiannually through the Program-Specific Reports (PSRs). Consequently, recognition of clinically relevant changes in clinical outcomes may be delayed, limiting the success of quality improvement efforts.

Statistical process control charts were originally developed to study industrial processes in the 1930s by W.A. Shewhart and his colleagues at Bell Laboratories (1). These charts measure performance over time and ‘signal’ if there is a deviation from accepted production standards. The CUSUM, or cumulative sum, chart was introduced in 1954 by Page and provided a very sensitive approach to monitoring a process and identifying changes in outcome (2). The purpose of these charts is to give timely and easily interpreted summaries of outcome data. The potential utility of CUSUMs in health care was recognized in the early 1970s when the paper, ‘Why don't doctors use CUSUM?’ was published in the *Lancet* (3). Several years later, the *New England Journal of Medicine* published a manuscript, which highlighted the value of CUSUM techniques in clinical applications (4). Broad acceptance of these techniques, however, was delayed initially by data collection limitations and subsequently by the inability to include meaningful risk adjustment.

Recent high-profile events, including a cluster of heart transplant deaths in Britain (5) and the Institute of Medicine's report, ‘Crossing the Quality Chasm’ (6), have contributed to a heightened awareness of the need to monitor surgical outcomes. Public interest and recent improvements in the analytic methods have led to rapid increases in the utilization of CUSUMs to track surgical outcomes. In a 2007 review, Biau and colleagues identified 31 studies which utilized CUSUMs to track surgical outcomes in cardiac, general and ENT surgery (7). Other work on CUSUM methods and applications in medical studies can be found in various statistical and medical journals (8–10).

Application of CUSUMs to the management of transplant centers offers physician leaders the opportunity to track outcomes in a real-time, risk-adjusted manner. A previous retrospective analysis demonstrated that CUSUMs identified changes in clinical practice sooner and with higher sensitivity than current center monitoring techniques (11). Recent improvements in chart construction are based on survival analysis techniques and allow the incorporation of outcomes as they occur, rather than after passage of a specific time period (e.g. 1-year posttransplant) as had previously been the case (12). Furthermore, these CUSUM charts are risk adjusted using comprehensive models currently employed by the SRTR to adjust outcomes for patient and donor characteristics and are adjusted for patient mix. In general, the CUSUM compares observed outcomes with expected results; it increases in value as graft failures or patient deaths occur and decreases during periods with no failures. If too many failures occur over time (compared to what would be expected) the value of the CUSUM will exceed a predetermined threshold value and ‘signal’ that a process review should be initiated.

This article provides a brief overview of the construction and application of CUSUM charts for transplant professionals. We begin with a brief summary of the construction and interpretation of CUSUM charts, including a number of examples. Following this, we briefly review the methods currently used by the SRTR to assess transplant programs in the PSRs. Next, we present a retrospective comparison of CUSUM monitoring with the techniques employed in the PSRs. Finally, we address the strengths as well as potential difficulties and risks of a broad application of this technique.

### Defining CUSUMs

A CUSUM chart for a given program or center presents a simple graphical comparison of observed and expected numbers of events over time. In their initial description of a clinically relevant, risk-adjusted CUSUM, Steiner et al. and Grigg et al. described methods for assessing performance of a clinical system that produced a binary outcome (e.g. death following cardiac surgery) (10,13,14). In these methods, the CUSUM is increased or decreased by a variable degree depending upon the observed and expected outcomes from the process. Axelrod et al. applied this approach to transplant data and developed CUSUM charts to monitor 1-year posttransplant survival for liver transplant recipients and 1-year allograft survival for kidney transplant recipients (11). Based on the method of Steiner et al. (13), a logistic regression model, which included several donor and recipient factors, was utilized for risk adjustment.

Binary outcome CUSUM charts are very useful tools for monitoring situations in which the outcome is binary and rapidly ascertained; for example, in monitoring conversion rates (the percentage of possible donors which actually result in transplantable organs), acceptance rates (the percentage of organ offers which are accepted by a program) or mortality rates over a short, fixed period of time. They do have a disadvantage: in monitoring longer term survival outcomes such as 1- or 2-year mortality rates, the data on any given individual cannot be used until the corresponding period has elapsed.

In 2008, Biswas and Kalbfleisch developed a method to create risk-adjusted CUSUM charts that are based on a continuous time survival analysis approach and are able to incorporate deaths or graft failures as they occur (12). These charts have a substantial advantage in the monitoring of longer term survival endpoints and are more consistent with the Cox model-based risk-adjusted methods used by the SRTR. This method can be utilized to construct two types of CUSUM methods: a one-sided chart in which the value is restricted to nonnegative sums and a two-sided or O-E chart, as described below; a more complete description of the calculation of the CUSUMs is included in the Appendix.

**The one-sided CUSUM: ** It is constructed principally to assess for a clinically significant excess of allograft failures or patient deaths. The CUSUM is restricted to positive values and so is bounded below at a value of zero. Thus, a center performing at, or better than, the expected performance (and thus having fewer observed failures than expected) would have a chart which tends to stay relatively close to zero. It would increase with any failure, and then return to zero in an ensuing period without failures. Conversely, if the center's outcomes are much poorer than the national average, the number of failures will lead to a substantial increase in the CUSUM and this will eventually result in a signal. The one-sided CUSUM signals when the plot line crosses a horizontal line, termed the control limit, which defines the signaling threshold. The height of this line (L) reflects the balance between rapid signaling that will very quickly identify centers with poor outcomes (sensitivity) and the desirability of avoiding false positive results (specificity) and signaling when the center's performance is actually consistent with the national average. The value of L can be adjusted for center volume to ensure that the sensitivity is kept at a suitably low level for all centers. Figures 1–3 (bottom panels) provide examples of the one-sided CUSUM. In our analysis, if a kidney transplant center's volume is 40 transplants/year or more and its true rate is twice the adjusted national average, the chance of a signal over a 3-year period is 90% or more. If the center's rates are the same as the national average, a period of 30 years would be expected before the first false positive signal.

**The two-sided or O-E CUSUM: ** For a given center, this simplest of CUSUM plots, as a function of time, the difference between the observed number of deaths and the number of deaths that would be expected based on the risk-adjusted national average. This CUSUM can be viewed as being updated daily by adding to its previous value the observed number of deaths on that day less the expected number. The expected number of deaths is estimated from a survival model based on national data and is adjusted for the particular patient mix at the center. Thus, the O-E chart traces out an approximately horizontal path (slope = 0) if the death rate at the center is close to the risk-adjusted national average. An upward trend of the O-E plot over a specified time interval indicates that the center has worse outcomes than the adjusted national average. Conversely, a downward trend of the plot corresponds to better outcomes than the adjusted national average. Figures 1–3 (top panel) provide examples of O-E plots and are discussed further below. As indicated by the arrows superimposed on the plots, the ‘slope’ of the plot gives an estimate of the approximate relative risk (RR), which is the ratio of the death rate at the center to that for the adjusted national average.

It is possible to use the two-sided CUSUM to provide a signal when there is statistical evidence that a center's outcomes are different from the national average. The two-sided CUSUM signals if the ‘slope’ of the plot exceeds a predetermined value over an extended period. The signal is obtained by systematically checking the slope of the plot at each successive time using a V-mask as introduced by Barnard (15,16) and discussed in the Appendix. As in the height of L in the one-sided chart, the angle of slope, which is considered significant in the two-sided chart, can be adjusted to balance sensitivity and specificity.

The principal advantage of the O-E plot is that the slope of the plot over a given interval gives an immediate picture of the relative rate of outcomes within the center of interest compared to expected results. In the one-sided CUSUM, the slope of the chart is more difficult to interpret and immediate comparisons to the national average or expected results are more difficult to see. We find that both charts provide useful and complementary information, as the examples in the next subsection illustrate.

### Examples of CUSUM charts and interpretation

A sample of CUSUM charts over a 3-year period for liver transplant programs, labeled Center A, Center B and Center C, is described here. For each center, the one-sided CUSUM and O-E CUSUM charts are presented for 1-year patient survival. Similar charts could be constructed for graft survival or for other outcomes such as 1-month or 2-year survival.

**Center A: ** From the O-E chart, the failure rate for the 1-year survival in Center A is close to the national average for the first year, as is suggested by the nearly horizontal plot line (slope approximately 0) (Figure 1). For the second period, from 1 to 3.5 years, the death rate exceeded the national average, as illustrated by an increase in the slope of the O-E CUSUM. From the one-sided CUSUM chart (Figure 1), we see that these trends would have led to a signal at the end of the second year. If the chart had been in place, this signal suggests a review of center practices may be appropriate.

**Center B: ** The O-E chart in Figure 2 illustrates that Center B experienced death rates very close to the national average (adjusted for patient mix) over the first 2 years of the CUSUM period. During the last 1.5 years, the center had considerably better 1-year outcomes than the adjusted national average. Correspondingly, the one-sided CUSUM chart (Figure 2) did not signal.

**Center C: ** From the O-E chart (Figure 3), we see that the 1-year mortality at the center was higher than the national average for the first year or so. After that period, the 1-year death rates were considerably lower than the adjusted national rates through the end of the 3.5-year period. From the one-sided CUSUM (Figure 3), we see that the higher death rates observed early on were not sufficient to lead to a signal.