Until recently, the American College of Surgeons (ACS) National Surgery Quality Improvement Program (NSQIP) measured outcomes across an array of broad surgical categories (eg general surgery, vascular surgery). Recently, the program has moved toward more targeted procedures and indications,1 and a consortium has been developed (ie, the Oncology NSQIP National Cancer Institute Center Consortium) that will focus only on surgical oncology cases.2 However, ACS NSQIP has been critiqued on its ability to predict cancer surgery outcomes,3, 4 partly because the program does not contain cancer-specific variables (eg, American Joint Committee on Cancer [AJCC] stage). Currently, ACS NSQIP only collects 2 cancer variables: whether the patient had disseminated cancer and whether the patient received chemotherapy or radiation within a limited time frame before surgery.
An increasing body of literature supports the need to validate and update prediction models. A model should include reliable and standardized data and variables that are clinically relevant to the population in which it is to be used.5-7 To date, no study has systematically evaluated the predictive capabilities of cancer-specific variables on short-term outcomes after cancer surgery.8 Because the ACS NSQIP has not collected cancer-specific variables, there has been no way to determine the effect of cancer variables on short-term outcomes. Therefore, we linked ACS NSQIP to the National Cancer Data Base (NCDB), the largest oncology registry in the United States. Our objectives were: 1) to compare the staging and neoadjuvant therapy variables in the ACS NSQIP and in the NCDB and 2) to assess the relative importance and predictive capabilities of cancer stage and neoadjuvant therapy variables in short-term outcome models after colorectal cancer surgery. We focused on colorectal cancer surgery because it is a common cancer procedure and is associated with appreciable morbidity and mortality, making it an ideal target for quality-improvement initiatives.9-11
MATERIALS AND METHODS
The ACS NSQIP and the NCDB were linked to obtain a data set that contained patient demographics, comorbidities, and short-term outcomes data from ACS NSQIP with cancer-specific staging and treatment data from the NCDB. Details regarding both programs were described extensively in previous reports.12-15 In brief, the ACS NSQIP is a surgical quality-improvement program that originated in the Veterans Health Administration in 1994 and subsequently was transferred to the non-Veterans Affairs hospitals in 2001. The program is based on highly standardized and robust clinical data collected by trained and audited surgical clinical reviewers.16 Patients are followed for 30 days after their index procedure, and complications are captured irrespective of whether they occur during the patients' initial hospital stay or after discharge or whether they are readmitted to the same hospital or to another facility.
The NCDB is a joint program of the ACS Commission on Cancer, and the American Cancer Society. The NCDB is the largest cancer registry in the world, capturing >70% of all newly diagnosed malignancies in the United States from approximately 1500 hospitals.17 All Commission on Cancer-accredited hospitals are required to meet specific standards of care in the management of patients with cancer. Data are abstracted by trained cancer registrars who are periodically audited to ensure data reliability.
Study Cohort and Data Linkage
Patients were identified from the ACS NSQIP (2006-2008) if they underwent a colorectal resection for cancer based on Current Procedure Terminology18 (CPT) and International Classification of Disease, Ninth Revision19 (ICD-9) codes. Patients from the NCDB with colorectal cancer based on International Classification of Diseases for Oncology, Third Revision20 (ICD-O-3) codes who underwent resection were identified. Because the ACS NSQIP and NCDB datasets do not contain unique patient-level identifiers, patients were linked using a probabilistic matching algorithm based on the following variables: American Hospital Association number, patient age, sex, and date of operation. To focus on elective cancer surgery, patients who underwent emergency operations or who were classified as ASA class V (ie, moribund) were excluded from the analysis.
American College of Surgeons National Surgical Quality Improvement Program and National Cancer Data Base Variables
The ACS NSQIP collects >130 preoperative, operative, and postoperative outcome variables, including 2 cancer-related variables: the presence of preoperative disseminated cancer and preoperative chemotherapy (within 30 days of surgery) or radiotherapy (within 90 days of surgery). Neoadjuvant therapy variables were examined only for rectal cancer, and, for the purposes of the current analysis, these variables were combined. Other patient demographic variables that were considered have been extensively described previously and are identified in Table 1.21
Table 1. Characteristics of 11,405 Patients Undergoing Colorectal Resection for Cancer: 2006 to 2008
|Age: Median [IQR], y||63 [53-73]|
|Sex: Women||5532 (48.5)|
| White||8802 (77.2)|
| Black||1065 (9.3)|
| Hispanic||277 (2.4)|
| Other||1261 (11.1)|
|ASA class|| |
| I/II||5276 (46.3)|
| III||5551 (48.7)|
| IV/V||578 (5.0)|
|Functional status|| |
| Independent||10,708 (93.9)|
| Partially dependent||614 (5.4)|
| Totally dependent||83 (0.7)|
|Weight loss||737 (6.5)|
|Albumin <3.0 g/dL||848 (7.4)|
| Underweight||344 (3.0)|
| Normal||3589 (31.8)|
| Overweight||3873 (34.3)|
| Obese||3486 (30.9)|
| Previous cardiac event||1534 (13.5)|
| CHF||150 (1.3)|
| HTN||6418 (56.3)|
| PVD||167 (1.5)|
| Dyspnea at rest||130 (1.1)|
| Current smoker||1771 (15.5)|
| COPD||685 (6.0)|
|Preoperative sepsis||364 (3.2)|
| Oral||1399 (12.3)|
| Insulin dependent||598 (5.2)|
|Renal failure||53 (0.5)|
|Steroid use||292 (2.6)|
|Procedure type|| |
| Open colectomy||6260 (54.9)|
| Laparoscopic colectomy||3418 (30)|
| Total proctocolectomy||178 (1.6)|
| Open proctectomy||816 (7.1)|
| Laparoscopic proctectomy||149 (1.3)|
| Abdominoperineal resection||584 (5.1)|
|AJCC stageb|| |
| I||3694 (30.7)|
| II||3151 (29.4)|
| III||3160 (29.5)|
| IV||1111 (10.4)|
|Neoadjuvant CRTb,c||913 (52.9)|
To evaluate and compare cancer variables common to both the ACS NSQIP and the NCDB, we evaluated 2 variables from the NCDB: cancer stage and neoadjuvant chemoradiation. Cancer stage from the NCDB is based on the AJCC staging schema and was updated to reflect the most recent edition.22 To systematically evaluate the importance of specific aspects of cancer staging, AJCC stage was categorized by creating the following groups: 1) metastatic cancer (stage IV vs stages I-III combined), 2) AJCC stage (stages I-IV), and 3) individual tumor (T) classification (T1-T4), lymph node (N)-classification (N0-N2), and metastasis (M)-classification (M0-M1). The NCDB defines chemoradiotherapy use based on Facility Oncology Registry Data Standards23 and captures preoperative therapy if it is received as the first course therapy for an index cancer.
The ACS NSQIP collects more than twenty 30-day postoperative complication variables, which have been defined elsewhere.21 To broadly evaluate the importance of cancer variables in ACS NSQIP complication models, the following outcomes were selected: mortality, serious morbidity, superficial surgical site infection (SSI), organ space infection, wound dehiscence, venous thromboembolism (VTE), and pneumonia. Serious morbidity included any of the following ACS NSQIP-defined postoperative complications: cardiac arrest requiring cardiopulmonary resuscitation, myocardial infarction, deep vein thrombosis, pulmonary embolism, sepsis/septic shock, deep incisional SSI, organ space SSI, wound dehiscence, unplanned reintubation, pneumonia, renal failure/progressive renal insufficiency, urinary tract infection, and unplanned reoperation. Patients were precluded from being categorized with postoperative pneumonia, urinary tract infection, renal failure/insufficiency, reintubation, or a prolonged ventilation adverse event if a related condition was present at the time of the index operation.
The predictive strength of cancer staging variables was examined in patients with colorectal cancer, whereas neoadjuvant chemoradiotherapy variables were evaluated only in patients with rectal cancer. Variable agreement between the ACS NSQIP and NCDB data sets was compared using the Cohen kappa coefficient to assess whether variable agreement (ie, similar in ACS NSQIP and NCDB) was poor (kappa, <0.2), fair (kappa, 0.2-0.4), moderate (kappa, 0.4-0.6), or very good (kappa, >0.6). By using the ACS NSQIP modeling approach,12 multivariable logistic regression models were developed using a forward stepwise selection process (entrance criteria, P < .05). First, to evaluate the importance of cancer stage among all patients in the study, 4 models were developed. In addition to standard ACS NSQIP risk-adjustment variables, each of the 4 models included 1 of the 4 cancer stage variables (ie, disseminated cancer [from the NSQIP], and metastatic cancer, AJCC stage, TNM stage [from the NCDB]). Second, for each outcome assessed, 2 additional models were developed among patients who underwent rectal surgery that separately included the neoadjuvant chemoradiotherapy variables from each database. For this study, we used a forward stepwise modeling process, such that the order of variable selection could be assessed and compared (variables that were selected earlier have stronger model explanatory power and are considered “more important”). If variables are selected, they are deemed to add explanatory power to the model; whereas, if the variable is not selected, then that variable is not important for that model. Odds ratios (ORs) and 95% confidence intervals (CIs) were included for the mortality and serious morbidity models. Cancer variables were tested for significant interaction between other explanatory variables based on a priori clinical knowledge. Models also were assessed by evaluating model discrimination (c-statistic), calibration (Hosmer-Lemeshow chi-square), and a combined discrimination and calibration metric (Akaike information criterion [AIC]).24, 35 All statistical analyses were performed using the SAS version 9.3 software package (SAS Institute Inc., Cary, NC). Northwestern University Institutional Review Board approved this study.
From 146 hospitals, 11,405 patients were identified who underwent surgery for colorectal cancer (colon, 85%; rectum, 15%). In merging the patients from the ACS NSQIP with those in the NCDB, 83.1% of eligible patients in the ACS NSQIP were identified in the NCDB. The 3 most common operations performed were open colectomy (55% of patients), laparoscopic colectomy (30% of patients), and open proctectomy (7% of patients). Patient characteristics are listed in Table 1.
Importance of Cancer Staging Variables
Among all patients with colorectal cancer, 761 patients (6.7%) from the ACS NSQIP had preoperative disseminated cancer reported, and 1111 patients (10.4%) from the NCDB had metastatic cancer (kappa, 0.454), as would be expected given their somewhat different definitions. On multivariable analysis, both ACS NSQIP and NCDB cancer variables were important in certain complication models (Table 2). For mortality, the ACS NSQIP disseminated cancer variable (OR, 2.11; 95% CI, 1.40-3.17) and the NCDB metastatic cancer variables (stage IV vs. stage I-III: OR, 2.18; 95% CI, 1.51-3.15; stage IV vs stage I: OR, 2.28; 95% CI, 1.39-3.75) were included in the models (examined separately). The NCDB metastatic cancer variable was selected earlier (step 5 of 15) in the logistic regression, forward stepwise process compared with model testing the ACS NSQIP disseminated cancer variable (step 9 of 15) and the overall AJCC 4-level cancer stage variable (step 8 of 15). For serious morbidity, only NCDB cancer variables were selected. T-classification (step 13 of 18) was selected as important, but only T4 versus T1 was statistically significant (OR, 1.46; 95% CI, 1.17-1.81). Metastatic cancer (M1 vs M0) was selected at step 16 of 18. With respect to wound occurrences, the only variable selected was NCDB AJCC N-classification for superficial SSI, whereas neither ACS NSQIP nor NCDB variables were selected in organ space SSI or wound dehiscence models. For venous thromboembolism, only the NCDB metastatic cancer variable was selected (step 3 of 7); and, for pneumonia, only the NCDB T-classification variable was selected (step 10 of 14). Across all cancer stage variables and complication models, c-statistics, Hosmer-Lemeshow chi-square values, and AIC values were very similar, if not identical.
Table 2. The Importance of Cancer Staging Variables From the National Surgical Quality Improvement Program and the National Cancer Data Base in American College of Surgeons National Surgical Quality Improvement Program Complication Models After Colorectal Resection for Cancer (n = 11,405)
|NSQIP|| || || || || || || |
| Disseminated cancer|| || || || || || || |
| Step selected||Step 9 of 15a||Not selected||Not selected||Not selected||Not selected||Not selected||Not selected|
|NCDB|| || || || || || || |
| Metastatic cancer|| || || || || || || |
| Step selectedb||Step 5 of 15a||Step 16 of 18a||Not selected||Not selected||Not selected||Step 3 of 7a||Not selected|
|AJCC stage (I-IV)|| || || || || || || |
| Step selectedb||Step 8 of 15a,c||Not selected||Not selected||Not selected||Not selected||Not selected||Not selected|
|AJCC TNM stage|| || || || || || || |
| T-classification|| || || || || || || |
| Step selectedb||Not selected||Step 13 of 18a,d||Not selected||Not selected||Not selected||Not selected||Step 10 of 14a,e|
| N-classification|| || || || || || || |
| Step selectedb||Not selected||Not selected||Step 8 of 10a,f||Not selected||Not selected||Not selected||Not selected|
| M-classification|| || || || || || || |
| Step selectedb||Step 5 of 15a||Not selected||Not selected||Not selected||Not selected||Step 3 of 7a||Not selected|
Importance of Neoadjuvant Therapy Variables
To examine neoadjuvant therapy variables, patients who underwent resection for rectal cancer (n = 1727) were identified. On the basis of somewhat different definitions for neoadjuvant therapy receipt, 622 patients (36%) from the ACS NSQIP and 913 patients (52.9%) from the NCDB were identified as having received neoadjuvant therapy (kappa, 0.570). Among patients with stage II/III disease, 45.1% in the ACS NSQIP versus 65.3% in the NCDB were identified as having received neoadjuvant therapy.
On multivariable analysis, both ACS NSQIP and NCDB neoadjuvant therapy variables were important in certain models. Neoadjuvant therapy was not important for 30-day mortality (ie, it was not selected in the model). For serious morbidity, only the NCDB neoadjuvant therapy variable (OR, 1.55; 95% CI, 1.33-1.81) was important, but it was selected at step 10 of 11 (Table 3). For superficial SSI, both ACS NSQIP and NCDB neoadjuvant variables were selected at the same early step in the forward stepwise selection process (step 2 of 7); whereas, for organ space infection, only the NCDB neoadjuvant therapy variable was included. Receipt of neoadjuvant therapy was not selected in wound dehiscence, VTE, or pneumonia models. Across all models, whether the neoadjuvant therapy variable from either data set was selected or excluded, the model fit was very similar if not identical (Table 3).
Table 3. Importance of Neoadjuvant Chemoradiotherapy Variables from the American College of Surgeons National Surgical Quality Improvement Program and the National Cancer Data Base in American College of Surgeons National Surgical Quality Improvement Program Complication Models After Rectal Resection for Cancer (n = 1727)
|NSQIP|| || || || || || || |
| Neoadjuvant CRTa|| || || || || || || |
| Step selectedb||Not selected||Not selected||Step 2 of 7c||Not selected||Not selected||Not selected||Not selected|
|NCDB|| || || || || || || |
| Neoadjuvant CRTd|| || || || || || || |
| Step selectedb||Not selected||Step 10 of 11c||Step 2 of 7c||Step 3 of 3c||Not selected||Not selected||Not selected|
The linkage of colorectal cancer patients from the ACS NSQIP and NCDB data created a data set that united rich clinical data (ACS NSQIP) with cancer-specific treatment and stage information (NCDB). This data set represents a unique and powerful tool with distinct advantages over administrative or cancer registry data alone. Unlike administrative or cancer registry data, the ACS NSQIP includes detailed preoperative patient details and postoperative complication information collected by trained clinical data abstractors using standardized definitions. Short-term outcomes are either unavailable or unreliable in administrative and cancer registry data sources.25, 26, 27 In addition, cancer-specific data in the NCDB are collected by the same cancer registrar using the same coding manuals used by the better known Surveillance, Epidemiology, and End Results Program. However, the NCDB is not limited to select geographic locations like the Surveillance, Epidemiology, and End Results Program and has several enhancements, including the collection of chemotherapy information.17 By using this merged data set, we observed that metastatic cancer and neoadjuvant therapy variables were somewhat important for assessing short-term outcomes after cancer surgery, whereas T-classification and N-classification were not consistently important. Despite differences in variable selection and inclusion for each outcome assessed, overall model performance was very similar for all models.
Importance of Cancer Staging Variables
Cancer stage has clear importance for long-term recurrence and survival outcome assessment; yet, whether cancer stage influences short-term outcomes after colorectal resection is not established. In the current study, we observed that cancer variables from both data sets, and particularly cancer-specific variables from the NCDB, were important for certain outcomes. Similarly, our prior work indicated that T and N classification was not important in predicting 60-day mortality for any of 15 operations using NCDB data alone.28
Among all outcomes assessed, we observed that 30-day mortality was particularly influenced by cancer stage. Specifically, the ACS NSQIP disseminated cancer variable and the NCDB stage IV variable were statistically significant predictors of 30-day mortality after colorectal resection. These data suggest that, after controlling for other preoperative factors, such as functional status, ASA class, weight loss, and albumin, stage I through III or T-classification and N-classification variables generally are not important for 30-day mortality assessment; whereas the consideration of metastatic cancer is somewhat important. In addition, it should be recognized that, although the NCDB metastatic cancer variable was selected before the other staging variables, metrics of model quality did not differ meaningfully between any of the mortality models (AIC range, 1829-1835), indicating similar model-based predictive ability.
TNM staging elements were important in certain ACS NSQIP models, specifically, serious morbidity (T-classification), pneumonia (T-classification), and superficial SSI (N-classification). Yet, for all models, T and N classification did not appear to be substantially important, because it was selected very late compared with the other ACS NSQIP variables, indicating a lack of explanatory power of T and N classification in these models. For example, N-classification was selected at step 8 of 10 in the superficial SSI model, and T-classification was not included in the top 10 variables selected in the serious morbidity model. To our knowledge, this is the first study to systematically and comprehensively evaluate the influence of specific stage categories on short-term outcomes after colorectal resection for cancer.
Importance of Neoadjuvant Therapy Variables
Among patients with locally advanced, nonmetastatic rectal cancer, chemoradiotherapy followed by surgery has become the preferred treatment strategy.29 Some reports have linked the receipt of preoperative therapy with postoperative complications, such as wound occurrences, whereas other studies reported no such associations. For example, 2 recent investigations that used ACS NSQIP data and focused on laparoscopic versus open proctectomy did not identify an association between the receipt of neoadjuvant therapy and complications.30, 31 By comparison, other publications, including randomized trial data, have identified a significant association between neoadjuvant therapy and postoperative complications.32-34
We compared the importance of neoadjuvant therapy variables from the ACS NSQIP and the NCDB. Our findings indicated that neoadjuvant therapy variables from either data set were important for superficial SSI, but only the NCDB variable was important in serious morbidity and organ space SSI models. Similar to models that evaluated cancer stage variables, model quality was very similar if not identical whether the neoadjuvant therapy variable from either data set was selected or excluded.
An explanation for why the NCDB neoadjuvant therapy variable appeared to be preferentially selected likely rests in its definition. Until recently, the ACS NSQIP only considered a patient to have received preoperative radiotherapy if it was delivered within 90 days and chemotherapy if it was delivered within 30 days of surgery. By comparison, the NCDB considers patients to have received preoperative therapy if it is received as their first course of therapy for the index cancer. This difference was evident in the current study, in which 291 more patients (17%) in the NCDB were coded as having received preoperative treatment. Nevertheless, the current ACS NSQIP procedure targeted platform has expanded its preoperative time window to 90 days for both chemotherapy and radiation therapy, which will improve its capture of patients receiving neoadjuvant therapy and, perhaps, its importance in short-term outcome models.
Relevance of Cancer Variables on Model-Based Risk Prediction
The current findings are noteworthy. Improving predictive capability is an important aspect of surgical outcome assessment. Because data sources are limited to the variables they collect, it is important to search for and evaluate new variables and update definitions of old variables to optimize short-term outcome models. Historically, surgical quality-improvement programs, namely, the Veterans Affairs NSQIP and the ACS NSQIP, have focused on broad evaluations by measuring outcomes across many different procedure types (eg, general surgery, vascular surgery). In this context, more globally encompassing variables have a distinct purpose—that is, to account for unique patient-level risk across a wide spectrum of surgical procedures and indications. For example, the disseminated cancer variable originally was created to account for the patient with widespread cancer who was undergoing any surgical procedure. This was intentionally distinct from AJCC stage, which was designed to stratify patients for treatment and long-term prognostic purposes separately for each individual cancer. Although cancer-specific variables clearly are important for long-term outcomes (eg, survival, recurrence), it was unknown whether those variables could improve model-based predictions for short-term outcomes. We observed that, although certain cancer variables had statistically significant associations with many outcomes, model-based prediction was essentially unchanged by any of the cancer variables, suggesting that age, ASA class, comorbidities, and the other patient risk factors are considerably more important than disease stage and neoadjuvant therapy in predicting short-term outcomes after surgery. Therefore, for the identified outcomes assessed, we do not believe the tested cancer variables are necessary for the purposes of risk prediction among patients with colorectal cancer who undergo surgery.
These findings are directly applicable to the current ACS NSQIP modeling approach, which has been criticized for not accounting for cancer-specific variables.3 We demonstrate that the current ACS NSQIP method provides sufficient adjustment for short-term outcomes in patients with colorectal cancer. Nevertheless, the program is aware that additional variables may be important, and an Oncology NSQIP Working Group is currently further developing a system to optimize cancer surgery modeling and reporting. Future research and policy initiatives should consider this work when developing models for risk-adjusted cancer surgery outcomes.
Our results should be considered with regard to certain limitations. First, the methods used to assign variable importance included stepwise logistic regression. This method has been criticized because, theoretically, it may underestimate the importance of certain combinations of variables and may lead to model over fitting.35 Nevertheless, we used it in this study because it is the method generally used by the ACS NSQIP and because it standardized the manner in which variables were selected, thus allowing for variable comparisons between data sets. Second, 1 noteworthy difference between these methods and standard ACS NSQIP methods is that we restricted procedure-mix and operative indication to merge ACS NSQIP and NCDB data. By only considering cancer patients, patient-level homogeneity is increased,36 which may lead to alternative results compared with the standard ACS NSQIP colorectal model, which includes all indications for surgery. Third, there may be additional cancer-specific variables that were not investigated in the current study that are important in risk-adjusted, short-term outcomes. For example, T-classification may have substantial explanatory power for positive radial margin rates after rectal resection. In 2013, the ACS NSQIP will begin collecting additional cancer-specific variables that will allow additional assessment in subsequent years. However, the purpose of this study was to compare and contrast the importance of variables currently included in the ACS NSQIP with those from the NCDB. The ACS NSQIP oncology pilot will work to identify, define, and collect additional variables that we believe are important for risk adjustment. Finally, this study focused only on colon and rectal surgery, therefore; the findings cannot be generalized to other malignancies.
In conclusion, using linked ACS NSQIP and NCDB data, we observed that cancer variables from both data sets were important in many ACS NSQIP models. Overall, it appeared that advanced stage (eg, metastatic cancer) variables had the most explanatory power, and T-classification and N-classification were not consistently important in most risk-adjustment models. In addition, among the patients who underwent resection for rectal cancer, the receipt of neoadjuvant therapy appeared to be somewhat important for assessing postoperative outcomes. Finally, model-based quality statistics were similar irrespective of the cancer variables included, indicating similar model-based predictive ability.
This study was supported by the American Cancer Society, the Agency for Healthcare Research and Quality, and the Northwestern Institute for Comparative Effectiveness Research in Oncology.
CONFLICT OF INTEREST DISCLOSURES
The authors made no disclosures.