SEARCH

SEARCH BY CITATION

Keywords:

  • colorectal cancer;
  • outcomes;
  • risk-adjustment;
  • outcomes;
  • American College of Surgeons National Surgical Quality Improvement Program;
  • National Cancer Data Base;
  • cancer stage;
  • neoadjuvant therapy

Abstract

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. FUNDING SOURCES
  8. REFERENCES

BACKGROUND:

The American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) generally has not collected cancer-specific variables. Because increasing numbers of studies are using ACS NSQIP data to study cancer surgery, the objectives of the current study were 1) to examine differences between existing ACS NSQIP variables and cancer registry variables, and 2) to determine whether the addition of cancer-specific variables improves modeling of short-term outcomes.

METHODS:

Data from patients in the ACS NSQIP and National Cancer Data Base (NCDB) who underwent colorectal resection for cancer were linked (2006-2008). By using regression methods, the relative importance of cancer staging and neoadjuvant therapy variables were assessed along with their effects on morbidity, serious morbidity, and mortality.

RESULTS:

From 146 hospitals, 11,405 patients were identified who underwent surgery for colorectal cancer (colon, 85%; rectum, 15%). The NCDB metastatic cancer variable and the ACS NSQIP disseminated cancer variables agreed marginally (Cohen kappa coefficient, 0.454). For mortality, only the ACS NSQIP disseminated cancer variable and the NCDB stage IV variable were identified as important predictors; whereas the variables stage I through III, tumor (T)-classification, and lymph node (N)-classification were not selected. Cancer stage variables were inconsistently important for serious morbidity (stage IV, T-classification), superficial surgical site infection (N-classification), venous thromboembolism (metastatic cancer), and pneumonia (T-classification). With respect to neoadjuvant therapy, ACS NSQIP and NCDB variables agreed moderately (kappa, 0.570) and predicted superficial surgical site infection, serious morbidity, and organ space surgical site infection. The model fit was similar regardless of the inclusion of stage and neoadjuvant therapy variables.

CONCLUSIONS:

Although advanced disease stage and neoadjuvant therapy variables were predictors of short-term outcomes, their inclusion did not improve the models. Cancer 2013. © 2012 American Cancer Society.


INTRODUCTION

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. FUNDING SOURCES
  8. REFERENCES

Until recently, the American College of Surgeons (ACS) National Surgery Quality Improvement Program (NSQIP) measured outcomes across an array of broad surgical categories (eg general surgery, vascular surgery). Recently, the program has moved toward more targeted procedures and indications,1 and a consortium has been developed (ie, the Oncology NSQIP National Cancer Institute Center Consortium) that will focus only on surgical oncology cases.2 However, ACS NSQIP has been critiqued on its ability to predict cancer surgery outcomes,3, 4 partly because the program does not contain cancer-specific variables (eg, American Joint Committee on Cancer [AJCC] stage). Currently, ACS NSQIP only collects 2 cancer variables: whether the patient had disseminated cancer and whether the patient received chemotherapy or radiation within a limited time frame before surgery.

An increasing body of literature supports the need to validate and update prediction models. A model should include reliable and standardized data and variables that are clinically relevant to the population in which it is to be used.5-7 To date, no study has systematically evaluated the predictive capabilities of cancer-specific variables on short-term outcomes after cancer surgery.8 Because the ACS NSQIP has not collected cancer-specific variables, there has been no way to determine the effect of cancer variables on short-term outcomes. Therefore, we linked ACS NSQIP to the National Cancer Data Base (NCDB), the largest oncology registry in the United States. Our objectives were: 1) to compare the staging and neoadjuvant therapy variables in the ACS NSQIP and in the NCDB and 2) to assess the relative importance and predictive capabilities of cancer stage and neoadjuvant therapy variables in short-term outcome models after colorectal cancer surgery. We focused on colorectal cancer surgery because it is a common cancer procedure and is associated with appreciable morbidity and mortality, making it an ideal target for quality-improvement initiatives.9-11

MATERIALS AND METHODS

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. FUNDING SOURCES
  8. REFERENCES

Data Sources

The ACS NSQIP and the NCDB were linked to obtain a data set that contained patient demographics, comorbidities, and short-term outcomes data from ACS NSQIP with cancer-specific staging and treatment data from the NCDB. Details regarding both programs were described extensively in previous reports.12-15 In brief, the ACS NSQIP is a surgical quality-improvement program that originated in the Veterans Health Administration in 1994 and subsequently was transferred to the non-Veterans Affairs hospitals in 2001. The program is based on highly standardized and robust clinical data collected by trained and audited surgical clinical reviewers.16 Patients are followed for 30 days after their index procedure, and complications are captured irrespective of whether they occur during the patients' initial hospital stay or after discharge or whether they are readmitted to the same hospital or to another facility.

The NCDB is a joint program of the ACS Commission on Cancer, and the American Cancer Society. The NCDB is the largest cancer registry in the world, capturing >70% of all newly diagnosed malignancies in the United States from approximately 1500 hospitals.17 All Commission on Cancer-accredited hospitals are required to meet specific standards of care in the management of patients with cancer. Data are abstracted by trained cancer registrars who are periodically audited to ensure data reliability.

Study Cohort and Data Linkage

Patients were identified from the ACS NSQIP (2006-2008) if they underwent a colorectal resection for cancer based on Current Procedure Terminology18 (CPT) and International Classification of Disease, Ninth Revision19 (ICD-9) codes. Patients from the NCDB with colorectal cancer based on International Classification of Diseases for Oncology, Third Revision20 (ICD-O-3) codes who underwent resection were identified. Because the ACS NSQIP and NCDB datasets do not contain unique patient-level identifiers, patients were linked using a probabilistic matching algorithm based on the following variables: American Hospital Association number, patient age, sex, and date of operation. To focus on elective cancer surgery, patients who underwent emergency operations or who were classified as ASA class V (ie, moribund) were excluded from the analysis.

American College of Surgeons National Surgical Quality Improvement Program and National Cancer Data Base Variables

The ACS NSQIP collects >130 preoperative, operative, and postoperative outcome variables, including 2 cancer-related variables: the presence of preoperative disseminated cancer and preoperative chemotherapy (within 30 days of surgery) or radiotherapy (within 90 days of surgery). Neoadjuvant therapy variables were examined only for rectal cancer, and, for the purposes of the current analysis, these variables were combined. Other patient demographic variables that were considered have been extensively described previously and are identified in Table 1.21

Table 1. Characteristics of 11,405 Patients Undergoing Colorectal Resection for Cancer: 2006 to 2008
CharacteristicNo. of Patients (%)a
  • Abbreviations: AJCC, American Joint Committee on Cancer; ASA, American Society of Anesthesiology; BMI, body mass index; CAD, cardiovascular disease; CFH, congestive heart failure; COPD, chronic obstructive pulmonary disease; CRT, chemoradiation; HTN, hypertension; IQR, interquartile range; PVD, peripheral vascular disease.

  • a

    Values were missing for BMI in 113 patients (1.0%), and for AJCC stage in 680 patients(5.9%).

  • b

    Data for this variable were obtained from the National Cancer Data Base.

  • c

    This applied only to patients with rectal cancer.

Age: Median [IQR], y63 [53-73]
Sex: Women5532 (48.5)
Race 
 White8802 (77.2)
 Black1065 (9.3)
 Hispanic277 (2.4)
 Other1261 (11.1)
ASA class 
 I/II5276 (46.3)
 III5551 (48.7)
 IV/V578 (5.0)
Functional status 
 Independent10,708 (93.9)
 Partially dependent614 (5.4)
 Totally dependent83 (0.7)
Weight loss737 (6.5)
Albumin <3.0 g/dL848 (7.4)
BMI 
 Underweight344 (3.0)
 Normal3589 (31.8)
 Overweight3873 (34.3)
 Obese3486 (30.9)
Cardiovascular 
 Previous cardiac event1534 (13.5)
 CHF150 (1.3)
 HTN6418 (56.3)
 PVD167 (1.5)
Pulmonary 
 Dyspnea at rest130 (1.1)
 Current smoker1771 (15.5)
 COPD685 (6.0)
Preoperative sepsis364 (3.2)
Diabetes 
 Oral1399 (12.3)
 Insulin dependent598 (5.2)
Ascites131 (1.2)
Renal failure53 (0.5)
Steroid use292 (2.6)
Procedure type 
 Open colectomy6260 (54.9)
 Laparoscopic colectomy3418 (30)
 Total proctocolectomy178 (1.6)
 Open proctectomy816 (7.1)
 Laparoscopic proctectomy149 (1.3)
 Abdominoperineal resection584 (5.1)
AJCC stageb 
 I3694 (30.7)
 II3151 (29.4)
 III3160 (29.5)
 IV1111 (10.4)
Neoadjuvant CRTb,c913 (52.9)

To evaluate and compare cancer variables common to both the ACS NSQIP and the NCDB, we evaluated 2 variables from the NCDB: cancer stage and neoadjuvant chemoradiation. Cancer stage from the NCDB is based on the AJCC staging schema and was updated to reflect the most recent edition.22 To systematically evaluate the importance of specific aspects of cancer staging, AJCC stage was categorized by creating the following groups: 1) metastatic cancer (stage IV vs stages I-III combined), 2) AJCC stage (stages I-IV), and 3) individual tumor (T) classification (T1-T4), lymph node (N)-classification (N0-N2), and metastasis (M)-classification (M0-M1). The NCDB defines chemoradiotherapy use based on Facility Oncology Registry Data Standards23 and captures preoperative therapy if it is received as the first course therapy for an index cancer.

Outcomes

The ACS NSQIP collects more than twenty 30-day postoperative complication variables, which have been defined elsewhere.21 To broadly evaluate the importance of cancer variables in ACS NSQIP complication models, the following outcomes were selected: mortality, serious morbidity, superficial surgical site infection (SSI), organ space infection, wound dehiscence, venous thromboembolism (VTE), and pneumonia. Serious morbidity included any of the following ACS NSQIP-defined postoperative complications: cardiac arrest requiring cardiopulmonary resuscitation, myocardial infarction, deep vein thrombosis, pulmonary embolism, sepsis/septic shock, deep incisional SSI, organ space SSI, wound dehiscence, unplanned reintubation, pneumonia, renal failure/progressive renal insufficiency, urinary tract infection, and unplanned reoperation. Patients were precluded from being categorized with postoperative pneumonia, urinary tract infection, renal failure/insufficiency, reintubation, or a prolonged ventilation adverse event if a related condition was present at the time of the index operation.

Statistical Analysis

The predictive strength of cancer staging variables was examined in patients with colorectal cancer, whereas neoadjuvant chemoradiotherapy variables were evaluated only in patients with rectal cancer. Variable agreement between the ACS NSQIP and NCDB data sets was compared using the Cohen kappa coefficient to assess whether variable agreement (ie, similar in ACS NSQIP and NCDB) was poor (kappa, <0.2), fair (kappa, 0.2-0.4), moderate (kappa, 0.4-0.6), or very good (kappa, >0.6). By using the ACS NSQIP modeling approach,12 multivariable logistic regression models were developed using a forward stepwise selection process (entrance criteria, P < .05). First, to evaluate the importance of cancer stage among all patients in the study, 4 models were developed. In addition to standard ACS NSQIP risk-adjustment variables, each of the 4 models included 1 of the 4 cancer stage variables (ie, disseminated cancer [from the NSQIP], and metastatic cancer, AJCC stage, TNM stage [from the NCDB]). Second, for each outcome assessed, 2 additional models were developed among patients who underwent rectal surgery that separately included the neoadjuvant chemoradiotherapy variables from each database. For this study, we used a forward stepwise modeling process, such that the order of variable selection could be assessed and compared (variables that were selected earlier have stronger model explanatory power and are considered “more important”). If variables are selected, they are deemed to add explanatory power to the model; whereas, if the variable is not selected, then that variable is not important for that model. Odds ratios (ORs) and 95% confidence intervals (CIs) were included for the mortality and serious morbidity models. Cancer variables were tested for significant interaction between other explanatory variables based on a priori clinical knowledge. Models also were assessed by evaluating model discrimination (c-statistic), calibration (Hosmer-Lemeshow chi-square), and a combined discrimination and calibration metric (Akaike information criterion [AIC]).24, 35 All statistical analyses were performed using the SAS version 9.3 software package (SAS Institute Inc., Cary, NC). Northwestern University Institutional Review Board approved this study.

RESULTS

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. FUNDING SOURCES
  8. REFERENCES

From 146 hospitals, 11,405 patients were identified who underwent surgery for colorectal cancer (colon, 85%; rectum, 15%). In merging the patients from the ACS NSQIP with those in the NCDB, 83.1% of eligible patients in the ACS NSQIP were identified in the NCDB. The 3 most common operations performed were open colectomy (55% of patients), laparoscopic colectomy (30% of patients), and open proctectomy (7% of patients). Patient characteristics are listed in Table 1.

Importance of Cancer Staging Variables

Among all patients with colorectal cancer, 761 patients (6.7%) from the ACS NSQIP had preoperative disseminated cancer reported, and 1111 patients (10.4%) from the NCDB had metastatic cancer (kappa, 0.454), as would be expected given their somewhat different definitions. On multivariable analysis, both ACS NSQIP and NCDB cancer variables were important in certain complication models (Table 2). For mortality, the ACS NSQIP disseminated cancer variable (OR, 2.11; 95% CI, 1.40-3.17) and the NCDB metastatic cancer variables (stage IV vs. stage I-III: OR, 2.18; 95% CI, 1.51-3.15; stage IV vs stage I: OR, 2.28; 95% CI, 1.39-3.75) were included in the models (examined separately). The NCDB metastatic cancer variable was selected earlier (step 5 of 15) in the logistic regression, forward stepwise process compared with model testing the ACS NSQIP disseminated cancer variable (step 9 of 15) and the overall AJCC 4-level cancer stage variable (step 8 of 15). For serious morbidity, only NCDB cancer variables were selected. T-classification (step 13 of 18) was selected as important, but only T4 versus T1 was statistically significant (OR, 1.46; 95% CI, 1.17-1.81). Metastatic cancer (M1 vs M0) was selected at step 16 of 18. With respect to wound occurrences, the only variable selected was NCDB AJCC N-classification for superficial SSI, whereas neither ACS NSQIP nor NCDB variables were selected in organ space SSI or wound dehiscence models. For venous thromboembolism, only the NCDB metastatic cancer variable was selected (step 3 of 7); and, for pneumonia, only the NCDB T-classification variable was selected (step 10 of 14). Across all cancer stage variables and complication models, c-statistics, Hosmer-Lemeshow chi-square values, and AIC values were very similar, if not identical.

Table 2. The Importance of Cancer Staging Variables From the National Surgical Quality Improvement Program and the National Cancer Data Base in American College of Surgeons National Surgical Quality Improvement Program Complication Models After Colorectal Resection for Cancer (n = 11,405)
VariableMortalitySerious MorbiditySuperficial SSIOrgan Space SSIWound DehiscenceVTEPneumonia
  • Abbreviations: AIC, Akaike information criterion; AJCC, American Joint Committee on Cancer; H-L, Hosmer-Lemeshow chi-square statistic; NCDB, National Cancer Data Base; NSQIP, National Surgical Quality Improvement Program; SSI, surgical site infection; VTE, venous thromboembolism.

  • a

    Steps were selected in a logistic, forward stepwise regression process (entrance criteria, P < .05).

  • b

    If the step was earlier (lower), then that variable was more important.

  • c

    Only stage IV (vs stage I) was statistically significant.

  • d

    Only T3 and T4 tumors were significant (vs T1).

  • e

    Only T4 tumors were significant (vs T1).

  • f

    Only N1 status was significant (vs N0).

NSQIP       
 Disseminated cancer       
  Step selectedStep 9 of 15aNot selectedNot selectedNot selectedNot selectedNot selectedNot selected
   C-statistic0.8540.6600.6240.6440.7460.6580.752
   H-L5.937.173.1414.3910.542.4710.70
   AIC1835.29952.16420.04300.81585.01389.92678.3
NCDB       
 Metastatic cancer       
  Step selectedbStep 5 of 15aStep 16 of 18aNot selectedNot selectedNot selectedStep 3 of 7aNot selected
   C-statistic0.8560.6630.6240.6440.7460.6830.752
   H-L6.715.413.1414.3910.544.7310.70
   AIC1829.69948.76420.04300.81585.01386.42678.30
AJCC stage (I-IV)       
  Step selectedbStep 8 of 15a,cNot selectedNot selectedNot selectedNot selectedNot selectedNot selected
   C-statistic0.8570.6600.6240.6440.7460.6580.752
   H-L7.087.173.1414.3910.542.4710.70
   AIC1831.99952.16420.04300.81585.01389.92678.3
AJCC TNM stage       
 T-classification       
  Step selectedbNot selectedStep 13 of 18a,dNot selectedNot selectedNot selectedNot selectedStep 10 of 14a,e
 N-classification       
  Step selectedbNot selectedNot selectedStep 8 of 10a,fNot selectedNot selectedNot selectedNot selected
 M-classification       
  Step selectedbStep 5 of 15aNot selectedNot selectedNot selectedNot selectedStep 3 of 7aNot selected
   C-statistic0.8560.6240.6260.6440.7460.6830.755
   H-L6.717.173.3614.3910.544.7310.55
   AIC1829.610,074.26416.24300.81585.01386.42686.6

Importance of Neoadjuvant Therapy Variables

To examine neoadjuvant therapy variables, patients who underwent resection for rectal cancer (n = 1727) were identified. On the basis of somewhat different definitions for neoadjuvant therapy receipt, 622 patients (36%) from the ACS NSQIP and 913 patients (52.9%) from the NCDB were identified as having received neoadjuvant therapy (kappa, 0.570). Among patients with stage II/III disease, 45.1% in the ACS NSQIP versus 65.3% in the NCDB were identified as having received neoadjuvant therapy.

On multivariable analysis, both ACS NSQIP and NCDB neoadjuvant therapy variables were important in certain models. Neoadjuvant therapy was not important for 30-day mortality (ie, it was not selected in the model). For serious morbidity, only the NCDB neoadjuvant therapy variable (OR, 1.55; 95% CI, 1.33-1.81) was important, but it was selected at step 10 of 11 (Table 3). For superficial SSI, both ACS NSQIP and NCDB neoadjuvant variables were selected at the same early step in the forward stepwise selection process (step 2 of 7); whereas, for organ space infection, only the NCDB neoadjuvant therapy variable was included. Receipt of neoadjuvant therapy was not selected in wound dehiscence, VTE, or pneumonia models. Across all models, whether the neoadjuvant therapy variable from either data set was selected or excluded, the model fit was very similar if not identical (Table 3).

Table 3. Importance of Neoadjuvant Chemoradiotherapy Variables from the American College of Surgeons National Surgical Quality Improvement Program and the National Cancer Data Base in American College of Surgeons National Surgical Quality Improvement Program Complication Models After Rectal Resection for Cancer (n = 1727)
VariableMortalitySerious MorbiditySuperficial SSIOrgan Space SSIWound DehiscenceVTEPneumonia
  • Abbreviations: AIC, Akaike information criterion; H-L, Hosmer-Lemeshow chi-square statistic; NCDB, National Cancer Data Base; CRT, chemoradiotherapy; NSQIP, National Surgical Quality Improvement Program; SSI, surgical site infection; VTE, venous thromboembolism.

  • a

    Neoadjuvant CRT was defined by the NSQIP as receipt of chemotherapy within 30 days of surgery and receipt of radiotherapy within 90 days of surgery.

  • b

    If the step was earlier (lower), then that variable was more important.

  • c

    Steps were selected in a logistic, forward stepwise regression process (entrance criteria, P < .05).

  • d

    Neoadjuvant CRT was defined by the NCDB as CRT as part of first course therapy for the index cancer.

NSQIP       
 Neoadjuvant CRTa       
  Step selectedbNot selectedNot selectedStep 2 of 7cNot selectedNot selectedNot selectedNot selected
  C-statistic0.9030.6710.7040.5660.7070.6820.750
  H-L15.873.978.640.0020.7101.743.6
  AIC117.91670.01156.5681.0291.4231.1336.6
NCDB       
 Neoadjuvant CRTd       
  Step selectedbNot selectedStep 10 of 11cStep 2 of 7cStep 3 of 3cNot selectedNot selectedNot selected
  C-statistic0.9030.6750.7010.5920.7070.6820.750
  H-L15.872.286.720.190.7101.743.6
  AIC117.91483.61153.9679.0291.4231.1336.6

DISCUSSION

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. FUNDING SOURCES
  8. REFERENCES

The linkage of colorectal cancer patients from the ACS NSQIP and NCDB data created a data set that united rich clinical data (ACS NSQIP) with cancer-specific treatment and stage information (NCDB). This data set represents a unique and powerful tool with distinct advantages over administrative or cancer registry data alone. Unlike administrative or cancer registry data, the ACS NSQIP includes detailed preoperative patient details and postoperative complication information collected by trained clinical data abstractors using standardized definitions. Short-term outcomes are either unavailable or unreliable in administrative and cancer registry data sources.25, 26, 27 In addition, cancer-specific data in the NCDB are collected by the same cancer registrar using the same coding manuals used by the better known Surveillance, Epidemiology, and End Results Program. However, the NCDB is not limited to select geographic locations like the Surveillance, Epidemiology, and End Results Program and has several enhancements, including the collection of chemotherapy information.17 By using this merged data set, we observed that metastatic cancer and neoadjuvant therapy variables were somewhat important for assessing short-term outcomes after cancer surgery, whereas T-classification and N-classification were not consistently important. Despite differences in variable selection and inclusion for each outcome assessed, overall model performance was very similar for all models.

Importance of Cancer Staging Variables

Cancer stage has clear importance for long-term recurrence and survival outcome assessment; yet, whether cancer stage influences short-term outcomes after colorectal resection is not established. In the current study, we observed that cancer variables from both data sets, and particularly cancer-specific variables from the NCDB, were important for certain outcomes. Similarly, our prior work indicated that T and N classification was not important in predicting 60-day mortality for any of 15 operations using NCDB data alone.28

Among all outcomes assessed, we observed that 30-day mortality was particularly influenced by cancer stage. Specifically, the ACS NSQIP disseminated cancer variable and the NCDB stage IV variable were statistically significant predictors of 30-day mortality after colorectal resection. These data suggest that, after controlling for other preoperative factors, such as functional status, ASA class, weight loss, and albumin, stage I through III or T-classification and N-classification variables generally are not important for 30-day mortality assessment; whereas the consideration of metastatic cancer is somewhat important. In addition, it should be recognized that, although the NCDB metastatic cancer variable was selected before the other staging variables, metrics of model quality did not differ meaningfully between any of the mortality models (AIC range, 1829-1835), indicating similar model-based predictive ability.

TNM staging elements were important in certain ACS NSQIP models, specifically, serious morbidity (T-classification), pneumonia (T-classification), and superficial SSI (N-classification). Yet, for all models, T and N classification did not appear to be substantially important, because it was selected very late compared with the other ACS NSQIP variables, indicating a lack of explanatory power of T and N classification in these models. For example, N-classification was selected at step 8 of 10 in the superficial SSI model, and T-classification was not included in the top 10 variables selected in the serious morbidity model. To our knowledge, this is the first study to systematically and comprehensively evaluate the influence of specific stage categories on short-term outcomes after colorectal resection for cancer.

Importance of Neoadjuvant Therapy Variables

Among patients with locally advanced, nonmetastatic rectal cancer, chemoradiotherapy followed by surgery has become the preferred treatment strategy.29 Some reports have linked the receipt of preoperative therapy with postoperative complications, such as wound occurrences, whereas other studies reported no such associations. For example, 2 recent investigations that used ACS NSQIP data and focused on laparoscopic versus open proctectomy did not identify an association between the receipt of neoadjuvant therapy and complications.30, 31 By comparison, other publications, including randomized trial data, have identified a significant association between neoadjuvant therapy and postoperative complications.32-34

We compared the importance of neoadjuvant therapy variables from the ACS NSQIP and the NCDB. Our findings indicated that neoadjuvant therapy variables from either data set were important for superficial SSI, but only the NCDB variable was important in serious morbidity and organ space SSI models. Similar to models that evaluated cancer stage variables, model quality was very similar if not identical whether the neoadjuvant therapy variable from either data set was selected or excluded.

An explanation for why the NCDB neoadjuvant therapy variable appeared to be preferentially selected likely rests in its definition. Until recently, the ACS NSQIP only considered a patient to have received preoperative radiotherapy if it was delivered within 90 days and chemotherapy if it was delivered within 30 days of surgery. By comparison, the NCDB considers patients to have received preoperative therapy if it is received as their first course of therapy for the index cancer. This difference was evident in the current study, in which 291 more patients (17%) in the NCDB were coded as having received preoperative treatment. Nevertheless, the current ACS NSQIP procedure targeted platform has expanded its preoperative time window to 90 days for both chemotherapy and radiation therapy, which will improve its capture of patients receiving neoadjuvant therapy and, perhaps, its importance in short-term outcome models.

Relevance of Cancer Variables on Model-Based Risk Prediction

The current findings are noteworthy. Improving predictive capability is an important aspect of surgical outcome assessment. Because data sources are limited to the variables they collect, it is important to search for and evaluate new variables and update definitions of old variables to optimize short-term outcome models. Historically, surgical quality-improvement programs, namely, the Veterans Affairs NSQIP and the ACS NSQIP, have focused on broad evaluations by measuring outcomes across many different procedure types (eg, general surgery, vascular surgery). In this context, more globally encompassing variables have a distinct purpose—that is, to account for unique patient-level risk across a wide spectrum of surgical procedures and indications. For example, the disseminated cancer variable originally was created to account for the patient with widespread cancer who was undergoing any surgical procedure. This was intentionally distinct from AJCC stage, which was designed to stratify patients for treatment and long-term prognostic purposes separately for each individual cancer. Although cancer-specific variables clearly are important for long-term outcomes (eg, survival, recurrence), it was unknown whether those variables could improve model-based predictions for short-term outcomes. We observed that, although certain cancer variables had statistically significant associations with many outcomes, model-based prediction was essentially unchanged by any of the cancer variables, suggesting that age, ASA class, comorbidities, and the other patient risk factors are considerably more important than disease stage and neoadjuvant therapy in predicting short-term outcomes after surgery. Therefore, for the identified outcomes assessed, we do not believe the tested cancer variables are necessary for the purposes of risk prediction among patients with colorectal cancer who undergo surgery.

These findings are directly applicable to the current ACS NSQIP modeling approach, which has been criticized for not accounting for cancer-specific variables.3 We demonstrate that the current ACS NSQIP method provides sufficient adjustment for short-term outcomes in patients with colorectal cancer. Nevertheless, the program is aware that additional variables may be important, and an Oncology NSQIP Working Group is currently further developing a system to optimize cancer surgery modeling and reporting. Future research and policy initiatives should consider this work when developing models for risk-adjusted cancer surgery outcomes.

Limitations

Our results should be considered with regard to certain limitations. First, the methods used to assign variable importance included stepwise logistic regression. This method has been criticized because, theoretically, it may underestimate the importance of certain combinations of variables and may lead to model over fitting.35 Nevertheless, we used it in this study because it is the method generally used by the ACS NSQIP and because it standardized the manner in which variables were selected, thus allowing for variable comparisons between data sets. Second, 1 noteworthy difference between these methods and standard ACS NSQIP methods is that we restricted procedure-mix and operative indication to merge ACS NSQIP and NCDB data. By only considering cancer patients, patient-level homogeneity is increased,36 which may lead to alternative results compared with the standard ACS NSQIP colorectal model, which includes all indications for surgery. Third, there may be additional cancer-specific variables that were not investigated in the current study that are important in risk-adjusted, short-term outcomes. For example, T-classification may have substantial explanatory power for positive radial margin rates after rectal resection. In 2013, the ACS NSQIP will begin collecting additional cancer-specific variables that will allow additional assessment in subsequent years. However, the purpose of this study was to compare and contrast the importance of variables currently included in the ACS NSQIP with those from the NCDB. The ACS NSQIP oncology pilot will work to identify, define, and collect additional variables that we believe are important for risk adjustment. Finally, this study focused only on colon and rectal surgery, therefore; the findings cannot be generalized to other malignancies.

In conclusion, using linked ACS NSQIP and NCDB data, we observed that cancer variables from both data sets were important in many ACS NSQIP models. Overall, it appeared that advanced stage (eg, metastatic cancer) variables had the most explanatory power, and T-classification and N-classification were not consistently important in most risk-adjustment models. In addition, among the patients who underwent resection for rectal cancer, the receipt of neoadjuvant therapy appeared to be somewhat important for assessing postoperative outcomes. Finally, model-based quality statistics were similar irrespective of the cancer variables included, indicating similar model-based predictive ability.

FUNDING SOURCES

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. FUNDING SOURCES
  8. REFERENCES

This study was supported by the American Cancer Society, the Agency for Healthcare Research and Quality, and the Northwestern Institute for Comparative Effectiveness Research in Oncology.

CONFLICT OF INTEREST DISCLOSURES

The authors made no disclosures.

REFERENCES

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. FUNDING SOURCES
  8. REFERENCES
  • 1
    American College of Surgeons, National Surgical Quality Improvement Program. Semiannual Report, July 2011. Chicago, IL: American College of Surgeons; 2011.
  • 2
    Merkow RP, Bilimoria KY. Currently available quality improvement initiatives in surgical oncology. Surgical oncology clinics of North America. 2012;21:367375, vii.
  • 3
    Borja-Cacho D, Parsons HM, Habermann EB, Rothenberger DA, Henderson WG, Al-Refaie WB. Assessment of ACS NSQIP's predictive ability for adverse events after major cancer surgery. Ann Surg Oncol. 2010; 17: 2274-2282.
  • 4
    Shah R, Velanovich V, Syed Z, Swartz A, Rubinfeld I. Limitations of patient-associated co-morbidity model in predicting postoperative morbidity and mortality in pancreatic operations. J Gastrointest Surg. 2012; 16: 986-992.
  • 5
    Kansagara D, Englander H, Salanitro A, et al. Risk prediction models for hospital readmission: a systematic review. JAMA. 2011; 306: 1688-1698.
  • 6
    Mu Y, Edwards JR, Horan TC, Berrios-Torres SI, Fridkin SK. Improving risk-adjusted measures of surgical site infection for the national healthcare safety network. Infect Control Hosp Epidemiol. 2011; 32: 970-986.
  • 7
    Moons KG, Kengne AP, Grobbee DE, et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart. 2012; 98: 691-698.
  • 8
    Spinks TE, Walters R, Feeley TW, et al. Improving cancer care through public reporting of meaningful quality measures. Health Aff (Millwood). 2011; 30: 664-672.
  • 9
    Schilling PL, Dimick JB, Birkmeyer JD. Prioritizing quality improvement in general surgery. J Am Coll Surg. 2008; 207: 698-704.
  • 10
    Cohen ME, Bilimoria KY, Ko CY, Hall BL. Development of an American College of Surgeons National Surgery Quality Improvement Program: morbidity and mortality risk calculator for colorectal surgery. J Am Coll Surg. 2009; 208: 1009-1016.
  • 11
    The National Quality Forum. Risk-Adjusted Colon Surgery Outcome Measure (NQF 0706). Available at: http://www.qualityforum.org/Home.aspx. Accessed October 5, 2011.
  • 12
    Raval MV, Cohen ME, Ingraham AM, et al. Improving American College of Surgeons National Surgical Quality Improvement Program risk adjustment: incorporation of a novel procedure risk score. J Am Coll Surg. 2010; 211: 715-723.
  • 13
    Khuri SF, Henderson WG, Daley J, et al. Successful implementation of the Department of Veterans Affairs' National Surgical Quality Improvement Program in the private sector: the Patient Safety in Surgery study. Ann Surg. 2008; 248: 329-336.
  • 14
    Khuri SF. The NSQIP: a new frontier in surgery. Surgery. 2005; 138: 837-843.
  • 15
    Khuri SF. Safety, quality, and the National Surgical Quality Improvement Program. Am Surg. 2006; 72: 994-998; discussion 1021-1030, 1133-1048.
  • 16
    Shiloach M, Frencher SK Jr, Steeger JE, et al. Toward robust information: data quality and inter-rater reliability in the American College of Surgeons National Surgical Quality Improvement Program. J Am Coll Surg. 2010; 210: 6-16.
  • 17
    Bilimoria KY, Stewart AK, Winchester DP, Ko CY. The National Cancer Data Base: a powerful initiative to improve cancer care in the United States. Ann Surg Oncol. 2008; 15: 683-690.
  • 18
    American Medical Association. CPT 2011: Current Procedural Terminology. Chicago, IL: American Medical Association; 2011.
  • 19
    Department of Health and Human Services. The International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9 CM). Washington, DC: Government Printing Office; 1998.
  • 20
    World Health Organization. International Classification of Disease for Oncology. 3rd ed. Geneva, Switzerland: World Health Organization; 2000.
  • 21
    American College of Surgeons. National Surgical Quality Improvement Program. Chicago, IL: American College of Surgeons; 2011. Available at http://www.facs.org/acsnsqip.html. Accessed January 1, 2012.
  • 22
    Compton CC, Fritz AG, Greene FL, Trotti A, eds. AJCC Cancer Staging Manual. 7th ed. New York: Springer, 2010.
  • 23
    Commission on Cancer. Facility Oncology Registry Data Standards. Chicago, IL: Commission on Cancer; 2003-2011. Available at: http://www.facs.org/cancer/ncdb/cocmanuals.html. Accessed April 15, 2011.
  • 24
    Hosmer DW, Lemeshow S. Applied Logistic Regression. 2nd ed. New York: John Wiley & Sons, Inc.; 2000.
  • 25
    Koch CG, Li L, Hixson E, Tang A, Phillips S, Henderson JM. What are the real rates of postoperative complications: elucidating inconsistencies between administrative and clinical data sources. J Am Coll Surg. 2012; 214: 798-805.
  • 26
    Steinberg SM, Popa MR, Michalek JA, Bethel MJ, Ellison EC. Comparison of risk adjustment methodologies in surgical quality improvement. Surgery. 2008; 144: 662-667; discussion 662-667.
  • 27
    Lawson EH, Louie R, Zingmond DS, et al. A Comparison of Clinical Registry Versus Administrative Claims Data for Reporting of 30-Day Surgical Complications. Annals of surgery. 2012.
  • 28
    Bilimoria KY, Bentrem DJ, Talamonti MS, Stewart AK, Winchester DP, Ko CY. Risk-based selective referral for cancer surgery: a potential strategy to improve perioperative outcomes. Ann Surg. 2010; 251: 708-716.
  • 29
    National Comprehensive Cancer Network (NCCN). NCCN Guidelines in Oncology: Rectal Cancer. Version 1.2012. Fort Washington, PA: NCCN; 2011. Available at: http://www.nccn.org. Accessed October 1, 2011.
  • 30
    El-Gazzaz G, Kiran RP, Lavery I. Wound complications in rectal cancer patients undergoing primary closure of the perineal wound after abdominoperineal resection. Dis Colon Rectum. 2009; 52: 1962-1966.
  • 31
    Greenblatt DY, Rajamanickam V, Pugely AJ, Heise CP, Foley EF, Kennedy GD. Short-term outcomes after laparoscopic-assisted proctectomy for rectal cancer: results from the ACS NSQIP. J Am Coll Surg. 2011; 212: 844-854.
  • 32
    Kapiteijn E, Marijnen CA, Nagtegaal ID, et al. Preoperative radiotherapy combined with total mesorectal excision for resectable rectal cancer. N Engl J Med. 2001; 345: 638-646.
  • 33
    Bullard KM, Trudel JL, Baxter NN, Rothenberger DA. Primary perineal wound closure after preoperative radiotherapy and abdominoperineal resection has a high incidence of wound failure. Dis Colon Rectum. 2005; 48: 438-443.
  • 34
    Degrate L, Garancini M, Misani M, et al. Right colon, left colon, and rectal surgeries are not similar for surgical site infection development. Analysis of 277 elective and urgent colorectal resections. Int J Colorectal Dis. 2011; 26: 61-69.
  • 35
    Livingston E, Cao J, Dimick JB. Tread carefully with stepwise regression. Arch Surg. 2010; 145: 1039-1040.
  • 36
    Merkow RP, Hall BL, Cohen ME, et al. Relevance of the c-statistic in risk-adjustment models in surgery. J Am Coll Surg. 2012; 214: 822-830.