External validation of a pre-operative nomogram predicting peri-operative mortality risk after liver resections for malignancy


  • This paper was presented at the American Hepato-Pancreato-Biliary Association 11th Annual Meeting, Miami, FL, USA.

Chandrakanth Are, Associate Professor, Division of Surgical Oncology, Department of Surgery/Genetics, Cell Biology and Anatomy, Program Director – General Surgery Residency, University of Nebraska Medical Center, Omaha, NE 68198, USA. Tel: 402 559 8941. Fax: 402 559 7900. E-mail: care@unmc.edu


Aim:  A pre-operative nomogram using a population-based database to predict peri-operative mortality risk after liver resections for malignancy has recently been developed. The aim of the present study was to perform an external validation of the nomogram using data from a high volume institution.

Methods:  The National Inpatient Sample (NIS) database (2000–2004) was used initially to construct the nomogram. The dataset for external validation was obtained from a high volume centre specializing in hepatobiliary surgery. Validation was performed using calibration plots and concordance index.

Results:  A total of 794 patients who underwent liver resection from the years 2000–2010 at the external institute were included in the validation set with an observed mortality rate of 1.6%. The mean total points for this sample of patients was 124.9 [standard error (SE) 1.8, range 0–383] which translates to a nomogram predicted mortality rate of 1.5%, similar to the actual observed overall mortality rate. The nomogram concordance index was 0.65 [95% confidence interval (CI) 0.46–0.82] and calibration plots stratified by quartiles revealed good agreement between the predicted and observed mortality rates.

Conclusions:  The present study provides an external validation of the pre-operative nomogram to predict the risk of peri-operative mortality after liver resection for malignancy.


Hepatic resection is a well-accepted modality in the treatment algorithm of patients with primary and secondary malignancies involving the liver. Several previous studies have demonstrated that the number of hepatic resections being performed for malignancy have significantly increased over the past decade.1,2 In spite of the increasing number of procedures being performed, hepatic resections are major operations that are associated with significant morbidity and mortality.2–4 The majority of these operations are also performed in middle-aged to elderly individuals who have multiple pre-existing co-morbidities. The pre-operative counselling to discuss and determine the likely rate of peri-operative mortality associated with these high-risk procedures remains an important part of the management algorithm.

Several recent studies have proposed different tools to predict peri-operative outcomes after hepatic resections.5–10 A nomogram was recently devised by our institute using easily available pre-operative variables to enable prediction of peri-operative mortality after hepatic resections for malignancy.11 This nomogram was constructed from data derived from a national database comprising heterogenous institutions and low to high volume surgeons. It is not clear whether predictive tools developed from this database are reliable and accurate when applied to specific centres. The aim of the present study was, therefore, to undertake an external validation of a population-based dataset derived nomogram predicting mortality after liver resection for malignancy by utilizing data derived from a high volume single academic centre.


The National Inpatient Sample(NIS) database (2000–2004) was initially used to develop the nomogram which included age, race, gender, liver primary, coagulopathy, renal failure, congestive heart failure (CHF), cardiac arrhythmias and other major co-morbidities (Fig. 1).11 The dataset for external validation was obtained from the University of Pittsburgh Liver Cancer Centre. Data were obtained from a prospectively maintained dataset of all patients undergoing liver resections for malignancy at this institute. Peri-operative mortality for both datasets was defined as the mortality during the same hospital admission.

Figure 1.

Representative figure of the nomgram (published with permission from Springerlink. Originally published in Dhir et al. J Gastrointest Surg 2010)11. CHF, Congestive heart failure; COPD, Chronic obstructive pulmonary disease; Unk, Unknown

Statistical methods

SAS software (SAS Institute Inc., Cary, NC, USA) was used for all statistical analysis. The nomogram was initially constructed using the previously described techniques, using the NIS dataset from 2000 to 2004.12,13 Each variable was assigned points based the multivariate logistic regression. Depending on the number of variables/factors present in the case of an individual patient, the total number of points was calculated for each person in the NIS 2000 to 2004 dataset. The median total points for this dataset were 116 with a range of 0 to 469 which corresponds to a mortality rate of 1.3%.11 The overall observed mortality rate in this (NIS 2000–2004) dataset was 4.1%.11

The distribution of patient characteristics in the external validation dataset was compared with the populate values estimated from the NIS dataset using one sample tests for proportions, using a two-sided exact test. Validation was performed using data derived from the external institute utilizing calibration plots and concordance index. Briefly, the concordance index was calculated by comparing the patients who died to those who were alive. All possible pairs were constructed between dead and alive patients. For each pair, if the nomogram assigned a higher probability of death to the patient who died compared with the ones alive, then the model matched the data and the pair was said to be concordant. The concordance index is the probability of being concordant out of all possible dead/alive patient pairs. A 95% confidence interval (CI) was calculated for the concordance index based on 10 000 bootstrapped samples. A calibration plot was constructed by plotting predicted probabilities from the nomogram versus the actual probabilities. Quartiles of the predicted probabilities were delineated and observed mortality proportions were determined for the quartiles along with 95% CIs, and plotted. A perfectly predictive nomogram should result in the observed and expected probabilities aligned along a 45 degree line.


A total of 795 patients underwent liver resections for malignancy from 2000–2010. One person was excluded from the analysis as data on in-hospital mortality were not available. Median age for all patients was 65 years [standard deviation (SD) 12.5, range 18–92]. Approximately half (445/794, 56%) of the patients were males and the median length of stay was 6 days (range 0–39). The distributions of the relevant variables are summarized in Table 1. There were significant differences between the two datasets i.e. the NIS dataset and the external validation dataset with regards to demographic variables, diagnoses, procedure types and various co-morbidities. Briefly, the patients operated at the University of Pittsburgh Medical Centre (UPMC) (external validation dataset) were older, had shorter lengths of hospital stay, had larger volume resections and more often underwent resections for primary hepatobiliary malignancies.

Table 1.  Comparison of demographic characteristics, diagnoses, procedure types and co-morbidities between the National Inpatient Sample (NIS) (years 2000–2004) dataset and the external validation dataset
  NIS (years 2000–2004) datasetExternal validation datasetP-value
Weighted frequencyPercentageFrequencyPercentage
  1. CHF, Congestive heart failure; COPD, Chronic obstructive pulmonary disease.

Age70 or less1473175.854568.6<0.001
Over 70469224.224931.4 
Length of stay10 days or less1599282.369087.1<0.001
More than 10 days343117.710212.9 
Admission typeElective1519878.277097.0<0.001
Liver proceduresLobectomy or (wedge + lobectomy)883645.541251.9<0.001
Wedge only1058754.538248.1 
Liver primaryNo1341069.064080.6<0.001
Cardiac arrhythmiaNo1785191.974593.80.047
Other Neurological DisordersNo1926399.278699.00.610
Renal failureNo1928099.377998.10.001
Liver diseaseNo1645284.770889.2<0.001
Fluid and electrolyte disordersNo1687686.972691.4<0.001
Inpatient deathNo1860695.978198.4<0.001

The nomogram was validated using the external validation dataset. The median total points for the external validation dataset were 115 with a range of 0 to 383. The median total points in this dataset correspond to a predicted mortality rate of 1.2%. The overall observed mortality rate in the external validation dataset was 1.6% (13/794). The concordance index calculated with the external validation dataset was found to be 0.65 (95% CI 0.46–0.82).

Calibration of the nomogram was tested using the observed mortality and the model predicted mortality. First, the quartiles of the predicted probabilities for patients who died in the external dataset were determined (Table 2). The observed mortality rates were calculated for the predicted probability deciles along with 95% CIs and plotted against the predicted probabilities (Fig. 2). The 45 degree line on the plot shows an imaginary line depicting perfect agreement between the observed probabilities and the nomogram predicted probabilities. The observed probabilities closely approximate the 45 degree imaginary line depicting predicted probability and therefore shows a good agreement.

Table 2.  Mortality distribution in the external validation dataset
Quartile of predicted mortalityObserved mortalityLower 95% CIUpper 95% CI
  1. Confidence interval.

10.92%0. 19%2.66%
21.25%0. 34%3.17%
33.61%0. 75%10.2%
44.69%0. 98%13.09%
Figure 2.

Calibration plot for the external validation dataset


Hepatic resection still remains one of the main options available to patients with malignancies of the liver.1,2 Over the past decade the number of hepatic resections have been increasing in number owing to the expanding criteria for resectability as well as increasing the number of treatment strategies available for the treatment of primary and secondary malignancies of the liver.1,2,14–20 Similarly, the number of complex resections as well as the number of resections being performed in elderly individuals with significant co-morbidities has been increasing.2–4 Being major procedures these operations are often fraught with significant morbidity and mortality.2–4 Therefore, discussion of peri-operative mortality remains an important part of thorough pre-operative patient counselling and informed consent.

In spite of the increasing number of resections being performed, the reported peri-operative mortality rates remain highly variable in the published literature.2,3,21–26 Variability in the published mortality rates make the discussion of patient-specific peri-operative mortality difficult at the time of informed consent and often leave the individual patients uncertain about their mortality rates. In order to overcome these limitations several authors have attempted to devise prediction tools such as risk scores and nomograms to predict peri-operative outcomes for patients undergoing hepatic resections.5,7–10,27 Although risk score systems are useful, it is thought that nomograms are superior to risk scores or risk grouping systems in predicting probabilities tailored to an individual patient.28,29 Additionally, most of the tools devised to predict peri-operative outcomes lack an external validation which limits their wide clinical use.8–10,27 Recently, a pre-operative nomogram to predict patient-specific peri-operative mortality after hepatic resections for malignancy has been constructed.11 The present study was undertaken to perform an external validation of this pre-operative nomogram to testify its applicability in a different clinical setting.

The current nomogram was designed using a nationwide population-based dataset NIS, which is heterogeneous in nature comprising of patients treated at different hospitals including rural vs. urban hospitals, small to medium vs. large hospitals, teaching vs. non-teaching hospitals, etc. However, for the purpose of validation, the dataset on the other end of spectrum i.e. high volume, single institute, centre of excellence was chosen, to test the robustness of the nomogram in a distinctly dissimilar setting. These factors may have contributed to the significant differences in the demographic factors, diagnoses and procedure types and various tested co-morbidities as shown in Table 1. Peri-operative mortality in the NIS dataset was found to be 4.1% which is similar to mortality in other population-based datasets as previously reported.21 This mortality was almost two and half times the mortality reported in an external validation set which is representative of mortality at high volume centres. In spite of these differences, the nomogram shows a good agreement between the observed and predicted probabilities between the two datasets with a moderate concordance index.

One major criticism of the tools derived from population-based administrative datasets is their limited applicability to high volume centres which have significantly lower mortality when compared with population-based mortality rates.21 Previous studies have internally validated the nomogram using a validation set from a different year from the same NIS dataset (year 2005).11 The present study further upholds the applicability of the nomogram to data derived from high volume single centres. In spite of these strengths there are several limitations to the present study. Drawing the estimates from a more heterogeneous population may add to their generalizability but may yield estimates with a higher degree of error for each patient.8 The concordance index observed in the current study showed only a moderate agreement and there may be under and over estimation of mortality for certain subgroups of patients. Although the calibration plots for the validation set looked good the CIs were wide. These could be as a result of extremely small number of deaths (thirteen) in the external validation dataset. Again, this underscores the fact that the nomogram is meant to be used as an additional tool and is not meant to replace the surgeons assessment based on adequate clinical parameters. Some of the other limitations and strengths of our nomogram have been discussed previously.11

In conclusion, the present study validates the use of the nomogram in the setting of a single high volume institute. The current nomogram is a simple tool which is patient specific, is available in the pre-operative setting and is applicable to both primary and secondary liver malignancies. The ease of use of this nomogram will make it an adjunctive clinical tool in the pre-operative setting and combined with good clinical judgment make it useful for patient counselling, obtaining informed consent and optimizing peri-operative care. To the authors’ knowledge, this nomogram may be the only clinical tool which has been externally validated to predict the pre-operative mortality after liver resections for malignancy. The validation of this nomogram in both the population-based setting11 and currently in a single institute setting makes it potentially applicable in broader and wider clinical settings.

Conflicts of interest

None declared.