A deep learning nomogram kit for predicting metastatic lymph nodes in rectal cancer

Abstract Background Preoperative diagnoses of metastatic lymph nodes (LNs) by the most advanced deep learning technology of Faster Region‐based Convolutional Neural Network (Faster R‐CNN) have not yet been reported. Materials and Methods In total, 545 patients with pathologically confirmed rectal cancer between January 2016 and March 2019 were included and were randomly allocated with a split ratio of 2:1 to the training and validation sets, respectively. The MRI images for metastatic LNs were evaluated by Faster R‐CNN. Multivariate regression analyses were used to develop the predictive models. Faster R‐CNN nomograms were constructed based on the multivariate analyses in the training sets and were validated in the validation sets. Results The Faster R‐CNN nomogram for predicting metastatic LN status contained predictors of age, metastatic LNs by Faster R‐CNN and differentiation degrees of tumors, with areas under the curves (AUCs) of 0.862 (95% CI: 0.816‐0.909) and 0.920 (95% CI: 0.876‐0.964) in the training and validation sets, respectively. The Faster R‐CNN nomogram for predicting LN metastasis degree contained predictors of metastatic LNs by Faster R‐CNN and differentiation degrees of tumors, with AUCs of 0.859 (95% CI: 0.804‐0.913) and 0.886 (95% CI: 0.822‐0.950) in the training and validation sets, respectively. Calibration plots and decision curve analyses demonstrated good calibrations and clinical utilities. The two nomograms were used jointly as a kit for predicting metastatic LNs. Conclusion The Faster R‐CNN nomogram kit exhibits excellent performance in discrimination, calibration, and clinical utility and is convenient and reliable for predicting metastatic LNs preoperatively. Clinical trial registration: ChiCTR‐DDD‐17013842.


| INTRODUCTION
Rectal cancer constitutes a large part of gastrointestinal tumors, and the mortality rate of colorectal cancer remains high despite its decline over the last 2 decades. [1][2][3] Among the metastatic pathways of rectal cancer, lymph node (LN) metastasis is the most important, as it usually leads to a poor prognosis due to a high rate of local recurrence. [4][5][6] Moreover, preoperative evaluation of metastatic LNs is critical for determining the optimal treatment strategies for rectal cancer patients. 7,8 Magnetic resonance imaging (MRI) has been widely used in clinical practice for diagnosing metastatic LNs in rectal cancer and is considered superior to computed tomography (CT) for its better soft-tissue discrimination. 9,10 However, radiologists usually take considerable time to identify metastatic LNs by observing their shapes, boundaries, and signal intensities. 11 Moreover, different radiologists often have different conclusions with regard to LN metastasis even on the same MRI image. [12][13][14][15][16] To improve the performance of preoperative metastatic LN diagnosis in colorectal cancer, different kinds of predictive models have been developed, including gene-or serum miRNA-based models and image-based models. [17][18][19] Of image-based models, radiomics derived from CT images are predominant. Radiomics features with different coefficients are first selected from radiological images of a patient and then, are linearly combined for a radiomics score; this work is time-consuming. The following questions arise: (1) as MRI is superior to CT in diagnosing metastatic LNs, how are the performances of models derived from MRI images, and (2) whether there are any other image-based methods for rapid and accurate evaluation of metastatic LNs? During the past decade, deep learning technologies have been deeply developed for image recognition; deep learning auto recognition of images has significantly advanced the identification and marking of interesting areas by methods of automatic sketching and 3D reconstruction. It has been used in diagnoses of solid tumors with regard to the skin, lung, breast, and prostate. [20][21][22][23][24] However, compared with solid tumors, metastatic LNs are more difficult to recognize due to their large quantities and tiny differences. Of deep learning technologies, the most advanced deep learning technology is Faster Region-based Convolutional Neural Network (Faster R-CNN), which was developed by Ross b. Girshick in recent years. Faster R-CNN integrates feature and proposal extractions, bounding box regression, and classification into a complete network and considerably advances the deep learning performance. In the current study, Faster R-CNN was applied for diagnosing metastatic LNs in rectal cancer and was incorporated into mathematical nomograms for predicting metastatic LNs preoperatively.

| Ethics
The Ethics Committee of the Affiliated Hospital of Qingdao University approved the study. The patients gave informed consent. The registration number of the study is ChiCTR-DDD-17013842. Patients did not participate in the design, conduct, reporting, or dissemination plans of the study.

| Patients
Patients with pathologically confirmed rectal cancer and who met the selection criteria between January 2016 and March 2019 were enrolled in the study, and their clinical data were obtained from the electronic medical system. The inclusion criteria were patients who (1) had pathologically confirmed rectal cancer; (2) had complete clinicopathological data; and (3) received preoperative endoscopic biopsies. The exclusion criteria were patients who (1) were diagnosed with other concurrent malignant tumors; (2) had undergone surgeries for other tumors; (3) were diagnosed with advanced rectal cancer and had not undergone surgeries; (4) received preoperative chemotherapies; and (5) received preoperative radiotherapies. In total, 545 patients were included in the study and were randomly allocated with a split ratio of 2:1 to the training and validation sets, respectively; 362 patients composed the training set, and 183 patients composed the validation set.
Clinicopathological data, including age, sex, differentiation degrees of tumors by preoperative pathology, metastatic LNs by MRI, metastatic LNs by postoperative pathology, carcinoembryonic antigen (CEA) levels, and carbohydrate antigen 199 (CA199) levels, were obtained from the electronic medical system. The levels of CEA and CA199 were analyzed by biochemical tests within 7 days before surgery. A CEA level of ≤5 ng/ml was considered normal, and >5 ng/ml was considered abnormal; a CA199 level of ≤37 U/ml was considered normal, and >37 U/ml was considered abnormal.

R-CNN
The MRI images of every individual patient with rectal cancer were separately recognized for metastatic LNs by Faster R-CNN. The architecture of Faster R-CNN includes detectors that consist of a feature extraction network of feature structures of images in ImageNet and region of interest eigenvectors, as well as regional proposal networks (RPNs). MRI images and a location marking dataset in the training database were used during the training process. ImageNet was initialized by the VGG16 model, and transfer training was performed. The weight initialization of the regional generation networks and region of interest eigenvectors shows a zero mean Gaussian distribution with a 0.01 deviation for the initial random weights. Other training parameters of Faster R-CNN are shown in Table 1. Detectors and regional generation networks perform endto-end training by stochastic gradient descent and back propagation. The loss function values of the RPN and detectors, as well as the total loss function value of Faster R-CNN, were output at every iteration of training. More details on the training and validation of Faster R-CNN are shown in our previous study. 25

| Statistical analysis
In univariate analyses, normally distributed continuous variables are presented as the mean (standard deviation, SD), and nonnormally distributed continuous variables are expressed as the median (quartile spacing, Q). Group differences were examined by bilateral t test for normally distributed data and rank sum test for nonnormally distributed data. Number and percentage (%) were used to express categorical variables, and group differences were examined by chi-squared tests (Pearson or continuous correction). In multivariate analyses, logistic regression analyses were performed; the risk degrees of predictors are expressed by the odds ratio (OR) and 95% confidence interval (CI). R software was used for statistical analyses. A two-tailed p value of <0.05 indicates a significant difference.

| Nomogram construction
Data from the training set were used for nomogram construction. Using LN metastasis as an outcome variable, the training set included 362 patients with rectal cancer; using LN metastasis degree (at stage N1 or N2) as an outcome variable, the training set included 280 rectal cancer patients with metastatic LNs. Univariate analyses and multivariate analyses were separately performed to identify the significant predictors of an outcome variable. A nomogram for predicting the risk probability of an outcome variable was developed on the basis of multivariate analysis.

| Nomogram validation
Using LN metastasis as an outcome variable, the validation set included 183 patients with rectal cancer; using LN metastasis degree (at stage N1 or N2) as an outcome variable, the validation set included 153 rectal cancer patients with metastatic LNs. The performance of a nomogram was validated with regard to capabilities of discrimination, calibration and clinical utility. The area under the curve (AUC) of the receiver operating characteristic (ROC) curve were used to verify the discrimination capability of a nomogram. The calibration plot graphically presenting the predictive probabilities and the observations was used to verify the calibration capability of a nomogram based on resamples of 1000 bootstraps. 26 Decision curve analysis (DCA) was used to verify the clinical utility of a nomogram, considering realistic threshold probabilities. 27 In total, 545 patients were included, including 338 males (62.0%) and 207 females (38.0%), with a mean (SD) age of 58.55 (12.57) years ranging from 26 to 86 years. No difference in any characteristic was observed between groups (all p > 0.05; see, Table 2).

Univariate analyses
Predictors of age, metastatic LNs by MRI, metastatic LNs by Faster R-CNN, differentiation degrees of tumors, and CEA levels were considered significant for LN metastasis (all P＜0.05; see Table 3)   Figure 1A). In the validation set, good performances of the proposed nomograms were also observed, as evidenced by the AUCs and 95% CIs of 0.914 (0.867-0.960) and 0.920 (0.876-0.964), respectively. A significant difference in the AUCs was found by DeLong's test between the MRI1 nomogram and Faster R-CNN1 nomogram (Z = −2.179, p = 0.029, see, Figure 1B). Therefore, the Faster R-CNN1 nomogram was considered preferable for predicting metastatic LN status by a linear combination of three predictors (see, Figure 2A). The calibration plots demonstrated good consistency between the observations and the predictive probabilities of the Faster R-CNN1 nomogram (see, Figure 3A). The DCA showed considerable net benefit of the Faster R-CNN1 nomogram, along with the threshold probabilities ( Figure 3B). In total, 433 patients were included in the study, including 260 males (60.0%) and 173 females (40.0%), with a mean (SD) age of 59.86 (12.26) years ranging from 28 to 86 years. No difference in any characteristic was observed between groups (all p > 0.05; see, Table 5).

| Predictors for LN metastasis degree in rectal cancer patients with metastatic LNs-Univariate analyses
Predictors of metastatic LNs by Faster R-CNN, differentiation degrees of tumors, CEA levels, and CA199 levels were significantly associated with LN metastasis degree (all p < 0.05; see, Table 6).

| Predictors for LN metastasis degree in rectal cancer patients with metastatic LNs-Multivariate analyses
Here, potential predictors of metastatic LNs by Faster R-CNN and metastatic LNs by MRI were also separately incorporated with other potential predictors into stepwise multivariate logistic regression analyses. As shown in

| Development and validation of nomograms for predicting LN metastasis degree in rectal cancer patients with metastatic LNs
Significant predictors identified in MRI-based and Faster R-CNN-based multivariate analyses were separately included for the development of predictive nomograms, and they were the MRI1 nomogram and Faster R-CNN1 nomogram, respectively.  Figure  1C). In the validation set, good performances of the proposed nomograms were also observed, as evidenced by the AUCs and 95% CIs of 0.784 (0.694-0.874) and 0.886 (0.822-0.950), respectively. A significant difference in the AUCs was found by DeLong's test between the MRI2 nomogram and Faster R-CNN2 nomogram (Z = −3.210, p < 0.001, Figure 1D). Therefore, the Faster R-CNN2 nomogram was considered optimal for predicting LN metastasis degree by calculating two predictors with respective scores (see, Figure 2B). The calibration plots demonstrated good consistency between the observations and the predictive probabilities of the Faster R-CNN1 nomogram (see, Figure 3C). The DCA showed considerable net benefit of the Faster R-CNN1 nomogram, along with the threshold probabilities ( Figure 3D).

| DISCUSSION
In this study, the most advanced deep learning technology, that is, Faster R-CNN, and the mathematical statistical model of the nomogram were merged to predict metastatic LNs preoperatively. A nomogram integrates independent factors of different weights for predicting the risk probability of a clinical outcome visually. 29 However, Faster R-CNN nomograms for predicting metastatic LNs have not yet been reported. The study mainly included two aspects of predicting metastatic LNs: the preoperative predictions of metastatic LN status in rectal cancer patients and LN metastasis degree in rectal cancer patients with metastatic LNs.
Predicting metastatic LN status in rectal cancer patients. No significant difference in any characteristic was found between the training (n = 362) and validation (n = 183) sets. The results indicate that selection bias did not impact the results. In univariate analyses for the training set, age,  discrimination capacities, with corresponding AUCs of 0.914 and 0.920, respectively, and the Faster R-CNN1 nomogram was still considered superior to the MRI1 nomogram. Therefore, the Faster R-CNN1 nomogram was considered the best model and was graphically presented to predict the risk probability of metastatic LN status by a linear combination of three independent variables with different scores (see, Figure 2A). In addition, the calibration plots demonstrated good consistency between the observations and the predictive probabilities of the Faster R-CNN1 nomogram.
The results confirmed that the nomogram possesses excellent discrimination and calibration. However, a nomogram with excellent discrimination and calibration is not necessarily suitable for clinical application due to its impractical net benefit threshold probabilities. 30 Therefore, the DCA curves, with threshold probabilities and net benefits on the horizontal and vertical axes, respectively, were plotted to examine the practical value of the nomogram and showed that the Faster R-CNN1 nomogram yielded clinical benefits. There have been some reported researches using artificial intelligence to predict LN metastasis in rectal cancer. Tse DM developed a computer algorithm by combining morphological features of LNs on MRI to predict LN status in a small sample of 17 patients, and the maximum accuracy was merely 0.86 against radiologists' diagnoses. 31 Compared with Faster R-CNN, its generalization ability and detection effect are much poorer, for most of its learned features are shallow. Moreover, it takes much more time in training. Zhou YP established an automatic recognition system for metastatic LNs of rectal cancer using convolution neural network, and the AUC of the system was 0.89 with reference to radiologists' diagnoses. 32 The deep learning technology of this study is on par with ours', but the incorporation of mathematical nomogram kit into our method makes our prediction on metastatic LNs more accurate, as evidenced by the AUC of 0.92 with reference to pathologists' diagnoses in the validation set. The Journal of Clinical Oncology reported a CT-based radiomics nomogram for predicting metastatic LN status in colorectal cancer and showed a relatively good C-index of 0.736, smaller than the 0.920 of the nomogram proposed by the current study. 19 In the CT-based radiomics model, the feature extraction process focuses on the whole tumor outline other than a single metastatic LN as Faster R-CNN concerns. Some studies reported gene-related nomograms and miRNA-related nomograms for the risk prediction of metastatic LN status in colorectal cancer and showed discrimination capabilities of 0.700 and 0.883, respectively. 17,18 The proposed Faster R-CNN nomogram is comparable to the gene-or miRNA-related nomogram with regard to the predictive accuracy and even surpasses them. Moreover, all the previous nomograms require plenty of time to extract radiomics features or to conduct gene-related examinations. However, it takes only 20 s other than the radiologists' 600 s per case for the Faster R-CNN to diagnose metastatic LNs. 25 In addition, all the previous models only predict the risk probability of whether LN metastasis occurs or not; it does not predict the LN metastasis degree (at stage N1 or N2). Predicting LN metastasis degree (stage N1 or N2) in rectal patients with metastatic LNs. No significant difference was found between the training (n = 280) and validation (n = 153) sets in any characteristic, indicating well-controlled selection bias. Logistic regression analyses showed that metastatic LNs by MRI and differentiation degrees of tumors were significant predictors of metastatic LN degree in the MRI-based analyses, and metastatic LNs by Faster R-CNN and differentiation degrees of tumors were significant predictors in the Faster R-CNN-based analyses in rectal cancer patients with metastatic LNs. Then, the MRI2 nomogram and Faster R-CNN2 nomogram were separately constructed on the basis of the multivariate analyses. In the training set, both of the proposed nomograms demonstrated excellent discrimination capacities, with AUCs of 0.770 and 0.859, respectively; the AUC of the Faster R-CNN2 nomogram was statistically larger than that of the MRI2 nomogram. The results suggest that the Faster R-CNN2 nomogram is preferable. In the validation set, both of the proposed nomograms also demonstrated excellent discrimination capacities, with their corresponding AUCs of 0.784 and 0.886, and the Faster R-CNN2 nomogram was still considered superior to the MRI2 nomogram. The Faster R-CNN2 nomogram is graphically presented for predicting LN metastasis degree by calculating two independent variables with different scores (see, Figure 2B). In addition, good calibration and clinical utility were confirmed in the nomogram by the calibration plot and the DCA, respectively.
In clinical use, the Faster R-CNN nomograms for the risk prediction of metastatic LN status and LN metastasis degree should be jointly used as a kit by multiplicative effect methods. For example, a 45-year-old man was preoperatively diagnosed with rectal cancer of moderate differentiation by endoscopic biopsy, and the number of metastatic LNs identified by Faster R-CNN was 6. First, according to the Faster R-CNN1 nomogram, the total risk score of the patient is calculated to be 135 by sums of the respective points of three predictors, and the corresponding risk probability is considered 1.0 for LN metastasis. Second, the risk probability of metastatic LN degree is calculated to be 0.45 at stage N2 and 0.55 at stage N1 according to the Faster R-CNN2 nomogram. Therefore, the total risk probability of the patient is 0.45 (1 multiplied by 0.45) at stage N2 and 0.55 (1 multiplied by 0.55) at stage N1. According to the decision curves for the two nomograms, the optimal high risk thresholds are 0.80 for nomogram 1 and 0.25 for nomogram 2, such that the total optimal high risk threshold is 0.20 (0.80 multiplied by 0.25) for stage N2. Therefore, it is preferable to predict the patient at stage N2 for the most clinical benefit.
In addition, there are some limitations to declare. First, the comprehensive performance of Faster R-CNN and its related nomogram for predicting metastatic LNs are limited to some extent because Faster R-CNN is based on MRI images other than pathological images. In fact, when the lymph nodes are cut from the body for pathological diagnoses, it is difficult to label each metastatic LN on MRI images. We are looking forward to solving these problems in future studies. Second, the study was performed in one medical group. Though it contains three hospital regions and each region is dozens of kilometers apart, more centers should be involved.
In conclusion, the proposed Faster R-CNN nomogram kit exhibits excellent comprehensive performance and is convenient and reliable for predicting metastatic LNs preoperatively.