Fax: (212) 794-5851

Original Article

# Clinical benefits of a multivariate prediction model for bladder cancer^{†}

### A decision analytic approach

Article first published online: 12 OCT 2009

DOI: 10.1002/cncr.24615

Copyright © 2009 American Cancer Society

Additional Information

#### How to Cite

Vickers, A. J., Cronin, A. M., Kattan, M. W., Gonen, M., Scardino, P. T., Milowsky, M. I., Dalbagni, G., Bochner, B. H. and for The International Bladder Cancer Nomogram Consortium (2009), Clinical benefits of a multivariate prediction model for bladder cancer. Cancer, 115: 5460–5469. doi: 10.1002/cncr.24615

^{†}See editorial on pages 5368-70, this issue.

#### Publication History

- Issue published online: 19 NOV 2009
- Article first published online: 12 OCT 2009
- Manuscript Accepted: 2 MAR 2009
- Manuscript Revised: 25 FEB 2009
- Manuscript Received: 14 NOV 2008

- Abstract
- Article
- References
- Cited By

### Keywords:

- bladder cancer;
- adjuvant chemotherapy;
- prognosis;
- decision support;
- outcomes

### Abstract

#### BACKGROUND:

It has been demonstrated that multivariate prediction models predict cancer outcomes more accurately than cancer stage; however, the effects of these models on clinical management are unclear. The objective of the current study was to determine whether a previously published multivariate prediction model for bladder cancer (“bladder nomogram”) improved medical decision making when referral for adjuvant chemotherapy was used as a model.

#### METHODS:

Data were analyzed from an international cohort study of 4462 patients who underwent cystectomy without chemotherapy from 1969 to 2004. The number of patients eligible for chemotherapy was determined using pathologic stage criteria (lymph node-positive disease or pathologic T3 [pT3] or pT4 tumor classification) and for 3 cutoff levels on the bladder nomogram (10%, 25%, and 70% risk of recurrence with surgery alone). The number of recurrences was calculated by applying a relative risk reduction to the baseline risk among eligible patients. Clinical net benefit was then calculated by combining recurrences and treatments and weighting the latter by a factor related to drug tolerability.

#### RESULTS:

A nomogram cutoff outperformed pathologic stage for chemotherapy in every scenario of drug effectiveness and tolerability. For a drug with a relative risk of 0.80, with which clinicians would treat ≤20 patients to prevent 1 recurrence, use of the nomogram was equivalent to a strategy that resulted in 60 fewer chemotherapy treatments per 1000 patients without any increase in recurrence rates.

#### CONCLUSIONS:

The authors concluded that referring patients who undergo cystectomy to adjuvant chemotherapy on the basis of a multivariate model is likely to lead to better patient outcomes than the use of pathologic stage. Further research is warranted to evaluate the clinical effects of multivariate prediction models. Cancer 2009. © 2009 American Cancer Society.

Many decisions in oncology depend implicitly or explicitly on predictions. These predictions normally are thought of in terms of “risk”; typically, we act when a patient is deemed at sufficiently high risk that the benefits of intervention, in terms of reducing risk, outweigh the harms, in terms of toxicities. Most commonly, decisions that involve prediction are based on risk categories. The most common system for risk categorization in cancer is cancer stage, with more aggressive treatments reserved for patients with higher stage disease. Stage can influence the extent of surgery, such as in breast cancer or bladder cancer; the intensity of chemotherapy, such as in lymphoma; and whether adjuvant therapy is indicated, such as in bladder cancer or colon cancer.

Recent years have seen an upsurge of interest in multivariate prediction models. Typically, these models provide a numerical estimate of risk in the form of a probability on the basis of several tumor and patient characteristics. The well known “Kattan nomogram,” for example, provides the probability of prostate cancer recurrence after radical prostatectomy on the basis of stage, grade, and prostate-specific antigen level.1 Numerous similar models have been published for a variety of different cancers2-6 and for specific treatment decisions, such as adjuvant chemotherapy after breast cancer.7 It appears reasonable that such models might predict more accurately than simple staging systems, because they include additional prognostic information. For example, a patient with high-grade, organ-confined prostate cancer most likely is at a higher risk of recurrence than a patient at a similar stage but with low-grade disease. Empirical studies have confirmed that multivariate models provide more accurate predictions than American Joint Committee on Cancer (AJCC) staging or other simple risk groupings in a wide variety of cancers, including melanoma,8 gastric cancer,9 pancreatic cancer,10 and prostate cancer.11 In the case of bladder cancer, the topic of the current report, the predictive accuracy of a multivariate model for recurrence after radical cystectomy (the “bladder cancer nomogram”) was a concordance index of 0.75 compared with only 0.68 for the AJCC TNM classification and 0.62 for standard pathologic stage.12

Nonetheless, the clinical implications of an improved concordance index are not immediately obvious. We may be able to predict better who is at high risk of death from bladder cancer; however, this may make little practical difference to patient care. We are interested in defining better whether using the bladder cancer nomogram would improve clinical decision making, such as whether a patient should receive adjuvant chemotherapy. To address this question, we examined a recent multivariate prediction model, the bladder cancer nomogram, using decision analytic methods.

### MATERIALS AND METHODS

#### Patients

The collection of data for the International Bladder Cancer Nomogram Consortium has been described previously.12 In total, data on 9064 patients who underwent radical cystectomy were collected from 13 institutions in 6 countries. Because we wanted to estimate the risk of recurrence after radical cystectomy alone, we excluded patients who had received systemic adjuvant chemotherapy, neoadjuvant chemotherapy, or definitive pelvic radiotherapy (n = 2001) and patients for whom pelvic radiotherapy status was unknown (n = 319). Of the remaining patients, 871 were not followed for recurrence, and 1411 were excluded because of missing data on variables that were included in the nomogram. Therefore, our sample consisted of 4462 patients who were managed by radical cystectomy only, who were followed for disease recurrence, and who had complete data for all predictors: sex, age, pathologic stage, histology, lymph node status, grade, and time between diagnosis and cystectomy. The proportion of patients by institution was similar to that previously reported.12

#### Adjuvant Chemotherapy for Bladder Cancer

Our interest in this study was adjuvant rather than salvage chemotherapy. Adjuvant chemotherapy commonly is given to high-risk bladder cancer patients after radical cystectomy. Two separate groups have published meta-analyses of randomized trials and reported similar findings suggesting a survival benefit to adjuvant chemotherapy. Ruggeri et al pooled data from 5 phase 3 trials and reported a statistically significant improvement in disease-free survival in patients who received adjuvant chemotherapy (hazard ratio, 0.65; 95% confidence interval [95% CI], 0.54-0.78) for disease-free survival.13 The Advanced Bladder Cancer Meta-Analysis Collaboration14 included an additional trial and performed individual patient data analysis: Those investigators reported a hazard ratio of 0.68 (95% CI, 0.53-0,89). These numbers are approximately equivalent to a relative risk (RR) of recurrence at 5 years of approximately 0.75. However, given the relatively limited number of patients included in these meta-analyses (<500), and the accordingly wide 95% CIs, we planned to use a variety of different estimates of RR in our analyses.

#### Statistical Analysis

Eligibility for adjuvant chemotherapy depends on a decision rule for the identification of high-risk patients. With respect to bladder cancer, patients who have pathologic T3 (pT3) or pT4 disease or those with positive lymph nodes generally are considered at high risk for disease recurrence and, thus, according to standard guidelines, are eligible for chemotherapy; lymph node-negative patients with organ-confined disease (<pT3) are re followed by observation only after surgery.15 We wanted to compare this eligibility criterion with 1 of 3 prespecified rules based on the nomogram-predicted risk of recurrence at 5 years: 10%, 25%, and 70%.

Our overall statistical approach follows a previously published methodology.16 In brief, to calculate sensitivity and specificity for survival time data, we first define *x* = 1 if the patient is classified as being at high risk and *x* = 0 otherwise; *s*(*t*) is the Kaplan-Meier survival probability at time *t*, which we predefined as 5 years. According to the method described by Begg et al,17 we used the following formula for sensitivity and specificity:

and

Assuming a constant RR, the proportion of patients that recurs with a particular intervention scenario *i* can be given as:

Because we are treating recurrence as a binary event, RR is defined as the risk of recurrence at 5 years with treatment divided by 5-year recurrence risk without treatment. Our primary analyses assumed that the RR was constant across risk groups, so that, for example, given a chemotherapy regimen associated with an RR of 0.9, a patient who has a 50% probability of disease recurrence without treatment would have a risk of 45% on chemotherapy, and patient who has a 10% baseline risk would have a 9% risk.

For sensitivity analyses, we considered 3 scenarios in which the treatment was most effective for patients at low risk, average risk, and high risk. A patient's risk was estimated by using the nomogram probability, which is justified because it has been demonstrated that the nomogram is well calibrated.12 Figure 1 is an illustration of how RR varied by absolute risk for each scenario. The mean RR was kept constant between scenarios, such that the total number of disease recurrences was identical regardless of the presumed relation between absolute risk and RR.

The proportion of patients that would be treated under each strategy is estimated by counting the proportion of patients in our dataset that meets each criterion. Knowing the treatment and recurrence rates for each strategy does not necessarily identify the optimal approach. Often, when comparing 2 strategies, 1 approach will be associated with a lower rate of treatment but also with a higher rate of disease recurrence. To calculate whether the reduction in the number of patients receiving chemotherapy offsets the increase in recurrence rates, we need to consider the maximum number of patients a clinician would consider treating to prevent 1 recurrence. This is known as the “number-needed-to-treat threshold” (NNT_{T}) and is a clinical judgment that can vary from clinician to clinician and from patient to patient.18 The NNT_{T} is reciprocal of the minimum, clinically significant difference, a concept necessary to design and interpret randomized trials.19, 20 NNT_{T} is a measure of drug tolerability in which an agent that is easy to take, inexpensive, and associated with low toxicity would have a high NNT_{T}; and a toxic, inconvenient, or expensive drug would have a low NNT_{T}. Note that the NNT_{T} is different from the usually reported number needed to treat (NNT): NNT is calculated from the results of a study and tells us how many patients would need to be treated to prevent 1 event; NNT_{T} is a clinical consideration independent of the results of any study and tells us how a physician weights the harms of treatment against the benefits of avoiding an event. Then, we can define “clinical net benefit”16 as follows:

Because net benefit includes both treatments and recurrences, the optimal treatment strategy is the 1 with the highest net benefit, irrespective of the size of the difference between strategies. Statistical analyses were conducted using Stata 9.2 statistical software (StataCorp, College Station, Tex).

### RESULTS

Baseline characteristics of the 4462 patients in our sample are summarized in Table 1. The median age at cystectomy was 62 years (interquartile range, 51-70 years), and the majority of patients were men (77%). Approximately half were either lymph node positive or had disease classified as ≥pT3 (n = 2466; 55%); 4081 patients (91%), 1835 patients (41%), and 365 patients (8%) had a ≥10%, ≥25%, and ≥70% 5-year nomogram probability of recurrence, respectively.

Characteristic | No. of Patients (%) or Median [Interquartile Range] |
---|---|

^{}pTis indicates pathologic tumor in situ; TCC, transitional cell carcinoma; SCC, squamous cell carcinoma.
| |

Age at cystectomy, y | 62 [51-70] |

Elapsed time from diagnosis to cystectomy, mo | 3.4 [1.2-9.5] |

5-y nomogram probability of recurrence, % | 21 [15-36] |

Sex | |

Men | 3457 (77) |

Women | 1005 (23) |

Pathologic tumor classification | |

pT0 | 72 (2) |

pTis | 124 (3) |

pTa | 111 (2) |

pT1 | 765 (17) |

pT2 | 1035 (23) |

pT3 | 1878 (42) |

pT4 | 477 (11) |

Pathologic grade | |

Low | 706 (16) |

High | 3676 (82) |

Unknown | 80 (2) |

Histology | |

TCC | 3429 (77) |

SCC | 853 (19) |

Adenocarcinoma | 180 (4) |

Lymph node status | |

Negative | 3668 (82) |

Positive | 542 (12) |

Unknown | 252 (6) |

Year of cystectomy | |

Before 1985 | 359 (8) |

1985-1989 | 814 (18) |

1990-1994 | 999 (22) |

1995-1999 | 1401 (31) |

2000-2004 | 889 (20) |

There were 1068 recurrence events. The median follow-up for recurrence-free patients was 3.8 years. The Kaplan-Meier estimate of the 5-year probability of disease recurrence for the entire sample was 28% (95% CI, 26%-29%).

Table 2 shows the number of patients treated using each treatment strategy and the expected number of disease recurrences given various levels of treatment effectiveness. Compared with the standard approach, using a nomogram probability of 25% as eligibility for postoperative chemotherapy reduces the number of patients treated by nearly 25% at the cost of a slight increase in disease recurrence rates; using a 10% nomogram probability increases the number of patients treated by approximately 65%, but this is associated with a large reduction in rates of disease recurrence. Table 3 shows the net benefit for each cutoff level given a range of NNT_{T} and RR values. The strategy with the highest net benefit will have the optimal clinical results. When drugs are very effective (RR, 0.6 or 0.7) or tolerable (NNT_{T}, 35 or 50), the highest net benefit is for a nomogram cutoff level of 10%. When drugs are of more marginal benefit and poor tolerability (RR, 0.9; NNT_{T}, 20), the nomogram cutoff of 70% is optimal. For the remaining scenarios, a nomogram cutoff of 25% provides the highest net benefit. The differences between strategies are not trivial. For example, at an RR of 0.80 and an NNT_{T} of 20, the use of a 25% threshold has a net benefit 0.003 higher than the standard pathologic groups. This is equivalent to a strategy that led to 3 fewer recurrences per 1000 patients without any change in the number of patients who received chemotherapy or to a strategy associated with 60 fewer treatments per 1000 patients without any increase in the rate of disease recurrence.

Treatment Strategy | Patients Treated, % | Sensitivity, %* | Specificity, %† | Proportion of Patients With Event (Reduction in Event Rate From Treating None), % | |||
---|---|---|---|---|---|---|---|

RR, 0.60 | RR, 0.70 | RR, 0.80 | RR, 0.90 | ||||

^{}RR indicates relative risk. - *
Sensitivity is the proportion of patients who would develop disease recurrence by 5 years who would be treated with a particular treatment strategy. - †
Specificity is the proportion of patients who would be free of disease recurrence by 5 years and would be spared treatment with a particular treatment strategy.
| |||||||

All | 100 | 100 | 0 | 16.8 (11.2) | 19.5 (8.4) | 22.3 (5.6) | 25.1 (2.8) |

Standard | 56 | 78 | 53 | 19.2 (8.7) | 21.4 (6.5) | 23.6 (4.4) | 25.7 (2.2) |

Nomogram cutoff 10% | 92 | 99 | 11 | 16.9 (11.1) | 19.6 (8.3) | 22.4 (5.5) | 25.2 (2.8) |

Nomogram cutoff 25% | 42 | 72 | 70 | 19.9 (8) | 21.9 (6) | 23.9 (4) | 25.9 (2) |

Nomogram cutoff 70% | 9 | 23 | 97 | 25.4 (2.6) | 26.0 (1.9) | 26.6 (1.3) | 27.3 (0.6) |

None | 0 | 0 | 100 | 27.9 | 27.9 | 27.9 | 27.9 |

Treatment Strategy | Net Benefit | |||
---|---|---|---|---|

RR, 0.60 | RR, 0.70 | RR, 0.80 | RR, 0.90 | |

^{}RR indicates relative risk; NNT _{T}, the number-needed-to-treat threshold.
| ||||

NNT_{T} 20 | ||||

All | 61.7 | 33.8 | 5.9 | −22.1 |

Standard | 59.3 | 37.5 | 15.7 | −6 |

Nomogram cutoff 10% | 64.7 | 37.1 | 9.4 | −18.2 |

Nomogram cutoff 25% | 59.6 | 39.5 | 19.4 | −0.8 |

Nomogram cutoff 70% | 21.4 | 15 | 8.6 | 2.1 |

NNT_{T} 35 | ||||

All | 83.1 | 55.2 | 27.3 | −0.6 |

Standard | 71.2 | 49.4 | 27.7 | 5.9 |

Nomogram cutoff 10% | 84.4 | 56.7 | 29.1 | 1.4 |

Nomogram cutoff 25% | 68.5 | 48.4 | 28.3 | 8.2 |

Nomogram cutoff 70% | 23.2 | 16.8 | 10.4 | 4 |

NNT_{T} 50 | ||||

All | 91.7 | 63.8 | 35.9 | 7.9 |

Standard | 76 | 54.2 | 32.4 | 10.7 |

Nomogram cutoff 10% | 92.2 | 64.6 | 36.9 | 9.3 |

Nomogram cutoff 25% | 72.1 | 52 | 31.9 | 11.8 |

Nomogram cutoff 70% | 24.0 | 17.6 | 11.1 | 4.7 |

To explain these findings, Figure 2 provides an illustration of the distribution of predicted probabilities within each pathologic grouping. A nomogram cutoff of 25% includes all patients with lymph node-positive disease. However, a little more than 25% of patients with pT3 or pT4 disease or lymph node-negative disease are at low risk (<25%) according to the nomogram and, thus, would not receive chemotherapy; conversely, just under 25% of patients with less than pT3 disease (organ-confined tumors) are considered high risk according to the nomogram.

We then repeated our analysis for all possible combinations of RR (0.50-0.99) and NNT_{T} (1-250 patients) and recorded which strategy had the highest net benefit. The results are provided in Figure 3. For example, Table 3 shows that, for an NNT_{T} of 20 and an RR of 0.8, the highest net benefit is obtained by treating only patients with at least a 25% risk of disease recurrence; the point corresponding to 0.8 on the *x* axis and 20 on the *y* axis in Figure 3 is medium gray, indicating the 25% threshold as the optimal strategy. When adjuvant therapy is highly effective or tolerable, as expected, either all patients or all patients except those with a very low risk (nomogram probability, <10%) should be treated; when therapy is of moderate effectiveness or tolerability, either no patients or only those at the highest risk (nomogram probability ≥70%) should be treated. For the remaining patients, the 25% cutoff is optimal. It is important to note that for no combination of effectiveness or tolerability is the highest net benefit associated with the standard eligibility for chemotherapy. In other words, the nomogram outperforms the standard guideline for every plausible scenario of drug effect and tolerability: Whatever a clinician believes about a particular drug, he or she should use a nomogram cutoff to decide which patients to treat.

We excluded 2320 patients who were receiving chemotherapy or radiotherapy or other, unknown treatments, including 19% (679 of 3551 patients) who had low disease classification (≤pT2 and lymph node-negative disease), 32% (1536 of 4875 patients) who had high disease classification (≥pT3 or lymph-node positive disease), and 16% (105 of 638 patients) who had unknown stage. Although a higher proportion of patients with high stage were excluded, our cohort comprised the vast majority of patients with both high and low pathologic stages. Accordingly, we see no reason to believe that treatment selection would have an important impact on our findings.

Our primary analyses involved an assumption that the RR reduction associated with treatment was constant across risk groups. Therefore, we performed sensitivity analyses to verify that our results were robust to this assumption. Results of these sensitivity analyses are displayed in Table 4. Although it appears initially that conclusions regarding the optimal cutoff level sometimes depend on our assumptions of how RR varies with absolute risk, the assumption of constant RR analyses yielded good results. Generally, there were only small differences in net benefit noted between the optimal strategy under constant RR and the optimal strategy in which the RR varied by baseline risk. Most important, the strategy chosen under the assumption of a constant RR was superior to standard pathologic risk groups across all but 2 of the sensitivity analyses that we conducted; and, in both of those analyses, the advantage was trivial (1 or 3 recurrences per 10,000 patients). Thus, our conclusion that the nomogram improves clinical outcomes holds true regardless of the relation between absolute risk and RR reduction.

Treatment Strategy | Optimal Treatment Strategy† | |||
---|---|---|---|---|

RR, 0.60 | RR, 0.70 | RR, 0.80 | RR, 0.90 | |

^{}RR indicates relative risk; NNT _{T}, the number-needed-to-treat threshold; Cutoff, nomogram cutoff.- *
The different scenarios for how the effectiveness of treatment varies with baseline risk are illustrated in Figure 1. - †
Numbers in parentheses represent the improvement in net benefit per 1000 patients treated if the optimal treatment strategy under the main analysis had been implemented instead of the conventional strategy under the respective sensitivity analysis.
| ||||

NNT_{T} 20 | ||||

Main analysis: Constant RR | 10% Cutoff (5.4) | 25% Cutoff (1.9) | 25% Cutoff (3.6) | 70% Cutoff (8.2) |

Sensitivity analysis 1: Treatment most effective for those at average risk | All (3.7) | 25% Cutoff (2.4) | 25% Cutoff (4.0) | 70% Cutoff (11.9) |

Sensitivity analysis 2: Treatment most effective for those at low risk | All (4.0) | 25% Cutoff (2.4) | 25% Cutoff (4.1) | 70% Cutoff (13.1) |

Sensitivity analysis 3: Treatment most effective for those at high risk | 10% Cutoff (6.8) | 25% Cutoff (1.5) | 25% Cutoff (3.1) | 25% Cutoff (3.4) |

NNT_{T} 35 | ||||

Main analysis: Constant RR | 10% Cutoff (13.1) | 10% Cutoff (7.3) | 10% Cutoff (1.4) | 25% Cutoff (2.3) |

Sensitivity analysis 1: Treatment most effective for those at average risk | All (11.5) | All (5.7) | All (−0.1) | 25% Cutoff (2.7) |

Sensitivity analysis 2: Treatment most effective for those at low risk | All (11.8) | All (5.7) | All (−0.3) | 70% Cutoff (2.8) |

Sensitivity analysis 3: Treatment most effective for those at high risk | 10% Cutoff (14.5) | 10% Cutoff (8.8) | 10% Cutoff (3.1) | 25% Cutoff (1.8) |

NNT_{T} 50 | ||||

Main analysis: Constant RR | 10% Cutoff (16.2) | 10% Cutoff (10.4) | 10% Cutoff (4.5) | 25% Cutoff (1.1) |

Sensitivity analysis 1: Treatment most effective for those at average risk | All (14.6) | All (8.8) | All (3.0) | All (1.5) |

Sensitivity analysis 2: Treatment most effective for those at low risk | All (14.9) | All (8.8) | All (2.8) | All (1.6) |

Sensitivity analysis 3: Treatment most effective for those at high risk | 10% Cutoff (17.6) | 10% Cutoff (11.9) | 10% Cutoff (6.2) | 25% Cutoff (0.6) |

The nomogram was developed with the exclusion of patients who had received chemotherapy or radiotherapy—patients who likely have the most unfavorable tumor characteristics. Consequently, the predictions from the nomogram may underestimate the overall risk of recurrence. Therefore, in a second sensitivity analysis, we added a constant to the nomogram prediction for every patient (varying the constant from 2% to 5%) and repeated all analyses, and no results were changed.

In a third sensitivity analysis, we repeated all analyses using only patients who had transitional cell carcinoma, the predominant histology for bladder cancer in Europe and North America (see Fig. 3). Although there were some differences in results, our key finding was unaffected: One or another nomogram cutoff was superior to the standard eligibility criteria for chemotherapy for every combination of drug tolerability and effectiveness.

Our final sensitivity analysis accounted for the competing risk of death; that is, we defined the probability of recurrence by using the cumulative incidence function instead of the Kaplan-Meier estimate.21 The optimal treatment strategy under the competing risk analysis was the same as that under the primary analysis for all but 1 combination of NNT_{T} and RR, as shown in Table 3.

### DISCUSSION

Adjuvant chemotherapy after bladder cancer is subject to some debate, in particular, data from randomized trials, although positive,13, 14 are limited by inadequate numbers of patients. Nonetheless, adjuvant therapy often is given after radical cystectomy (it is recommended in standard guidelines15), and pathologic stage is the most common criterion to determine which patients receive it. Our analyses suggest that determining eligibility for adjuvant chemotherapy after radical cystectomy on the basis of a multivariate model will yield superior clinical results compared with determining eligibility on the basis of pathologic characteristics alone. In a typical comparison, use of the nomogram would reduce the number of patients subjected to chemotherapy by 14% with only a small increase in disease recurrences (0.4%). This improvement in outcome is obtained purely by changing a decision rule: there are no additional tests, procedures, or treatments. The superior clinical performance of the nomogram appears to result from reclassification of an important proportion of patients: Some patients with pT3 or pT4 disease would not be eligible for chemotherapy on the basis of the nomogram because of the absence of any other risk factor (such as a long time from diagnosis to treatment); comparably, some patients with <pT3 disease are eligible for chemotherapy according to the nomogram, because they are defined as high risk for a reason other than pathologic stage.

Although previous studies have demonstrated that multivariate models improve predictive accuracy compared with staging systems,8-11 we believe that we are the first to demonstrate that use of such models would have beneficial effects with respect to a therapeutic decision. This finding has important consequences for cancer care. An enormous range of decisions concerning the care of the cancer patient is based on risk, with patients thought to be a higher risk subject to more intensive treatment or monitoring. Currently, however, nearly all such decisions are based on simple risk stratifications, such as stage. We hypothesize that multivariate risk prediction models could be used to replace many of the decisions currently made on the basis of stage, including whether a patient receives surgery, chemotherapy, or radiotherapy; how aggressively the patient is treated; the intensity of post-treatment follow-up; and eligibility for clinical trials. We have demonstrated that changing the criteria to make decisions has important clinical consequences; therefore, we also hypothesize that the use of such models would improve cancer care either by decreasing the number of patients subject to unnecessary treatment or by decreasing event rates as a result of better identification of which patients require intervention.

Multivariate models have 2 additional advantages over crude risk stratifications based on criteria such as stage. First, multivariate models allow for the individualization of care. Patients may differ with respect to the relative value they place on treatment toxicities and disease recurrence, and a model allows patients to vary with respect to the thresholds they use for treatment. Second, multivariate models allow for the addition of prognostic markers when and if it is demonstrated that they have benefit. In the case of bladder cancer, for example, it has been suggested that the status of markers such as cyclin E1, p53, p21, retinoblastoma protein, and p27 may distinguish more aggressive tumors from less aggressive tumors22, 23; similarly, genomic analyses have indicated that certain patterns of gene expression may be associated with cancer outcome.24 If these markers are validated, then it will remain unclear how they could be incorporated into a staging system without an unmanageable expansion of multiple categories (high stage/lymph node-negative/low p53/low genomic risk; high stage/lymph node-negative/high p53/low genomic risk; and so on). Conversely, such markers easily can be incorporated into multivariate models.

There are several possible limitations to our study. First, the dataset was used to generate the predictive model that was used to assess it. That said, we do not believe that this resulted in statistical over fit on the grounds that there was a very large number of events. Indeed, we conducted some preliminary analyses to estimate statistical “optimism”25 and observed that this was close to zero (eg, using a cutoff probability of 25% and an RR of 0.8, the optimism for the net benefit from the nomogram was 0.0001). It also is possible that patients who are treated in the community differ systematically from patients who are treated at academic centers, who contribute data to the nomogram. But even if this is true, it is not clear that this would favor either the nomogram or the standard criteria for chemotherapy. For example, if stage is assessed poorly in a community setting or if lymph node removal is less extensive, then it would reduce the predictive accuracy of the nomogram, but it also would affect the predictive value of a decision rule based on stage and lymph node status alone.

A clear limitation of this study, as discussed above, is that, despite being a National Comprehensive Cancer Network guideline standard of care, adjuvant therapy for bladder cancer is not unequivocally considered to be of benefit. However, it can be argued that the quality of evidence for adjuvant therapy is not directly relevant to our findings unless one takes the position that adjuvant therapy does not and cannot possibly work for bladder cancer and should never be subjected to a clinical trial. In the absence of such a position, a decision will have to be made regarding which patients receive adjuvant therapy, whether as a clinical decision rule or the as eligibility criteria for a trial. Our Figure 3 illustrates that, irrespective of the characteristics of the adjuvant agent—whether its toxicity is high or low and whether it is of great or only moderate effectiveness—the predicted clinical results will be superior if the nomogram rather than standard pathologic criteria are used to determine which patients receive treatment.

Although it may be desirable in principle, subjecting a nomogram-based strategy for chemotherapy referral to prospective experimental trial is of doubtful feasibility. For example, imagine that we wanted to demonstrate that a nomogram-based strategy would lead to a recurrence rate similar to that achieved by a standard nomogram but required that fewer patients receive chemotherapy. A noninferiority trial designed to test whether the nomogram did not increase recurrence rates by >1% well may require >25,000 patients.

This article is presented in the spirit of “proof of principle.” We believe that we are the first to demonstrate clearly that use of a prediction model to inform decisions regarding chemotherapy would improve clinical outcome. That said, there are several steps that would need to be taken before the prediction model could be used in the clinic. First, we believe that the prediction model itself should be updated. The original bladder nomogram was developed for general use rather than for the specific purpose of chemotherapy referral. Consequently, the dataset that was used to develop the nomogram included patients who would not be considered for adjuvant chemotherapy, such as those of advanced age or those diagnosed with squamous cell carcinoma. Second, in this report, we chose illustrative cutoff levels for the nomogram (10%, 25%, and 70%). The alternative would be to choose optimal cutoff levels for each of several scenarios of drug effectiveness and tolerability. Third, any statistical model would have to be implemented in a user-friendly format, perhaps in a web version similar to Adjuvantonline.7

In the current study, we have demonstrated that referring patients to chemotherapy on the basis of a multivariate model is likely to lead to better patient outcomes than the use of pathologic groups. Given the importance of this finding—that we can improve outcome merely by changing the basis on which we make decisions—we recommend research on other multivariate prediction models to determine their clinical effects.

### Conflict of Interest Disclosures

This research was funded by a P50-CA92629 Specialized Program of Research Excellence (SPORE) grant from the National Cancer Institute.

### References

- 1Postoperative nomogram predicting the 10-year probability of prostate cancer recurrence after radical prostatectomy. J Clin Oncol. 2005; 23: 7005-7012., , , et al.
- 2Preoperative probability model for predicting overall survival after resection of pulmonary metastases from colorectal cancer. Br J Surg. 2004; 91: 112-120., , , .
- 3Postoperative nomogram for disease-specific survival after an R0 resection for gastric carcinoma. J Clin Oncol. 2003; 21: 3647-3650., , , .
- 4A nomogram for predicting the likelihood of additional nodal metastases in breast cancer patients with a positive sentinel node biopsy. Ann Surg Oncol. 2003; 10: 1140-1151., , , et al.
- 5Multi-institutional validation of a new renal cancer-specific survival nomogram. J Clin Oncol. 2007; 25: 1316-1322., , , et al.
- 6Prognostic nomogram and index for overall survival in previously untreated patients with chronic lymphocytic leukemia. Blood. 2007; 109: 4679-4685., , , et al.
- 7Computer program to assist in making decisions about adjuvant therapy for women with early breast cancer. J Clin Oncol. 2001; 19: 980-991., , , et al.
- 8A nomogram that predicts the presence of sentinel node metastasis in melanoma with better discrimination than the American Joint Committee on Cancer staging system. Ann Surg Oncol. 2005; 12: 282-288., , , .
- 9Validation of a nomogram for predicting disease-specific survival after an R0 resection for gastric carcinoma. Cancer. 2005; 103: 702-707., , , et al.
- 10Validation of a postresection pancreatic adenocarcinoma nomogram for disease-specific survival. J Clin Oncol. 2005; 23: 7529-7535., , , , , .
- 11Pretreatment nomogram for predicting the outcome of 3-dimensional conformal radiotherapy in prostate cancer. J Clin Oncol. 2000; 18: 3352-3359., , , , , .
- 12Postoperative nomogram predicting risk of recurrence after radical cystectomy for bladder cancer. J Clin Oncol. 2006; 24: 3967-3972., , .
- 13Adjuvant chemotherapy in muscle-invasive bladder carcinoma: a pooled analysis from phase III studies. Cancer. 2006; 106: 783-788., , , et al.
- 14Advanced Bladder Cancer (ABC) Meta-analysis Collaboration. Adjuvant chemotherapy in invasive bladder cancer: a systematic review and meta-analysis of individual patient data. Advanced Bladder Cancer (ABC) Meta-analysis Collaboration. Eur Urol. 2005; 48: 189-199; discussion 199-201.
- 15National Comprehensive Cancer Network Guidelines. Bladder Cancer: Clinical Practice Guidelines in Oncology (v 1.2007). Available at http://www.nccn.org.
- 16Selecting patients for randomized trials: a systematic approach based on risk group [serial online]. Trials. 2006; 7: 30., , .
- 17Comparing tumour staging and grading systems: a case study and a review of the issues, using thymoma as a model. Stat Med. 2000; 19: 1997-2014., , , .
- 18When should an effective treatment be used? Derivation of the threshold number needed to treat and the minimum event rate for treatment. J Clin Epidemiol. 2001; 54: 253-262., , , , .
- 19Estimating sample sizes for binary, ordered categorical, and continuous outcomes in 2 group comparisons. BMJ. 1995; 311: 1145-1148., , .
- 20Sufficiently important difference: expanding the framework of clinical significance. Med Decis Making. 2005; 25: 250-261., , , .
- 21A class of K-sample tests for comparing the cumulative incidence of a competing risk. Ann Stat. 1988; 16: 1141-1154..
- 22Accumulation of nuclear p53 and tumor progression in bladder cancer. N Engl J Med. 1994; 331: 1259-1264., , , et al.
- 23Multiple biomarkers improve prediction of bladder cancer recurrence and mortality in patients undergoing cystectomy. Cancer. 2008; 112: 315-325., , , et al.
- 24Bladder cancer stage and outcome by array-based comparative genomic hybridization. Clin Cancer Res. 2005; 11( 19 pt 1): 7012-7022., , , et al.
- 25Regression Modelling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. New York, NY: Springer; 2001..