Fax: (713) 563-4246
Prognostic differences between participants and nonparticipants
Article first published online: 25 APR 2006
Copyright © 2006 American Cancer Society
Volume 106, Issue 11, pages 2452–2458, 1 June 2006
How to Cite
Elting, L. S., Cooksley, C., Bekele, B. N., Frumovitz, M., Avritscher, E. B. C., Sun, C. and Bodurka, D. C. (2006), Generalizability of cancer clinical trial results. Cancer, 106: 2452–2458. doi: 10.1002/cncr.21907
Presented in part at the 38th Annual Meeting of the American Society of Clinical Oncology, May 18-21, 2002, Orlando, Florida.
The funding entities were not involved in the design or conduct of the study, analysis of results, interpretation of findings, or preparation of this report.
- Issue published online: 18 MAY 2006
- Article first published online: 25 APR 2006
- Manuscript Accepted: 19 JAN 2006
- Manuscript Revised: 9 DEC 2005
- Manuscript Received: 7 OCT 2005
- William Randolph Hearst Foundations
- National Cancer Institute. Grant Number: 5-P30 CA16672-27
- National Institutes of Health
- clinical trial participation;
- clinical trial recruitment;
- bias in clinical trials
The generalizability of clinical trial results is questionable, because fewer than 5% of cancer patients participate. The authors examined the comparability of clinical trial participants and nonparticipants and the potential impact of differences.
A retrospective cohort of 19,340 cancer patients who were diagnosed between January 1990 and December 1997 was characterized by trial participation. The distributions of prognostically important factors among trial participants were compared with the distributions among nonparticipants and the population of patients diagnosed during the same period in the Surveillance, Epidemiology, and End Results population. The impact of these factors on survival was examined by using a Cox proportional hazards analysis.
Trial participants were younger and had better performance status and fewer comorbid conditions compared with nonparticipants. However, participants were more likely to have locally advanced disease, positive lymph node status, poorly differentiated tumors, liver metastases, and multiple metastatic sites. The former factors were associated with significantly longer survival, whereas the later factors were associated with significantly shorter survival.
The lack of comparability between trial participants and nonparticipants called into question the generalizability of clinical trial results. Although selective recruitment for clinical trials is justified, the authors encourage the use of population-based trials of effectiveness in “all comers.” Cancer 2006. © 2006 American Cancer Society.
Clinical equipoise requires a reasonable likelihood that investigational treatments, at the least, will be as effective as standard therapy.1, 2 Failing this, the recruitment of participants in clinical trials would be unethical in all circumstances except those in which no effective therapy exists. Generally, these hopes for effectiveness are realized, and the majority of patients enjoy benefit from standard and investigational treatments. However, there is a widespread perception that participation in clinical trials confers benefits in addition to those derived from treatment.3, 4 This hypothesis has been investigated in a number of studies.5–11 However, a recent report summarizing previous research on this topic detailed the shortcomings of those studies.4 In particular, that review pointed to the failure of previous studies to account for important prognostic differences between clinical trial participants and nonparticipants.
These prognostic differences can have important implications. Ethically, the exposure of patients to any risks posed by clinical trials is justified only on the basis of expectations of benefit from the knowledge gained. A condition of gaining knowledge from clinical trials is the generalizability of the results to the population at large. Bias in recruitment to clinical trials reduces generalizability and, thus, the benefit of the knowledge gained from trials. Fewer than 5% of cancer patients participate in clinical trials; therefore, it is no surprise that the generalizability of the results has been questioned and studied.12–22
We evaluated the comparability of clinical trial participants and nonparticipants in a large cohort of patients who were treated at a comprehensive cancer center with the objective of examining the generalizability of the study findings. We also compared the characteristics of clinical trial participants with the characteristics in the Surveillance, Epidemiology, and End Results (SEER) population of cancer patients who were diagnosed during the same period. Finally, we examined the impact of the observed differences on survival.
MATERIALS AND METHODS
We constructed a cohort that consisted of all 62,562 patients with newly diagnosed cancer who presented at The University of Texas M. D. Anderson Cancer Center (M. D. Anderson) between January 1, 1990 and December 31, 1997 and were residents of the U.S. This time frame allowed for a minimum 7-year follow-up of clinical trial participation for all patients. From this cohort, we excluded 5097 patients who had multiple primary cancers; 20,515 patients who had received surgical or hormone therapy, chemotherapy, or radiotherapy for their cancer prior to registration at M. D. Anderson; 11,185 patients who were seen for second opinions only; and 571 patients who had squamous cell and basal cell tumors of the skin. We also extracted patients with newly diagnosed cancer from the SEER Cancer Incidence public use data base (August 2000 submission) using the same exclusion criteria. Clinical trial participants who were treated at M. D. Anderson were compared with nonparticipants who were treated at M.D. Anderson and with patients from the SEER data base (for whom clinical trial participation was not known). The Institutional Review Board at M. D. Anderson approved the study protocol and granted waivers of informed consent and authorization.
We characterized patients from M. D. Anderson as early trial participants if they were enrolled on a treatment clinical trial during the 4 months after registration at M. D. Anderson by using the definition of initial treatment employed by all tumor registries. Those who participated in a clinical trial >4 months after registration were considered late trial participants. Patients who participated only in psychosocial, behavioral, or diagnostic studies and those who never participated in a treatment clinical trial were considered nonparticipants. Participation information was obtained from the Protocol Data Management System. Registration of every patient who participates in a clinical trial is mandatory. Nonparticipants were treated according to disease-specific and stage-specific institutional guidelines that reflected the current standard of care.
Survival was compared among participants and nonparticipants with solid tumors who were treated at M. D. Anderson. It was computed from the date of registration to the date of death or December 31, 2001, whichever came first. Because all patients were newly diagnosed, this definition of survival closely approximated overall survival. The date of death was obtained from the M. D. Anderson Tumor Registry, which continuously is updated prospectively as new information becomes available. This information is supplemented by death certificate information obtained from a monthly search of the records of all deaths from the Bureaus of Vital Statistics from Texas and surrounding states. Follow-up of inactive patients is conducted through annual phone calls or letters.
Patients with newly diagnosed cancer are a heterogeneous population, and their clinical and socioeconomic characteristics vary in important ways (Table 1). Therefore, we categorized patients by site and stage of cancer. There also is considerable within-stage variation among patients with cancer. Thus, we also accounted for histology, grade, size of primary solid tumors, and extension to regional lymph nodes. Patients with solid tumors were categorized according to the number of metastatic sites and the presence or absence of liver metastases.
|Characteristic||Total No. of Patients (n = 19,340)||No. Ever on Trial (n = 6321)||Percent on Trial (95% CI)|
|Non-Hispanic white||14,611||4885||33 (33–34)|
|African American||2086||538||26 (24–28)|
|Other race||406||131||32 (28–37)|
|Not married||6549||1929||29 (28–31)|
|Zubrod performance status|
|Hematologic malignancy||2700||2031||75 (74–77)|
|Solid tumor||16,440||4290||26 (25–26)|
|Limited local disease||3733||434||12 (11–13)|
|Locally advanced disease||1687||469||28 (26–30)|
|Local with direct extension||2656||555||21 (19–22)|
|Regional lymph nodes||3765||1330||35 (34–37)|
|Distant metastases||4799||1502||31 (30–37)|
|Liver metastases||1640||629||38 (36–41)|
|Single metastatic site||1028||272||26 (24–29)|
|>1 metastatic site||3772||1230||33 (31–34)|
Although cancer status is an important predictor of survival, measurement of the patient's clinical condition adds an important dimension to the analysis. At the time of registration, we characterized each patient according to their age and performance status by using the Zubrod score23 and the Dartmouth-Manitoba version of the Charlson comorbidity score.24, 25 Finally, we measured socioeconomic factors that have been shown to be associated with variation in mortality,26 including gender, marital status, race, ethnicity, and medical indigence (which we defined as an absence of medical insurance).
We compared the characteristics of participants and nonparticipants by using 2-tailed tests of significance. We then examined the impact of the observed differences between participants and nonparticipants on survival. For this analysis, we separated the 2 major diagnostic groups among solid tumors, localized tumors, and metastatic tumors. First, we examined the impact of each factor on survival univariately, then multivariately, using Cox proportional hazards analysis. We then examined survival in trial participants and nonparticipants, adjusting for factors that demonstrated a significant association with survival in earlier analyses. All analyses were computed by using Stata software (Stata Press, College Station, TX).
In total, 19,340 patients at M. D. Anderson met all eligibility criteria. Among these, males (51%), non-Hispanic whites (75%), and patients with solid tumors (86%) predominated (Table 1). However, 11% of patients were African American, 12% were Hispanic, and 14% had hematologic malignancies. Among the solid tumors, breast, prostate, and lung cancers were most common (Fig. 1).
Clinical trial participation varied significantly by gender and by marital status, although the magnitudes of the differences were not large (Table 1). African Americans were far less likely to participate in clinical trials than their white, Hispanic, or other-race counterparts. Participation decreased with increasing age, although 20% of patients older than age 70 years participated in trials. Among patients with solid tumors who had limited local disease, for whom conventional therapy provides a high likelihood of cure, participation was less common (12%) than among patients who had lymph node involvement (35%) or distant metastases (31%), for whom conventional therapy may be ineffective. Patients with hematologic malignancies were significantly more likely to participate in clinical trials than their counterparts with solid tumors, regardless of disease stage (75% vs. 26%; P<.001).
Are Participants and Nonparticipants Comparable?
The differences in participation rates led to significant differences in the distribution of important confounding factors between participants and nonparticipants (Table 2). Overall, participants were in better health, but they had more extensive cancer than their nonparticipant counterparts. Specifically, participants were younger, less likely to have chronic comorbid conditions, and had better performance status than nonparticipants. However, participants were more likely to have lymph node involvement and distant metastases. Among those with metastatic disease, participants were more likely to have liver metastases and had more metastatic sites than nonparticipants. Among those without metastatic disease, participants were more likely to have locally advanced disease and/or extension to regional lymph nodes than nonparticipants. Early and late trial participants were very similar.
|Characteristic||No. of Patients (%)||P Value|
|Early Participants (n = 5122)||Late Participants (n = 1199)||Nonparticipants (n = 13,019)|
|Male gender||2747 (54)||644 (54)||6552 (50)||<.001|
|Non-Hispanic white||3975 (78)||910 (76)||9726 (74)||<.001|
|African American||426 (8)||112 (9)||1548 (12)||<.001|
|Hispanic||614 (12)||153 (13)||1470 (11)||.03|
|Other race||107 (2)||24 (2)||275 (2)||<.001|
|Married||3596 (70)||795 (66)||8400 (64)||<.001|
|Medically indigent||765 (15)||219 (18)||1938 (15)||.07|
|<20||147 (3)||63 (5)||416 (3)||<.001|
|20–54||2481 (47)||607 (51)||4582 (35)||<.001|
|55–70||1916 (38)||430 (36)||5381 (42)||<.001|
|>70||578 (11)||99 (8)||2640 (20)||<.001|
|Comorbid condition||753 (15)||143 (12)||2339 (18)||.001|
|Zubrod performance status >0||435 (8)||69 (6)||1488 (11)||.03|
|Solid tumor||3338 (65)||952 (80)||12,350 (93)||<.001|
|Limited local disease||307 (9)||127 (13)||3299 (27)||<.001|
|Locally advanced||394 (12)||75 (8)||1218 (10)||<.001|
|Local with direct extension||427 (13)||128 (13)||2101 (17)||<.001|
|Regional lymph nodes||1031 (31)||299 (32)||2435 (20)||<.001|
|Distant metastases||1179 (35)||323 (34)||3297 (26)||<.001|
|Liver metastases||516 (44)||113 (35)||1011 (31)||<.001|
|>1 metastatic site||953 (81)||277 (86)||2541 (77)||<.001|
Are Participants Comparable to All Cancer Patients?
The comparability of participants with nonparticipants in the same institution probably is less important than their comparability with all cancer patients at the population level. However, the differences observed were more extreme, particularly with respect to age and stage of disease. Participants were more likely to be male (54% vs. 49%) and married (69% vs. 60%) than newly diagnosed patients from the SEER data. Participants were less likely to be age >70 years (14% vs. 32%) compared with the SEER population, and they were more likely to have metastatic disease (35% vs. 25%).
Impact of Differences on Survival
Examination of the impact of the observed differences on survival revealed the potential importance to generalizations of clinical trial results to the population level. Tumor site and characteristics, measures of extent of disease, and measures of clinical status were most important, but sociodemographic factors like gender, race, and insurance status also were associated significantly with survival (Table 3).
|Factor||Hazard Ratio (95% CI)|
|Solid Tumors||Hematologic Malignancies (n = 2633)|
|Local Disease (n = 11,838)||Metastatic Disease (n = 4552)|
|Male gender||1.16 (1.08–1.25)||1.11 (1.03–1.19)†||1.20 (1.06–1.36)†|
|Age >70 y||1.64 (1.52–1.77)||1.13 (1.03–1.24)†||1.79 (1.53–2.10)|
|African-American race||1.18 (1.08–1.31)||1.13 (1.02–1.24)†||1.32 (1.08–1.61)†|
|Medically indigent||1.17 (1.07–1.29)||1.16 (1.06–1.26)||NS|
|Zubrod performance status >0||3.27 (2.94–3.65)||2.27 (2.12–2.44)||2.92 (2.51–3.41)|
|Comorbid condition||1.51 (1.40–1.63)||1.21 (1.11–1.31)||1.95 (1.69–2.25)|
|Poorly differentiated tumor||1.38 (1.29–1.48)||1.29 (1.21–1.38)||NA|
|Locally advanced tumor||1.65 (1.53–1.77)||NA||NA|
|Positive regional lymph nodes||1.93 (1.80–2.07)||NS||NA|
|Liver primary||2.63 (2.09–3.31)||1.77 (1.43–2.20)||NA|
|Esophageal primary||2.28 (1.96–2.66)||1.44 (1.21–1.72)||NA|
|Pancreas primary||3.34 (2.88–3.89)||1.94 (1.65–2.28)||NA|
|Gastric primary||1.43 (1.16–1.76)||1.63 (1.35–1.61)||NA|
|Nonsmall cell lung cancer||1.90 (1.73–2.09)||1.48 (1.36–1.61)||NA|
|Breast cancer||0.39 (0.34–0.45)||0.52 (0.45–0.60)||NA|
|Prostate cancer||0.20 (0.16–0.24)||0.42 (0.33–0.54)||NA|
|Ovarian primary||0.60 (0.32–0.99)†||0.55 (0.47–0.65)||NA|
|Liver metastases||NA||1.21 (1.12–1.30)||NA|
|>1 metastatic site||NA||1.14 (1.07–1.22)||NA|
|Non-Hodgkin lymphoma||NA||NA||2.03 (1.40–2.94)|
|Hodgkin disease||NA||NA||0.58 (0.36–0.94)†|
|Multiple myeloma||NA||NA||2.89 (1.93–4.3)|
|Other leukemias||NA||NA||2.37 (1.72–3.27)|
Even after adjusting for the influence of the observed confounding factors, it appeared that important differences remained. Clinical trial participants with localized solid tumors had significantly shorter survival compared with nonparticipants (hazard ratio [HR], 1.37; 95% confidence interval [95% CI], 1.29-1.45 [P<.001]). The survival difference was most pronounced among late trial participants (Fig. 2). In contrast, among patients with metastatic solid tumors, trial participation was associated with significantly longer survival (HR, 0.80; 95% CI, 0.76-0.85 [P<.001]).
We observed prognostically important differences between participants and nonparticipants and between participants and the cancer population at large. These findings call into question the generalizability of clinical trial results. Specifically, we observed that patients who had unfavorable prognostic factors were significantly less likely to participate in trials. This observation is not unique to our study but has been reported previously in numerous studies of patients with cancer.12, 13, 15, 19, 20, 22 The literature is not consistent with respect to this finding, however. Population-based studies with high participation rates have shown that participants and nonparticipants were very comparable.7, 18, 21 In particular, as great as 60% of pediatric cancer patients participate in clinical trials; thus, the participants are more comparable to the population at large.
We also found evidence of unmeasured bias with important prognostic implications. After controlling for all measured clinical and socioeconomic factors, we observed that patients with metastatic solid tumors who were treated on clinical trials enjoyed superior survival compared with patients who were treated off trials. This finding has been reported previously by Lennox et al. among children with nephroblastoma,5 by Boros et al. among adults with leukemia,8 and by Davis et al. among patients with nonsmall cell lung cancer.6 In contrast, we observed that, among patients with localized solid tumors and patients with hematologic malignancies, survival was superior for patients who were treated off clinical trials.
It is possible that clinical trial participation, with its frequent monitoring and access to new, breakthrough drugs, benefits patients with metastatic disease and that no such benefits accrue to patients with local disease. However, it appears almost implausible that clinical trial participation would lead to poorer outcomes. Furthermore, it is noteworthy that biased recruitment may produce the same results. Among patients with metastatic disease, if the oldest and sickest patients (poor performance status and many comorbid conditions) were excluded generally from clinical trials, the patients who were treated on clinical trials would be expected to have superior survival. Likewise, among patients with localized disease, if clinical trials were reserved for patients who were unlikely to respond to standard therapy, then superior survival would be expected among patients who were treated off clinical trials.
There are good reasons to believe that such bias occurred. The majority of clinical trials exclude patients with serious hepatic or renal dysfunction, and most are reserved for patients with expected survival of at least 3 months. These standard eligibility criteria bias accrual in clinical trials among patients with metastatic disease in favor of patients with superior survival. Similarly, among patients with local disease, those whose primary cancers are small often respond well to surgery, adjuvant chemotherapy, and radiation therapy. Clinical trials typically focus on patients for whom these standard therapies are unlikely to produce durable remissions. Based on these observations, we believe that biased recruitment to clinical trials is the more likely explanation for the observed differences in the current study.
Although some previous authors have attributed better outcomes to a “trial effect,” others have concluded that differences were because of bias in recruitment. Antman et al. reported shorter survival among patients with sarcoma who were treated off trial than among patients who were treated on trial, but those treated off trial were at higher risk.12 Bertelsen also reported biased recruitment of patients with ovarian cancer who had a better prognosis that resulted in inferior survival of patients off trial who received the same treatment.14
These observations point to limitations of this and previous studies. Most studies of this topic, including ours, are retrospective, observational, and involve the analysis of data collected for purposes other than testing the study hypotheses. Therefore, the data are likely to be less reliable.
Despite their limitations, the results of the current study have important implications. Most notably, these results point to the perils of inferring treatment effectiveness (in a population of “all comers”) from trial efficacy (observed in a selected subpopulation). Although randomization may eliminate bias associated with treatment assignment among trial participants, it does not eliminate biases caused by stringent eligibility criteria, failure to offer clinical trials, or refusal to participate. Our current results suggest that bias toward the recruitment of patients with more favorable prognoses (in those with metastatic disease) or less favorable prognoses (in those with localized disease) than the average may lead to corresponding overestimations or underestimations of the benefit that will be observed in the general population. Furthermore, even after controlling for the effects of prognostically important clinical and socioeconomic factors, evidence of bias remained.
We suggest that recruitment for treatment efficacy clinical trials, quite appropriately, is an inherently biased exercise with the objective of comparing the best attainable outcomes of standard and investigational treatments. This objective has intrinsic value and requires no additional benefit to justify continued research. Furthermore, recruitment of higher risk patients, such as the elderly, those with comorbid conditions, or those with widely metastatic disease, probably is inappropriate during the early stages of testing new treatments, because their inclusion confounds this testing. However, information regarding the outcomes of new treatments in such populations is critically important to the everyday practice of oncology. After initial testing, large population-based effectiveness trials of all comers, including those who may not be eligible for early clinical trials, are needed to provide realistic estimates of the benefits of treatment in general oncology practice.
The authors acknowledge with gratitude the efforts of the personnel who develop and maintain The University of Texas M. D. Anderson Cancer Center Tumor Registry and the Protocol Data Management System. Without their efforts, this project would not have been possible.
- 19Impact of selection process on response rate and long-term survival of potential high-dose chemotherapy candidates treated with standard-dose doxorubicin-containing chemotherapy in patients with metastatic breast cancer [see comment]. J Clin Oncol. 1997; 15: 3171–3177., , , et al.
- 22A comparison of elderly patients with aggressive histology lymphoma who were entered or not entered on to a randomized Phase II trial. Leuk Lymphoma. 2000; 38(3–4): 327–334., , .
- 23Appraisal of methods for the study of chemotherapy of cancer in man: comparative trial of nitrogen mustard and trimethylene thiophosphoramide. J Chronic Dis. 1960; 11: 7–33., , , et al.