Generalizability of cancer clinical trial results

Prognostic differences between participants and nonparticipants


  • Linda S. Elting Dr.P.H.,

    Corresponding author
    1. Department of Biostatistics and Applied Mathematics, The University of Texas M. D. Anderson Cancer Center, Houston, Texas
    • Department of Biostatistics and Applied Mathematics, The University of Texas M. D. Anderson Cancer Center, 1515 Holcombe Boulevard, Unit 196, Houston, TX 77030===

    Search for more papers by this author
    • Fax: (713) 563-4246

  • Catherine Cooksley Dr.P.H.,

    1. Department of Biostatistics and Applied Mathematics, The University of Texas M. D. Anderson Cancer Center, Houston, Texas
    Search for more papers by this author
  • B. Nebiyou Bekele Ph.D.,

    1. Department of Biostatistics and Applied Mathematics, The University of Texas M. D. Anderson Cancer Center, Houston, Texas
    Search for more papers by this author
  • Michael Frumovitz M.D.,

    1. Department of Gynecological Oncology, The University of Texas M. D. Anderson Cancer Center, Houston, Texas
    Search for more papers by this author
  • Elenir B. C. Avritscher M.D.,

    1. Department of Biostatistics and Applied Mathematics, The University of Texas M. D. Anderson Cancer Center, Houston, Texas
    Search for more papers by this author
  • Charlotte Sun Dr.P.H.,

    1. Department of Gynecological Oncology, The University of Texas M. D. Anderson Cancer Center, Houston, Texas
    Search for more papers by this author
  • Diane C. Bodurka M.D.

    1. Department of Gynecological Oncology, The University of Texas M. D. Anderson Cancer Center, Houston, Texas
    Search for more papers by this author

  • Presented in part at the 38th Annual Meeting of the American Society of Clinical Oncology, May 18-21, 2002, Orlando, Florida.

  • The funding entities were not involved in the design or conduct of the study, analysis of results, interpretation of findings, or preparation of this report.



The generalizability of clinical trial results is questionable, because fewer than 5% of cancer patients participate. The authors examined the comparability of clinical trial participants and nonparticipants and the potential impact of differences.


A retrospective cohort of 19,340 cancer patients who were diagnosed between January 1990 and December 1997 was characterized by trial participation. The distributions of prognostically important factors among trial participants were compared with the distributions among nonparticipants and the population of patients diagnosed during the same period in the Surveillance, Epidemiology, and End Results population. The impact of these factors on survival was examined by using a Cox proportional hazards analysis.


Trial participants were younger and had better performance status and fewer comorbid conditions compared with nonparticipants. However, participants were more likely to have locally advanced disease, positive lymph node status, poorly differentiated tumors, liver metastases, and multiple metastatic sites. The former factors were associated with significantly longer survival, whereas the later factors were associated with significantly shorter survival.


The lack of comparability between trial participants and nonparticipants called into question the generalizability of clinical trial results. Although selective recruitment for clinical trials is justified, the authors encourage the use of population-based trials of effectiveness in “all comers.” Cancer 2006. © 2006 American Cancer Society.

Clinical equipoise requires a reasonable likelihood that investigational treatments, at the least, will be as effective as standard therapy.1, 2 Failing this, the recruitment of participants in clinical trials would be unethical in all circumstances except those in which no effective therapy exists. Generally, these hopes for effectiveness are realized, and the majority of patients enjoy benefit from standard and investigational treatments. However, there is a widespread perception that participation in clinical trials confers benefits in addition to those derived from treatment.3, 4 This hypothesis has been investigated in a number of studies.5–11 However, a recent report summarizing previous research on this topic detailed the shortcomings of those studies.4 In particular, that review pointed to the failure of previous studies to account for important prognostic differences between clinical trial participants and nonparticipants.

These prognostic differences can have important implications. Ethically, the exposure of patients to any risks posed by clinical trials is justified only on the basis of expectations of benefit from the knowledge gained. A condition of gaining knowledge from clinical trials is the generalizability of the results to the population at large. Bias in recruitment to clinical trials reduces generalizability and, thus, the benefit of the knowledge gained from trials. Fewer than 5% of cancer patients participate in clinical trials; therefore, it is no surprise that the generalizability of the results has been questioned and studied.12–22

We evaluated the comparability of clinical trial participants and nonparticipants in a large cohort of patients who were treated at a comprehensive cancer center with the objective of examining the generalizability of the study findings. We also compared the characteristics of clinical trial participants with the characteristics in the Surveillance, Epidemiology, and End Results (SEER) population of cancer patients who were diagnosed during the same period. Finally, we examined the impact of the observed differences on survival.


We constructed a cohort that consisted of all 62,562 patients with newly diagnosed cancer who presented at The University of Texas M. D. Anderson Cancer Center (M. D. Anderson) between January 1, 1990 and December 31, 1997 and were residents of the U.S. This time frame allowed for a minimum 7-year follow-up of clinical trial participation for all patients. From this cohort, we excluded 5097 patients who had multiple primary cancers; 20,515 patients who had received surgical or hormone therapy, chemotherapy, or radiotherapy for their cancer prior to registration at M. D. Anderson; 11,185 patients who were seen for second opinions only; and 571 patients who had squamous cell and basal cell tumors of the skin. We also extracted patients with newly diagnosed cancer from the SEER Cancer Incidence public use data base (August 2000 submission) using the same exclusion criteria. Clinical trial participants who were treated at M. D. Anderson were compared with nonparticipants who were treated at M.D. Anderson and with patients from the SEER data base (for whom clinical trial participation was not known). The Institutional Review Board at M. D. Anderson approved the study protocol and granted waivers of informed consent and authorization.

Trial Participation

We characterized patients from M. D. Anderson as early trial participants if they were enrolled on a treatment clinical trial during the 4 months after registration at M. D. Anderson by using the definition of initial treatment employed by all tumor registries. Those who participated in a clinical trial >4 months after registration were considered late trial participants. Patients who participated only in psychosocial, behavioral, or diagnostic studies and those who never participated in a treatment clinical trial were considered nonparticipants. Participation information was obtained from the Protocol Data Management System. Registration of every patient who participates in a clinical trial is mandatory. Nonparticipants were treated according to disease-specific and stage-specific institutional guidelines that reflected the current standard of care.


Survival was compared among participants and nonparticipants with solid tumors who were treated at M. D. Anderson. It was computed from the date of registration to the date of death or December 31, 2001, whichever came first. Because all patients were newly diagnosed, this definition of survival closely approximated overall survival. The date of death was obtained from the M. D. Anderson Tumor Registry, which continuously is updated prospectively as new information becomes available. This information is supplemented by death certificate information obtained from a monthly search of the records of all deaths from the Bureaus of Vital Statistics from Texas and surrounding states. Follow-up of inactive patients is conducted through annual phone calls or letters.

Confounding Factors

Patients with newly diagnosed cancer are a heterogeneous population, and their clinical and socioeconomic characteristics vary in important ways (Table 1). Therefore, we categorized patients by site and stage of cancer. There also is considerable within-stage variation among patients with cancer. Thus, we also accounted for histology, grade, size of primary solid tumors, and extension to regional lymph nodes. Patients with solid tumors were categorized according to the number of metastatic sites and the presence or absence of liver metastases.

Table 1. Characteristics of Patients and Correlations with Trial Participation
CharacteristicTotal No. of Patients (n = 19,340)No. Ever on Trial (n = 6321)Percent on Trial (95% CI)
  1. 95% CI: 95% confidence interval.

 Male9943339134 (33–35)
 Female9397293031 (30–32)
 Non-Hispanic white14,611488533 (33–34)
 African American208653826 (24–28)
 Hispanic223776734 (32–36)
 Other race40613132 (28–37)
Marital status   
 Married12,791439134 (34–35)
 Not married6549192929 (28–31)
Medically indigent   
 Yes292298434 (32–35)
 No16,418533733 (32–33)
Age, y   
 <2062621034 (30–37)
 20–547670290038 (37–39)
 55–707727234630 (29–31)
 >70331767720 (19–22)
Comorbid condition   
 Yes331197229 (28–31)
 No16,029534933 (33–34)
Zubrod performance status   
 >0212850424 (22–26)
 017,212568133 (32–34)
 Hematologic malignancy2700203175 (74–77)
 Solid tumor16,440429026 (25–26)
 Limited local disease373343412 (11–13)
 Locally advanced disease168746928 (26–30)
 Local with direct extension265655521 (19–22)
 Regional lymph nodes3765133035 (34–37)
 Distant metastases4799150231 (30–37)
  Liver metastases164062938 (36–41)
  Single metastatic site102827226 (24–29)
  >1 metastatic site3772123033 (31–34)

Although cancer status is an important predictor of survival, measurement of the patient's clinical condition adds an important dimension to the analysis. At the time of registration, we characterized each patient according to their age and performance status by using the Zubrod score23 and the Dartmouth-Manitoba version of the Charlson comorbidity score.24, 25 Finally, we measured socioeconomic factors that have been shown to be associated with variation in mortality,26 including gender, marital status, race, ethnicity, and medical indigence (which we defined as an absence of medical insurance).

Statistical Considerations

We compared the characteristics of participants and nonparticipants by using 2-tailed tests of significance. We then examined the impact of the observed differences between participants and nonparticipants on survival. For this analysis, we separated the 2 major diagnostic groups among solid tumors, localized tumors, and metastatic tumors. First, we examined the impact of each factor on survival univariately, then multivariately, using Cox proportional hazards analysis. We then examined survival in trial participants and nonparticipants, adjusting for factors that demonstrated a significant association with survival in earlier analyses. All analyses were computed by using Stata software (Stata Press, College Station, TX).


In total, 19,340 patients at M. D. Anderson met all eligibility criteria. Among these, males (51%), non-Hispanic whites (75%), and patients with solid tumors (86%) predominated (Table 1). However, 11% of patients were African American, 12% were Hispanic, and 14% had hematologic malignancies. Among the solid tumors, breast, prostate, and lung cancers were most common (Fig. 1).

Figure 1.

Tumors are illustrated according to their distribution. CNS indicates central nervous system; GYN, gynecologic; GU, genitourinary.

Clinical trial participation varied significantly by gender and by marital status, although the magnitudes of the differences were not large (Table 1). African Americans were far less likely to participate in clinical trials than their white, Hispanic, or other-race counterparts. Participation decreased with increasing age, although 20% of patients older than age 70 years participated in trials. Among patients with solid tumors who had limited local disease, for whom conventional therapy provides a high likelihood of cure, participation was less common (12%) than among patients who had lymph node involvement (35%) or distant metastases (31%), for whom conventional therapy may be ineffective. Patients with hematologic malignancies were significantly more likely to participate in clinical trials than their counterparts with solid tumors, regardless of disease stage (75% vs. 26%; P<.001).

Are Participants and Nonparticipants Comparable?

The differences in participation rates led to significant differences in the distribution of important confounding factors between participants and nonparticipants (Table 2). Overall, participants were in better health, but they had more extensive cancer than their nonparticipant counterparts. Specifically, participants were younger, less likely to have chronic comorbid conditions, and had better performance status than nonparticipants. However, participants were more likely to have lymph node involvement and distant metastases. Among those with metastatic disease, participants were more likely to have liver metastases and had more metastatic sites than nonparticipants. Among those without metastatic disease, participants were more likely to have locally advanced disease and/or extension to regional lymph nodes than nonparticipants. Early and late trial participants were very similar.

Table 2. Comparison of Baseline Characteristics of Early and Late Participants and Nonparticipants
CharacteristicNo. of Patients (%)P Value
Early Participants (n = 5122)Late Participants (n = 1199)Nonparticipants (n = 13,019)
Male gender2747 (54)644 (54)6552 (50)<.001
Non-Hispanic white3975 (78)910 (76)9726 (74)<.001
African American426 (8)112 (9)1548 (12)<.001
Hispanic614 (12)153 (13)1470 (11).03
Other race107 (2)24 (2)275 (2)<.001
Married3596 (70)795 (66)8400 (64)<.001
Medically indigent765 (15)219 (18)1938 (15).07
Age, y    
 <20147 (3)63 (5)416 (3)<.001
 20–542481 (47)607 (51)4582 (35)<.001
 55–701916 (38)430 (36)5381 (42)<.001
 >70578 (11)99 (8)2640 (20)<.001
Comorbid condition753 (15)143 (12)2339 (18).001
Zubrod performance status >0435 (8)69 (6)1488 (11).03
Solid tumor3338 (65)952 (80)12,350 (93)<.001
 Limited local disease307 (9)127 (13)3299 (27)<.001
 Locally advanced394 (12)75 (8)1218 (10)<.001
 Local with direct extension427 (13)128 (13)2101 (17)<.001
 Regional lymph nodes1031 (31)299 (32)2435 (20)<.001
 Distant metastases1179 (35)323 (34)3297 (26)<.001
  Liver metastases516 (44)113 (35)1011 (31)<.001
  >1 metastatic site953 (81)277 (86)2541 (77)<.001

Are Participants Comparable to All Cancer Patients?

The comparability of participants with nonparticipants in the same institution probably is less important than their comparability with all cancer patients at the population level. However, the differences observed were more extreme, particularly with respect to age and stage of disease. Participants were more likely to be male (54% vs. 49%) and married (69% vs. 60%) than newly diagnosed patients from the SEER data. Participants were less likely to be age >70 years (14% vs. 32%) compared with the SEER population, and they were more likely to have metastatic disease (35% vs. 25%).

Impact of Differences on Survival

Examination of the impact of the observed differences on survival revealed the potential importance to generalizations of clinical trial results to the population level. Tumor site and characteristics, measures of extent of disease, and measures of clinical status were most important, but sociodemographic factors like gender, race, and insurance status also were associated significantly with survival (Table 3).

Table 3. Factors Associated with Mortality*
FactorHazard Ratio (95% CI)
Solid TumorsHematologic Malignancies (n = 2633)
Local Disease (n = 11,838)Metastatic Disease (n = 4552)
  • 95% CI indicates 95% confidence interval; NS, not significant; NA, not applicable; AML, acute myeloid leukemia; ALL, acute lymphocytic leukemia.

  • *

    This table shows the results of a multivariate Cox proportional hazards analysis of all factors except clinical trial participation.

  • P value: .05>P>.001; all other factors were significant at the P<.001 level unless otherwise specified.

Male gender1.16 (1.08–1.25)1.11 (1.03–1.19)1.20 (1.06–1.36)
Age >70 y1.64 (1.52–1.77)1.13 (1.03–1.24)1.79 (1.53–2.10)
African-American race1.18 (1.08–1.31)1.13 (1.02–1.24)1.32 (1.08–1.61)
Medically indigent1.17 (1.07–1.29)1.16 (1.06–1.26)NS
Zubrod performance status >03.27 (2.94–3.65)2.27 (2.12–2.44)2.92 (2.51–3.41)
Comorbid condition1.51 (1.40–1.63)1.21 (1.11–1.31)1.95 (1.69–2.25)
Poorly differentiated tumor1.38 (1.29–1.48)1.29 (1.21–1.38)NA
Locally advanced tumor1.65 (1.53–1.77)NANA
Positive regional lymph nodes1.93 (1.80–2.07)NSNA
Liver primary2.63 (2.09–3.31)1.77 (1.43–2.20)NA
Esophageal primary2.28 (1.96–2.66)1.44 (1.21–1.72)NA
Pancreas primary3.34 (2.88–3.89)1.94 (1.65–2.28)NA
Gastric primary1.43 (1.16–1.76)1.63 (1.35–1.61)NA
Nonsmall cell lung cancer1.90 (1.73–2.09)1.48 (1.36–1.61)NA
Breast cancer0.39 (0.34–0.45)0.52 (0.45–0.60)NA
Prostate cancer0.20 (0.16–0.24)0.42 (0.33–0.54)NA
Ovarian primary0.60 (0.32–0.99)0.55 (0.47–0.65)NA
Liver metastasesNA1.21 (1.12–1.30)NA
>1 metastatic siteNA1.14 (1.07–1.22)NA
Non-Hodgkin lymphomaNANA2.03 (1.40–2.94)
Hodgkin diseaseNANA0.58 (0.36–0.94)
Multiple myelomaNANA2.89 (1.93–4.3)
AMLNANA2.15 (1.78–2.59)
ALLNANA1.75 (1.38–2.22)
Other leukemiasNANA2.37 (1.72–3.27)

Even after adjusting for the influence of the observed confounding factors, it appeared that important differences remained. Clinical trial participants with localized solid tumors had significantly shorter survival compared with nonparticipants (hazard ratio [HR], 1.37; 95% confidence interval [95% CI], 1.29-1.45 [P<.001]). The survival difference was most pronounced among late trial participants (Fig. 2). In contrast, among patients with metastatic solid tumors, trial participation was associated with significantly longer survival (HR, 0.80; 95% CI, 0.76-0.85 [P<.001]).

Figure 2.

Survival is illustrated among patients on study and off study who had localized and metastatic solid tumors. The results were adjusted for gender, age, race, indigence, Zubrod performance status, comorbidities, primary site, histology and grade, number of metastatic sites, and presence of liver metastases. Circles: nonparticipants; squares: early participants; triangles: late participants.


We observed prognostically important differences between participants and nonparticipants and between participants and the cancer population at large. These findings call into question the generalizability of clinical trial results. Specifically, we observed that patients who had unfavorable prognostic factors were significantly less likely to participate in trials. This observation is not unique to our study but has been reported previously in numerous studies of patients with cancer.12, 13, 15, 19, 20, 22 The literature is not consistent with respect to this finding, however. Population-based studies with high participation rates have shown that participants and nonparticipants were very comparable.7, 18, 21 In particular, as great as 60% of pediatric cancer patients participate in clinical trials; thus, the participants are more comparable to the population at large.

We also found evidence of unmeasured bias with important prognostic implications. After controlling for all measured clinical and socioeconomic factors, we observed that patients with metastatic solid tumors who were treated on clinical trials enjoyed superior survival compared with patients who were treated off trials. This finding has been reported previously by Lennox et al. among children with nephroblastoma,5 by Boros et al. among adults with leukemia,8 and by Davis et al. among patients with nonsmall cell lung cancer.6 In contrast, we observed that, among patients with localized solid tumors and patients with hematologic malignancies, survival was superior for patients who were treated off clinical trials.

It is possible that clinical trial participation, with its frequent monitoring and access to new, breakthrough drugs, benefits patients with metastatic disease and that no such benefits accrue to patients with local disease. However, it appears almost implausible that clinical trial participation would lead to poorer outcomes. Furthermore, it is noteworthy that biased recruitment may produce the same results. Among patients with metastatic disease, if the oldest and sickest patients (poor performance status and many comorbid conditions) were excluded generally from clinical trials, the patients who were treated on clinical trials would be expected to have superior survival. Likewise, among patients with localized disease, if clinical trials were reserved for patients who were unlikely to respond to standard therapy, then superior survival would be expected among patients who were treated off clinical trials.

There are good reasons to believe that such bias occurred. The majority of clinical trials exclude patients with serious hepatic or renal dysfunction, and most are reserved for patients with expected survival of at least 3 months. These standard eligibility criteria bias accrual in clinical trials among patients with metastatic disease in favor of patients with superior survival. Similarly, among patients with local disease, those whose primary cancers are small often respond well to surgery, adjuvant chemotherapy, and radiation therapy. Clinical trials typically focus on patients for whom these standard therapies are unlikely to produce durable remissions. Based on these observations, we believe that biased recruitment to clinical trials is the more likely explanation for the observed differences in the current study.

Although some previous authors have attributed better outcomes to a “trial effect,” others have concluded that differences were because of bias in recruitment. Antman et al. reported shorter survival among patients with sarcoma who were treated off trial than among patients who were treated on trial, but those treated off trial were at higher risk.12 Bertelsen also reported biased recruitment of patients with ovarian cancer who had a better prognosis that resulted in inferior survival of patients off trial who received the same treatment.14

These observations point to limitations of this and previous studies. Most studies of this topic, including ours, are retrospective, observational, and involve the analysis of data collected for purposes other than testing the study hypotheses. Therefore, the data are likely to be less reliable.

Despite their limitations, the results of the current study have important implications. Most notably, these results point to the perils of inferring treatment effectiveness (in a population of “all comers”) from trial efficacy (observed in a selected subpopulation). Although randomization may eliminate bias associated with treatment assignment among trial participants, it does not eliminate biases caused by stringent eligibility criteria, failure to offer clinical trials, or refusal to participate. Our current results suggest that bias toward the recruitment of patients with more favorable prognoses (in those with metastatic disease) or less favorable prognoses (in those with localized disease) than the average may lead to corresponding overestimations or underestimations of the benefit that will be observed in the general population. Furthermore, even after controlling for the effects of prognostically important clinical and socioeconomic factors, evidence of bias remained.

We suggest that recruitment for treatment efficacy clinical trials, quite appropriately, is an inherently biased exercise with the objective of comparing the best attainable outcomes of standard and investigational treatments. This objective has intrinsic value and requires no additional benefit to justify continued research. Furthermore, recruitment of higher risk patients, such as the elderly, those with comorbid conditions, or those with widely metastatic disease, probably is inappropriate during the early stages of testing new treatments, because their inclusion confounds this testing. However, information regarding the outcomes of new treatments in such populations is critically important to the everyday practice of oncology. After initial testing, large population-based effectiveness trials of all comers, including those who may not be eligible for early clinical trials, are needed to provide realistic estimates of the benefits of treatment in general oncology practice.


The authors acknowledge with gratitude the efforts of the personnel who develop and maintain The University of Texas M. D. Anderson Cancer Center Tumor Registry and the Protocol Data Management System. Without their efforts, this project would not have been possible.