Fax: (713) 563-9586
The dangers of “cross-trial” and “cross-retrospective experience” comparisons
Examples employing data in the peer-reviewed ovarian cancer literature
Article first published online: 23 MAR 2007
Copyright © 2007 American Cancer Society
Volume 109, Issue 10, pages 1929–1932, 15 May 2007
How to Cite
Markman, M. (2007), The dangers of “cross-trial” and “cross-retrospective experience” comparisons. Cancer, 109: 1929–1932. doi: 10.1002/cncr.22645
- Issue published online: 25 APR 2007
- Article first published online: 23 MAR 2007
- Manuscript Accepted: 15 FEB 2007
- Manuscript Revised: 7 FEB 2007
- Manuscript Received: 18 DEC 2006
Consider the following analyses involving previously published data in the peer-reviewed medical literature comparing outcomes across 1) ovarian cancer trials and 2) retrospective institutional experiences in this malignancy:
Several institutions have reported their individual data regarding long-term (>5 years) risk for recurrence in patients with advanced ovarian cancer who previously had achieved a surgically confirmed complete response to primary chemotherapy.1–3 Table 1 shows that, after cisplatin-based treatment, this rate exceeded 50%,1, 2 whereas for patients who received alkylating agent chemotherapy, the risk of recurrence was <20% (even with follow-up >8 years).3 These data would appear to support the conclusion that alkylating agent therapy is the superior primary chemotherapy strategy in advanced ovarian cancer. Why is this class of antineoplastic agents not the “standard of care” in the management of the malignancy?4
|Chemotherapy||Patients with recurrence, %|
|Alkylating agent chemotherapy†||<20|
In a randomized Phase III trial that was conducted by the Gynecologic Oncology Group (GOG) exploring the clinical utility of primary platinum-based chemotherapy in large-volume, residual, advanced ovarian cancer (stage IV disease and stage III disease with at least a single residual tumor nodule >1 cm in maximum dimension), the median progression-free and overall survival of patients who received cisplatin plus paclitaxel were 14 months and 26 months, respectively, which clearly was not different from the GOG experience in an identical patient population that received cisplatin plus cyclophosphamide (Table 2).5, 6 Based on this analysis, what was the justification for the GOG, other cooperative groups, and practicing oncologists to consider cisplatin plus paclitaxel the new “standard of care” in this clinical setting?4
|Variable||Cisplatin and cyclophosphamide||Cisplatin and paclitaxel|
|Median PFS, mo||13||14|
|Median OS, mo||24||26|
Two independent investigative groups reported objective response rates of >40% associated with the administration of a cisplatin plus gemcitabine combination chemotherapy regimen to a group of patients with ovarian cancer who were defined by the researchers as having platinum-resistant disease (Table 3).7, 8 It is noteworthy that previously published Phase II and Phase III trials that involved multiple antineoplastic agents delivered in this setting have revealed objective response rates of <20%.9, 10 One interpretation of the cisplatin plus gemcitabine data is that they represent a substantial improvement in activity compared with alternative programs in this clinical setting.
|Phase II agents (multiple)||5–20|
|Phase II trials of cisplatin and gemcitabine||>40|
Although the specific “facts” in the 3 “analyses” presented above are stated accurately, the profoundly flawed conclusions result from inappropriate comparisons made between patient populations outside the setting of a well-designed, prospective, randomized trial. Now, consider the following “reanalyses” of each of the 3 scenarios.
The stated recurrence rates noted in Table 1 after platinum-based and alkylating agent therapy reflect the observed institutional experiences with these agents.1–3 However, they do not represent an appropriate evaluation of patients who received the specific regimens that differed only in the drugs employed. “Standard” cisplatin-based treatment generally has included from 5 to 8 months of drug delivery with a second-look surgical evaluation (if this management approach is employed) performed approximately 1 or 2 months after the completion of chemotherapy.5, 11 In sharp contrast, in the past, patients who were administered alkylating agent-based therapy who did not experience disease progression typically received the drug for ≥18 months before surgical reassessment was undertaken (Table 4),3, 12 Thus, during this more prolonged observation phase, a far larger proportion of individuals with persistent disease had the opportunity to exhibit biologic progression that became clinically evident or that, at least, was of sufficient quantity/volume to become detectable (e.g., biopsy-positive, second-look procedure) in contrast to the much shorter time interval from diagnosis to the second surgery associated with cisplatin treatment. Consequently, the overall smaller numbers of patients who had a “negative” reassessment after alkylating agent chemotherapy (both because of the inferiority of nonplatinum-based regimens13, 14 and because of the timing of the evaluation3, 12) appeared to have a superior ultimate prognosis. It is highly likely (although only actual data can address the issue definitively) that, if second-look surgeries were performed routinely in platinum-treated ovarian cancer patients from 18 to 24 months after the initiation of primary chemotherapy, then those individuals found to be “without documented disease” would experience a similarly low risk of recurrence (from that point onward), as noted in the alkylating agent-treated population.
|Variable||Patients with recurrence, %||Approximate timing of second-look assessment (from initiation of chemotherapy), mo|
|Alkylating agent chemotherapy||<20||>18|
The analysis (Table 2), although it involved the management of patients by the same group of gynecologic oncologists employing identical eligibility criteria, represents a cross-trial comparison of populations treated with the 2 chemotherapy programs on consecutive GOG studies (cisplatin/cyclophosphamide on GOG 111 and cisplatin/paclitaxel on GOG 132).5, 6 In fact, the actual differences between the 2 regimens in progression-free survival (P < .001) and overall survival (P < .001) within the same trial (Table 5) were substantial5 and led (along with the results of a second Phase III randomized study15) to an appropriate change in the management paradigm for advanced ovarian cancer.4 What caused the major difference between the outcomes associated with the same cisplatin/paclitaxel study regimens in the 2 studies, in what appear to be “identical” entry criteria for the consecutive trials? Although it is not possible to provide a definitive answer to this provocative question, it is logical to assume that there were unanticipated and unappreciated differences in the patient populations that participated in the 2 specific trials. One hypothesis is that the legitimate concern for “sudden death” and “severe cardiac toxicity” that surrounded the very early experience with paclitaxel led to a relative decrease in participation in the initial randomized trial of women with clinically relevant comorbidities in addition to known cardiac dysfunction (which was an exclusion criteria for the trial).16, 17 When this Phase III study revealed the absence of serious cardiac events5 and it became clear that paclitaxel-associated hypersensitivity was a manageable drug-related toxicity, it is possible that the subsequent trial (with the lower survival in the paclitaxel-containing arm) included women with a spectrum of comorbid conditions that were not evident (at least in the same proportion) in the first study and that had a negative influence on the outcome.6 It is important to state again that this is only a hypothesis and nothing more.
|Variable||Cisplatin and cyclophosphamide||Cisplatin and paclitaxel*||(Cisplatin and paclitaxel)*|
|Median PFS, mo||13||14||18|
|Median OS, mo||24||26||38|
|GOG trial||GOG 111||GOG 132||GOG 111|
It is well recognized in oncology that the objective response rates in small Phase II, antineoplastic drug studies frequently are not replicated when the identical regimens are examined in multicenter, randomized trials.18 Differences in the extent of disease and prior therapy, location and size of the tumor(s), the presence or absence of significant comorbidity, and the criteria employed for determining a response all may play an important role in this common observation. Thus, it should not come as a surprise to learn that, when the GOG conducted its own Phase II trial of the combination of cisplatin plus gemcitabine in platinum-resistant ovarian cancer, the observed objective response rate was substantially lower than that noted in the noncooperative group setting19 and was within the range reported for agent-single agent gemcitabine20, 21 and other cytotoxic agents employed in this clinical setting (Table 6).9, 10
|Phase II agents (multiple)||5–20|
|Phase II trials of cisplatin and gemcitabine||>40|
|(Phase II GOG trial of cisplatin and gemcitabine)||(16)|
This commentary has not been written to suggest that an examination of differences in outcomes between individual trials or institutional experiences is always without merit. On the contrary, such efforts may lead to provocative hypotheses that subsequently may result in the initiation of prospective, randomized, Phase III studies. For example, the rather prolonged survival of a subset of women with platinum-resistant ovarian cancer who were treated with single-agent paclitaxel on a National Cancer Institute-sponsored Phase II trial, compared with an “expected outcome” that was suggested by historic data in this clinical setting,22 led to the development of a Phase III study exploring this agent as a “maintenance” strategy in patients with advanced ovarian cancer.23
However, it is absolutely critical that the substantial limitations (this type of analysis can be only hypothesis-generating) and inherent dangers (profoundly inappropriate claims for the validity of the results of such comparisons) of this academic exercise be understood clearly. Furthermore, an appreciation for the importance of this issue must be shared by journal editors, peer-reviewers, and all who attempt to evaluate the objectivity and relevance of articles that appear in the oncology literature. Finally, it is essential that the education and mentoring of our clinical (and clinical research) trainees include thorough discussions regarding the fundamental concerns associated with “cross-trial” and “cross-retrospective experience” comparisons as well as the potential role of such efforts in the generation of novel hypotheses to be tested in future, well-considered investigative initiatives.
- 10Ovarian carcinoma: management after first-line therapy. J Pelvic Surg. 2000; 6: 27–33..
- 23Phase III randomized trial of 12 versus 3 months of maintenance paclitaxel in patients with advanced ovarian cancer after complete response to platinum and paclitaxel-based chemotherapy: a Southwest Oncology Group and Gynecologic Oncology Group trial. J Clin Oncol. 2003; 21: 2460–2465., , , et al.