There has been considerable discussion in the medical literature regarding the need to measure the quality of care delivered based on objective indicators of outcome.1 This discussion has included the provider (eg, physicians, hospitals, health maintenance organizations), payer (eg, Medicare, insurance industry, major employers), and healthcare consumer communities.
In the oncology arena, a variety of possible endpoints might be considered to satisfy the requirements for the provision of care of acceptable quality, including the proportion of patients receiving treatment by a given provider or group that adhered to evidence-based guidelines or the use of established cancer screening strategies by primary care physicians.
The relevance of overall survival as a measurement of the effectiveness of oncologic care is self-evident. In fact, several peer-reviewed publications have documented the favorable impact on mortality associated with the ‘experience’ of cancer surgeons and hospitals with specific complex and technically demanding procedures (eg, pancreatic resection for carcinoma of the pancreas).2, 3 In this situation, the frequent surrogate employed for an objective evaluation of experience is the number of procedures of a particular type performed in a defined period of time (eg, calendar year).
The overall validity and impact of these results come not only from the actual data published in the peer-reviewed literature, but also from the simple statement that it is rational to hypothesize that survival after a potentially complex and morbid surgery might be substantially influenced by such individual surgical and institutional experience. In addition, hospitals in which more complex surgeries are performed most likely have more equipment and experienced support services with which to manage the complications associated with these particular procedures.
A similar conclusion can be drawn regarding the reasonableness of a previously published analysis that revealed that women in Quebec (Canada) with invasive lymph node-negative breast cancer who were treated based on ‘practice guidelines’ experienced superior survival compared with women not managed with therapy driven by evidence-based data.4 Furthermore, it is rational to argue that it would be appropriate to use this factor as 1 feature of a ‘measure of quality-of-care’ provided by a single physician or an oncology group.
However, caution is strongly advised in any potential assumption that simply demonstrating a ‘statistically significant’ association between: 1) a particular clinical feature in a specific setting, and 2) overall survival actually implies any cause-and-effect relation between the events. Stated even more directly, an observed correlation between parameters does not necessarily mean the presence of 1 factor resulted in, or led to, the secondary outcome.
In this regard it is important to examine the overall strength of any proposed association. For example, data generated from a meta-analysis of multiple phase 3 randomized trials should be given greater weight in public discussions of these issues compared with 1 (or even several) relatively small observational studies.
The potential risk of overly simplistic conclusions was highlighted in a recent report discussing the influence of individual surgical volume on overall survival in breast cancer. Several previously reported studies had suggested that the volume of cases or number of individual surgeries performed by breast surgeons could significantly influence the ultimate outcome, apparently independent of the stage of disease at the time of presentation or documentation of how the patient was subsequently managed.5-7
In this situation, and in sharp contrast to the previous statement regarding the potential for severe treatment-related morbidity and possible mortality associated with high-risk oncologic surgical procedures, it would be difficult to argue that the nature and complexity of current breast cancer surgery itself would substantially influence survival. Furthermore, one could question the fairness of an assumption that surgeons who perform fewer breast surgeries are less likely to refer their patients to other physicians who then administer state-of-the-art chemotherapy or, if these surgeons maintain direct responsibility for this component of care, they would not be capable of delivering appropriate systemic treatment.
Therefore, is it possible that the demonstrated differences in survival noted in these studies are the result of factors largely if not completely unrelated to the actual care delivered by these surgeons? A recent report examining this precise issue has suggested that, in fact, this may be the case.8
In the analysis that involved the National Cancer Institute (NCI)'s Surveillance, Epidemiology, and End Reports (SEER) program and national payment data for >12,000 women (age ≥66 years) in the U.S. with stages I or II breast cancer, the patient population cared for by ‘higher volume’ surgeons did experience a superior survival compared with women whose cancers were initially managed by ‘lower volume’ surgeons.8 However, the patient population of the ‘higher volume surgeons’ was also found to possess other characteristics (eg, younger age, white race, less comorbidity, residence in a more affluent neighborhood) that might have explained some, if not all, of these observed differences. Thus, these data suggest that although it may be factually correct that, as a group, women with breast cancer cared for by ‘lower volume’ surgeons have a ‘statistically significant’ inferior survival, the actual care delivered by the individual surgeon may not have been the direct cause of this less than optimal outcome.
Another example of the potential for ‘statistically significant’ correlations within specific datasets to lead to unjustifiable conclusions is a published report that examined the survival of a group of women with advanced ovarian cancer who were stated to have received their chemotherapy from either a gynecologic oncologist or a medical oncologist.9 Using information from the NCI SEER program, which provides limited nondetailed descriptions regarding aggregate patient populations, their clinical characteristics, and outcome, the investigators remarkably concluded that: 1) because there was no ‘statistically significant’ difference in survival between the patients who were apparently reported in the database to have received chemotherapy primarily from 1 of the 2 groups of providers, and 2) the patients managed with medical oncology received on average more cycles of chemotherapy and appeared to experience a somewhat higher incidence of reported treatment-related side effects (the implications of which were not discussed), it could be argued that the gynecologic oncologists may have possessed a better understanding of the course of disease and therefore were better able to treat women without undue morbidity or excess treatment.9
The authors also state that from the perspective of the patient, the results therefore raise the issue of whether patients should choose physicians based on whether they train and practice in a more technique-specific versus disease-specific paradigm. Is it reasonable to suggest that the results should lead to an objective nonbiased observer to ask that particular question?
Finally, strong support for the assertion that the caution advised in this commentary is warranted is provided by a recent report examining the utility of using a minimum number of lymph nodes removed at the time of resection of colon cancer as a valid indicator of the appropriateness of the surgical procedure.10-12 It should be noted that this strategy has already been accepted by several organizations as an important objective marker for the quality of care received by patients in this clinical setting.
Justification for the use of this parameter comes from several observational studies reported over the past few years that have strongly suggested that individuals with colon cancer who had more lymph nodes removed at the time of resection experienced superior survival compared with patients in whom fewer lymph nodes were resected during the primary surgical procedure.10-12 Although to my knowledge a completely acceptable biologic explanation for this observation has yet to be provided, one might speculate that more extensive surgery could remove micrometastatic cancer within these local and regional lymph nodes, resulting in a superior outcome, or this strategy might enhance the clinical information provided during staging, optimizing the subsequent use of ‘evidence-based’ adjuvant chemotherapy.
However, it currently remains unknown whether the presence of more lymph nodes actually indicates the performance of a superior surgical procedure, or if this finding is in fact more related to some other biologic factor (eg, an enhanced local immunoregulatory anticancer response) that itself is the reason for the more favorable survival outcome.
The clinical relevance of the minimum number of lymph nodes removed defining acceptable surgery in this setting has now been challenged by the results of a retrospective analysis of NCI SEER data that revealed that although the average number of lymph nodes obtained at the time of colon cancer surgery did vary considerably between hospitals, there was no difference noted with regard to the percentage of individuals with positive lymph nodes, the use of adjuvant chemotherapy, or overall survival based on the average number of lymph nodes removed at the time of surgery at the institutions with documented relatively ‘high’ versus ‘low’ total lymph node counts.13
Importantly, this observation suggests that although the number of lymph nodes may in some cases certainly be correlated to the extent of surgery performed, this factor does appear to influence either the subsequent use of evidence-based treatment or the patient's ultimate survival from the cancer. Thus, the question to be asked is: in the absence of a clearly defined relation between this parameter and an important outcome measure, is it legitimate to claim that the number of lymph nodes removed is an appropriate indicator of the quality of care provided?
In summary, although the search for meaningful objective measures of quality of care in oncology continues, it is essential that this highly laudable societal goal not be permitted to obscure the complexity of the task, and the critical need to fully understand both the biologic and clinical relevance of any proposed correlations, including the influence of important confounding variables (eg, patient comorbidity), before the indicator becomes the ‘standard’ by which quality is defined.1, 9, 14 Finally, it is reasonable to conclude that individual physicians, and the medical establishment itself, will be far more likely to accept both the need for, and the specifics of, an objective evaluation of quality if the actual parameters to be examined are clearly demonstrated to be unbiased, and their clinical significance above reproach.