Received from the Department of Medicine (BPL, ATE, BMR, YVK, KP, LGR, JAA, TMA, CAS, JPS), John H. Stroger, Jr. Hospital of Cook County and Rush Medical College, Chicago, Ill.
Presented at the National Society of General Internal Medicine Meeting, May 2002 (finalist for the Milton W. Hamolsky Award).
Address correspondence and requests for reprints to Dr. Lucas: 1901 West Harrison Street, Room 7560, Chicago, IL 60612 (e-mail: email@example.com).
OBJECTIVE: Previous studies have shown that most medical inpatients receive treatment supported by strong evidence (evidence-based treatment), but they have not assessed whether and how physicians actually use evidence when making their treatment decisions. We investigated whether physicians would change inpatient treatment if presented with the results of a literature search.
DESIGN: Before-after study.
SETTING: Large public teaching hospital.
PARTICIPANTS: Random sample of 146 inpatients cared for by 33 internal medicine attending physicians.
INTERVENTIONS: After physicians committed to a specific diagnosis and treatment plan, investigators performed standardized literature searches and provided the search results to the attending physicians.
MEASUREMENTS AND MAIN RESULTS: The primary study outcome was the number of patients whose attending physicians would change treatment due to the literature searches. These changes were evaluated by blinded peer review. A secondary outcome was the proportion of patients who received evidence-based treatment before and after the literature searches. Attending physicians changed treatment for 23 (18%) of 130 eligible patients (95% confidence interval, 12% to 24%) as a result of the literature searches. Overall, 86% of patients (112 of 130) received evidence-based treatments before the searches and 87% (113 of 130) after the searches. Changes were not related to whether patients were receiving evidence-based treatment before the search (P = .6). Panels of peer reviewers judged the quality of patient care as improved or maintained for 18 (78%) of the 23 patients with treatment changes.
CONCLUSIONS: Searching the literature could improve the treatment of many medical inpatients, including those already receiving evidence-based treatment.
Physicians and pundits have long debated the scientific basis of clinical practice. As recently as 1990, skeptics opined that most medical treatments were not supported by strong evidence.1–4 Since then, a number of studies have debunked this notion.5 All three studies involving medicine inpatients (from England, Canada, and Sweden) found that over 80% of inpatients received treatments supported by either randomized controlled trials or convincing nonexperimental evidence.6–8 Thus, skeptics’ concerns about therapeutic quackery seem unfounded.
However, to conclude from these studies that most medical inpatients receive evidence-based treatment is potentially misleading.9,10 None of the studies assessed whether or how physicians actually used evidence when making their treatment decisions. Nor did they determine whether, as a result, patients received the best available treatment. (For particular patients, some evidence-based treatments are better than others.) All of these considerations—physicians’ judgments, patients’ unique clinical circumstances, and the evidence relevant to various treatment options—are essential to the practice of evidence-based medicine.11
In order to explore these important issues further, we sought to measure the potential efficacy of actually practicing evidence-based medicine. In the present study, we assessed the impact of standardized literature searches on attending physicians’ treatment decisions for medical inpatients.
We performed a before-after study of physicians’ treatment decisions for a random sample of patients admitted to a general medicine inpatient service at one hospital. Before performing a standardized literature search, we elicited from the attending physician the patient's primary clinical problem (the principal diagnosis requiring inpatient treatment) and the specific treatment being given for it. After searching the medical literature about the patient's problem, we presented the search findings to the attending physician and asked whether he or she would change the patient's treatment as a result of the literature search. Figure 1 outlines the study design within the conceptual framework of therapeutic decision making. This model assumes that physicians typically consider various treatment options before committing to a specific treatment. We hypothesized that a literature search would cause physicians to change their treatments if the search identified treatment options not previously considered or provided data that altered their evaluation of treatment options.
The study was performed at Cook County Hospital, a 700-bed public teaching hospital, from March through May 2001. Each month, 12 attending physicians and their respective teams of residents and students are responsible for all patients admitted to the general medicine inpatient service, which excludes only medical patients admitted to the medical intensive care unit, the coronary care unit, or the HIV subspecialty ward. More than 80% of all general medicine admissions come from the emergency department, and all admissions are distributed in sequence to three admitting teams on call throughout a 24-hour period. Each team admits approximately 12 to 14 patients every fourth day. The team's attending physician serves as the team leader and the physician ultimately responsible for daily care of all team patients throughout their hospitalization.
The hospital's department of medicine has for many years promoted evidence-based medicine in the daily morning report, the residency curriculum, and faculty development programs.12–14 All attending and resident physicians have in-hospital access to computers capable of searching Best Evidence, Medline, UpToDate, and other evidence-based resources.
The subjects of study were a random sample of patients admitted to the inpatient teams of 33 different attending physicians during the 3-month study period. Patients admitted to the teams of study investigators were excluded. We prospectively selected 20 admission days within the 3-month study period to ensure that all attending physicians had a roughly equal likelihood of having their patients included in the study (approximately 2 admission days per attending physician). Using random number tables and a daily unique starting point, we randomly sampled 3 patients from the admission lists of each attending physician at the end of the admission day. Patients were eligible if they were still under the attending physician's care at the end of the admission day.
After receiving approval from the institutional review board, investigators conducted an interview, physical examination, and review of medical records for each study patient within 8 hours of sampling (end of the admission day). On that first day, and on every subsequent hospital day until the attending physician assented, investigators asked the attending physician if he or she had made a “final” treatment decision. This was defined as the attending physician's assertion that no pending investigations or consultations remained that might alter the patient's primary diagnosis or the treatment being given for it.
Standardized Literature Search
Within 24 hours after the attending physician committed to a treatment plan, an investigator performed a standardized search of the medical literature and delivered the results to the attending physician. The focus of each literature search was the patient's primary clinical problem—identified and targeted for treatment by the attending physician. The goal was to find the best scientific evidence about all available treatments for the clinical problem in question, whether those treatments had been prescribed by the attending physician or not. Investigators who performed the literature searches used a procedures manual that adhered to published recommendations15,16 and specified in detail the standardized sequence of the search protocol (Appendix 1). A unique search was performed for each patient; investigators did not recycle searches for similar clinical problems. Search findings given to attending physicians included text, tables, figures, and abstracts from the various sources listed in Appendix 1 but did not include commentary or analysis by the investigators. All investigators were board-certified internists, previously trained in the standardized search protocol, whose searching skills had been verified during pilot testing.
Primary Outcome—Change in Treatment Due to the Literature Search
Within 36 hours of delivering the search results, investigators conducted scripted interviews with attending physicians using a standardized data collection instrument. Because searches were performed only after a final treatment plan had been decided, we anticipated that the time required to complete the entire study protocol (including the interview) might exceed some patients’ length of hospital stay. Therefore, we measured “change in treatment” whenever physicians reported that they did change or would change treatment. Physicians were asked explicitly whether or not their change in treatment was primarily, or in part, due to the literature search results they had been provided.
Using criteria defined by Ellis6 and later adopted by others,5,7,8 we classified patients’ treatments (both before and after any search-prompted changes) as evidence-based or not. Thus, a treatment was considered evidence-based if either: 1) its “value (or nonvalue) had been established in one or more randomized trials or overviews of randomized trials,” or, 2) its “face validity is so great that randomized trials were unanimously judged by the [investigators] to be both unnecessary and, if a placebo would have been involved, unethical” (convincing nonexperimental evidence).6 Because the treatment for some patients included multiple components, we evaluated the evidence for each treatment component separately. In these cases, treatment was considered evidence-based if: 1) at least one treatment component met the above criteria, and, 2) convincing evidence was not found demonstrating harm or lack of efficacy for any of the other treatment components.
The clinical value of physicians’ search-prompted treatment changes was assessed by a panel of local experts. We presented written summaries of each case to 5 peer faculty reviewers (2 generalists and 3 specialists with expertise in the relevant clinical area; a total of 29 different reviewers). Reviewers were instructed to assume that the given diagnosis was correct and that only 2 treatment options were available (the presearch treatment and the postsearch treatment). Blinded to physicians’ actual treatment choices, reviewers independently assessed which treatment “would provide the higher quality of care.” (A third choice allowed reviewers to indicate that the 2 treatment options provided equal quality of care.) Peer review disagreement with a physician's search-prompted treatment change was defined a priori as a majority of reviewers (at least 3 of 5) choosing the presearch treatment as providing higher quality of care.
We planned for a sample size of 130 study patients so that the width of the 95% confidence interval (CI) for our primary outcome measure (proportion of patients whose treatment changed as a result of the literature search) would be no greater than 15 percentage points, assuming that the point estimate for the main outcome was less than 25%. We explored associations between the primary outcome and characteristics of patients, physicians, investigators, and the literature searches. Because patients were clustered within attending physicians, the outcomes were not independent. We account for this clustering using robust standard errors and generalized estimating equations when reporting adjusted proportions and confidence intervals.
A generalized κ statistic was calculated using the method of Fleiss17 to measure the agreement above chance among the 5 members of the expert review panel. The panelists’ responses were first dichotomized: the first treatment (before change) provided superior quality of care versus the second treatment (after change) provided superior or equal quality of care. The reliability for the majority opinion of the 5-member expert panel was estimated using the Spearman-Brown formula.18 All analyses were conducted with SPSS version 10 (SPSS, Inc., Chicago, Ill) and STATA version 7 (STATA Corporation, College Station, Tex).
During the 20-day study period, 660 patients were admitted to the inpatient general medicine service. Among these, 124 patients were admitted to ineligible physicians (study investigators) and 37 other patients were ineligible because they were no longer under the care of the attending physician at the end of the admission day. From the remaining 499 eligible patients, 146 patients were randomly sampled (Table 1). Sampled patients had a high prevalence of adverse social, behavioral, and clinical characteristics but their mean length of hospital stay was only 3 days. Their primary clinical problems warranting hospitalization are depicted in Table 2.
Table 1. Characteristics of Attending Physicians and Patients
Among 146 sampled patients, 15 (10%) of the 146 patients received only diagnostic investigations and no inpatient treatment for their primary problem, and 1 additional patient left the hospital against medical advice before being treated. The remaining 130 sampled patients who actually received inpatient treatment for their primary problem comprised the study group. Study group patients were admitted to 33 different attending physicians who, as a group, had a median of 8 years’ experience as inpatient attending physicians and reported frequently using evidence-based literature sources when caring for inpatients (Table 1).
Primary Outcome—Treatment Changes Due to Literature Searches
Attending physicians reported that they would change treatment in 23 of 130 patients as a result of the literature searches (18%; 95% CI, 12% to 24%[adjusted for clustering within physician]; Table 3; Appendix 2). These 23 patients were cared for by 17 (52%) of the 33 attending physicians. This primary outcome measure can also be represented as a “number-needed-to-search” (NNS), the average number of searches needed to change one patient's treatment: NNS = 1/0.18 = 6 patients (95% CI, 4 to 8). Treatment was changed in 6 other patients, but the physicians reported that these changes were not due to the searches. Among the remaining 101 study patients (78%) whose treatment did not change, searches in 23 cases identified alternative treatment options that were rejected by the attending physicians because they believed the search evidence supporting the alternative treatment was not applicable to the particular patient (n = 13) or not convincing (n = 7), or because a consultant (n = 2) or the patient (n = 1) preferred the presearch treatment.
Table 3. Primary and Secondary Study Outcomes
Physicians reported that changes in treatment were primarily due to the search in 18 patients and partly due to the search in 5 patients.
In 23 of these 101 patients (18% of the total 130 patients), attending physicians rejected alternative treatments identified by the literature searches and did not change their treatment. Physicians’ reasons for rejecting the treatment options identified by the search included: evidence not applicable to particular patient (n = 13), evidence not convincing enough to change (n = 7), subspecialty consultant prefers initial treatment (n = 2), and patient prefers initial treatment (n = 1).
Evidence-based Treatment. Physicians’ presearch treatments were evidence-based in 86% (112 of 130) of all study patients: 51% (66 of 130) were supported by randomized controlled trials or meta-analyses and 35% (46 of 130) by convincing nonexperimental evidence (Table 3). After reviewing the literature searches and changing treatment in 23 patients, physicians’ postsearch treatments were evidence-based in 87% (113 of 130) of all study patients. Thus, literature searches resulted in changing only 1 patient from non-evidence-based treatment to evidence-based treatment (Case 1, Appendix 2). Changes in treatment due to the literature searches occurred in 17% of all patients whose presearch treatment was evidence-based and 22% of all patients whose initial treatment was not evidence-based (P = .6; Table 3).
Peer Review of Treatment Changes. In 18 of 23 patients (78%), panels of peer reviewers agreed with the treatment changes made as a result of the literature searches; that is, they judged the postsearch treatment superior or equal to the presearch treatment. In the remaining 5 patients (22%), the presearch treatment was judged superior by a majority of the peer review panel members (Appendix 2). Although the chance-corrected agreement for any pair of peer reviewers was weak (κ 0.2), the reliability of the majority opinion of 5 peer reviewers (the panel size in each case) was stronger (κ 0.56).
Literature Searches. Searches were completed successfully in all 130 study patients (100%) in a median time of 42 minutes. Table 4 depicts the frequency with which information from the 4 literature sources were included in the search results given to attending physicians as well as physicians’ ratings of the helpfulness of this information in the clinical management of patients. Overall, physicians rated the search packets as very helpful or somewhat helpful in 49% of all patients.
Table 4. Characteristics and Evaluations of the Literature Searches
Frequency Evidence Source Represented in Search Results Given to Physician (N = 130)
Pages Included in Search Results (N = 130)
Frequency Physician Rated Search Results Very or Somewhat Helpful for Patient Management (N = 127†)
Includes time to read and compare sources.
Denominator is 127 because only 127 of the 130 literature searches were read by an attending physician.
Median Minutes (25th, 75th Percentiles)
Median Pages (25th, 75th Percentiles)
15 (10, 20)
3 (2, 5)
20 (11, 27)
2 (0, 3)
3 (1, 10)
0 (0, 2)
7 (0, 12)
0 (0, 1)
42 (31, 53)
7 (4, 10)
We found that a standardized literature search convinced experienced attending physicians to change their treatment in 18% (95% CI, 12% to 24%) of a random sample of patients admitted to the general medicine inpatient service. This suggests that routinely searching the literature for relevant evidence (even after physicians have committed to a specific treatment plan) may improve the treatment of many medical inpatients. For example, in our hospital, where 15,000 patients are admitted annually to the medical service, 2,700 inpatients per year (95% CI, 1,800 to 3,600) might benefit from this practice. These conclusions must be tentative, given the limitations of our study design. But our findings provide the first published evidence that searching the literature in real time—actually practicing evidence-based medicine—could improve inpatient care.
Prior studies have focused on a different question using different methods.5–8 Investigators from Great Britain,6 Canada,7 and Sweden8 conducted studies to challenge the opinion that only 10% to 20% of physicians’ therapeutic interventions are supported by strong evidence and demonstrated that 82% to 84% of medical inpatients in fact received “evidence-based treatment”—randomized trials or meta-analyses were found to support 53% to 57% of patients’ treatments, and convincing nonexperimental evidence supported an additional 27% to 29%. Data from the present study agree remarkably with those findings, using identical criteria: 86% of our patients were receiving evidence-based treatment before the literature searches were performed. But what happened after the literature searches were performed explores new territory and contributes new knowledge about the challenges of practicing evidence-based medicine.
As predicted by the conceptual model (Fig. 1), the literature searches identified potential alternative treatment options (35% of all patients) and provided the information physicians needed to decide whether to change to these alternative treatments (in 18% of patients). The reasons physicians rejected the alternative treatments exemplify the multiple, complex tasks clinicians perform when choosing the best evidence-based treatments for their patients: compare the magnitude of treatment benefits and harms; evaluate the quality of evidence; assess its applicability (relevance) to the individual patient; elicit the patient's personal treatment preferences and values; and, integrate all of these considerations with physician's own (and consultant’s) clinical judgment (Table 3). Our study does not allow an accurate assessment of how well physicians performed these tasks, but it does illustrate the multidimensional process of treatment decision making.
The complexity of these decisions may be highlighted by the deliberations of our peer review panels about the quality of physicians’ search-prompted treatment changes. Overall, the 5-member panels agreed with the majority (78%) of treatment change decisions, with moderately strong interpanel reliability (κ 0.56 for the majority opinion of the 5 reviewers). But individual peer reviewers disagreed with or equivocated about treatment changes in 35% of all cases (Appendix 2), and agreement between pairs of individual peer reviewers was only slightly better than chance (κ 0.2). We suspect that some of the discordance among peer reviewers reflects the fact that the peer reviewers, unlike the attending physicians, had no contact with the patients themselves. It is also possible that some peer reviewers were more expert than others. Our study data do not allow more detailed analysis of this important issue. But they illustrate the fact that therapeutic decision making, even when conscientiously “evidence-based,” is an uncertain, complex, multidimensional process.19,20
Viewed from this perspective, we find it unimportant that our literature searches had little impact on the number of patients who received evidence-based treatment (as defined by previous investigators): 86% of patients before versus 87% after the literature searches. Far more telling is the fact that 17 different attending physicians changed patients’ treatments because they judged alternative treatments identified by the literature searches better for those patients. This outcome may be “softer” than previous researchers’ measure of evidence-based treatment, but it is also more meaningful clinically.21,22
Future research, in our view, should adopt this broader perspective. It should also evaluate a more sophisticated practice of evidence-based medicine than we did. After all, we evaluated “routine” literature searches, generated by protocol from each patient's diagnosis, regardless of the attending physician's perceived “need for information.” In contrast, the evidence-based medicine orthodoxy recommends literature searches that are physician-generated, patient-centered, and precisely tailored to specific clinical uncertainties.15 It will be interesting to compare the efficacy and feasibility of these very different approaches. In this regard, we note that strategies to improve the time efficiency of evidence-based practice should receive high priority in future research.23–26 Our results support a longstanding concern that lack of time is a major impediment to practicing evidence-based medicine. We do not recommend our search protocol for practical application because it was designed for a specific research purpose. Time savers that seem promising include instant resource books (compendia of previously performed searches) and physician extenders such as electronic librarians. For the present, we do not advocate clinicians’ abdicating these responsibilities entirely to “evidence experts”; however, persuasive arguments may support that strategy in the future.27–29
Our study has several limitations. We studied only one group of physicians at one hospital; physicians with different job descriptions in different settings may behave differently. Our before-after study design used patients and their physicians as their own controls; hence our analyses lack an independent control group. Moreover, our primary outcome measure relied on physicians’ self-reports and we cannot completely exclude a Hawthorne effect due to investigators’ interactions with attending physicians. However, we believe that experienced attending physicians do not easily admit the inferiority of their own treatment decisions and suspect that any related bias would favor the null hypothesis. Finally, we cannot claim that our patients’ treatment changes resulted in the best available treatment as measured by some external gold standard because, in most cases, no such standard exists.20
These limitations can be addressed best by randomized controlled trials of literature searches used as interventions to assist physicians’ decisions. Such trials will not be simple to design nor easy to perform. But the present study suggests that practicing evidence-based medicine can make a real difference in patient care.
Funding was provided by the Department of Medicine, John H. Stroger, Jr. Hospital of Cook County.
Table 5. Literature Search Protocol
Details of the literature search protocol are available from the authors on request.
Only searched if the primary clinical problem was a malignancy or a complication arising from a malignancy.
Focus of the literature search:
The primary clinical problem identified and targeted for treatment by the attending physician. (Example: for a woman treated with aspirin and heparin for unstable angina, the search would focus on “unstable angina.”)
• Clinical Evidence (either most recent edition of the book or the website)
• UpToDate (either the most recent version on CD-ROM or the website)
• PDQ/CancerNet† (National Cancer Institute website for health professionals)
• EBM reviews (using Ovid website: ACP Journal Club, DARE, Best Evidence, and the Cochrane Collaboration database)
• Medline (using Ovid website: initially limit to meta-analyses and guidelines; expand to include clinical trials and review articles)
Format of the search results:
Print relevant text, tables, figures, and abstracts according to the following principles:
• Include all information that a conscientious physician might find useful in making a treatment decision for the specific individual patient.
• Give priority to studies with the most rigorous scientific methods.
• Include high-quality studies about treatments the attending physician has prescribed, as well as treatments not provided.
Table 6. Clinical Problems and Treatments for the 23 Patients Whose Treatment Changed Due to the Literature Searches
Primary Clinical Problem
Treatment Before Search
Treatment Changes After Search
Peer Review Judgments About Quality of Treatments*
The first number represents the number of blinded peer reviewers who thought that the treatment before the search provided better quality of care. The second number represents reviewers who thought the two treatments were probably of equal quality. The third number represents the number of reviewers who thought that treatment after the search provided a higher quality of care.
In case 1, the treatment after the search but not the treatment before the search was supported by randomized trials or systematic overviews.
Cases where the majority of peer reviewers (3 out of 5) did not disagree with the treatment change, that is, did not think that treatment before the search provided better quality of care.
In cases 7–11, 15–17, and 22–23, treatment plans were supported by randomized trials or systematic overviews before and after the literature search. In cases 6, 13–14, and 18–21, treatment plans were supported by convincing nonexperimental evidence before and after the literature search. In cases 5 and 12, treatment plans were supported by convincing nonexperimental evidence before, and randomized trials or systematic overviews after, the literature search.
Before Search: Not Evidence-based; After Search: Evidence-based†
Pre search better
Post search better
Transarterial chemoembolization with adriamycin and ethadiol
stop Transarterial chemoembolization add Palliative care