• internship and residency;
  • evidence-based medicine;
  • information storage and retrieval;
  • medical education;
  • medline


  1. Top of page
  2. Abstract
  6. Acknowledgments

OBJECTIVE: To determine if a simple educational intervention can increase resident physician literature search activity.

DESIGN: Randomized controlled trial.

SETTING: University hospital–based internal medicine training program.

PATIENTS/PARTICIPANTS: Forty-eight medical residents rotating on the general internal medicine service.

INTERVENTIONS: One-hour didactic session, the use of well-built clinical question cards, and practical sessions in clinical question building.

MEASUREMENTS AND MAIN RESULTS: Objective data from the library information system that included the number of log-ons to medline, searching volume, abstracts viewed, full-text articles viewed, and time spent searching. Median search activity as measured per person per week (control vs intervention): number of log-ons to medline (2.1 vs 4.4, P < .001); total number of search sets (24.0 vs 74.2, P < .001); abstracts viewed (5.8 vs 17.7, P = .001); articles viewed (1.0 vs 2.6, P = .005); and hours spent searching (0.8 vs 2.4, P < .001).

CONCLUSIONS: A simple educational intervention can markedly increase resident searching activity.

In the current health care environment, it is expected that individual patient decisions will be made using clinical judgment, expertise, and information from well-designed clinical trials. There is evidence that caring for patients using information from valid clinical trials can improve patient outcomes.1–3 However, the process of moving from an individual patient encounter to practicing evidence-based medicine (EBM) requires explicit skills that are rarely taught. Specifically, clinicians must identify the need for information, formulate questions, search for information, appraise literature, and then apply scientific evidence to the clinical decision at hand.

Most physicians, either while in training or in practice, have been exposed to the skills and knowledge needed to critically appraise a scientific report. For example, the User's Guide to the Medical Literature series has clearly outlined how to efficiently appraise most types of scientific reports and trials.4–7 These skills have been used to effectively teach evaluation of the medical literature.8 Established teaching traditions, such as medical journal clubs, have been used to motivate critical reading in house officers.9 However, the critical appraisal process is dependent on asking the right questions and finding the relevant evidence. These important first steps in the EBM process are rarely discussed or practiced.

Physicians often form questions from patient encounters,10–12 but the generation of a question does not necessarily lead to a search of the literature.11 One impediment to searching is the lack of a focused and searchable question. The difficulties in formulating questions and moving to the search are troubling, because evidence does exist to treat patients12 and patient outcomes are enhanced when the best available evidence is applied to their care.1–3 Therefore, physicians in training need to be taught the skills of EBM, and studies are needed to show how these educational interventions affect behavior.

We designed a randomized controlled trial to objectively measure the effect of an educational intervention on the first steps of the EBM process, moving from a clinical question to a search of the medical literature. We sought to teach residents how to build clinically relevant questions that would facilitate searching. Without an appropriate question, searching is not likely to be fruitful. Our primary hypothesis was that residents taught to frame effective clinical questions would increase their use of medline.


  1. Top of page
  2. Abstract
  6. Acknowledgments

Study Design

Forty-eight medical residents in the Department of Medicine at Duke University Medical Center, Durham NC, were randomly assigned in blocks by coin flip to a teaching intervention or to a control group. Randomization was blocked so that all 8 residents on the inpatient general medicine rotation at any one time were assigned to the same group. The trial, which began in September 1998 and concluded in May 1999, encompassed 6 inpatient general medicine rotations, each lasting from 6 to 8 weeks. The study received Institutional Review Board approval and all participants signed informed consent.

Inpatient General Medicine Educational Environment

The overall training program is structured such that each medical resident, while on the inpatient general medicine rotation, meets individually with the chief resident to discuss patients admitted to the general medicine service. These meetings take place during the evening of the long call nights. The purpose of these meetings is to discuss the patients admitted to the service with a focus on education in relation to diagnosis and therapeutic decision making. During a typical 6- to 8-week inpatient general internal medicine rotation, each resident has approximately 12 to 14 long call nights with 5 to 7 admissions typical for each long call night.

The chief residents receive specific training in EBM both through our training program and through workshop attendance. The chief residents attend the McMaster University workshop on the principles of teaching EBM. This workshop combines didactic lessons with small group discussion sections. In addition, practice teaching sessions with role playing are used to facilitate the development of practical skills in teaching EBM to learners such as medical residents.

All residents, both the intervention groups and control groups, experienced the chief resident sessions while on general medicine. In addition, all residents were exposed to the typical educational environment while on the general medicine rotation. A main component of this environment is daily interaction with the on-service attending to discuss patient care specifically in relation to the medical literature. In addition, the educational environment includes daily morning report, daily conferences, and weekly sessions with a medical librarian.

The medical librarian sessions were 30-minute lessons on the different resources available to locate important medical information. These sessions took place just before morning report 1 day per week. These sessions followed a structured curriculum that included the use of medical subject headings and subheadings, text words, EBM filters, and other electronic resources such as Best Evidence and the Cochrane database of systematic reviews. Each week the medical librarian provided pragmatic examples of how the answers to medical questions could be answered utilizing specific techniques and electronic medical resources.

Regardless of the randomized group, all residents experienced the same overall general internal medicine education environment.

Educational Intervention

The intervention consisted of 3 components: 1-hour didactic sessions, well-built clinical question (WBCQ) cards,13 and practical experience building questions. The first part of the intervention was a 1-hour didactic session devoted to the key components of a well-built clinical question13 and how this structure can translate into effective search strategies. The principle investigator (CHC) and the medical librarian (CS) conducted the 60-minute didactic teaching sessions during the first week of each intervention rotation. The curriculum (Fig. 1) was designed prior to the start of the trial and care was taken to deliver a consistent session for each group. The second aspect of the intervention was the use of WBCQ cards. These 3 × 5 cards (Fig. 2) were used to record clinical questions generated from each admission.


Figure 1. Standardized curriculum used for didactic session. The standardized curriculum was delivered during a 1-hour didactic session to the intervention group. It was delivered in a consistent fashion over time.

Download figure to PowerPoint


Figure 2. The well-built clinical question card. The cards were used by the intervention group to record clinical questions. In the first column, the patient/population or problem was recorded. Next, the intervention or exposure was added. If an appropriate comparison existed, then this information was placed in the bottom half of the second column. Finally, the outcome of interest was recorded in the third column. The cards were then stored for searching at a more convenient time.

Download figure to PowerPoint

The third aspect of the intervention focused on the use of the WBCQ cards to build searchable questions generated from admissions to the general medicine service. The cards were used each long call night during the regular session with the chief resident in a real time, learner-centered, case-based teaching session. During these sessions, the residents presented the admissions as well as clinical questions derived from the patient encounter to the chief resident. In this way, each resident received reinforcement and repetition in the skill of clinical question building, and feedback was given on the question-building process. In addition, the relationship between the development of a clinical question, the use of the WBCQ cards, and how this process aids in the expedient searching of the medical literature was repeatedly reinforced.

The residents kept the cards and used them to aid in formulating their search strategies in medline, our primary database for current biomedical literature. In an effort to control confounding, the residents' real-time use of the WBCQ card was not tracked, nor was there any further intervention after the long call teaching session. As is typically expected of medical residents caring for patients, the residents were encouraged to seek answers to their questions, to search each of these questions, and to bring the data back to their teams to aid in patient care.

The control group did not receive the 3-part educational intervention. All residents, regardless of study group assignment, participated in the general medicine rotation in the usual fashion. The only experiential difference between groups was the delivery of the intervention. In particular, the chief resident sessions were similar for each group except for the time spent on practical experience formulating well-built clinical questions and the use of the WBCQ cards in the intervention group. In addition, although the chief resident sessions for both groups emphasized the importance of the medical literature in the care of patients, it was only in the intervention group that a direct link between recording a well-built clinical question and searching the medical literature was emphasized and reinforced. All resident study participants and faculty on the inpatient general medicine service were blinded to the study question.

Outcome Measures

All house officers in the training program have an individual identification (ID) code to access on-line resources provided by the medical center library including the medline database through Ovid (Ovid Technologies, Inc., New York, NY). Use of the personal ID allowed the Ovid system to track specific information for each of our primary and secondary outcome measures. These data were collected continuously from the first day of the rotation through the last day of the rotation. These data were downloaded directly into a data set for analyses.

We collected the following data directly from the Ovid system for each individual user: number of log-ons, time spent searching, total searching volume, total number of abstracts viewed, and total number of full-text articles viewed. Our a priori primary outcome was the number of times that each resident logged on to Ovid during his or her inpatient general medicine rotation. The total number of search statements generated during each searching session was used as a measure of total searching volume.

We utilized a self-report questionnaire to collect basic demographic information, personal usage of medline, perceived barriers to searching, and a self-assessment of searching skill.

A random subset of the intervention group was evaluated in order to collect data on the quality of the clinical question and the link to the data retrieved in the search. There were no validated tools available to assess clinical question and search quality; therefore, this subset of data was used to generate a search strategy assessment tool (Fig. 3).


Figure 3. Search strategy assessment tool. The tool was developed after review of clinical questions and search results.

Download figure to PowerPoint

Statistical Methods

Descriptive statistics for continuous variables were summarized as medians and interquartile ranges (IQRs). Categorical variables were reported as the number and percent of physicians with the characteristic. Differences between the control and intervention groups were evaluated by means of Fisher's exact tests for categorical variables and Wilcoxon rank sum tests for continuous variables. To evaluate the extent to which the number of self-reported log-ons and objective log-ons reported by Ovid differed, the difference in these measures was obtained for each physician. These differences were then tested separately for the control and intervention groups using Wilcoxon signed rank tests. A Wilcoxon rank sum test was used to evaluate whether the control and intervention groups differed in the extent to which they misreported their number of log-ons. A 2-sided P value of less than .05 was considered significant for all statistical tests. All statistical analyses were done with the use of SAS software (SAS Institute, Inc., Cary, NC).


  1. Top of page
  2. Abstract
  6. Acknowledgments

Baseline Characteristics

Our sample groups consisted of internal medicine residents at Duke University Hospital. Baseline data and self-report surveys were collected on all 48 residents. Data directly from Ovid were collected on 47 of 48 residents. The data on 1 person in the control group were lost due to a systems error. There were no statistical differences in baseline characteristics in the 2 groups (Table 1).

Table 1.  Baseline Data
 Control (N = 24)Intervention (N = 24)
  • *Categorical variables are reported as number and percent affected.

  • Age is reported as the median and interquartile range.

  • There were no statistical differences between groups. PGY, postgraduate year.

Level of training
 PGY2, n (%)14 (58.3)17 (70.8)
 PGY3, n (%)10 (41.7)6 (25.0)
Age29.3 (3.2)29.0 (2.9)
Female, n (%)10 (41.7)8 (33.3)
Future plans, n (%)n = 23n = 24
 General medicine private practice5 (21.7)5 (20.8)
 Subspecialty fellowship14 (60.9)15 (62.5)
 General medicine fellowship1 (4.4)2 (8.3)
 Geriatrics1 (4.4)1 (4.2)
 Other2 (8.7)1 (4.2)

Primary and Secondary End Points

Our primary end point was the number of times each resident logged on to the Ovid system. The median number of log-ons per person per week was 2-fold higher in the intervention group (2.1 vs 4.4 log-ons; P < .001) (Table 2). We examined a variety of secondary end points that represented more detailed usage of medline (Table 2). For each of these parameters, there was a consistent 2- to 3-fold increase in activity in the intervention group (Table 2). Total searching volume, number of abstracts and full-text articles viewed, and total search time were all significantly greater in the intervention group (Table 2).

Table 2.  Ovid Outcomes
Outcome Measure (Per Person/Week)Control Median, n (IQR) (N = 23)Intervention Median, n (IQR) (N = 24)P ValueRelative Increase in Activity
  1. IQR, interquartile range.

Log-ons2.1 (2.3)4.4 (2.2)<.0012.1
Total sets in search strategies24.0 (30.1)74.2 (60.8)<.0013.1
Total abstracts viewed5.8 (12.0)17.7 (19.8).0013.1
Total full-text articles viewed1.0 (1.5)2.6 (2.9).0052.6
Time spent searching (hr)0.8 (1.3)2.4 (1.0)<.0013.0

Comparison Between Self-report Survey and Objective Data

In self-report, residents in both groups overestimated their number of log-ons each week (P < .001 for both groups). To examine this overestimation, the average number of self-reported log-ons per week was subtracted from the Ovid log-ons for each individual to create a difference score. The median difference score was 2.9 in the control group and 2.4 in the intervention group. Therefore, both groups significantly overestimated their log-on activity (2.9 vs 2.4 more reported log-ons than observed log-ons) but the extent of overestimation was not significantly different between groups (P > .2).

Barriers to Searching and Retrieving Articles

There were no statistical differences between the 2 groups in responses to the self-report questions regarding the barriers to searching and retrieving articles. Both groups perceived that difficulty with strategies, question formation, and physical fatigue accounted for the majority of their searching problems (Table 3).

Table 3.  Most Important Barriers to Searching and Retrieving Articles
 Control, n (%) (N = 24)Intervention, n (%) (N = 24)
  1. There were no statistical differences between groups. Participants were asked to select all responses that were important barriers.

Barriers to searching
 Poor question11 (46)9 (37)
 Strategy did not work11 (46)13 (54)
 Too tired9 (37)9(37)
 Interruptions6 (25)6 (25)
 Other3 (12)6 (25)
 Connection problems3 (12)2 (8)
 Equipment problems0 (0)1 (4)
Barriers to retrieving articles
 Time17 (71)14 (59)
 Patient responsibilities18 (75)11 (46)
 Couldn't get to the library8 (33)8 (33)
 Library closed7 (30)14 (59)
 Article not worth it4 (17)7 (29)
 Other4 (17)3 (13)

The perceived barriers to retrieving articles were also similar between groups. Inpatient responsibilities, time, and access to the library accounted for the majority of the barriers to retrieving articles (Table 3).

Search Strategy Assessment Tool

Data on the clinical question and the associated search strategy were collected from a random subset of the intervention group. To determine the effectiveness of the strategy, the medical librarian (CS) reran the searches so that the project team could review the relevance of the retrieval. On the basis of this information, a search strategy assessment tool was generated (Fig. 3). While the purpose of this study was not to analyze the quality of the question or the search strategy, preliminary review of this data shows that a good clinical question may be essential to a successful search strategy.


  1. Top of page
  2. Abstract
  6. Acknowledgments

We examined the effect of an educational intervention that emphasized question building on the use of medline by medical residents. We have shown a 2-fold increase in residents' access to medline as well as a 3-fold increase in measures that reflect their on-line activity. Residents in the intervention group were on the system longer, generated more queries, and viewed more abstracts and full-text articles.

Prior work has focused on ways to increase utilization of information resources through enhanced access. In 1990, Haynes et al. were able to show that unrestricted online access to medline increased the amount of searching done by medical residents.14 Similarly, Sackett and Straus reported on the utility of an “evidence cart” that made evidence immediately available during rounds.15 Implicit in these findings is the assumption that when evidence is readily accessible, it will be used. In our study, even when access was universal there were significant differences in search activity. Thus, our study expands prior work by showing that access to information resources is necessary but not sufficient to maximize use of information technology for clinical care.

The marked increase in searching activity may be related to the way in which the intervention was designed and delivered. The intervention coupled a didactic session that focused on knowledge building regarding the link between a well-built clinical question and an executable search. In addition, the intervention focused on skill acquisition through real time, learner-centered, case-based practice that included reinforcement of concepts, repetition of skills, and process feedback. The WBCQ card facilitated the transition from question building to literature search. In addition, the success of the intervention may be related to its root in adult learning theory. As shown by Green and Ellis,16 when EBM curricula incorporate important aspects of adult learning theory, measurable success is possible. Our multifaceted intervention helped the medical residents understand why knowledge of EBM and the question-building process is necessary. The question-building activities facilitated responsibility in the process and provided experience. Finally, the use of the WBCQ card to direct medline searching provided the opportunity for repetition in life tasks that brought the process full circle.

Residents reported that the major barriers to successful searching were problems with question building and search strategies. Interestingly, there were no differences between groups in the most important barriers to searching and retrieving articles. For instance, despite the intervention, residents were challenged by clinical question formation. In addition, patient care and time pressures curtailed their ability to retrieve articles. One explanation for these similarities may be that after the intervention and significant amount of practice, the intervention group was more cognizant of their own limitations. Identifying barriers is an important first step in implementing change. Educational programs can be designed that balance the inherent time pressures of training with the skills necessary to move from the patient encounter to the literature and back to the bedside.

Once barriers are identified, others could implement the educational program used in this study without difficulty. The curriculum was concise and deliverable in a timely and consistent fashion (Fig. 1). Similarly, the WBCQ card and the expectation of searching are simple, exportable interventions that contributed to a substantial change in measured behavior.

This study has several limitations. First, we used a block randomization design to attempt to control for contamination bias, the time of year, and experience of the medical residents. Although study participants were blinded to the study question, it is impossible to know if contamination between groups was totally prevented. The data collected from the first week of each rotation were not used for analysis to reduce contamination between groups at the time of rotation changes. Second, we believe that residents utilized their personal ID codes to access the Ovid system. However, there are other ways to search medline that are separate from our library system and we had no way to collect this information. Finally, measures of search volume and activity may not correlate with the quality and relevance of information retrieved. Our research question focused on quantitative measurements of search activity. During the course of this study, we collected qualitative information on the individual clinical questions and search strategies that were used to design an assessment tool (Fig. 3). Ongoing work will focus on using this tool to assess the quality of clinical questions and searches and whether the literature retrieved is used to impact patient care.

In summary, this study advances the understanding of resident physician use of information technology. We have shown that in a setting of uniform access to information technology, a simple, exportable educational intervention markedly increased resident searching activity. However, in the classic paradigm of evidence-based medicine, the process must always begin and end with the patient. Future work will need to focus on the quality of information retrieved and whether the application of this information will lead to better care for our patients.


  1. Top of page
  2. Abstract
  6. Acknowledgments

This study could not have been performed without the assistance from the outstanding house staff in the Department of Medicine at Duke University Medical Center, in particular Kristen Tyszkowski, MD, Joseph Poku, MD, Mark Strong, MD, and Steven Crowley, MD.

Dr. Cabel is a Joseph C. Greenfield Jr. Scholar.


  1. Top of page
  2. Abstract
  6. Acknowledgments
  • 1
    Callahan CM, Dittus RS, Katz BP. Oral corticosteroid therapy for patients with stable chronic obstructive pulmonary disease. Ann Intern Med. 1991;114:21623.
  • 2
    Sontag SJ, Hirschowitz BI, Holt S, et al. Two doses of omeprazole versus placebo in symptomatic erosive gastritis: the U.S. multicenter study. Gastroenterology. 1992;102:10918.
  • 3
    Chen J, Radford MJ, Wang Y, Marciniak TA, Krumholz HM. Do “Americas Best Hospitals” perform better for acute myocardial infarction? N Engl J Med. 1999;340:28692.
  • 4
    Oxman AD, Sackett DL, Guyatt GH. Users' guides to the medical literature: I. How to get started. JAMA. 1993;270:20935.
  • 5
    Guyatt GH & Rennie D. Users' guides to the medical literature. JAMA. 1993;270:20967.
  • 6
    Bucher HC, Guyatt GH, Cook DJ, Holbrook A, McAlister FA. Users' guides to the medical literature: XIX. Applying clinical trial results. A. How to use an article measuring the effect of an intervention on surrogate end points. JAMA. 1999;282:7718.
  • 7
    McAlister FA, Laupacis A, Wells GA, Sackett DL. Users' guides to the medical literature: XIX. Applying clinical trial results. B. Guidelines for determining whether a drug is exerting (more than) a class effect. JAMA. 1999;282:13717.
  • 8
    Bennett KJ, Sackett DL, Haynes RB, Neufeld VR, Tugwell P, Roberts R. A controlled trial of teaching critical appraisal of the clinical literature to medical students. JAMA. 1987;257:24514.
  • 9
    Linzer M, Brown JT, Frazier LM, DeLong ER, Siegel WC. Impact of a medical journal club on house-staff reading habits, knowledge, and critical appraisal skills. JAMA. 1988;260:253741.
  • 10
    Osheroff JA, Forsythe DE, Buchanan BG, Bankowitz RA, Blumenfeld BH, Miller RA. Physicians' information needs: analysis of questions posed during clinical teaching. Ann Intern Med. 1991;114:57681.
  • 11
    Gorman PN & Helfand M. Information seeking in primary care: how physicians choose which clinical questions to pursue and which to leave unanswered. Med Decis Making. 1995;15:1139.
  • 12
    Ellis J, Mulligan I, Rowe J, Sackett DL. Inpatient general medicine is evidence based. Lancet. 1995;346:40710.
  • 13
    Richardson WS, Wilson MC, Nishikawa J, Hayward RS. The well-built clinical question: a key to evidence-based decisions. ACP J Club. 1995;123:A123.
  • 14
    Haynes RB, McKibbon KA, Walker CJ, Ryan N, Fitzgerald D, Ramsden MF. Online access to MEDLINE in clinical settings. Ann Intern Med. 1990;112:7884.
  • 15
    Sackett DL & Straus SE. Finding and applying evidence during clinical rounds: the “evidence cart.” JAMA. 1998;280:13368.
  • 16
    Green ML & Ellis PJ. Impact of evidence-based medicine curriculum based on adult learning theory. J Gen Intern Med. 1997;12:74250.