Introduction of Evidence-based Medicine into an Ambulatory Clinical Clerkship

Authors


Address correspondence to Dr. Thomas: 1830 East Monument St. Room 9033, Baltimore, MD 21205 (e-mail: pathomas@welch.jhu.edu).

Abstract

Evidence-based medicine (EBM) has emerged has a critical clinical competency in the 21st century. Medical schools usually introduce students to critical appraisal in the preclinical years, but there have been few evaluated interventions in teaching EBM in the clinical years. We describe a strategy to encourage students to practice EBM during a required ambulatory medicine clerkship. During this clerkship, our students are required to submit an EBM report, which is prompted by an individual case, and structured with a 5-step approach. One small-group session is devoted to modeling this approach with a case of chest pain. Using a checklist to grade 216 consecutive EBM reports, we found that students were quite successful with the exercise, achieving on average 89.6% of possible checklist points. Students who followed the structure of the exercise closely were more likely to extend their discussions beyond that required and to suggest potential further areas of investigation or design.

Evidence-based medicine (EBM) refers to the use of best current evidence in clinical decision making applied to individual patients. Recognizing that EBM is increasingly important as a clinical competency, medical schools and residency programs have sought effective strategies to teach EBM in an already crowded curricula.1–6 To effectively practice EBM, trainees need at least 3 skill sets: the ability to identify knowledge deficiencies generated by specific patient problems, the ability to search efficiently for best evidence, and the ability to critically appraise the evidence and apply it to the clinical problem. Medical school curricula most often introduce critical appraisal in biostatistics and clinical epidemiology courses.5,7 In a review of published literature on teaching critical appraisal skills, Norman and Shannon8 noted that these undergraduate programs invariably result in improved knowledge and attitudes in critical appraisal.

Helping students acquire the additional EBM skills of formulating questions and applying evidence to specific situations requires that EBM be incorporated into clinical clerkships as well. Previous efforts to do this included interactive seminars or mini-courses implemented during core clerkships, which have been evaluated by student self-assessment or performance on paper-based examinations.1,2,9,10 We sought to move these exercises from the classroom to the bedside and increase the learner-centeredness of the process during our ambulatory medicine clerkship. We assessed student competency in EBM by requiring a graded written report addressing a patient problem that had arisen during the clerkship.

This paper describes our experience with this educational strategy. We asked the following questions regarding the exercise: (1) For what types of clinical questions were students seeking evidence? (2) What elements of the exercise predicted successful application of EBM to the clinical question? (3) Did the exercise promote additional critical thinking by students? and ( 4) How did the students value the exercise as compared to other educational methods in the clerkship?

METHODS

The Ambulatory Medicine Clerkship is a required 4-week clinical clerkship for third- or fourth-year students, using a combination of clinical outpatient experience, small-group interactive sessions, and self-directed learning projects. Students arrive to the clerkship with background instruction in critical appraisal and searching techniques from the year 2 biostatistics and clinical epidemiology course. All students complete a practicum on literature searching during year 2, which is also available on-line on the student intranet. At our institution, students have access to online searching, primarily through Ovid during this study period.

To encourage the use of EBM during the clerkship, each rotation of students received the following intervention (Table 1). At orientation for the clerkship, students were given instructions in the structure of an EBM report, adapted from the 5 steps of EBM of Sackett et al.11(Table 2). Second, at a small-group conference in the first week of the clerkship, facilitated by one of the authors (JC), the 5-step EBM approach was reviewed and demonstrated through a case of chest pain. Students were then asked to formulate a specific question based on an actual patient interaction, and research available evidence to help answer the question. Finally, students presented their findings to their colleagues at a conference convened on the last day of the clerkship. These reports were used to trigger discussion points relevant to clinical epidemiology, such as the concepts of absolute risk and number-needed-to-treat.

Table 1.  Program Description: The Evidence-based Medicine (EBM) Exercise
Educational InterventionTimelineResources
Student orientation to 5 elements of EBM report (see Table 2)Orientation for the clerkshipClerkship director; syllabus
Small group session models the 5-step approach with a case of chest pain90-minute small-group session during week 1Clerkship faculty
Students select a patient case with a diagnostic or therapeutic dilemma from the clerkship experienceAnytime throughout the clerkshipClinic preceptors, patients, library on-line support
Students search for best evidence, submit report, and present findings to peers90-minute small-group session during week 4Clerkship director and clerkship faculty
Students receive graded checklist evaluation of reportAt completion of clerkshipClerkship director
Table 2.  The Evidence-based Medicine Report* for the Ambulatory Clerkship in Medicine
  1. *Adapted from Sackett et al.11

I. The Question: The clinical question is generated by 1 of the cases you have seen with your preceptor, and should include 3 elements: (1) patient type, (2) an intervention (preventive, diagnostic, or therapeutic), and (3) the patient outcome of interest.
II. The Search: Outline your search strategy and results.
III. The Answer: Choose one best paper that addresses your question. Briefly summarize this study as follows: objective of the study, design, setting, patients who participated, intervention, main outcome measures, and main results.
IV. Critical Appraisal: Do you accept the conclusions of the study? Why or why not?
V. Apply: Does the study apply to your clinical situation? Why or why not?

To score the reports, the clerkship director developed a 12-item checklist, which included the components suggested for the report, rated from 0 (not done) to 3 (completely done). In earlier rotations, many students had spontaneously expanded the exercise, suggesting further probes of the question. We wondered if elements of the exercise prompted this creative thinking. For this reason, an additional item was added to the checklist, “going beyond the exercise,” rated 0 (not done) to 3 (extensively done). Examples of going beyond the exercise include suggestions for further research studies or alternative designs that might answer issues which arose in the report's discussion. Lastly, the rater noted the topic of the report; whether a diagnostic, therapeutic, or physician-patient relationship issue was addressed; and what type of outcome was addressed.

For program evaluation, students anonymously complete an end-of-clerkship assessment using a Likert-type scale (0 = “not at all useful” to 5 = “extremely useful”) to rate the value of different educational interventions used in the clerkship. They also self-assess their personal growth and understanding in a variety of learning objectives, using a 6-point scale (0 = “none, clerkship didn't address” to 5 = “excellent, feel competent”).

Descriptive statistics, correlations and significance of correlation coefficients were generated with Stata statistical software (Small Stata 6.0; Stata Corporation, College Park, Tx). Differences between groups were measured with the Student t test.

RESULTS

Evidence-based Medicine Reports

Two hundred sixteen students (148 third-year and 68 fourth-year) completed the clerkship during the study period, and all submitted an EBM report (see Appendix). Fifty-eight topic categories were identified in these reports. The most frequent topics addressed by student questions were diabetes mellitus (20), hypercholesterolemia (17), cancer screening (12), smoking cessation (11), and hypertension (11). Fifty-six percent addressed topics not included in the clerkship curriculum of training problems, either in the workbook or at the small-group meetings. Eighty-one percent addressed therapeutic interventions, 16% diagnostic interventions. Of the therapeutic interventions, 82% addressed pharmaceutical interventions, 10% addressed surgical interventions, and 8% lifestyle interventions. The studies reviewed addressed morbidity and intermediate outcomes in 67%, mortality outcomes in 26%, and quality of life outcomes in 9%.

The checklist scoring system resulted in mean ratings of 2.5 to 2.9, out of possible maximum of 3, for 11 of the 13 individual items (Table 3). Third-year students did not differ from fourth-year students in their performance of the exercise. The mean score of third-year students was 34.43 (SD, 4.06) (maximum score possible was 39), and the mean score of fourth-year students was 34.80 (SD, 3.97; P = NS). Items which received the lowest ratings were “stating a patient outcome in the original clinical question” and “going beyond the exercise.” Evidence-based medicine reports that addressed a diagnostic intervention had lower ratings for the “going beyond” item, (mean, 1.52) than reports that addressed a therapeutic intervention (mean, 2.05; P < .001).

The ability to apply the paper to the case in question was correlated with 3 items: “patient type specified in the original question”(r = .39, P < .001), “description of the intervention clearly stated” (r = .19, P < .01), and “main outcomes clearly stated” (r = .21, P = .001). “Going beyond the exercise” was correlated with “applying the study to the case in question” (r = .17, P < .01).

Postclerkship Evaluations

Because the postclerkship evaluations were anonymous, they could not be matched with individual EBM reports. In a review of 186 postclerkship evaluations, (response rate, 86%), the mean ratings by students for the usefulness to their learning of the EBM report were 3.62 ± 1.07. These were lower than the values assigned to the clerkship syllabus (4.22 ± .90, P < .001) and to the clinical sessions with the preceptor (4.65 ± 0.64, P < .001), but higher than assigned to developing a learning plan (3.11 ± 0.97, P < .001). Students rated their perception of improved competence during the clerkship as follows: diagnostic decision making, 3.70 ± 0.73; EBM, 3.59 ± 0.79, (P < .001); and self-directed learning, 4.04 ± 0.75 (P < .001).

DISCUSSION

The EBM exercise was designed as a limited but feasible intervention which would build on earlier learning from the preclinical clinical epidemiology and biostatistics courses, and demonstrate an approach that students could practice during clinical years. The evaluation of the exercise did not focus on cognitive objectives for an entire curriculum in critical appraisal, but rather on the skill of application of EBM to a specific case. We believe this is the first publication describing a detailed evaluation of student use of EBM in actual clinical work.

Our experience builds on earlier work that has shown that even brief seminars in the third year improve students' self-assessed skills and attitudes. One of the earliest efforts to teach students EBM was a controlled trial,1 which found improved critical appraisal skills in paper-based problems after 16 hours of tutorial-based teaching during the core medicine clerkship. In 1987, Landry et al.2 reported that 2 interactive seminars during a medicine core clerkship improved student knowledge and attitudes, but did not increase the use of medical literature in blinded reviews of student write-ups. Ghali et al.9 found that a 4-session minicourse in EBM given to third-year students resulted in changes in student's self-assessed skills and attitudes, but did not objectively evaluate student competence. Ellis et al.10 used two 60-minute workshops in an ambulatory clerkship and also required a written project, but did not formalize the evaluation of these projects. Only one study has directly assessed student competency in structuring questions and searching: Burrows and Tylman12 used an objective structured clinical examination station to evaluate student performance in EBM, and found that only 5% of student search strategies and 26% of articles cited were ranked excellent or good.

A similar lack of evaluation was noted in Green's6 systematic review of residency curricula in EBM. He noted that resident curricula often focused on critical appraisal to the exclusion of other EBM skills, and called for more meaningful evaluations in future studies.

Our students' experience with the exercise has directed our teaching of EBM to emphasize the structure of clinical questions and the importance of articulating an outcome of interest. We found that students struggle with appropriate framing of clinical questions. Noting the patient type in the question was correlated with the ability to apply the evidence to the problem in our students' work. For example, students who failed to note the age or race of their patient were more likely to generalize the results of a study which would not have included their patient in the study population and to underestimate the limitations of this generalization. The analysis indicates that those students who successfully structured their original clinical question were indeed more successful in applying conclusions from the study to their patient situation and were more likely to expand the exercise beyond the original requirements.

The EBM topics chosen by students were of interest both to illuminate perceived needs and to demonstrate the self-directed nature of the exercise. Over half of our students chose topics that were not otherwise addressed in the syllabus or small-group discussions. The remainder of topics reflected the clerkship training problems, but students often queried new issues related to these problems, such as bupropion for smoking cessation and influenza vaccination in young adults. Traditional textbooks were unlikely to provide necessary information for these questions. While no formal curriculum can prepare students for every patient problem encountered, our experience makes a strong argument for students to begin using the skills of EBM early in their clinical training to address knowledge deficiencies as they arise in the context of their clinical experiences.

The content of the reports raises other questions, however. Despite the early stage of clinical training, our students seemed more concerned about therapeutic interventions than diagnostic interventions. This may relate to the nature of the clerkship, ambulatory medicine, but could also suggest that students are focusing on management issues before mastery of diagnostic skills. Alternatively, randomized controlled trials of therapeutic interventions may better fit the format of the exercise; those students who chose questions of diagnostic intervention had more difficulty with successful completion of the exercise.

Students had varying reactions to the exercise. Although on average the students rated it as useful, students did not rate it as highly as a structured syllabus or the clinical experience itself. By design the exercise was not comparable in time commitment or breadth to these activities, which may have affected the student ratings. Our feedback instrument was insufficient to distinguish whether students were rating components of the exercise, such as the small-group meeting and report development, or the student perception of EBM in general.

One concern is that the addition of a written exercise would be perceived as “busy work” in a clinical clerkship, but students rarely commented that this was the case. A second issue for students is their inefficiency in finding the “best evidence.” We did not formally evaluate search strategies or time required by students to complete the search, although we suspect that the ongoing upgrade of on-line searching resources to our medical library, including MDConsult,13The Cochrane Library,14 and Evidence Based Medicine Review,15 has resulted in less student reliance on literature searches and more on clinical guidelines to answer questions. As a result of student feedback, we now introduce the exercise with a computer laboratory facilitated by a medical librarian and informatics faculty to demonstrate search strategies and streamlined techniques using other databases and Web sites. An interesting further area of study will be to compare student performance in this exercise when these alternative databases are used.

One exercise could not be used to evaluate a student's general skills of searching and critical reading of the literature. The use of the checklist facilitates specific structured feedback to the students. This method needs to be further validated by testing the interrater reliability of the form, but since the checklist followed the structure of the exercise closely, we feel it has high content validity. The finding that fourth-year students did not perform better with this exercise than third-year students may be a ceiling effect of the checklist, or may indicate a need to extend these exercises throughout the clinical curriculum to better reinforce skill development.

This paper describes an approach to introducing EBM into clinical clerkships, using a structured exercise to stimulate bedside use of EBM by students as well as to assess their EBM skills. A number of factors might affect student use of EBM in future clinical experiences, despite their observed competency in this exercise, such as the perceived time or difficulty in completing this exercise, the preceptors' support of an EBM approach, and student maturation. A controlled trial with a long-term outcome would help to understand the importance of these factors as well as the impact of the intervention.

Acknowledgments

The authors are grateful for permission to use the appended EBM report by Charlotte Moore, MS IV, Johns Hopkins University School of Medicine.

Appendix

Evidence-Based Case Report Prepared by Charlotte Moore, MS III, Johns Hopkins University School of Medicine, June 1999

Question: Should an 86-year-old postmenopausal woman with diagnosed osteoporosis be treated with bisphosphonates or with hormone replacement therapy (HRT)?

Article: Wimalawansa SJ: A four-year randomized controlled trial of hormone replacement and bispohosphonate, alone or in combination, in women with postmenopausal osteoporosis. Am J Med. 1998;104:219–26.

Objective: To compare the effectiveness of HRT, bisphosphonates, and combined HRT plus etidronate on bone mineral density (BMD) in postmenopausal women with established osteoporosis.

Design: Prospective double-blind clinical trial with controls.

Seventy-three postmenopausal women with established osteoporosis were selected to participate in the double-blind clinical trial. Enrolled patients were randomly assigned to 1 of 4 treatment groups and followed for 4 years. The four treatment groups were: (1) HRT group (n = 18): premarin 0.625 mg daily and norgestrel 150 mcg for 12 days each month; (2) bisphosphonate group (n = 18): 400 mg disodium etidronate for 14 days each month; (3) combined therapy (n = 19): HRT and etidronate, as described above; and (4) controls (n = 18). All patients enrolled in the study, including controls, received 1.0 gm elemental calcium and 400 units of vitamin D per day.

Bone mineral density was measured in the lumbar spine and in the neck of the femur at the initiation of the study, after 2 years, and again after 4 years of treatment. Bone mineral density was measured using dual x-ray absorptiometry bone density scanning (DXA). In addition, new vertebral fractures were documented and changes in hei8ght were recorded.

Treatment groups were compared with respect to BMD data at baseline and with respect to relevant demographic data (including age, number of years since menopause, body mass index, and reported daily calcium intake). Baseline data was compared by 1-way analysis of variance (ANOVA). No significant difference in baseline characteristics was observed between the 4 treatment groups.

Changes in BMD from baseline were compared within each group after 2 and 4 years of treatment by the paired t test. Changes in BMD from baseline were compared between treatment groups after 2 and 4 years of treatment by ANOVA, correcting for multiple comparisons. The number of new vertebral fractures and changes in height between treatment groups were compared by the χ2 test. Rates of new vertebral fractures and height loss were compared with the Kruskal-Wallis (within each group) and with the Wilcoxon rank-sum test (between each group). An uncorrected P value of less than .05 was considered significant.

Setting: 1997. Patients selected from postmenopausal women attending metabolic bone disease outpatient clinics in London, England.

Patients: Of 350 patients screened, 73 postmenopausal women with established osteoporosis were selected for the study. Osteoporosis was defined as at least 1 atraumatic thoracic vertebral crush fracture and a spine BMD reported to be 2 standard deviations below the reference range mean established by health women.

The women were all Caucasian, with a mean age of 64.9 ± 0.5 years. Women with surgical menopause or secondary menopause were excluded from the study. Women treated with anabolic steroids, glucocorticoids, calcitonin, fluoride, HRT, or bispohsphonates at any time since menopause were also excluded.

Outcome Measures: Bone Mineral Density (BMD): determined by DXA at the second to the fourth lumbar vertebrae (spine) and at the neck of the femur (hip).

Vertebral Fractures: determined by lateral radiographs of the thoracic (T4-T12) and the lumbar (L1-L4) spine, defined as a reduction by at least 20% of the anterior, middle, or posterior height, with a reduction of at least 15% in the area of a previously unaffected vertebra.

Results: After 4 years of therapy, the results were as follows:

With respect to BMD:

  • HRT: BMD increased in the spine by 7.0% and in the hip by 4.8%.

  • Bisphosphonate: BMD increased in the spine by 7.0% and in the hip by 0.9%.

  • (3) Combined: BMD increased in the spine by 10.4% and in the hip by 7.0%.

  • (4) Control: BD decreased in the spine by −2.5% and in the hip by −4.4%. All values were significantly significant.

With respect to vertebral fracture: There was no statistically significant difference in incidence of new fractures or fracture rates between treatment groups. However, the group receiving combined therapy showed fewer absolute fractures, and an observed trend may indicate a lower fracture rate in this group.

With respect to height loss: All 4 treatment groups lost height during the 4-year study period. Height loss was significantly less in the combined therapy group than in the groups receiving single HRT or bisphosphonate therapy (P < .02 and P < .001, respectively). Height loss in all 3 active treatment groups proved to be significantly less than when compared to the control group.

Critical Appraisal: The results of the study suggest that combining HRT with bisphosphonate therapy will produce an additive effect with respect to increasing BMD and decreasing vertebral fractures in postmenopausal women with established osteoporosis. This study attempts to fill an important void in current clinical research on osteoporosis, namely the lack of rigorous research of tertiary prevention. As the authors mention, most osteoporosis research focuses on primary and secondary prevention. It is important that attention be paid to women with established bone disease, in an attempt to prevent further bone loss and complications of low bone density.

The double-blind design of the study ensures that study-related bone density testing and radiographic readings will not be biased by clinical researchers. The unbiased results show statistically significant data, privileging combined HRT plus bisphosphonate therapy over single-drug regimens.

Unfortunately, while the data may be statistically significant, it may not be clinically significant. A 3% further increase in the BMD of the spine, or a 2.2% further increase in the BMD of the hip (when comparing combined therapy to HRT alone), may not justify the extra complications, potential risks or significant expense associated with taking multiple drugs. Issues of cost and convenience are not discussed in this study, but are important in the clinical decision-making process. This is especially true in scenarios applying to elderly persons, often with limited financial means and long-standing, significant polypharmacy.

In addition, the sample size is small (n = 73). The smallness of the sample size is further exaggerated when the limited number of participants are further divided into 4 treatment groups. A large-scale study would be desirable to confirm the roposed results.

Applicability: The results of this study may be applied to my clinical situation for several reasons. Similar to the patients enrolled, my patient is a postmenopausal Caucasian woman with established osteoporosis. Unfortunately, the mean age of the study participants was 64.9 years, a full 20 years younger than my patient. Very few clinical trials involve women over 80. As more people live to be octogenarians, more clinical research must be devoted to this demographic. On all other points of biologic or clinical comparison, the study is applicable to this case.

Making a clinical decision based strictly on the outcomes of this study, I would prescribe combined HRT plus bisphosphonate to my patient. However, as mentioned above, clinical decisions are not based solely on biologic or clinical phenomena.

Bisphosphonates are expensive drugs, significantly more expensive than HRT. Bisphosphonates are difficult drugs to take, requiring that the drug be administered independently of other drugs or food, and requiring a 30-minute wait period, during which time the patient must remain upright, and no food or other necessary drugs may be taken. This is especially problematic in frail, elderly persons with significant polypharmacy—the very target population for the drug.

Examining the results of the clinical trial, and knowing that my patient has limited financial resources and knowing that she already must self-administer a complicated regime of expensive drugs, I would look to the documented success of HRT alone at building BMD and preventing fracture, and prescribe only HRT.

Ancillary