Presented at the Society for Academic Emergency Medicine midwest regional meeting, Ann Arbor, MI, September 2009; the Council of Emergency Medicine Residency Directors Academic Assembly, Orlando, FL, March 2010; the Illinois College of Emergency Physicians resident research competition, Chicago, IL, April 2010; and the Society for Academic Emergency Medicine annual meeting, Phoenix, AZ, June 2010.
Background: Principles of evidence-based medicine (EBM) may be inconsistently applied to clinical decision-making due to lack of practice-based training, experience, and time.
Objectives: The authors sought to design, implement, and test the feasibility of an experiential learning model for senior emergency medicine (EM) residents to apply EBM principles during real-time clinical practice.
Methods: Targeted program evaluation of this learning model was conducted through a prospective observational cohort study involving EM residents at a large, urban, 4-year EM residency program. The curriculum development of a case-based knowledge translation shift followed Kern’s six-step design process. Subjects asynchronously completed a 1-hour EBM tutorial and were then assigned to clinical shifts in which they contributed to the care of emergency department (ED) patients by completing formal literature searches related to active management questions. Pre- and post-intervention self-assessments of practice norms and attitudes were used to evaluate the effect of this experiential learning model for individual residents. Self-assessments of the likelihood that the experience would result in future practice change were reported on a five-point Likert scale (1 = greatly impeded, 2 = somewhat impeded, 3 = no change, 4 = somewhat improved, 5 = greatly improved). Subjects presented available evidence to the primary ED team, formally disseminated their findings as a brief “EBM rounds” at sign-out and completed an “EBM consult note” and case log to document shift performance. Changes in patient management and/or disposition were recorded. EBM search questions and resultant findings were entered in a local database.
Results: Of the 45 eligible senior EM resident shifts, 91% resulted in complete sets of performance data and self-assessments. A total of 80 patient encounters were documented during 45 scheduled shifts over a 3-month study period. Literature review took a mean (±SD) of 36.2 (±26.4) minutes per case. During the 3-hour interval before or after shift sign-out, residents completed a mean (±SD) of 2.11 (±1.4) literature searches and recorded a mean (±SD) of 3.0 (±1.5) articles for each case. Alterations in ED management for 13 of 80 patient encounters (16.3%) were documented to be the direct result of on-shift literature searches.
Conclusions: Case-based knowledge translation shifts for senior EM residents can provide opportunities to practice EBM skills in the ED. This experiential learning model may result in future practice change by resident learners, as well as affect the management of active patients in the ED.
Knowledge translation is the transfer of evidence-based information to systems of care.1 The World Health Organization defines knowledge translation as “the synthesis, exchange, and application of knowledge by relevant stakeholders to accelerate the benefits of global and local innovation in strengthening health systems and improving people’s health.”2 Knowledge translation at the individual provider level has been defined as “clinical shifts with a focus on real-time, need-to-know, knowledge acquisition, dissemination, and translation.”3 To appropriately apply evidence at the bedside, a provider must have the ability to generate a focused question from variable clinical stimuli, the knowledge of and access to appropriate database resources to answer the question, the competency to critique available information, and an understanding of how to apply that information in clinical practice.4,5
The efficient use of information technology to rapidly access and apply evidence has become an essential element of the practice of medicine.6,7 Specialty organizations and expert panels have endorsed several recommended innovations in knowledge translation research and education.3,4 Although evidence-based websites, data repositories, and educational curricula have increased in complexity and popularity, some clinicians still lack experience using even basic search engines in their daily practice. A survey of general practitioners found that while over 40% of physicians were familiar with The Cochrane Database, only 15% had ever applied it during clinical practice.8 An examination of barriers to evidence-based decision-making in the United States and in the Netherlands demonstrated that 30% to 40% of patients do not receive care according to available, accepted guidelines.9 There are gaps between accessible technology, training opportunities, and knowledge translation in the clinical environment. This article describes the design and testing of an experiential pilot curriculum in case-based knowledge translation used to provide senior emergency medicine (EM) residents with opportunities to practice evidence-based medicine (EBM) skills while on shift in the emergency department (ED).
This curriculum development project used a six-step design approach.10 Targeted program evaluation was conducted through a prospective observational cohort study. The institutional review board approved the study protocol.
Study Setting and Population
The educational program was designed for EM residents at an urban, academic medical center with an annual ED census of 81,000 patient visits. Subjects included all 24 senior EM residents (postgraduate years [PGY] 3 and 4) in this 4-year training program. All subjects were required to participate in the curriculum, but completion of pre- and postintervention self-assessments was optional. Residents were informed that their anonymous shift performance data and self-assessment responses may be used for educational research purposes if they provided written, informed consent prior to participation in the curriculum.
The study team used Kern’s six-step approach to design an experiential curriculum in knowledge translation.10 This is a well-established model for curriculum design that has been shown to lead to successful implementation and long-term sustainability.11
1) Problem Identification and General Needs Assessment. Investigators performed a detailed review of the existing residency program for elements of formal or informal instruction in evidence-based decision-making. Each of the four investigators met with residency leadership to create a consensus needs assessment prior to the design of the knowledge translation curriculum. This was compared with other descriptions of knowledge translation curricula for EM residents available through literature review.4,12–17
2) Local Needs Analysis. Senior EM residents, including PGY-3 and PGY-4, were queried for additional curricular elements using an anonymous, preintervention self-assessment of their practice norms and attitudes regarding the use of EBM in our ED (Data Supplement S1, available as supporting information in the online version of this paper). This self-assessment was created by the investigators and included questions that were based on findings of the general needs assessment and literature review. As this was a pilot study, neither content knowledge nor skills were assessed in this local needs analysis, and validation of the tool was not performed. Data were collected using SurveyMonkey (http://www.surveymonkey.com, Oak Park, CA).
3) Educational Goals and Objectives. Findings of the needs analyses were summarized and three educational goals for the case-based knowledge translation curriculum emerged. The resultant goals were: 1) to improve EM resident knowledge of available EBM search engines, 2) to improve EM resident practice of knowledge translation skills by facilitating multiple opportunities for learners to formulate and answer clinical questions about ED patients in real time, and 3) to teach residents to communicate principles of knowledge translation to other ED providers while on shift.
4) Educational Strategies. The educational model selected to accomplish our local program’s goals was a scheduled knowledge translation shift for senior EM residents. This includes an asynchronous study of EBM tutorials made available online, practice of case-based knowledge translation skills in the ED, and reflective practice.
5) Implementation. The case-based knowledge translation shifts were pilot tested over a 12-week clinical schedule in July 2009 through September 2009. Third- and fourth-year EM residents were each scheduled for up to three 8-hour shifts during weekday, daytime hours in the study period. During their first shift, subjects were consented for analysis of their performance data and the previously completed self-assessment. Subjects then completed the Web-based course, “Introduction to Evidence-Based Medicine,” offered by the University of North Carolina (Chapel Hill, NC).18 A minimum of 1 hour was reserved for completion of this asynchronous learning module. Additional search engines were available for review on the online curriculum page. The remainder of time in the scheduled shifts was dedicated to the practice of case-based knowledge translation in the ED. Subjects were present in the ED and sought out cases in which a diagnostic or therapeutic management decision was under active consideration by the care team. Using the P-I-C-O framework (problem, identify, comparison, outcome), the subjects accessed Web-based search engines and identified evidence relevant to the clinical question. They documented their results as a brief “consult” note in the chart and presented the findings to the care team. The clinical questions and relevant literature citations were summarized by the subjects as an entry in a local database to be used by residents or faculty for teaching EBM at the bedside. At shift sign-out, the subjects conducted a brief EBM “teaching rounds,” in which they reviewed the clinical question, method, and source of the literature search, and the relevant evidence was used to answer the management question.
6) Program Evaluation. At the completion of their scheduled case-based knowledge translation shifts, subjects completed a postintervention self-assessment (Data Supplement S2, available as supporting information in the online version of this paper). This was designed to characterize their practice norms and attitudes regarding EBM. Case logs were used as performance measurements for their shifts. The self-assessment allowed for formal reflection of learned knowledge and skills; again, this tool was created by the investigators and was not validated.15 Subjects recorded qualitative and quantitative information about the type and number of literature searches they completed, any perceived effect on the medical decision-making by the care team in the ED, the length of time required to complete their searches, and demographic information. Data were again collected using SurveyMonkey. After the 3-month pilot was completed, the investigators evaluated the feasibility and effect of the case-based knowledge translation curriculum based on the resident self-assessments and shift performance data. A revised curriculum that included optimized shift goals, length, and activities was presented to residency leadership for incorporation as a sustainable experiential learning opportunity within the residency curriculum.
Shift performance data were recorded using open-ended fields to collect information on the content and length of the completed searches. Pre- and postintervention self-assessments used a five-point Likert scale to evaluate the effect of the case-based knowledge translation curriculum on resident practice of learned skills, as well as the potential for future practice change (1 = greatly impeded, 2 = somewhat impeded, 3 = no change, 4 = somewhat improved, 5 = greatly improved). Descriptive statistics were used to analyze performance and self-assessment data for the cohort. All statistical analyses were performed using Stata 9.0 (StataCorp, College Station, TX).
During the 3-month pilot of the curriculum, there were 45 scheduled case-based knowledge translation shifts, of which 41 (91%) resulted in complete sets of performance data and self-assessments. All 24 senior EM residents participated at least once in the scheduled shifts, and 96% of subjects consented to analysis of their responses; 54% were female, and 50% were PGY-3 residents. Subjects who provided complete data sets were identified as 73% female and 66% PGY-3 residents.
Prior to the case-based knowledge translation shifts, most residents self-reported that the proposed, experiential EBM curriculum might change their future clinical practice (mean ± SD = 3.57 ± 0.9 on the 5-point Likert scale). After participation in this pilot curriculum, there was no significant change in the number of subjects that anticipated further practice changes as a result of their experience (mean ± sd = 3.68 ± 0.8). Literature review took a mean (±SD) of 36.2 (±26.4) minutes per case. During the 3-hour interval before or after shift sign-out, residents completed a mean (±SD) of 2.11 (±1.4) literature searches relevant to actual patients in the ED, and recorded a mean (±SD) of 3.0 (±1.5) articles per case. In total, residents completed an EBM search on 80 patients, and 13 cases (16.3%) resulted in a change in management by the primary team based on the literature searches performed by the subjects. The change in management was a perceived change as reported by the subjects and was not reviewed by an attending physician. Data are summarized in Table 1.
Table 1. Resident Evaluation Data
Values are mean (95% CI).
EBM = evidence based medicine.
Will the knowledge you gained today change your future clinical practice? 5-point Likert scale
Did this shift change your skills for practicing EBM? 5-point Likert scale
How many articles did you use during your senior knowledge translation shift?
3.0 (2.5–3.5) articles
How many minutes did each EBM search take?
36.2 (27.5–44.9) minutes
For how many patients [during this shift] did you complete an EBM search and then apply your findings to an individual patient?
2.11 (1.65–2.56) patients/senior EBM shift
80 patients/pilot curriculum
How many patients [during this shift] had a change in their care as a result of your involvement in the case as the “knowledge translation shift senior resident”?
0.34 (0.12–0.56) patients/senior EBM shift
13 patients/pilot curriculum
For how many patients [during this shift] did the number of laboratory tests change as a result of your involvement in the case?
0.16 (0.01–0.30) patients/senior EBM shift
6 patients/pilot curriculum
For how many patients [during this shift] did the number of imaging tests change as a result of your involvement in the case?
0.05 (0–0.13) patients/senior EBM shift
2 patients/pilot curriculum
For how many patients [during this shift] did the disposition change as a result of your involvement in the case?
0 (0–0) patients/senior EBM shift
0 patients/pilot curriculum
The number of articles identified as a function of time in the study period is reported in Figure 1. As time progressed in the pilot curriculum, EM residents identified fewer articles per shift and were less likely to provide a complete case log of their activities.
The case-based knowledge translation shift designed in this project provided an experiential learning opportunity for senior EM residents to study and practice evidence-based decision-making in our ED. Self-assessments of practice norms and attitudes suggest that residents may incorporate learned skills in their future clinical practice. The self-reported change in management demonstrates that the curriculum had a modest effect on the real-time care of ED patients.
In 2006, the Institute of Medicine wrote that “a deeper commitment to evidence-based medicine can improve care … to accelerate the adoption and dissemination of effective, safe, and worthwhile interventions while avoiding those which are harmful or more costly than equally effective alternatives.”19 Although a multitude of resources exists for facilitating evidence-based decision-making in the clinical unit, physicians do not uniformly translate available knowledge into clinical practice.13
We chose an experiential model for teaching knowledge translation to residents to allow for the practice of learned skills. Previous studies have demonstrated that integrated approaches driven by self-motivation and clinical relevance result in behavioral change; passive approaches (e.g., attending a lecture) do not reliably affect behavior.20 The constructivist model of learning proposes that learners are best taught if they are at the center of the educational process.21 Knowles22 argued that the prime motivator for adult learning is the ability to apply knowledge and skills to an immediate problem. Lifelong learning is promoted through regular integration of new knowledge and skills in routine practice.13
It is critical to teach trainees how to efficiently access information technology that can aid in their clinical decision-making. Such instruction should begin in medical school or residency, while physician learners are more receptive to change.23 The Accreditation Council on Graduate Medical Education (ACGME) endorses such curricular content, mandating that “residents must be taught an understanding of basic research methodologies, statistical analysis, and critical analysis of current medical literature.”24
To the best of our knowledge, this pilot study is the first implementation of a case-based knowledge translation curriculum that encourages EM residents to teach newly learned EBM skills to colleagues while on shift in the ED. This model offers a novel method of active learning that can reinforce the integration and application of new knowledge by resident learners who must demonstrate a command of the material as they teach. Prior studies have described the effect of resident and attending shifts dedicated to bedside teaching in the ED,25,26 as well as a curriculum for developing resident teachers.27 While these studies demonstrated feasibility as innovative options for clinical teaching, they did not explicitly link practice-based learning and knowledge translation by trainees. A knowledge translation curriculum was previously described by educators at a pediatric residency program; however, that curriculum did not attempt to reinforce learning through real-time clinical decision-making or bedside teaching.12 Clinical integration of bedside EBM searches has been proprosed, yet some authors question whether or not it is possible to adequately apply medical literature in real time.20 We did not attempt to evaluate the quality of searches made by residents during this study; rather, we aimed to determine their ability and attitudes via their completion of the surveys.
During the pilot test of our curriculum, residents self-reported that their efforts on the case-based knowledge translation shift influenced a change in management by the primary ED team in 16.3% of cases. A majority of these cases involved a change in diagnostic testing. No changes in final disposition were cited as a direct result of the literature searches provided to the care teams. We did not record changes in length of ED stay or cost during this pilot project. Our findings are similar to those of another center that documented practice changes in the ED as a result of a focused didactic curriculum in EBM, noting that similarly significant educational efforts must be made to achieve favorable results.28
The documented number of articles reviewed per shift decreased over the 3-month pilot project, with a marked drop in documentation after senior residents completed an average of two shifts. This may represent learner fatigue with the educational experience after a certain amount of practice had been completed or fatigue with the process of recording case logs used for this study. Informal discussions with the residents suggested the former, as several residents felt that a 1-hour tutorial plus 12 hours of on-shift practice was adequate to reinforce content and skills. It is not surprising that the subjects devalued the shift over time. Knowles22 stated that learning and growth take place when “the next step is more intrinsically satisfying than the previous gratification, with which we have become familiar and even bored.”
The overall cost to sustain our case-based knowledge translation curriculum is minimal, with faculty effort primarily focused during the initial design and testing of the program. Since residents review selected course content in an asynchronous fashion, no additional faculty teaching time was required in the classroom. There are numerous EBM curricula available online at no cost; in some cases, content can be tailored to local needs. A brief faculty development session was presented during a regularly scheduled faculty meeting, to orient faculty to the project and facilitate an acceptable role for the resident learner.
A local repository of clinical questions and relevant high-quality literature has emerged as a significant benefit of the case logs used to document shift activities. This database offers program faculty a source of teaching cases and related articles for use during bedside rounds and didactic lectures. Several journal club selections emerged from our database this year.
Based on our findings in this study, we believe that the case-based knowledge translation shift is an effective method of teaching principles of EBM to senior EM residents outside of the classroom. We refined our final curriculum to include 16 hours of asynchronous learning: 4 hours of content review using electronic media and 12 hours of skills practice on shift in the ED. At our institution, two 8-hour knowledge translation shifts are now included in the clinical schedule for third-year residents in the fall.
There are several notable limitations to this feasibility study of our case-based knowledge translation curriculum. Selection bias results from a small sample of eligible subjects at a single institution, limiting generalization to all educational settings. Both recall bias and the Hawthorne effect are significant limitations of this study, as data were collected using self-assessment tools that relied on resident reporting of shift performance. The number of study participants was not powered to detect a significant difference in pre- and postintervention self-assessments, nor could we measure a significant change in the management, disposition, length of ED stay, or charges incurred by patients. The authors recognize that the quality of physician self-assessment is limited.29
The 4-year training program at our institution allowed for flexibility in scheduling that may not be possible in a 3-year setting. Although the ACGME requires that programs teach residents skills for lifelong learning, case-based knowledge translation shifts may be limited by work hour restrictions at certain residencies.3
We limited our analyses to learner attitudes and potential for individual behavioral change as markers of incorporating new knowledge and skills. These are essential steps to advancing adult learners past the early stages of learning (precontemplation, contemplation), but these markers do not address the adequacy of the curricular content that was taught.30 Future cycles of the curriculum will need to more explicitly test specific medical knowledge and decision-making.
Case-based knowledge translation shifts for senior EM residents can provide opportunities to practice evidence-based medicine skills and decision-making in the ED. The experiential learning model described in our pilot study may affect future individual practice change by our resident learners.
The authors are grateful to the senior emergency medicine residents who completed the pre- and post-test self-assessments.