ACADEMIC EMERGENCY MEDICINE 2011; 18:53–59 © 2011 by the Society for Academic Emergency Medicine
Objectives: Rotating residents represent a significant proportion of housestaff in academic emergency departments (EDs), yet they rarely receive targeted didactic education during their emergency medicine (EM) rotations. The goals of this study were: 1) to determine the effectiveness of an online didactic curriculum in improving EM knowledge among rotating residents and 2) to assess rotating resident satisfaction with this curriculum.
Methods: The authors created an online lecture series of six EM subject areas targeted to rotating residents called the Northwestern University Rotating Resident Curriculum (NURRC). All rotating residents at the study site were eligible, written consent was obtained, and the study was approved by the institutional review board. Consenting participants were pretested with a 42-question multiple-choice examination and then randomized to two groups: one with access to the NURRC during the first 2 weeks of the rotation (experimental) and one without (control). Halfway through the rotation, all participants were posttested with a different multiple-choice examination, and the controls were then granted NURRC access. The primary outcome was the difference between pretest and posttest scores (score delta). The t-test was used to compare mean scores, and a linear regression model was used to determine the association of NURRC access on score delta after adjustment for pretest type and resident type. A postintervention survey was administered at the end of the rotation to assess satisfaction with the NURRC and collect suggestions for improvement.
Results: Fifty-four rotating residents were enrolled: 29 in the experimental group and 25 in the control group. There was no significant difference in pretest scores between the two groups. Mean score delta was 17.3% in the experimental group and 1.6% in the control group, an absolute difference of 15.7% (95% confidence interval [CI] = 10% to 22%). After adjustment for resident type and pretest type, the only variable positively associated with the primary outcome was NURRC access. Third-year and preliminary-year internal medicine (IM) residents demonstrated the greatest absolute improvement in score delta when granted NURRC access. Eighty percent of the participants responded to the satisfaction survey. Over 80% of the survey respondents approved of each component lecture and of the NURRC overall.
Conclusions: After exposure to an online didactic curriculum, rotating residents demonstrated a significant increase in EM knowledge and reported a high level of satisfaction with the didactic program.
Curricular design for graduate medical education ideally incorporates a combination of clinical and didactic experiences.1 Since its official recognition as a specialty in 1979, emergency medicine (EM) has developed sophisticated core curricula for the training of its residents. However, EM residents are not the only housestaff who provide clinical service in the emergency department (ED).2–4 Rotations in EM are required by the Residency Review Committee for six different subspecialties and strongly recommended by another eight.5 Rotating residents comprise roughly a quarter of total housestaff in academic EDs.3 The ED provides a unique training environment to rotating residents by offering them access to patients with undifferentiated medical issues, continuous supervision by attending physicians, and opportunities for improving procedural competency.2,3,6,7 Additionally, the ED affords rotating residents exposure to unique clinical areas, including resuscitation, emergent airway management, toxicologic and environmental emergencies, and trauma.7 Internal medicine (IM) residents, in particular, benefit greatly from the educational opportunities in ambulatory medicine available in the ED.8
Since 1984, EM educators have advocated for organized didactics for rotating residents during their EM rotations.2,6,9,10 However, nearly one-third of EM residency programs provide no didactic education at all to rotating residents, and only 5% have a curriculum specific to their learning needs.3 Studies have shown multiple barriers to the implementation of such didactics, including lack of perceived need, lack of faculty resources, and difficulty predicting the educational needs of rotating residents.3,11,12
Optimal didactic curricular content for rotating residents during their EM rotation is slowly being defined.2,6,7 Kessler et al.6 recently proposed a standardized curriculum using EM and IM expert opinions. The efficacy of this or any other existing didactic curriculum at improving knowledge among rotating residents is unknown. While didactic sessions for medical students can help improve their clerkship test scores,13 discrete learning objectives and reading materials for rotating residents did not improve knowledge acquisition on a multiple-choice test in one study.10 In another study, Burnette et al.14 found that a Web-based, pediatric EM lecture series improved test scores of residents and medical students (mean improvement = 5.2%). However, the study was not randomized, only 25 of the 111 of the subjects were rotating residents, and the specific performance of the rotating residents (compared to the EM residents and medical students) was not described. The goal of this study was to test the effectiveness of an online didactic curriculum for rotating residents at improving knowledge acquisition of specific EM subjects.
This was a crossover study of an online didactic curriculum created to enhance the EM knowledge of rotating residents. We obtained written consent from the participants before the start of their EM rotation. The institutional review board at Northwestern University approved the study.
Study Setting and Population
Our study site was an urban, tertiary care, Level I trauma center with an annual ED census of 80,000 patients. All categorical and preliminary IM and surgical subspecialty rotating residents from December 2008 to August 2009 were eligible for the study. Obstetrics-gynecology residents were excluded because they began rotating in the ED after the study commenced. On completion of this study, participants were compensated with a $10 gift card to a local café.
We developed an online, narrated lecture series on EM subjects clinically relevant to rotating resident education entitled the “Northwestern University Rotating Resident Curriculum (NURRC).” We selected the lecture subjects (ears/nose/throat [ENT]/ophthalmology, environmental, obstetrics-gynecology, orthopedics, toxicology, and trauma) based on consultation with experienced IM educators at our institution. These IM educators reviewed lecture content to ensure relevance to the learning needs of IM residents. Visual slide content was formulated in PowerPoint (Microsoft Corp., Redmond, WA), audio narration was added in GarageBand (Apple, Inc., Cupertino, CA), and files were saved in a password-protected e-mail account as attached QuickTime (Apple, Inc.) movies. Each online lecture was 20–30 minutes in length and viewable on any computer with Internet access.
We developed two multiple-choice tests, labeled “A” and “B,” to assess medical knowledge. Each test had 42 questions divided equally among the six didactic subject areas. Both the subject areas and the test questions were initially formulated by an EM educational working group and approved by IM educators at our institution. Questions were then piloted in a sample of rotating residents for several months to gather quality and relevance feedback from test takers and IM residency leadership. This was done prior to finalization of the tests. Implementation of the education intervention was tested in a subsequent final sample.
Tests were administered to participants via SurveyMonkey (SurveyMonkey.com), and results were recorded with unique identifier numbers. We used an online program (http://www.random.org) to randomize participants to either A or B as the pretest, with the other serving as the posttest. All testing was performed in the presence of one of the study investigators. Participants were instructed not to use accessory materials while taking the tests and were informed that the scores would not affect their rotation evaluation or be reported to their respective residency directors. No EM faculty or residents other than the study investigators were aware of the didactic subject material present in the lectures or in the tests.
At the start of their EM rotations, all participants were pretested and then randomized into two groups: one receiving access to the NURRC (experimental group) and one not (control group). During the first 2 weeks of the rotation, the experimental group was granted unlimited access to the NURRC. Halfway through the rotation, we posttested all participants and then granted NURRC access to the controls to maintain the ethical crossover design of educational research. At the end of their EM rotation, a postintervention survey was administered to assess satisfaction with the NURRC. Before the posttest, the only education provided to the controls was clinical, on-shift teaching by the EM faculty and senior residents.
The primary outcome measure was the difference between a given participant’s pretest and posttest scores (score delta). The secondary outcome was satisfaction with the NURRC as measured by the anonymous, optional, postintervention survey. We developed the survey questions to assess resident satisfaction and recommendations by rotating residents for potential applicability to other academic EDs. Survey questions sampled participant ratings of the individual lectures, and of the NURRC as a whole relative to alternative didactic curricula. Respondents were also asked to provide free text feedback to give suggestions on how to improve the lectures.
We estimated that a meaningful difference between groups in test performance would be a 10% improvement in the score delta (posttest minus pretest). We assumed that the mean score delta would be 5% in the control group, compared to 15% in the experimental group. For detection of statistical significance, with a two-tailed mean comparison test with 90% power and alpha of 0.05, a sample size of at least 22 persons in each group was needed. This assumed a standard deviation (SD) of mean score delta to be 10% in both groups. The Student’s t-test was used to compare mean test scores. Before the posttest, members of the experimental group were asked whether they had actually viewed the lectures during the NURRC access period. Participants randomized to the experimental group who did not view the lectures were still included in the experimental group in an “intention-to-teach” analysis to preserve the effects of randomization. A linear regression model was created to determine the effect of group assignment, pretest type, and resident type on the primary outcome. Responses to the satisfaction survey were recorded on a five-point Likert scale. Positive approval ratings were based on combined percentage of responses of “5 = strongly like” and “4 = like” and are reported as proportions with 95% confidence intervals (CIs). All statistics were performed using Stata (Version 9, StataCorp, College Station, TX).
We enrolled 54 residents, including 18 preliminary IM, 18 Postgraduate Year (PGY)-2 IM, eight PGY-3 IM, three PGY-1 ENT, and seven PGY-1 orthopedics. Of the 59 residents eligible during the enrollment period (December 2008–August 2009), 54 (92%) completed the study (see Figure 1).
Fifty-four percent of the participants were randomized to the experimental group (see Table 1). There were no statistically significant differences in resident type between the groups. Mean pretest scores were similar between the experimental group (50.8%) and control group (52.6%, 95% CI for difference = −7% to 3%). Mean pretest scores stratified by resident type were also similar (Table 2).
|Experimental, n (%)||Control, n (%)||Percent Difference (95% CI)|
|Total participants||29 (53.7)||25 (46.3)||7.4 (−11.1 to 25.2)|
|Total medical||24 (82.8)||20 (80.0)||2.8 (−17.8 to 24.2)|
|Categorical IM||11 (37.9)||15 (60.0)||−22.1 (−44.6 to 4.3)|
|PGY-3||3 (10.3)||5 (20.0)||−9.7 (−30.0 to 9.9)|
|PGY-2||8 (27.6)||10 (40.0)||−12.4 (−35.6 to 12.2)|
|Preliminary IM||13 (44.8)||5 (20.0)||24.8 (0 to 45.7)|
|Total surgical||5 (17.2)||5 (20.0)||−2.7 (−24.2 to 17.8)|
|ENT||0 (0)||3 (12.0)||−12.0 (−30.0 to 2.1)|
|Orthopedics||5 (17.2)||2 (8.0)||9.2 (−10.3 to 27.5)|
|RR Type||Mean Pretest Score (%)||Mean Posttest Score (%)||Mean Score Delta (%)|
|Ctrl||Exp||Diff. (95% CI)||Ctrl||Exp||Diff. (95%CI)||Ctrl||Exp||Diff. (95% CI)|
|Total||52.6||50.8||−1.8 (−7 to 3)||54.2||68.1||13.9 (8 to 20)||1.6||17.3||15.7 (10 to 22)|
|PGY-3 IM||54.8||60.3||5.5 (−4 to 15)||51.9||81.0||29 (15 to 43)||−2.9||20.7||23.5 (12 to 35)|
|PGY-2 IM||48.8||50.3||1.5 (−7 to 10)||55.5||70.8||15.3 (6 to 25)||6.7||20.5||13.8 (3 to 25)|
|Prelim IM||49.0||50.2||1.2 (−8 to 11)||50.0||68.3||18.3 (6 to 30)||1.0||18.1||17.1 (5 to 30)|
|Ortho||60.7||47.6||−13.1 (−27 to 1)||57.1||55.7||−1.4 (−31 to 28)||−3.6||8.1||11.7 (−20 to 44)|
The primary outcome, mean score delta, was 17.3% in the experimental group and 1.6% in the control group, an absolute difference of 15.7% (95% CI = 10% to 22%). PGY-3 and preliminary IM residents demonstrated the greatest absolute improvement in mean score delta when exposed to the NURRC (23.5 and 17.1%, respectively). Orthopedics residents were the only resident type not to demonstrate a statistically significant improvement in mean score delta after NURRC exposure (11.7%, 95% CI = −20% to 44%).
After multivariate linear regression analysis, the only variable that was independently significantly associated with the primary outcome was assignment to the experimental group (see Table 3). Pretest type (A vs. B) and resident type were not independent predictors of score delta in the regression model.
|Predictor Variable||Coefficient||p-value||95% CI for Difference|
|Access to NURRC||15.6||<0.001||9.3 to 22.0|
|Pretest type||−5.8||0.062||−11.9 to 0.3|
|IM preliminary||−3.8||0.299||−11.1 to 3.5|
|IM PGY-3||−4.4||0.337||−13.6 to 4.7|
|ENT||−5.5||0.433||−19.5 to 8.5|
|Orthopedics||−12.0||0.015||−21.7 to −2.4|
Selected results of the NURRC satisfaction survey are shown in Figures 2 and 3. Forty-three of the 54 total study participants completed the survey (79.6% response rate). All NURRC lectures achieved an approval rating over 80%; the ENT/ophthalmology lecture had the highest approval rating at 91.9%, and the obstetrics-gynecology lecture the lowest at 80.6%, although the difference was not statistically significant (11.3%, 95% CI = −5.0% to 27.8%). Survey respondents felt that the three most important topics necessary to add to the NURRC were cardiopulmonary emergencies (41.7%), resuscitation (30.6%), and neurologic emergencies (27.8%).
A large majority of survey respondents felt that a didactic component during the EM rotation was essential for rotating resident training (83.7%, 95% CI = 70.0% to 91.9%) and that the NURRC enhanced their education (81.4%, 95% CI = 67.4% to 90.3%). Most respondents recommended the NURRC to other rotating residents both at our study site (86.0%, 95% CI = 72.7% to 93.4%) and at other academic institutions (72.1%, 95% CI = 57.3% to 83.3%). Only 7% (95% CI = 2.4% to 18.6%) would have preferred in-person lectures, and 2.3% (95% CI = 0% to 12.1%) journal or textbook readings over the NURRC.
This study established the efficacy of an online didactic curriculum in improving knowledge acquisition among rotating residents during their EM rotations. Rotating residents who received access to the NURRC for 2 weeks demonstrated more than a 15% mean absolute improvement on posttesting compared to those who were educated by clinical training alone. This improvement translates to over six questions on our 42-question test (an average of one question per didactic subject), which we believe is educationally significant. In our linear regression model, the only variable positively correlated with score delta was access to the NURRC.
We found a greater improvement in score delta than that seen by Burnette et al.14 in their evaluation of an online didactic lecture series. This is likely due to a number of differences in study design. Our study had a randomized, contemporaneous control group, included only rotating residents (as opposed to EM residents and medical students), had more resident participants (54 vs. 25), and contained didactic subjects in all areas of EM (as opposed pediatric-specific content).
The positive effect of the NURRC on knowledge acquisition is likely greater than that observed by our study. Five members of the experimental group indicated that they did not actually view the lectures. In our initial analysis, we used the “intention-to-teach” model by including these nonviewers in the experimental group as originally randomized. If the score deltas of the nonviewers are excluded from the analysis, the mean score delta for the experimental group would be 20.9%, yielding a difference between the experimental group and controls of 19.3% (95% CI = 13% to 25%).
It is interesting that orthopedic resident status was actually negatively associated with score delta. Three of the five orthopedics residents who were randomized to the experimental group were nonviewers. If these nonviewers were excluded, the mean score delta for the orthopedics residents in the experimental group would be 23.8%, compared to −3.6% for the controls (95% CI = −20% to 74%). Given the large magnitude of this difference, it is likely that the small number of orthopedic residents who actually viewed the NURRC lectures limited our ability to observe a statistically significant improvement in score delta.
The pretest/posttest design aimed to limit any inherent differences in baseline EM knowledge or multiple-choice test-taking skills among rotating residents. We did not measure baseline United States Medical Licensing Examination scores, previous EM clinical experience, or prior exposure to EM didactics. Nonetheless, the pretest scores were not statistically significantly different between the experimental and control groups, so we conclude that prior knowledge of EM topics produced only a minor effect at best.
A noted advantage of the NURRC over standard in-person lectures is that it requires minimal faculty time. The six NURRC lectures averaged 25 minutes in length, for a total of 2.5 hours of viewing time per 4-week academic block, none of which requires active faculty time. Before the NURRC was designed in an online format, six hour-long, in-person lectures were given during each academic block by one of the study investigators. Each online lecture required roughly the same amount of time to create as an in-person lecture. Although the effectiveness of the online format versus in-person format was not tested in this study, implementation of the NURRC saved a total of 78 faculty hours per academic year. Use of an online format also obviates the need for rotating residents to be released from clinical duties at a specific time and complements the EM Residency Review Committee’s recommendations on asynchronous didactic education. Furthermore, its online format makes the NURRC completely portable to any EM residency program that wants to offer didactic education to its rotating residents. Only 10% of EM programs without didactics for rotating residents would be averse to using a ready-made online curriculum if available.3
The response rate for the optional NURRC satisfaction survey was almost 80%. Overall, there was a markedly positive response to the NURRC. All lecture topics received a > 80% approval rating, and the free-text responses collected allowed for learner-based feedback to improve the curricular content. Over 80% of respondents felt that didactic education during their EM rotation was essential for their education. A similar proportion of EM residency leaders agree, but only 5% of academic EDs offer a targeted curriculum to its rotating residents.3 Only 7% of respondents would prefer attending the standard EM residency core conferences to the NURRC, a more resident-specific curriculum. Recent survey data show that 58% of rotating residents in academic EDs currently attend EM resident conferences.3
Cardiopulmonary emergencies and resuscitation were the two most commonly reported didactic subjects that rotating residents felt should be added to the NURRC. This fits with EM residency director opinions of these two subjects as important components of rotating resident didactic curricula.3 The NURRC was designed in conjunction with the Department of Internal Medicine at the study institution. Therefore, the selected topics were originally targeted to the learning needs of IM rotating residents. While the majority of the rotating residents in this study (81.5%) and around the country are IM, surgical rotating residents represent a sizable minority.3,4 It is difficult to predict didactic topics desired by rotating residents based solely on their respective medical specialties.12 A learner-centered approach,15 allowing the resident to pick from a selection of didactic subjects, may be an appropriate solution. Further studies should focus on eliciting the opinions of education experts in other specialties (e.g., ENT, orthopedics) regarding the optimal EM didactic program for their respective residents.
We aimed to evaluate whether a didactic program could improve knowledge acquisition as measured by scores on a multiple-choice test. We acknowledge that the goal of any curriculum is to achieve true competency among learners, only a facet of which is test-taking ability and medical knowledge. Further work may be done to determine whether greater didactic knowledge of EM topics among rotating residents results in improved competency during actual patient encounters in the ED.
It is possible that the overall mean score delta for the experimental group was elevated by the effect of a few participants who may have viewed the NURRC lectures frequently. We did not have the ability to measure how often the lectures were accessed. However, the majority of experimental group participants (18/29) demonstrated a significant (>10%) score delta, almost half (13/29) had more than 20%, and almost a quarter (7/29) more than 30%, although none more than 40%. This indicates a well-distributed improvement among experimental group participants and the absence of a small number of strongly positive outliers.
No study participant was blinded. The pretest was performed at the start of the rotation and contained questions pertaining to the same didactic material that the posttest did. Therefore, participants in either the experimental or the control group could have studied these lecture subjects independently of the NURRC after the pretest but before the posttest. Although no investigator (two residents, JBB and AWF, and two faculty, AZA and DMC) was blinded, the other EM residents (n = 46) and faculty (n = 25) at the study site were not informed of the study, lecture subjects, or test questions. Thus, the majority of interactions between the educational supervisors in the ED and the study participants likely occurred without confounding knowledge transmission. While there was no way to standardize clinical teaching in both groups, shift schedules for all participants were made based on routine scheduling requirements by a chief resident who was not involved in the study. Shift schedules for EM faculty were made by a faculty member who had no knowledge of the study or study participant shift schedule. Therefore, there is no reason to suspect that either group had any significant differences in clinical teaching or patient care experiences. We therefore feel that the educational experience of the control group was an accurate representation of learning by clinical practice and teaching alone.
Participants were not strictly observed during their entire test-taking session and could conceivably have accessed supplemental material during testing. We attempted to minimize this effect through multiple means: always having an investigator present for testing; instructing participants to refrain from using supplemental materials; and assuring them that their scores would be stored anonymously, not be used as part of their rotation evaluation, and not be reported to their respective residency directors. Additionally, even if supplemental materials were used during testing, there is no reason to suspect that this would occur with greater frequency on the posttest compared to the pretest or by the experimental group compared to the controls.
Low sample sizes could lead to skewed results in either direction. Indeed, orthopedics residents were the only group not to demonstrate an improvement in mean score delta (likely due to a combination of nonviewers and small sample size). However, despite only having 54 participants, the differences in the primary outcome between experimental and control groups still reached statistical and, we believe, educational significance. Further studies should include a larger sample size and multiple institutions to test whether these results may be generalized to the entire population of rotating residents in academic or community EDs.
The didactic subjects in our curriculum were planned in conjunction only with IM educators. We did not consult educators from the other departments of origin of our rotating residents (such as orthopedics or ENT). Future work with these surgical specialists could help tailor the didactic educational experience more appropriately for non-IM rotating residents.
Our postintervention satisfaction survey was not externally validated. To our knowledge, no preexisting, validated survey instrument exists to either assess learner satisfaction with rotating resident didactic curricula during EM rotations or assess the efficacy of didactic methods in this setting. Additionally, responses to the satisfaction survey were recorded anonymously. Therefore, we were unable to compare satisfaction data to score delta. Perhaps those participants who either had higher score deltas or higher absolute scores were more satisfied with the NURRC. Our survey data could, therefore, be confounded by test performance.
For practical reasons, we could not measure knowledge retention and decay. Contact is difficult to maintain with preliminary year and categorical PGY-3 IM residents, most of whom leave our institution after the academic year. Future work should be dedicated to determining whether knowledge gained from the NURRC can be retained over a longer period of time.
Rotating residents who received access to an online didactic curriculum during their EM rotation did better on tests measuring knowledge of specific subjects. The large majority of participants felt that the curriculum enhanced their education and would recommend it to others. Further studies, possibly incorporating simulation or actual patient encounters, should be performed to determine whether this improved didactic knowledge results in better patient care outcomes.
We thank Aashish Didwania, MD, for his expertise in helping to select appropriate topics for our sample didactic curriculum. We also thank the Emergency Medicine Foundation for its generous support of this study.