Evaluation of a Web-based Asynchronous Pediatric Emergency Medicine Learning Tool for Residents and Medical Students

Authors

  • Kreg Burnette MD,

    1. From the Department of Pediatric Emergency Medicine, University Medical Center, St Rose Dominican Hospital – Siena, (KB) Las Vegas, NV; Department of Pediatric Emergency Medicine, Akron Children’s Hospital Medical Center, Akron, OH, and Departments of Pediatrics and Emergency Medicine, Northeastern Ohio Universities College of Medicine and Pharmacy, (MR) Akron, OH; Department of Pediatrics, University of Louisville, (MS) Louisville, KY; Department of Emergency Medicine, Northeastern Ohio Universities College of Medicine and Pharmacy, Summa Health System, (MSB) Akron, OH.
    Search for more papers by this author
  • Maria Ramundo MD,

    1. From the Department of Pediatric Emergency Medicine, University Medical Center, St Rose Dominican Hospital – Siena, (KB) Las Vegas, NV; Department of Pediatric Emergency Medicine, Akron Children’s Hospital Medical Center, Akron, OH, and Departments of Pediatrics and Emergency Medicine, Northeastern Ohio Universities College of Medicine and Pharmacy, (MR) Akron, OH; Department of Pediatrics, University of Louisville, (MS) Louisville, KY; Department of Emergency Medicine, Northeastern Ohio Universities College of Medicine and Pharmacy, Summa Health System, (MSB) Akron, OH.
    Search for more papers by this author
  • Michelle Stevenson MD, MS,

    1. From the Department of Pediatric Emergency Medicine, University Medical Center, St Rose Dominican Hospital – Siena, (KB) Las Vegas, NV; Department of Pediatric Emergency Medicine, Akron Children’s Hospital Medical Center, Akron, OH, and Departments of Pediatrics and Emergency Medicine, Northeastern Ohio Universities College of Medicine and Pharmacy, (MR) Akron, OH; Department of Pediatrics, University of Louisville, (MS) Louisville, KY; Department of Emergency Medicine, Northeastern Ohio Universities College of Medicine and Pharmacy, Summa Health System, (MSB) Akron, OH.
    Search for more papers by this author
  • Michael S. Beeson MD, MBA

    1. From the Department of Pediatric Emergency Medicine, University Medical Center, St Rose Dominican Hospital – Siena, (KB) Las Vegas, NV; Department of Pediatric Emergency Medicine, Akron Children’s Hospital Medical Center, Akron, OH, and Departments of Pediatrics and Emergency Medicine, Northeastern Ohio Universities College of Medicine and Pharmacy, (MR) Akron, OH; Department of Pediatrics, University of Louisville, (MS) Louisville, KY; Department of Emergency Medicine, Northeastern Ohio Universities College of Medicine and Pharmacy, Summa Health System, (MSB) Akron, OH.
    Search for more papers by this author

  • Presented at the American College of Emergency Physicians Research Forum, Chicago, IL, October 27, 2008.

  • CoI: The author states that there are no conflicts of interest.

Address for correspondence and reprints: Michael S. Beeson, MD, MBA, e-mail: beesonM@summa-health.org.

Abstract

Objectives:  To examine the effectiveness of an asynchronous learning tool consisting of web-based lectures for trainees covering major topics pertinent to pediatric emergency medicine (PEM) and to assess resident and student evaluation of this mode of education.

Methods:  PEM faculty and fellows created a 21-lecture, web-based curriculum. These 20-minute online lectures used Microsoft PowerPoint with the voice-over feature. A 75-question test was created to assess the effectiveness of the web-based learning model, administered online before and after the rotation in the pediatric emergency department (PED). All fourth-year medical students and residents (across all specialties) rotating through the PED were required to complete 10 of the 21 lectures during their 1-month rotation. The main outcome variable was difference in score between pre- and post-rotation tests of participants who viewed no lectures and those who viewed at least one lecture. Evaluation of the program was assessed by anonymous survey using 5-point discrete visual analog scales. Responses of 4 or 5 were considered positive for analysis.

Results:  One hundred eleven residents and fourth-year medical students participated in the program. An initial 32 completed testing before implementation of the on-line lectures (March 2007–August 2007), and another five did not complete the on-line lectures after implementation (September 2007–February 2008). Seventy-one completed testing and on-line lectures, and all but three completed at least 10 on-line lectures during their rotation. Fourteen of 111 trainees did not complete the pre- or post-test (including two who viewed the lectures). The mean change in score was a 1% improvement from pre-test to post-test for trainees who viewed no lectures and a 6.2% improvement for those who viewed the lectures (mean difference = 5.2%, 95% confidence interval = 2.5% to 7.9%). In the linear regression model, the estimate of the coefficient was 0.43 (p < 0.001), meaning that, for each lecture viewed, post-test score rose by 0.43%. Sixty-nine of 75 test items (92%) had a point biserial correlation greater than 0.15. Thirty of the 72 trainees who completed the online lectures and testing (42%) returned surveys. All were comfortable using the Internet, and 87% (26/30) found the web-site easy to use. All felt that their educational goals were met, and 100% felt that the format would be useful in other areas of education.

Conclusions:  Although not a replacement for traditional bedside teaching, the use of web-based lectures as an asynchronous learning tool has a positive effect on medical knowledge test scores. Trainees were able to view online lectures on their own schedules, in the location of their choice. This is helpful in a field with shift work, in which trainees rarely work together, making it difficult to synchronously provide lectures to all trainees.

Introduction

The pediatric emergency department (PED) is a busy environment because of the high volume and acuity of pediatric patients who present for health care. This provides educational opportunities for trainees to learn how to manage children who are acutely ill or injured.1 Although rich in clinical opportunities for learning, this often chaotic and hectic environment can make it difficult to provide a uniform education to ensure that trainees have their educational needs met, have their performance properly assessed, and receive effective feedback regarding their performance.

Because of the different scheduling needs that residents and medical students (trainees) have when rotating in the PED, it is difficult to provide uniform formal educational programs to ensure that every trainee is provided with the learning opportunities necessary to competently care for children in the PED. Trainees work with many different faculty members because of the nature of shift work in the PED, so it is often difficult to consistently assess and evaluate them. Pediatrics, emergency medicine (EM), and family medicine residency programs require that trainees demonstrate competency, depending on their level of training, in the area of pediatric emergency medicine (PEM). Unless quantitative assessment tools are used, the information regarding the education that trainees receive in the area of PEM remains subjective at best.

An objective, quantitative, web-based tool was developed to educate and assess trainee competency in the area of medical knowledge in PEM. Trainees were given access to a web-site that consisted of lectures, a pre-test, and a post-test. The objective of this study was to assess the effectiveness of this asynchronous learning tool.

Methods

Study Design

This was a prospectively designed before-and-after study to assess improvement in medical knowledge of trainees before and after the implementation of required online lectures and pre- and post-testing. Akron Children’s Hospital’s Institutional Review Board reviewed this study, which was considered exempt.

Study Setting and Population

The implementation of the web-based lectures and testing began at the Akron Children’s Hospital ED in March, 2007. All rotating residents and medical students completing a 1-month rotation were eligible. This included pediatrics, EM, family medicine, internal medicine and pediatrics, and transitional residents, along with fourth-year medical students. A control group was established in the 6-month period before implementation of the online lectures. This group participated in pre- and post-rotation testing but did not view any online lectures. The intervention group began in September 2007, when the online lectures were made available, and continued for 6 months (through February 2008).

Study Protocol

Before the development of this project, a literature review was undertaken to locate work done in the area of web-based medical education and, more specifically, web-based education in the PED. A PubMed search was completed limiting the search with the terms [emergency medicine x education] and [emergency medicine x Internet]. Sixteen resources were found. Abstracts from all 16 were reviewed, and 13 were found to be relevant. Only one of these articles was related to PEM.2 The others were in general EM or related fields. Upon completion of the literature search, a needs assessment and an educational curriculum were created to define the goals and objectives of the project.3–6 The curriculum’s major goals were to:

  • 1Obtain a baseline level of knowledge trainees have in the area of PEM by using a web-based pre-test before starting the PEM rotation.
  • 2Use the results from the pre-test to help trainees identify areas of knowledge deficit so they could self-direct learning during the rotation.
  • 3Provide a series of web-based lectures that would allow trainees to review the material at a convenient time and location that best met individual trainee’s learning needs and schedule.
  • 4Upon completion of the PEM rotation, have trainees complete the web-based post-test of the material covered in the web-based lectures series.
  • 5Provide quantitative feedback about trainees’ ability to meet the competency of medical knowledge upon completion of the PEM rotation.
  • 6Provide feedback to the educators who developed the curriculum regarding its effectiveness in meeting trainees’ learning needs.

PEM faculty and fellows created a password-protected, 21-lecture, web-based curriculum (Table 1). The topics chosen were determined based on topics related to PEM that the general pediatric residency at Akron Children’s Hospital deemed essential to cover via formal lectures, as well as the most common diagnoses seen in the PED at Akron Children’s Hospital in 2005. These 20- to 40-minute lectures used Microsoft PowerPoint (Microsoft Corp., Redmond, WA) with the voice-over feature to standardize learning and were posted to a web-site created for this project (http://www.pedsemtests.com). The PEM core faculty reviewed all lectures for content before posting. A 75-question test was created to assess the effectiveness of the web-based learning model. The 75 questions were based on content from 18 of the 21 lectures, although not equally weighted.

Table 1. 
On-line Lecture Topics
  1. * No questions on the test covered these topics.

Pediatric trauma
Febrile infant
Pediatric respiratory emergencies
Pediatric surgical emergencies
Child abuse
Pediatric closed head injuries
Pediatric seizures
Altered mental status
Pediatric psychiatric emergencies
Pediatric resuscitation
Principles of wound care and dental trauma
Sickle cell emergencies
Pediatric urological emergencies
Pediatric orthopedic emergencies
Toxicology I
Toxicology II
Toxicology III
Toxicology IV
Injury prevention*
Burns and extremes of temperature*
Environmental emergencies*

Participating trainees were expected to have completed the online pre-test before the first shift of their 1-month rotation. Trainees were informed of this requirement using e-mail and through the online orientation. During the 1-month rotation, they were expected to view at least 10 of the 21 web-based lectures of their choosing. Trainees were aware that the web-site tracked the viewing of the lectures. Trainees were required to complete the online post-test and a post-rotation questionnaire during the last 3 days of the rotation or no later than 1 week after the rotation.

The main outcome variable was difference in score between pre- and post-rotation tests of participants who viewed no lectures (control group) and those who viewed at least one lecture (although 10 lectures were required). The same questions were administered for the pre-test and post-test. The order of the questions and the order of the potential answers within each question were scrambled each time a test was taken to decrease recall bias. The participants could not review the test and did not receive their scores until after completion of the post-test.

A post-test questionnaire was created to assess trainee satisfaction with web-based learning in general and this program specifically. The questionnaire used a 5-point discrete visual analog scale. Responses of 4 or 5 were considered positive for analysis. Data were collected on level of training, resident specialty, previous PEM rotations, and primary training site.

Table 2. 
Characteristics of Study Population
Characteristicn%
Sex
 Male6155
 Female5045
Program type
 Pediatric2422
 EM4641
 Family practice1413
 Transitional year1110
 4th-year medical student1614
Level of training
 4th-year medical student1614
 Post-graduate year 13733
 Post-graduate year 24238
 Post-graduate year 31614
Previous EM rotation4843

Data Analysis

A paired t-test was used to compare differences in scores between the two groups (those who did and did not view the lectures). Trainees were analyzed in the groups they were assigned to based on whether the online lectures were available. Linear regression was used to determine whether the number of lectures viewed was associated with improvement in test score. The point biserial correlation was calculated for question discrimination to internally validate the test. The point biserial is a correlation between performance on a specific item (correct vs incorrect) and overall test score. A cut-off value of 0.15 was arbitrarily chosen, being within the definition of small correlation by Cohen.7 All statistical analysis was performed using SAS Software (SAS Institute, Inc., Cary, NC).

Results

One hundred eleven residents and fourth-year medical students participated in the program. An initial 32 trainees completed testing before implementation of the on-line lectures (March 2007–August 2007), and another five trainees did not complete the on-line lectures after implementation (September 200 –February 2008). Seventy-one trainees completed testing and on-line lectures, and all but three of these completed at least 10 on-line lectures during their rotations. Fourteen of the 111 trainees did not complete the pre-test or post-test (including two who did view the lectures) (Figure 1).

Figure 1.

 Flowsheet of study participants.

The point biserial resulted in values ranging from –0.03 to 0.866 (range −1 to +1). Sixty-nine of 75 test items (92%) had a point biserial correlation greater than 0.15.

The mean change in pre- and post-test score was a 1% improvement for trainees who viewed no lectures and a 6.2% improvement for those who viewed lectures (mean difference 5.2%, 95% confidence interval = 2.5 to 7.9). In the linear regression model, the estimate of the coefficient was 0.43 (p < 0.001), meaning that, for each lecture viewed, the post-test score rose by 0.43%.

Thirty of 72 (42%) of the trainees who completed the online lectures and testing returned surveys. All of them were comfortable using the Internet, and 87% (26/30) found the site easy to use. Twenty-eight of 30 (93%) felt that the program would be helpful on in-training or board examinations. All felt that their educational goals were met and that the format would be useful in other areas of education.

Discussion

Web-based education and evaluation is a recent development in medicine. In reviewing the literature, there are many studies that have examined the use of web-based learning in medicine but few in EM and almost none in the PEM literature. To the best of our knowledge, this is the first study to examine the use of a web-based learning tool applied to trainees from multiple specialties and experiences.

Our study demonstrated a 5.2% difference in end-of-rotation test score between groups of trainees viewing online lectures (generally 10 or more) and those who did not have this option. Although a modest improvement, it suggests that the use of online lectures as an asynchronous learning tool will result in higher medical knowledge scores. These results may have been further improved if specific online lectures had been required of the trainees based on their pre-rotation test scores.

The results of the pre-test and post-test enable the trainee and the faculty to evaluate the trainee’s competence in the area of medical knowledge.8 The trainee and the program can use the results of these tests to structure additional self-learning experiences after completion of the rotation to correct any knowledge deficits. This curriculum also serves the PEM educators, because results from the post-test and trainee evaluations can be used to strengthen areas where the curriculum requires improvement.

Past work in the area of web-based education in the ED has examined the gradual shift of education from didactic delivery to problem-based learning and now to web-based virtual learning.9–11 In other directions, studies have focused on using web-based models for resident evaluation12 and, to take this one step further, to help identify residents needing early remediation.13 Studies have also looked at web-based education and testing to help teach specific ideas, such as guides for radiographic imaging in the ED,14 and to reduce medication errors, with the latter type being the only study found in the PEM literature using web-based education in the ED.2 That study used a web-based education program to teach proper use of the Broselow tape to improve dosing accuracy and decrease dosing time. A few studies have examined using an online curriculum to replace didactic lectures, such as one study that used case-based online interactive teaching programs to teach third-year medical students different case types,15 and as an adjunct curriculum to rotating medical students in the general ED.16

Our study fits into the current literature by taking web-based education one step further, by incorporating web-based education as the primary didactic educational tool of not only rotating medical students, but also residents. It also incorporates web-based testing and assessment.

Reviewing the data from the anonymous questionnaires, there was a favorable response to the online testing and lectures from those who had used it before and those who had not. Overall, the residents and students liked the ability to view the lectures on their own time and the ability to choose those lectures that they felt would benefit them the most, an important principle of adult learning.17,18

Future goals include making the pre-rotation test a needs-assessment mechanism, identifying specific topics of weakness. This will allow for targeting of online lectures to the trainee’s identified weaknesses, augmenting self-selection by the trainee. Based on point biserial scores for each question, future tests will be composed of questions with greater validity.

Limitations

There was no direct way to ensure that the lectures were being viewed in their entirety. We were able to track if they were opened, how many times, and at what time, but not for how long they were opened. There was no way to ensure that the trainees listened to the entire lecture. This is not significantly different, though, from a trainee not paying attention during a live lecture.

Another limitation is that the web-based lecture group was compared with a group receiving no lecture, so that a comparison could not be made between the two learning styles of traditional lectures and web-based. The two groups of trainees (online lectures and a control group of no online lectures) were not randomized. The control group was based on the date that the online lectures were made available to the trainees.

Although this study cannot be compared directly with traditional education, because the comparison groups received on-line lectures or no lectures, it shows a significant improvement in medical knowledge test scores in only 1 month with the web-based education. This should also be taken in the context that the majority of trainees viewed only half of the lectures available. Trainees were not told which lectures to watch. The requirement was that 10 or more lectures of the 21 total should be viewed (self-directed). This lack of control could have resulted in biased results. The medical knowledge score improvement could potentially be higher if all lectures had been viewed.

Limitations to the linear regression data are that most trainees viewed 10 lectures or none, and if calculated based only on those viewing lectures, it showed no improvement in score (p = 0.49). A linear regression would be more helpful if there were greater variation in the number of lectures viewed. We did not analyze question responses based on whether the lecture that each question came from was viewed. Hence, we cannot comment on any correlation between viewing a particular lecture and correctly answering questions based on that lecture.

Conclusions

Web-based education is an alternative to traditional teaching techniques in PEDs, where the trainees vary greatly in their year of training and background. It is a practical way to provide trainees who are working in a 24/7 environment with a uniform and consistent education. Our study demonstrated modest improvement in medical knowledge scores with the addition of online lectures.

Ancillary