Procedures Can Be Learned on the Web: A Randomized Study of Ultrasound-guided Vascular Access Training
This study was supported by a research grant from the Canadian Association of Emergency Physicians and a Fellowship for Studies in Education from the Royal College of Physicians and Surgeons of Canada.
Address for correspondence and reprints: Dr. Glen W. Bandiera; e-mail: firstname.lastname@example.org.
Objectives: Web-based learning has several potential advantages over lectures, such as anytime–anywhere access, rich multimedia, and nonlinear navigation. While known to be an effective method for learning facts, few studies have examined the effectiveness of Web-based formats for learning procedural skills. The authors sought to determine whether a Web-based tutorial is at least as effective as a didactic lecture for learning ultrasound-guided vascular access (UGVA).
Methods: Participating staff emergency physicians (EPs) and junior emergency medicine (EM) residents with no UGVA experience completed a precourse test and were randomized to either a Web-based or a didactic group. The Web-based group was instructed to use an online tutorial and the didactic group attended a lecture. Participants then practiced on simulators and live models without any further instruction. Following a rest period, participants completed a four-station objective structured clinical examination (OSCE), a written examination, and a postcourse questionnaire. Examination results were compared using a noninferiority data analysis with a 10% margin of difference.
Results: Twenty-one residents and EPs participated in the study. There were no significant differences in mean OSCE scores (absolute difference = −2.8%; 95% confidence interval [CI] = −9.3% to 3.8%) or written test scores (absolute difference = −1.4%; 95% CI = −7.8% to 5.0%) between the Web group and the didactic group. Both groups demonstrated similar improvements in written test scores (26.1% vs. 25.8%; p = 0.95). Ninety-one percent (10/11) of the Web group and 80% (8/10) of the didactic group participants found the teaching format to be effective (p = 0.59).
Conclusions: Our Web-based tutorial was at least as effective as a traditional didactic lecture for teaching the knowledge and skills essential for UGVA. Participants expressed high satisfaction with this teaching technology. Web-based teaching may be a useful alternative to didactic teaching for learning procedural skills.
The use of Web-based training in medical education has surged in popularity in recent years. Web-based educational resources are commonly used for undergraduate and postgraduate medical training, as well as for continuing medical education.1 The use of Web-based training has several potential advantages over didactic teaching, including the ability to access teaching materials at any time and almost any place, the inclusion of a wide variety of multimedia features, and the ability for interactive nonlinear navigation.2 Web-based education can be used to facilitate self-directed learning, leading to reduced time demands on clinical educators.3 The increasing use of Web-based teaching in medical education has been attributed to several factors, including improved design and quality of the online teaching resources and increased availability of high-speed Internet access.1
Despite its widespread adoption, there are few randomized controlled trials of Web-based education in medical education.4 A review of the literature concluded that Web-based learning was at least as effective as traditional teaching methods and was generally preferred by the users.4 However, most prior studies have only examined the use of Web-based education for facilitating knowledge gains, and there are few studies evaluating its effectiveness for teaching procedural skills. Unlike teaching new facts, procedural skill training involves a complex integration of procedural knowledge and psychomotor skill, which often includes live demonstrations by a clinical expert.5 With improvements in multimedia functions, such as streaming video and animation, Web-based learning has the potential to replace the need for live hands-on demonstration in the initial phases of teaching procedural skills.
Ultrasound-guided vascular access (UGVA) is a procedural skill that may be well suited for Web-based instruction. The use of ultrasound guidance has been shown to significantly reduce the complication rate and improve the success rate for the insertion of central lines.6–8 Several national organizations have recommended that ultrasound guidance should be the preferred method for all central line insertions.9,10 While many emergency physicians (EPs) have become comfortable with the use of bedside ultrasound for diagnostic purposes, its adoption for vascular access procedures remains low.11,12 A lack of available training resources may be contributing to the slow adoption of this procedural skill.12–14 Web-based education may be an ideal alternative for the large number of physicians without access to expert training. The purpose of this study was to determine whether a Web-based tutorial is at least as effective as a didactic lecture for teaching the knowledge and skills essential for UGVA to EPs and residents.
We conducted a randomized controlled noninferiority trial to evaluate the effectiveness of a Web-based tutorial compared with a didactic lecture for teaching UGVA. In consultation with several experts, we derived a set of learning objectives for teaching ultrasound guidance with central lines, peripheral intravenous lines, and arterial lines. We designed a training website according to accepted principles of high-quality Web-based educational material development.15,16 Active learning was promoted through the use of interactive features such as videos, animations, self-assessment quizzes, and a nonlinear navigation system. Once completed, the website was pilot tested by several experts and nonexperts in the area of UGVA, and modifications were made based on their feedback. This study was approved by the Research Ethics Board at the University of Toronto and all participants provided written informed consent.
Study Setting and Population
We invited all 13 junior emergency medicine (EM) residents and 124 staff EPs at the University of Toronto to participate in the study. To be eligible, participants must have completed an introductory ultrasound course, but must have had no previous training on UGVA. We selected the first 22 eligible participants who responded to the study invitation.
All participants completed a precourse written examination testing their knowledge of UGVA. The participants were subsequently randomized by a computer-generated random number sequence into either a Web-based learning group or a didactic group, stratified by certification level (resident or staff physician). Participants were blinded to the study hypothesis and were advised not to reveal their training method to other study participants. The Web-based group was provided access to the training website and instructed to spend approximately 1 hour reviewing the material. The didactic group attended a 1-hour classroom lecture that covered the same material as found on the website.
All participants subsequently spent 2 hours independently practicing their skills with no instructors present and without any further instruction. Participants used portable Sonosite MicroMaxx ultrasound machines with 10–5 MHz 38-mm linear array transducers (Sonosite Inc., Bothell, WA). Central line, arterial line, and peripheral intravenous line insertions were performed on neck and arm vascular phantoms (Blue Phantom, Kirkland, WA) using single-lumen central line kits (Arrow International Inc, Reading, PA) and 18-gauge angiocatheters (BD Inc., Franklin Lakes, NJ). In addition, participants practiced using ultrasound to identify anatomic features on live models.
Following the practice session, there was a 2-week rest period during which the Web group had access to the website and the didactic group had access to a study guide provided to them during the lecture. Following the rest period, both groups completed an objective structured clinical examination (OSCE) and a written test. All participants completed a questionnaire evaluating their satisfaction with the background training sessions.
The OSCE consisted of four 8-minute stations, with each station requiring the participant to complete a specific ultrasound-guided vascular procedure. The procedures included insertion of central lines, arterial lines, and peripheral intravenous lines using the same vascular phantoms and equipment as those used in the practice session. In addition, participants were required to identify landmarks on live models. The scoring template included a checklist of critical actions as well as an anchored global rating scale. The examiners for the OSCE were physicians with expertise in the technique and who were blinded to the participants’ training group. The OSCE was designed with input from the examiners and pilot tested prior to implementation.
The precourse and postcourse written examinations were identical and consisted of 20 short-answer questions. The written examinations were scored using an answer template by an examiner blinded to the intervention groups. Twenty percent of the examinations were double-scored by a second examiner, yielding an interrater reliability of 0.86. The satisfaction questionnaire addressed issues of accessibility and utility of the material rated on a 5-point Likert scale.
Study data were analyzed using SPSS 16.0 (SPSS Inc., Chicago, IL) using a noninferiority analysis. The expert examiners agreed by consensus that a minimum of 10% difference in the OSCE scores between groups would be clinically significant. This value is in keeping with prior studies.17 A sample size of 14 participants was required to detect a 10% difference in examination scores with a power of 80%. Differences in examination scores between groups and their two-sided 95% confidence intervals (CIs) were calculated using Student’s t-test. Using the Angoff method,18 we determined the passing score of the OSCE to be 70%. We compared the precourse and postcourse written exam scores using Student’s t-test, with statistical significance set at p = 0.05. Categorical qualitative data were analyzed using Fisher’s exact test. We used Google Analytics to track website usage.
All 124 staff EPs and 13 junior residents in the EM residency training program at the University of Toronto were invited to participate in the study. The first 22 respondents to the study invitation were included in the study (Figure 1). One participant subsequently withdrew, leaving 21 participants that completed the study (11 residents, 10 staff physicians). The baseline demographics were similar between the two groups (Table 1). The mean scores on the OSCE examination were 75.0% for the Web group and 77.8% for the didactic group (absolute difference = −2.8%; 95% CI = −9.3% to 3.8%; Table 2). The mean scores on the written examination were 78.8% for the Web group and 80.3% for the didactic group (absolute difference = −1.4%; 95% CI = −7.8% to 5.0%). Eighty-two percent (9/11) of the Web-based group achieved a passing score on the OSCE compared with 90% (9/10) of the didactic group (p = 1.0). Both the Web group and the didactic group demonstrated similar improvements in written test scores (26.1% vs. 25.8%; p = 0.95). The mean written examination scores for all study participants improved from 53.6% to 79.5% (p < 0.001).
Table 1. Baseline Characteristics of Study Participants
|Residents||5 (45)||4 (40)||1.0|
|Staff physicians||6 (55)||6 (60)||1.0|
|Gender (male)||8 (73)||5 (50)||0.39|
|Taken an introductory US course||11 (100)||10 (100)||1.0|
|Comfortable inserting central lines||4 (36)||4 (40)||1.0|
|Comfortable using bedside ultrasound||5 (45)||3 (30)||0.66|
|Internet access at home||11 (100)||10 (100)||1.0|
|Find internet resources useful||10 (91)||9 (90)||1.0|
|Precourse exam||52.8 (±13)||54.5 (±8)||0.72|
Table 2. Results of the OSCE and Written Examinations
|OSCE score||75.0 (±9.3)||77.8 (±3.6)||−2.8 (−9.3, 3.8)||0.39|
|Written examination||78.8 (±7.3)||80.3 (±6.6)||−1.4 (−7.8, 5.0)||0.65|
|Written score improvement||26.1 (±13.1)||25.8 (±8.5)||0.3 (−9.7, 9.2)||0.95|
Over the course of the study, participants in the Web-based group logged into the website an average of 3.4 times (range 1–7 times). Most participants (9/11, 82%) reported using the website for approximately 1 hour, with 1 participant (1/11, 9%) using the site for less than 30 minutes and 1 participant (1/11, 9%) using the site for more than 2 hours. The highest number of website visits occurred the day before the practice session (17 visits from 9 participants). All participants logged in using high-speed Internet access.
Data from the postcourse questionnaire revealed that there were no significant differences in the satisfaction level for the training format between the Web group and the didactic group (Table 3). The majority of participants found the overall training session to be useful, had enough time to practice, and felt ready to begin using their skills in clinical practice. All participants in the Web-based training group reported being able to access the website without difficulty, and all stated that they would use the website in the future to refresh their skills.
Table 3. Summary of Responses from the Postcourse Questionnaire
|Found the overall course to be useful||11 (100)||8 (80)||0.21|
|Found the teaching format (website, didactic lecture) to be effective||10 (91)||8 (80)||0.59|
|Had enough time to practice||9 (82)||9 (90)||1.0|
|Feel ready to begin using the skills in practice||9 (82)||8 (80)||1.0|
Our Web-based educational tutorial was at least as effective as a didactic lecture for teaching the knowledge and psychomotor skills essential for UGVA. Both study groups had similar scores on the practical OSCE examination and the written knowledge examination. The difference in examination scores between the two groups and their 95% CIs did not exceed the prespecified difference margin of 10%. Both groups exhibited significant improvements in knowledge of UGVA following the course and achieved high pass rates on the practical examination. Participants in the Web-based training group reported high satisfaction levels with the online tutorial and planned to use it again in the future.
The results of this study have several important implications. First, this study supports the use of Web-based education for learning complex technical procedures. Prior studies comparing Web-based education with didactic training primarily focused on the acquisition of new knowledge.4 However, to date there are few studies to demonstrate whether complex manual tasks can be efficiently learned via an online tutorial. In 1999, Carr et al.17 found that a Web-based tutorial was as effective as a didactic lecture for teaching epistaxis management to medical students. While their examination tested both psychomotor and cognitive skills, the psychomotor task in this study was limited in its complexity. Other studies have found computer-based video instruction to be effective for teaching suturing skills to medical students.19,20 However, these studies did not test remote instruction using a Web-based format.
Our study is unique as the psychomotor task being evaluated is complex, requiring a high degree of hand–eye coordination. Participants were required to guide their needle safely and efficiently into a simulated vessel while using a portable ultrasound device for real-time guidance. The Web-based learning group in this study was able to successfully learn the material by interacting with the website and subsequently practicing their skills without any instructor guidance. All participants used the website from a remote location on their own time prior to attending the practice session. Despite having no formal instructor-guided teaching, the Web-based group demonstrated a high level of proficiency on the practical examination. These findings provide new evidence to support the use of Web-based learning for teaching complex psychomotor tasks.
Another important implication of this study is that Web-based learning may facilitate the upgrading of skills among practicing physicians. Despite the evidence and practice guidelines supporting the use of UGVA, few EPs are using this technique in daily clinical practice.11,12 Part of the reason for the lack of adoption of this technique may be due to the lack of available qualified instructors and resources required to train large numbers of physicians.14 For this reason, the use of Web-based education may be well suited for teaching this relatively new procedural skill. Web-based learning provides access to high-quality education from remote locations and from regions where local expertise or equipment is not available. The results of this study demonstrate that Web-based learning compares favorably to expert didactic instruction and may be a feasible alternative to face-to-face direct instruction when expert instructors are not available.
In this study, we invited both junior EM residents and practicing EPs to participate. We felt that by including both groups of physicians, the results of our study may be more generalizable to the variety of learners that will be acquiring this skill in the coming years. Despite the lack of financial compensation and the requirement to attend two full afternoon sessions, we had a significant number of early staff responses to the call for registration for this study. This suggests that UGVA is a skill that many EPs want to learn, but do not currently have access to expert training.
We did not control for the inclusion of the practice session, which may have been a factor in learning this procedural skill. However, given the complexity of the procedural skill and the lack of instructor presence, the majority of the learning likely occurred prior to the practice session. The inclusion of a third noninstructed group would have allowed us to determine the effect of the practice session on the OSCE performance. Due to resource constraints, we were limited to a small sample size, which may limit the external validity of this study. However, our study was adequately powered to detect a 10% difference in scores between groups, which was determined a priori to be clinically significant. The use of parametric statistics may have been limited by the small sample size; however, our data appeared to be normally distributed. While the two groups were similar in terms of baseline skills and knowledge, there may have been differences between the groups that were not accounted for in this study. In addition, the small sample size did not allow for a comparison between residents and staff physicians.
Participants in the Web-based group were allowed access to the website to refresh their skills during the rest period. However, this simulates one of the benefits of using the website in actual practice. In addition, we provided participants in the didactic group with a comprehensive handout that could similarly be used to refresh their knowledge. Finally, we did not measure the interrater or intrarater reliability of our OSCE. However, it was designed with explicit scoring criteria, and the examiners completed a training session and practice OSCE to maximize reliability.
We demonstrated that a Web-based tutorial was at least as effective as a didactic lecture for teaching the knowledge and psychomotor skills essential for UGVA. Participants felt the Web-based tutorial was an appealing way to learn this important technical skill. Following the tutorial, most participants felt ready to begin implementing this skill as part of their regular clinical practice. This educational tool has the potential to reduce the time demands on clinical educators, which may facilitate training a large number of learners without compromising educational quality. It is possible that Web-based teaching tools will continue to have an expanding role for teaching technical skills as a part of distributed medical education.
The authors thank Kevin Thorpe, Muhammad Mamdani, and Drs Martin Horak, Anne-Marie Humniski, John Foote, and Martin Chapman for their assistance. The authors also thank all the residents and staff physicians who participated in this study.