Hybrid Simulation Combining a High Fidelity Scenario with a Pelvic Ultrasound Task Trainer Enhances the Training and Evaluation of Endovaginal Ultrasound Skills

Abstract

Objectives:  In this study, an endovaginal ultrasound (US) task trainer was combined with a high-fidelity US mannequin to create a hybrid simulation model. In a scenario depicting a patient with ectopic pregnancy and hemorrhagic shock, this model was compared with a standard high-fidelity simulation during training sessions with emergency medicine (EM) residents. The authors hypothesized that use of the hybrid model would increase both the residents’ self-reported educational experience and the faculty’s self-reported ability to evaluate the residents’ skills.

Methods:  A total of 45 EM residents at two institutions were randomized into two groups. Each group was assigned to one of two formats involving an ectopic pregnancy scenario. One format incorporated the new hybrid model, in which residents had to manipulate an endovaginal US probe in a task trainer; the other used the standard high-fidelity simulation mannequin together with static photo images. After finishing the scenario, residents self-rated their overall learning experience and how well the scenario evaluated their ability to interpret endovaginal US images. Faculty members reviewed video recordings of the other institution’s residents and rated their own ability to evaluate residents’ skills in interpreting endovaginal US images and diagnosing and managing the case scenario. Visual analog scales (VAS) were used for the self-ratings.

Results:  Compared to the residents assigned to the standard simulation scenario, residents assigned to the hybrid model reported an increase in their overall educational experience (Δ VAS = 10, 95% confidence interval [CI] = 4 to 18) and felt the hybrid model was a better measure of their ability to interpret endovaginal US images (Δ VAS = 17, 95% CI = 7 to 28). Faculty members found the hybrid model to be better than the standard simulation for evaluating residents’ skills in interpreting endovaginal US images (Δ VAS = 13, 95% CI = 6 to 20) and diagnosing and managing the case (Δ VAS = 10, 95% CI = 2 to 18). Time to reach a diagnosis was similar in both groups (p = 0.053).

Conclusions:  Use of a hybrid simulation model combining a high-fidelity simulation with an endovaginal US task trainer improved residents’ educational experience and improved faculty’s ability to evaluate residents’ endovaginal US and clinical skills. This novel hybrid tool should be considered for future education and evaluation of EM residents.

Endovaginal ultrasound (US) is considered the test of choice for the evaluation of abdominal pain or bleeding in first-trimester pregnancy patients.¹ The use of endovaginal US by emergency physicians (EPs) has demonstrated potential to decrease emergency department (ED) length of stay and to facilitate rapid assessment of patients with ruptured ectopic pregnancy.^2,3 During the past 10 years, acquiring and interpreting US images has become an accepted part of the training of emergency medicine (EM) residents. The American College of Emergency Physicians, the Society for Academic Emergency Medicine (SAEM), the Council of Emergency Medicine Residency Directors (CORD-EM), and the American Medical Association have all endorsed the training of EM residents in US skills.⁴

High-fidelity simulation is gaining acceptance as a training modality throughout the ranks of EM education.^5–7 This form of simulation utilizes sophisticated computer-driven electronic and pneumatic mannequins to provide realistic patients that breathe, respond to drugs, talk, and have vital sign outputs displayed on a monitor screen.⁶ The military, airline, and nuclear power industries have all seen reductions in human factor errors resulting from the use of simulation training.⁸ Participants at the Academic Emergency Medicine consensus conference on the science of simulation have suggested that high-fidelity simulation should be considered as a potential method for training and evaluating EM residents in life-saving, invasive diagnostic procedures without causing discomfort or risk to a patient.⁹

The use of patient simulation in residency training programs can decrease dependence on learning during real-time clinical care.^10–12 Use of multiple simulation modalities, such as high-fidelity mannequins, standardized patients, and task trainers, may be able to more fully represent real clinical care. Hybrid simulation is the combination of more than one simulation modality for a single teaching or evaluation exercise.

In this study, we combined a high-fidelity US mannequin with a recently developed pelvic US task trainer to create a hybrid model for training residents in the care of patients with ectopic pregnancy and hemorrhagic shock. At two institutions we compared residents training with this hybrid model with residents training with a standard high-fidelity simulation scenario. We hypothesized that residents would assess the hybrid model to be superior to the standard scenario in terms of their learning experience and the capability of the scenario to measure their ability to interpret endovaginal sonographic images. We also hypothesized that the hybrid model would increase the self-rated ability of faculty members to assess EM residents’ competence in the interpretation of endovaginal US images and in the care of patients with ectopic pregnancy and hemorrhagic shock.

Methods

Study Design

This was a prospective, randomized, cohort trial using EM residents and faculty at two institutions. The institutional review boards at both institutions approved this study.

Study Setting and Population

The study was conducted at two U.S. Level 1 trauma and tertiary care medical centers: Advocate Christ Medical Center (ACMC) and Georgetown University/Washington Hospital Center (GU/WHC). Both institutions support busy EDs that provide care to predominantly urban populations. Both institutions follow the guidelines for EM residency training programs approved by the Accreditation Council for Graduate Medical Education (ACGME).

All 45 EM residents at both institutions (33 at ACMC and 12 at GU/WHC) participated in the study as part of their regularly scheduled training. All residents had previously received instruction in the use of high-fidelity mannequins and had been exposed to multiple simulated patient scenarios as part of their curricula. Faculty members at both institutions (six at ACMC and three at GU/WHC) participated in the study. One other faculty member at each institution who was not involved in the study scheduled the residents for the training sessions.

Study Protocol

At each institution we prepared two formats of the same scenario: a patient with ectopic pregnancy and hemorrhagic shock. A faculty member (one at each institution) randomly divided the residents into two groups by scheduling them for a regular training session in one of the two rooms in which the scenarios were to be presented. The faculty member did not know the purpose or design of the study, did not participate in other areas of the study, and knew only that the residents were to be distributed equally with respect to their level of training.

As each resident participated in a scenario, his or her performance was video-recorded in digital format. Faculty members reviewed the videos of the other institution’s residents. These faculty members were blinded to the names of the residents they evaluated, their level of training, or any previously demonstrated clinical capabilities. During this study, the video recordings could be used by the participating residency programs for teaching or evaluation purposes, but not as criteria for advancement.

Each resident in either scenario was the lone physician who had to manage the case and identify the ectopic pregnancy sonographically. All residents agreed not to disclose the scenario to any other participant. Because participation in the study was for educational purposes only, there was little incentive not to abide by this agreement.

Scenario Development

A high-fidelity simulated scenario of a patient with ectopic pregnancy and hemorrhagic shock was initially developed by a board-certified EM faculty physician (DG). This scenario included scripted cues with regard to patient complaint, vital signs, physical exam findings, nurse statements, and response to therapy. The scenario was then piloted with two other board-certified EM faculty physicians (LC, RH), who refined the cues.

Before the study was started, faculty members from both institutions reviewed written, scripted versions of both scenarios along with video demonstrations to ensure standard portrayals to all participants. Faculty met and discussed the study design and made such changes as were necessary to reach consensus on all cues. Study investigators (DG, MA) also took part in e-mail and phone discussions with faculty, both before and during the study, on an as-needed basis. The main topic of these communications was maintenance of standardized prompts within the scenarios at both institutions.

The high-fidelity simulation mannequin (SimMan, Laerdal, Wappingers Falls, NY) that we used in both simulation formats has interchangeable male and female genitalia and can have a wig or false breasts applied. For our study, the mannequins simulated a female patient. Faculty members developing our scenario designed the mannequins’ physiologic parameters, namely, standardized initial vital signs, vital sign trends, and responses to therapeutic interventions, which they preprogrammed into the computer using the scenario-editing software. The software ran the simulations in exactly the same manner for each resident.

For the hybrid arm of the study, a pelvic US task trainer was placed between the legs of the high-fidelity mannequin to create a hybrid simulation model, which we covered with a sheet (Figure 1). The pelvic US task trainer (Blue Phantom, Kirkland, WA) is a molded rubber simulator in the shape of a human female pelvis. It is anatomically correct externally, and an endovaginal US probe can be placed into the vaginal orifice. Inside the pelvic task trainer are structures that can simulate the ultrasonographic appearance of either normal or abnormal pelvic findings. When residents in this arm of the study reached the point of making a diagnosis with endovaginal US, they were required to manually manipulate an US probe in the pelvic US task trainer and thereby generate sonographic images of simulated pelvic findings. From multiple possible images the residents had to identify the ectopic pregnancy and freeze and print the image.

Figure 1.  Hybrid combination of a high fidelity mannequin with an endovaginal ultrasound (US) task trainer.

For the control arm of the study, faculty members generated four static images from a pelvic US task trainer and printed hard copies for the scenario. One of these images was a diagnostic image of an ectopic pregnancy. When residents in this arm of the study reached the point of making a diagnosis with endovaginal US, they were required to identify the ectopic pregnancy from the four static pelvic US images handed to them.

Role Playing

Faculty members portrayed scenario characters in both arms of the study, practicing their presentations with each other before the study began. They were already experienced in using case simulations for oral exams and in representing simulated patients, ED personnel, consultants, and patients’ family members in a neutral manner.

Methods of Measurement

Data were collected in an anonymous fashion as part of the regularly scheduled EM resident training. After finishing a scenario, each resident underwent a period of debriefing and rated his or her own performance by completing a standardized data collection form. This form included two visual analog scales (VAS) for assessing residents’ responses to the questions: 1) Please rate the learning experience of this case (none to outstanding) and 2) Please rate how well this scenario tested your ability to interpret a diagnostic pelvic US image (not at all to extremely well).

Faculty members reviewing the video recordings of the residents’ performance also completed a standardized data collection form. This form included a VAS on which faculty self-rated their ability to evaluate 1) the residents’ skills in interpreting endovaginal US images and 2) the residents’ overall skills in diagnosing and managing the case. The data collection form also included the American Board of Emergency Medicine oral board grading scale to guide faculty with their evaluation of resident clinical skills.

Outcome Measures

Our primary outcome was the residents’ self-reported educational experience. For secondary outcomes we included 1) the residents’ self-reported rating of how well the hybrid or control simulation scenario measured their ability to interpret endovaginal sonographic images, 2) the faculty’s self-reported ability to evaluate residents’ competence in interpreting endovaginal US images, and 3) the faculty’s self-reported ability to evaluate residents’ competence in the overall care of patients with ectopic pregnancy and hemorrhagic shock.

Data Analysis

Data from the collection instruments were entered into an Excel spreadsheet (Microsoft Corp., Redmond, WA) and analyzed with SPSS Version 15 (SPSS Inc., Chicago, IL). Data are reported as means, with 95% confidence intervals (CI) for the differences between means.

Results

A total of 24 residents were assigned to the hybrid arm of the study, while 21 were assigned to the control arm. All 33 residents at ACMC completed their assignment in 1 day. The 12 residents at GU/WHC completed their assignment in blocks of 4 over a 2-week period.

Primary Outcome

Residents in the hybrid arm of the study rated their overall educational experience more highly than residents in the control arm (Δ VAS 10, 95% CI = 4 to 18; Figure 2).

Figure 2.  Visual analog scale (VAS) rating of residents’ overall educational experience. Error bars show 95% CIs. US = ultrasound.

Secondary Outcomes

Measure 1. In measuring their ability to interpret the endovaginal US images, residents gave higher ratings to the hybrid model than to the control scenario (Δ VAS 17, 95% CI = 7 to 28; Figure 3).

Figure 3.  Visual analog scale (VAS) rating of resident assessment of how well the scenario evaluated residents’ ability to interpret endovaginal ultrasound (US) Images. Error bars represent 95% CIs.

Measure 2. Faculty reviewers found the hybrid simulation to be better than the control simulation for evaluating residents’ skills in interpreting endovaginal US images (Δ VAS 13, 95% CI = 6 to 20; Figure 4).

Figure 4.  Visual analog scale (VAS) rating of faculty ability to evaluate resident skill interpreting endovaginal ultrasound (US) images. Error bars represent 95% CIs.

Measure 3. Faculty reviewers gave the hybrid model a higher rating than the control simulation for evaluating residents’ overall clinical care (Δ VAS 10, 95% CI = 2 to 18; Figure 5).

Figure 5.  Visual analog scale (VAS) rating of faculty ability to evaluate resident competence in the care of patients with ectopic pregnancy and hemorrhagic shock. Error bars represent 95% CIs.

Time to Diagnosis

Although residents assigned to the hybrid model took longer (an average of 602 seconds) than residents assigned to the control simulation (an average of 514 seconds) to diagnose the ectopic pregnancy, the difference was not significant (p = 0.053).

Discussion

The use of high-fidelity simulation in the training of EPs is increasing because it improves resident learning and provides better evaluative methods in the core competencies of patient care, communication, and professionalism.^7,13–18 For example, studies in anesthesiology, a specialty that was quick to adopt the use of high-fidelity simulation,¹⁹ describe the benefits of using high-fidelity simulated scenarios for resident core competency assesssment.²⁰ In Israel, high-fidelity simulation is used for national accreditation of paramedics, medical school admissions, and board certification in anesthesiology.²¹

Both high-fidelity simulation mannequins and task trainers have the potential to improve resident training and evaluation in many areas of patient care.^6,22,23 Hybrid simulations, in which two or more simulation modalities are combined, have been used in cardiology, anesthesiology, and obstetrics-gynecology.^24–26 Noeller et al.²⁷ used a hybrid simulation model to train EM residents in cardiac, pediatric, toxicology, and trauma resuscitations. They used different simulation modalities at separate training stations within a theme-based workshop and found a high level of resident satisfaction. Other EM investigators have used the newly developed endovaginal task trainer alone for endovaginal US training.^28,29

The hybrid model we developed differs from those in previous studies in that we combined a high-fidelity US mannequin and a new endovaginal US task trainer within a single scenario for the training and evaluation of residents in endovaginal US.³⁰ Task trainers alone are not optimal for simulating a clinical scenario that includes an undifferentiated patient presenting to an ED with critical vital sign alterations. However, simulating potentially life-saving procedures, such as central venous catheter placement and endovaginal US, are not possible with the current high-fidelity simulation mannequins and require the use of a separate task trainer to simulate the clinical action. Hybrid simulation with a high-fidelity simulation mannequin and a task trainer can synergize the advantages of both modalities.

Our study shows that a hybrid simulation approach can add fidelity to the overall simulation scenario. Residents were able to perform a pelvic US examination, interpret the US images, and incorporate their findings into the overall clinical picture, while faculty members were able to evaluate the residents in all these aspects.

Future research in this area could focus on the application of hybrid simulation techniques to training residents in other forms of resuscitation, such as shock requiring a central line, cardiac tamponade, or patients with meningitis. Refining and validating such tools for the training and evaluation of EM residents would be valuable. There is a need to demonstrate that hybrid simulation improves resident performance or decreases the number of patient care errors in real clinical situations compared to standard simulation formats or traditional didactic and bedside teaching.

Limitations

Although the pelvic US task trainer we used in our study simulates the feel and imaging characteristics of an actual endovaginal US exam, it does not incorporate such complications as bowel content and gas that often obscure the ovaries in a live patient. Likewise, the technical difficulties of encountering such problems as a full bladder or adjusting a poorly positioned probe are not fully replicated with this simulator.

At both institutions that participated in this study, the EM residents receive a structured US curriculum that includes both didactic and bedside clinical teaching and covers the entire spectrum of emergency ultrasonography. Residents learn to generate and interpret static as well as dynamic US images, but most of the clinical training at both institutions involves dynamic images. This might have made it more likely that the residents in the hybrid arm of this study would perform better than residents in the control arm, as the latter only viewed static images.

We collected detailed, action-specific (and timed) data. However, some data points are inherently subjective, which is part of the nature of resident evaluation in the core competency of patient care. Residency training programs do their best to minimize the subjective nature of patient care competency evaluations by gathering data from multiple sources and standardizing the evaluation criteria in structured clinical exams and high-fidelity simulation scenarios. The CORD-EM Consensus Group has called for explicit criteria and objective measurements for evaluating patient care competency.³¹ We gathered VAS scores of resident and faculty perceptions of the simulation scenarios and attempted to increase the objectivity of these measurements and thus attenuate one source of subjective bias, by gathering data at two institutions. This allowed us to have faculty from each institution evaluate the residents from the other institution. Faculty reviewers were therefore blinded to each resident’s year of training and were unlikely to have preconceived notions of residents’ skills.

We chose VAS as a measurement tool in this study on the basis of data from other studies evaluating new formats of simulation, in which VAS functioned well.^32–34 The optimal measurement tool for assessment of participant experience in simulation has not yet been identified.³⁵ The VAS appears to be a reasonable measure of the subjective components of simulation training at the present time.

In studies of task-oriented subjects, such as the EM residents in our study, some bias in favor of the more novel, active investigational arm is expected. In our study this bias would have tended to magnify our finding that both residents and faculty preferred the hybrid model, which included the endovaginal task trainer. However, prior theoretical evidence that such a hybrid simulation would serve both residents and faculty better than the standard high-fidelity simulation comes from an evaluation hierarchy described by Miller³⁶ in which “showing how” ranks higher than “knowing how.” In our study, instead of being able to demonstrate only that they knew the correct diagnostic image, residents using the hybrid model were able to demonstrate their knowledge by showing how to generate and identify the diagnostic image. Furthermore, the more closely an assessment context resembles actual clinical practice, the greater the likelihood that the assessment will predict performance as a practicing physician.³⁷

Conclusions

A hybrid simulation model that combines a standard high-fidelity simulation scenario with an endovaginal ultrasound task trainer improved the educational experience of emergency medicine residents and the ability of faculty to evaluate residents’ endovaginal ultrasound and clinical skills. This novel combination of two simulation modalities to create a hybrid tool should be considered for future education and evaluation of emergency medicine residents.

Acknowledgments

We would like to acknowledge the other Simulation/Ultrasound Research Group members that supported this study: Alexis Battista, MBA, Rahul Bhat, MD, and Sangeeta Wood, MD (Department of Emergency Medicine, Georgetown University/Washington Hospital Center) and Steve Delis, MD, Erik B. Kulstad, MD, MS, Jaime Thompson, MD, and Martha Villalba, MD (Department of Emergency Medicine, Advocate Christ Medical Center).