SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. Introduction
  4. Theoretical framework
  5. Methods
  6. Results
  7. Training in critical care and crisis management skills
  8. Discussion
  9. Acknowledgments
  10. References

Medical Education 2012

Context  High-fidelity simulators have enjoyed increasing popularity despite costs that may approach six figures. This is justified on the basis that simulators have been shown to result in large learning gains that may transfer to actual patient care situations. However, most commonly, learning from a simulator is compared with learning in a ‘no-intervention’ control group. This fails to clarify the relationship between simulator fidelity and learning, and whether comparable gains might be achieved at substantially lower cost.

Objectives  This analysis was conducted to review studies that compare learning from high-fidelity simulation (HFS) with learning from low-fidelity simulation (LFS) based on measures of clinical performance.

Methods  Using a variety of search strategies, a total of 24 studies contrasting HFS and LFS and including some measure of performance were located. These studies referred to learning in three areas: auscultation skills; surgical techniques, and complex management skills such as cardiac resuscitation.

Results  Both HFS and LFS learning resulted in consistent improvements in performance in comparisons with no-intervention control groups. However, nearly all the studies showed no significant advantage of HFS over LFS, with average differences ranging from 1% to 2%.

Discussion  The factors influencing learning, and the reasons for this surprising finding, are discussed.


Introduction

  1. Top of page
  2. Abstract
  3. Introduction
  4. Theoretical framework
  5. Methods
  6. Results
  7. Training in critical care and crisis management skills
  8. Discussion
  9. Acknowledgments
  10. References

There is growing interest in the use of realistic computer-controlled ‘high-fidelity’ simulation (HFS) in medical education, both for learning and assessment. Simulations have a number of potential advantages over traditional ward- or clinic-based learning. As Teteris et al.1 state:

‘Simulated learning experiences not only allow learners to practise and err, without patients suffering adverse clinical consequences, they also offer more control over the learning experience… The arguments in favour of simulation are so persuasive that some have proposed that the absence of clinical training is unethical.’

Central to the use of simulations are several assumptions.

1 Instruction on simulators results in meaningful learning.

This issue was recently elegantly addressed in a large systematic review conducted by Cook et al.,2 comprising a total of 609 studies with 35 000 trainees. The results of this review are unequivocal. Learning on simulators showed large gains in performance; typically, effect sizes of around 1.0 are observed for knowledge, time management skills, process skills and product skills.

However, only studies comparing simulator training with no training were examined. The authors2 deliberately excluded studies in which training on a simulator was compared with training on another simulator or an alternative instructional method.

2 Skills acquired on the simulation can be applied to real patients (i.e. they can be ‘transferred’).

A number of studies, reviewed by Teteris et al.,1 demonstrate that students who train on simulators show better performance in real situations in a variety of domains, including laparoscopy,3 catheter insertion,4 advanced cardiac life support (ACLS)5 and auscultation.6,7 Some of these gains have been shown in highly authentic transfer tasks, such as actual response to ACLS events.5

However, many of these studies compare the outcomes of formal simulator sessions with those of no formal training. Thus, although transfer is an important outcome of simulator training, there is little evidence that the gains observed are uniquely attributable to the HFS experience and not simply to the time spent in learning.

3 The closer to the ‘real world’ (and, commensurately, the more expensive the simulation), the better the transfer to real life.

This assumption appears as almost a self-evident truth. As Schuwirth and van der Vleuten state8:

‘Authenticity should have high priority when programmes for… assessment… are designed… The situation in which a candidate’s competence will be assessed should resemble the situation in which the competence will actually have to be used.’

The rationale for the assumption, and indeed for many educational practices, such as authentic assessment, situated cognition and cognitive apprenticeship,9 is the finding that the closer the context of learning to the context of practice, the better the learning.10 However, this is not a uniform finding within psychology11 and a number of studies have failed to show an advantage for matching context of learning and test. Even if we accept the argument that context similarity is desirable, a more difficult question refers to precisely what aspects of the context should be the focus of attention. This is explored in the next section.

4 Authenticity – the resemblance of the simulation to an equivalent real-life scenario – is the critical determinant of transfer.

Proximity to ‘real life’ may not be unidimensional. As Maran and Glavin12 have suggested, fidelity can be assessed on two levels: ‘engineering fidelity’ or authenticity (does the simulation look realistic?), and ‘psychological fidelity’ (does the simulator contain the critical elements to accurately simulate the specific behaviours required to complete the task?).13 These may be independent. As one example, early laparoscopic simulators were very realistic in their ability to simulate the visual field of the laparoscope and its change with movement. However, to learn to suture laparoscopically, it is critical to ‘feel’ the needle against the tissue using what is referred to as ‘haptic’ feedback. Thus, these simulators had good engineering fidelity but poor psychological fidelity. The latter might be better accomplished by using a beef tongue inside a box, although this would clearly lack engineering fidelity.

5 More complex skills demand more complex simulators.

Even if we recognise that basic skills may be taught and learned on relatively simple static simulators without concern for authenticity, it appears a universal assumption that complex skills require complex simulations:

‘Education in basic procedural skills like suturing… can be delivered using simple task trainers, devices that mimic body parts or regions… Complex clinical events… require training on much more sophisticated medical simulators.’14

Although it seems reasonable that more advanced learners performing more complex tasks may require more sophisticated simulators, there is little evidence to support the assertion.

Complex simulators do not come without cost. The METI cardiovascular simulator (Medical Education Technologies, Inc., Sarasota, FL, USA) costs US$80 000; the Harvey® heart sound simulator (Laerdal Medical Corp., Wappinger Falls, NY, USA) is similarly expensive. The cost of HFS has educational consequences. Few schools will have the resources to invest in a Harvey® simulator. Yet even if such a simulator is available and encouraged, student access to the simulator may be restricted simply by the number of learners. By contrast, if each student is given a CD at minimal cost, there is no economic restriction on the amount of practice time a student may accrue. Differentiating between the advantages afforded by each method may indicate a trade-off between substantially greater practice with lower fidelity and much less practice at higher fidelity. The extent to which additional fidelity, at additional cost, leads to more effective learning is a critical factor in this equation. It is certainly not a universal finding; aviation psychology has shown a law of diminishing returns in investment in simulators:

‘One [misconception is] that greater financial investment in a simulator facilitates training on that simulator […] Relatively low level simulations that capture the salient characteristics of the task are far more versatile for measuring and are very effective for improving performance.’15

The same may be true in medical simulation.

In this review, we will investigate the extent to which greater fidelity leads to greater transfer by examining studies that contrast learning based on low-fidelity simulation (LFS) with that based on HFS. To do this, we will essentially adopt the criteria of the respective authors. Although fidelity refers most commonly to the domain of ‘engineering fidelity’ (as defined earlier), exceptions arise. Further, the HFS may not necessarily be a computer-controlled manikin; in the examination of surgical techniques it is more common to compare the use of live or dead tissue with that of rubber or plastic. What does remain generally true is that HFS is more realistic (authentic) and more expensive (often much more expensive) than LFS.

Theoretical framework

  1. Top of page
  2. Abstract
  3. Introduction
  4. Theoretical framework
  5. Methods
  6. Results
  7. Training in critical care and crisis management skills
  8. Discussion
  9. Acknowledgments
  10. References

This review is not intended as a systematic review article. Such reviews have already been published, and one,16 in particular, is a primary source for the literature we will review. However, our goal differs from that of earlier reviewers; we do not aim to establish whether or not HFS ‘works’. That has been well established. Instead, our review is directed at examining two issues.

  • 1
     The relationship between performance on HFS and performance on a structured control intervention, often an LFS but occasionally an instructional video. It is of limited value to prove that an educational intervention is superior to nothing or to ‘usual care’, which often amounts to the same thing. As our specific interest is the relationship between fidelity and performance, we will primarily examine evidence that contrasts two types of structured intervention.
  • 2
     The relationship between learning on a simulator and performance outcomes. Ideally, this implies an outcome measured on real patients, on standardised patients (SPs) or on some credible measure that clearly involves some degree of transfer from the simulation. Assessing performance on an HFS using an outcome assessed with the same HFS says nothing about transfer because specific experience with the simulator is likely to bias any outcome assessed with the simulator. Such results are potentially interesting, however, if they can show that an LFS results in performance as good as those observed on the HFS because the experiment is biased against this outcome.

By implication, then, we are not interested in studies that use self-reported confidence, improvement in skill or any other self-report measure. The literature on self-assessment is uniform in its condemnation of conclusions drawn from such data.17 Further, we do not intend to examine such issues as the use of feedback or decay of skills; such issues are peripheral to our central question and have been addressed elsewhere.16

Methods

  1. Top of page
  2. Abstract
  3. Introduction
  4. Theoretical framework
  5. Methods
  6. Results
  7. Training in critical care and crisis management skills
  8. Discussion
  9. Acknowledgments
  10. References

The present study is not intended as a systematic review. As we indicated, a systematic review of HFS was published in 200516 and updated in 2010,14 and we carefully examined these reviews to identify any articles that contrasted HFS with LFS. We have supplemented articles mentioned in these reviews with other studies from other review articles1 and other studies identified by ourselves.

The present review includes evidence from 18 studies. Of these, seven primarily surgical interventions did not appear in the two systematic reviews, mainly because they involved static simulations.

The review will examine evidence in three broad domains: (i) auscultation skills and use of heart sound simulators; (ii) basic motor skills, and (iii) complex crisis management skills.

Results

  1. Top of page
  2. Abstract
  3. Introduction
  4. Theoretical framework
  5. Methods
  6. Results
  7. Training in critical care and crisis management skills
  8. Discussion
  9. Acknowledgments
  10. References

Training in auscultation skills

Perhaps the most comprehensive evidence of the role of HFS in skill acquisition refers to Harvey®, a computer-controlled manikin that has been used for about 20 years. As described in one review article,18 Harvey®:

‘…provides a comprehensive cardiology curriculum by realistically simulating 27 cardiac conditions. The physical findings programmed in the simulator for each disease include blood pressure, bilateral jugular venous, carotid and peripheral arterial pulses, precordial impulses in six different areas… auscultatory events… that are synchronised with the pulses and vary with respiration.’

However, although Harvey® has impressive engineering fidelity and simulates 30 different heart sounds, it contains only one example of 25 of the sounds, two examples of three sounds (mitral regurgitation, aortic regurgitation, acute myocardial infarction) and three examples of mitral stenosis. Thus, the opportunity to hear a variety of examples of each condition is severely constrained.

The originators of Harvey® and others have conducted a number of studies6, 7, 19 to demonstrate the validity of Harvey® in learning auscultation skills. However, with only one exception, outcomes using Harvey®21 have been compared with those in a no-intervention control group that received the usual haphazard bedside teaching of heart sounds.19

Perhaps the most widely cited of these studies is that by Issenberg et al.,20 in which students in an internal medicine rotation were randomly assigned (by rotation) to learn auscultation skills either in formal training sessions using Harvey® or by the usual ward-based instruction, which was neither standardised nor documented. At the end of the rotation, students in both groups were tested using heart sounds broadcast from the Harvey® simulator. The Harvey®-trained group showed a 33% gain in performance; the ‘usual care’ group showed a change of 5%.

Another study by Butter et al.7 used a similar design, in which outcomes in Year 3 students who had undertaken a simulation-based course were compared with those in untrained Year 4 students. The Year 3 group showed a large gain in accuracy (94% versus 74%) when tested with the same simulator.

Although these studies show impressive gains, they have two serious limitations to their ability to address our questions. Firstly, the control intervention is unspecified and is likely to be minimal. Although it might be argued that this represents the usual approach to auscultation instruction, nevertheless it is not useful in addressing the comparison between HFS and LFS. Secondly, the outcome was assessed on Harvey® and thus there is no evidence from this study that the skills are transferable.

Some reviews have claimed that Harvey® training does enhance transfer. As Issenberg et al.18 state:

‘Harvey has been rigorously tested to establish its educational efficacy… Students who used CPS [cardiology patient simulator] performed significantly better than the non-CPS group… not only on the CPS skills post-test (p < 0.001) but also on the patient skills post-test (p < 0.03).’

A study of transfer was conducted by Ewy et al.6 using final-year medical students. On real patients, the Harvey®-trained group achieved a mean score of 54.9 and the control group a mean score of 52.1. The difference of 2.8%, although statistically significant, was of little educational significance. Butter et al.7 also assessed transfer to a real patient, with a significant but small gain in the simulator group compared with the no-intervention control group (82% versus 75%).

All of these studies compared outcomes of training on a simulator with those of no formal training and such comparisons are not directly relevant to our review. However, despite this strong contrast in intensity of instruction, which is mirrored in large gains when trainees are tested on the simulator, the few reports of transfer to real patients show only modest improvement. Another study reveals one reason for this lack of transfer. Fraser et al.21 trained students to recognise a case of either symptomatic aortic stenosis or asthma on Harvey® and then tested them on both scenarios. Perhaps not surprisingly, the students showed large improvements in their ability to recognise and diagnose the training problem, but showed no improvement on either the alternative problem or a new problem in the same system (mitral regurgitation or pulmonary fibrosis). In short, transfer to a new problem, even when tested on the same simulator, did not occur.

One recent study by de Giovanni et al.19 made a specific comparison between HFS and LFS by comparing outcomes in a group of 37 Year 2 students who received 4 hours of instruction on Harvey® with those in a second group who received the same amount of training listening to a CD that included multiple sounds for each condition. Both groups were tested 6 weeks later in an objective structured clinical examination (OSCE) format with real patients with mitral stenosis, mitral regurgitation, aortic stenosis, aortic regurgitation or normal heart sounds. The Harvey®-trained group showed a small advantage in the detection of specific sounds (62% versus 49%), but no differences emerged in diagnostic accuracy (59% versus 57%), clinical skills (76% versus 79%) or communication skills (70% versus 70%) (the latter two categories were rated by a doctor-examiner). Hence, the opportunity to practise auscultation skills and learn heart sounds on the HFS did not translate into greater improvement in clinical or detection skills.

There is also correlational evidence for HFS versus LFS. Hatala et al.22 studied 28 internists in an OSCE with 12 stations. Heart sounds were portrayed by Harvey® in four stations, by an SP with heart sounds delivered by a laptop computer placed beside the patient (what Kneebone et al.9 call a ‘hybrid’ simulation) in four stations, and by real patients in four stations. The same four conditions (aortic stenosis, mitral stenosis, mitral regurgitation, normal) were present in all modalities. Overall diagnostic accuracy was 7.7 on Harvey® stations and 8.0 on SP-based stations, but only 6.7 on real patient-based stations (the differences between outcomes on real examples and those on simulation were significant, whereas the differences between outcomes on the two simulations were not). Furthermore, an examination of correlations between modalities, corrected for reliability, showed that the disattenuated correlation between the Harvey® and SP-based stations was 0.99, but correlations between each of these and real patient-based stations were 0.51 and 0.54, respectively. Thus, the HFS and LFS stations both appear to have been easier than real patient-based stations, and both appear to have tested equivalent skills, but the relationship between performance on either simulator and performance with real patients was moderate and equivalent.

Training in basic motor skills

A number of studies have examined the relative performance gains to be derived from HFS versus LFS in the development of basic motor, usually surgical, skills. As indicated earlier, although the HFS is not computer-controlled in many of these studies23–27 (the exception is virtual reality [VR] simulations for some skills), it does clearly differentiate from LFS in both realism and cost.

There is considerable uniformity in the approach; this is unsurprising as all the studies originated in the same centre in Toronto. All the studies used either two or three groups so that outcomes were compared in HFS versus LFS groups and occasionally in a control group of participants who read a text or watched a video, and outcomes were measured either on the HFS or on a more realistic object. Results are summarised in Table 1.

Table 1.   Characteristics of studies comparing outcomes of training using high-fidelity simulation with outcomes of training using low-fidelity simulation or other intervention
ReferenceDomainTreatment: high fidelityControl (1): low fidelityControl (2)OutcomeTreat-ment meanControl 1 meanControl 2 meanSignificance
  1. * This was a comparative study in an examination setting, not an intervention study. See text

  2. † Read from figure

  3. ‡ Significant difference for junior residents (p = 0.05), but not for global ratings or senior residents

  4. § Lower score is better

  5. ¶ Test of high- versus low-fidelity

  6. ** Raw scores arbitrarily divided by 25

  7. N/A = not applicable; VR = virtual reality; SP = standardised patient; HF = high-fidelity

Issenberg et al.19Heart soundsHarvey®N/ANothingSimulator diagnosis80%47% Yes
Butter et al.7Heart soundsHarvey®N/ANothingSimulator diagnosis94%74% Yes
Patient diagnosis82%75% Yes
Ewy et al.6Heart soundsHarvey®N/ANothingSimulator diagnosis68.2%58.6% Yes
Patient diagnosis54.9%52.1% Yes
de Giovanni et al.21Heart soundsHarvey®CD-ROMN/APatient diagnosis59%57% No
Hatala et al.22*Heart soundsHarvey®SP + laptopReal patientHarvey®77%N/AN/AN/A
SP + laptop80%
Patient67%
Anastakis et al.23Basic skillsCadaverBench modelN/ACadaver checklist70%68%61%No
Matsumoto et al.24UreteroscopyLimbs and thingsCoffee cup, plastic strawDidacticHigh-fidelity model checklist95%90%77%No
Grober et al.25Microsurgery anastomosisRat vas deferensSurgical tubingDidacticRat vas deferens88%84%73%No
Sidhu et al.27Vascular anastomosisHuman cadaver armPlastic model Live pig69%81%N/AYes/No
Chandra et al.26BronchoscopyVR simulatorWood boxN/AReal patient74%71%N/ANo
Munz et al.28LaparoscopyVR (LapSim®)Box trainerNoneMoves§5006001500No
Path§3500 cm4000 cm7000No
Economy§0.800.851.60No
Brydges et al.29i.v. insertionSimMan®Static plasticVRSP checklist**72%68%64%Yes
Morgan et al.30Critical careHF simulatorVideo demoN/AHF simulator85%85%N/ANo
Nyssen et al.31Critical care anaesthesiaComputer + manikinComputerN/AChecklist76%81%N/AYes
Baxter & Norman32Critical careSimMan®Video demoN/AGlobal rating72%67%52% 
Wenk et al.33InductionMETI simulatorProblem-based discussionN/AMETI simulator71%65% No
Bruppacher et al.34Cardiac weaningSimulatorInteractive seminar Real patient89%73% Yes

Anastakis et al.23 compared outcomes of a 4-hour training session on a cadaver with outcomes achieved by simple bench models (e.g. a coconut lined with plastic and a water-filled balloon for burr hole insertion) with outcomes achieved by reading a text for six different basic skills (Burr hole insertion, chest tube insertion, bowel anastamosis, wound closure, tendon repair, metacarpal fixation). Residents were tested on a cadaver in a realistic operating room (OR) environment a week later. Both simulations achieved significantly better outcomes than did the text, but there was no difference between the simulations.

Matsumoto et al.24 taught residents ureteroscopy using a didactic session, an LFS consisting of a Styrofoam coffee cup and plastic straws (based on task analysis) and a commercially available HFS costing US$3700. Testing was conducted using the HFS by expert observers using a global rating and checklist. Again, the LFS and HFS were superior to the didactic training, but there was no difference in outcomes between the LFS and HFS.

Grober et al.25 taught residents microsurgery of the vas deferens using either a live rat vas deferens or a surgical tubing bench model and then tested the residents using both modalities. Checklist and global scores for the live rat outcomes did not differ between the HFS and LFS groups, but both were higher than those achieved by a didactically taught group. This study also involved a ‘patient’ outcome: the patency of the rat vas deferens at 30 days. Again, outcomes in the LFS and HFS groups were superior to those in the didactic training group but did not differ between type of simulation.

Chandra et al.26 compared a VR bronchoscopy simulator (AccuTouch®; Immersion Medical, Inc., Gaithersburg, MD, USA) with a simple ‘choose the hole’ model, based on a careful task analysis. Outcomes were assessed on a real patient 1 week later. No control group was used as prior research by the group had shown little training benefit from didactic instruction. The results were consistent with those of other studies; no significant difference between the LFS and HFS groups emerged.

Sidhu et al.27 conducted a similar study of training in vascular anastomosis, comparing an LFS (plastic tube model) and an HFS (a human cadaver arm brachial artery). There was no didactic control group. Testing was carried out in a live pig a week later. By contrast with the previous studies, these authors27 found a difference between outcomes in the LFS and HFS groups; however, although this difference was significant for checklist scores, junior residents and ‘final product’ scores, it was not significant for global ratings or senior residents.

Finally, two studies directly manipulated engineering and psychological fidelity. Munz et al.28 compared two groups, one of which was trained with a box trainer using real laparoscopic instruments (costing about US$2000) and one of which was trained using a VR trainer (LapSim®; Surgical Science, Gothenburg, Sweden) costing about US$20 000 that gave realistic visual information but no haptic feedback. Training was given on three laparoscopic tasks (navigation, coordination, grasping) in three 30-minute instructional periods delivered over 3 weeks. Outcome was measured on the same trainer using objective measures of the number of movements, total distance, total time and economy, and changes in outcomes pre- and post-training were computed. Both groups showed improvement over a control group, but there was no difference between the two simulation groups in the number of movements, distance or economy of movement. However, this study did not include a transfer task, its instructional time was short (90 minutes) and its sample sizes were small (eight per group).

Brydges et al.29 compared three conditions in teaching intravenous (i.v.) catheterisation: LFS (a VR simulator costing US$12 000); HFS (SimMan® [Laerdal Medical Corp.], costing US$35 000), and ‘progressive’ fidelity, in which students progressed from low- to high-fidelity conditions. However, it is notable that the ‘low-fidelity’ simulator was an expensive single-purpose VR simulator, and the ‘medium-fidelity’ simulator (which was not studied separately) was an inexpensive static simulator costing US$600. The ordering was explicitly based on psychological fidelity; although the VR simulator provided better visual information, it lacked a realistic arm and so movements were not accurately simulated. In contrast with the other studies, the authors29 found a clear gradient with fidelity, with progressive fidelity showing the best performance on a global rating and checklist, and LFS showing the worst. These two studies28,29 provide some evidence that simulations that focus on psychological fidelity can provide performance gains equal to or greater than those of HFS at costs at least an order of magnitude lower.

Thus, of the studies examined, all but one showed that LFS resulted in performance gains, assessed on either an HFS or a real patient (animal or human), that are approximately equivalent to those observed with HFS, although outcomes of LFS appear to be consistently a little lower than those of HFS. The one exception showed a difference on checklist but not global scores.

Training in critical care and crisis management skills

  1. Top of page
  2. Abstract
  3. Introduction
  4. Theoretical framework
  5. Methods
  6. Results
  7. Training in critical care and crisis management skills
  8. Discussion
  9. Acknowledgments
  10. References

It is often assumed that although LFS may be adequate for the development of simple skills, as skills become more complex, the authenticity and fidelity of the simulation must increase in a manner commensurate with the skill.14 If this is true, studies of LFS versus HFS training in complex tasks such as advanced trauma life support or other crisis management situations may well show a clear superiority of HFS.

The design of such studies must, however, result in compromises. Firstly, it is not exactly clear what an LFS of a crisis management situation would look like. Perhaps for that reason, many of the comparative studies use a passive learning strategy, such as watching a video, as a control. Further, it would be extremely demanding to attempt to validate the training outcomes with real patients because the management of a real heart attack victim is not the best context in which to conduct educational research. One study that did look at formal ACLS training with simulators used a case–control design and measures of response to actual clinical events.14 However, the comparison was with ‘usual care’, not a formal LFS instruction. As a result, all the studies of complex skills that compare outcomes of HFS with those of some type of control actually test outcomes on the HFS. This leads to the obvious bias that transfer is not being examined for the HFS group and hence we would expect a bias toward a larger effect of HFS. Surprisingly, this was generally not observed.

Morgan et al.30 trained medical students in an anaesthesia rotation to manage myocardial ischaemia, anaphylaxis or hypoxaemia in a 1.5-hour session using either a full HFS with an instructor or a video demonstration by an instructor on the HFS. All students were then tested on the same scenario using the HFS. There were no significant overall differences between the two groups (mean score in the HFS group: 10.27; mean score in the video group: 10.21 [our calculation]).

Nyssen et al.31 trained students on an anaesthesia rotation on two simulators, comprising, respectively, a full HFS or a computer screen-based simulation. Performance was assessed on the HFS on the same or a different scenario. Strangely, these authors31 did not perform a statistical test of the difference between the two groups; however, on their ‘same scenario’ test the mean score of the computer-trained group (81.4) was significantly higher than that of the HFS group (75.8) (t = 2.35, p < 0.05 [our calculation]). On the ‘different scenario’ test, scores did not differ between the training groups (computer-trained group, mean score: 68.8; HFS group, mean score: 70.8).

Baxter and Norman32 examined students’ ability to respond in a critical care situation. After an initial orientation, they randomised 27 final-year nursing students to receive one of: no instruction; a video demonstration, and practice on an HFS (SimMan®). All students were then tested with a new simulation on the simulator and evaluated by a blinded assessor using a 7-point global rating scale. Both instructional groups significantly outperformed the control group (simulator group, 5.04; seminar group, 4.74; control group 3.64), but there was no difference between the instruction groups.

Wenk et al.33 compared a problem-based discussion with practice on an HFS in emergency induction with 32 students. The outcome was assessed 10 days later on a simulator. Although self-assessment scores (confidence) were higher in the HFS group (21.0 versus 19.4), an objective task-specific score was not (111 versus 123), although the effect size was reported at 0.52.

Bruppacher et al.34 conducted a randomised trial in which anaesthesiology residents were instructed in cardiac ‘weaning’ from the bypass and compared the outcomes of trainees instructed using an HFS with those of residents instructed in a 2-hour seminar. Twenty trainees were involved. Two weeks and 5 weeks after instruction, they were observed performing a weaning on an actual patient, using a global rating and a checklist. The simulation group performed significantly higher on both the post-test and the retention test (post-test global scores: 14.3 versus 11.8; retention test global scores: 14.1 versus 11.7; post-test checklist scores: 89.9 versus 75.4; retention test checklist scores: 93.2 versus 77.0).

Thus, in the learning of complex skills, four of five studies30–33 showed no significant advantage to the complex HFS. However, by contrast with the studies of basic motor skills, four of the five studies30, 32–34 used the passive viewing of a criterion performance on discussion as the LFS. Clearly, this involves no practice whatsoever. In addition, in four of the five studies, performance was assessed on the simulator30–33. Thus these studies have two potential biases in favour of the HFS group. Despite this, only one of the studies34 showed a significant effect of HFS over LFS, although this was also the single study that clearly assessed transfer to real patients.

The results are summarised in Table 1, in which we have abstracted the scores from each study and converted them to a percentage (except those in the study by Munz et al.28). These are summarised here.

Studies of training in heart sounds (6,7,19,21)

  • The average effect of training on the Harvey® simulator compared with no treatment when tested on the same simulator6,7,19 was 21%. On real patients, the effect of simulator-based training compared with no treatment6,7 was 4.5%; the effect of simulator-based training compared with LFS training was 2%.

Studies of training in basic surgical skills (23–29)

  • High-fidelity simulator training produced a gain of 12% compared with no active treatment23–25,29 and a gain of 1% compared with LFS training.23–27,29

Studies of training in critical care (30–34)

  • High-fidelity simulator training produced a gain of 4.4% compared with a passive demonstration.30–34 However, one study of weaning from bypass showed HFS training to have a positive effect of 16%. When this study was omitted, the average difference was 1.5%.

Discussion

  1. Top of page
  2. Abstract
  3. Introduction
  4. Theoretical framework
  5. Methods
  6. Results
  7. Training in critical care and crisis management skills
  8. Discussion
  9. Acknowledgments
  10. References

Despite a few anomalies, the results of this review appear quite consistent. HFS shows clear gains in performance and transfer to the real patient setting compared with typical opportunistic instruction. However, when the outcomes afforded by HFS are compared with those of LFS, the gains of HFS are more modest and, almost without exception, are not statistically significant.

The review may be faulted because the search was not based on explicit criteria and computer databases; it is not a ‘systematic review’. Although we cannot claim to have identified all the relevant literature, the results are sufficiently consistent that we have confidence in our conclusions.

The decision to choose HFS over LFS is not simply economic; if it were, one might well decide that the additional benefit, although small, can justify the additional cost, however large. However, although we have not conducted an exhaustive economic analysis, our impression is that any simulator that involves a dynamic computer typically costs an order of magnitude (factor of 10) more than an equivalent static simulator. No institution can afford an unlimited number of HFSs, yet furnishing even one simulator for each special purpose can become astronomically expensive.

The cost issue may ultimately translate into an educational issue. Expertise comes with deliberate practice; indeed, this axiom is the mantra of the simulation movement. However, the cost of the simulator can restrict the number of simulators available, which, in turn, can seriously limit the number of hours each student can access the simulator. It may be more effective, not just in terms of cost, to provide each student with unlimited access to an LFS rather than an hour or two on an HFS.

Why is it that increases in fidelity are not matched by equivalent gains in performance? There are a number of possible explanations.

The role of context

Clearly, HFS expends substantial resources on being as realistic as possible. Harvey® has a realistic face and extremities, lies on a real hospital gurney and has a chest that moves up and down to simulate breath sounds. SimMan® has built-in speakers so that the operator can communicate like a real patient. All of these strategies are aimed at making the context as realistic as possible. But is context that important? As we indicated in the introduction, studies of context effects in psychology are equivocal. It may be that we are more capable of moving from one context to another than has been thought.

Complexity and cognitive load

One assumption underlying the attraction of HFS is that practice in a realistic environment will lead to better transfer as a result of context similarity. This may not be the case for at least two reasons. Firstly, the relationship may not be linear. Using the reference standard of simulators, the cockpit simulator, a trainee must have considerable prerequisite skill to profit from the experience of a simulated Boeing 747 cockpit. If someone cannot fly a 747, placing him in a simulated situation in which he must try to land one with two engines out is unlikely to accomplish much. A related issue is cognitive complexity. Cognitive load theory35 demonstrates that many additions to the learning task may detract from learning because of our limited ability to process incoming information.

There is evidence in the motor skills literature36 that novices may well be better off with simpler models and should gradually move to more complex models as their skills improve, a strategy known as ‘progressive fidelity’. This was the thesis of the study by Brydges et al.29 described earlier. Dubrowski et al.36 showed that a group learning laparoscopic knot-tying under conditions of progressive fidelity did as well as a group of learners who spent equivalent time on the full complexity model. Notably, however, the progressive fidelity group did no better and therefore this study does not, of itself, provide evidence of the advantage of progressive fidelity. By contrast, the study by Brydges et al.29 does show superior performance in a progressive fidelity group; however, this may have resulted from mixed practice, not progression, which has been shown to have advantages.37

Fidelity

A number of authors have borrowed a page from aviation psychology and suggested that fidelity can be assessed on two levels: ‘engineering fidelity’, which refers to whether the simulation looks realistic, and ‘psychological fidelity’, which concerns whether the simulator contains accurate simulations of the critical elements that will demand specific behaviours to complete the task.12 The two types of fidelity can be dissociated. Moreover, there is some evidence from this review that psychological fidelity may be a more critical determinant of learning and transfer than engineering fidelity. The one study that specifically manipulated psychological fidelity29 showed a clear advantage for greater fidelity. Two studies that contrasted expensive commercial simulators with ‘kitchen table’ simulators24,26 (in both of the latter were based on careful task analysis and thus had high psychological fidelity) showed equivalent performance.

The nature of the task: ‘sensory’ versus ‘simple’ motor skills versus ‘complex management’

The studies we have reviewed illustrate the difficulty of generalisation. Although the findings appear quite uniform, closer inspection indicates that training in each group of studies is directed at the acquisition of very different skills. The studies of training in auscultation skills exemplify the use of simulation to acquire perceptual skills. Other examples are reading electrocardiograms, interpreting skin lesions and synthesising laboratory test values. From the perspective of psychological fidelity, practice must include multiple different examples of confusable categories. Yet, as we indicated earlier, Harvey® contains only one example of most conditions.

Training in so-called ‘simple’ motor skills generally requires considerable practice. The development of skills for suturing, for example, requires many hours of practice and psychological fidelity demands that the critical elements – the resistance of the tissue, the ‘feel’ of the instruments – must closely resemble these elements of the task performed with real tissue. It is probably less important that the simulated tissue has the right colour or shape.

Conversely, the development of ‘complex’ skills may actually require relatively little practice. If each element of an action is relatively straightforward or has been previously learned (e.g. starting an i.v. line, taking a blood pressure), mastering the correct approach to crisis management may be mainly a matter of remembering the exact sequence in which actions must be taken. Although it is often assumed that HFS has a central role in this situation, the opposite may be true. Practice may involve nothing more than mental rehearsal of the steps.

The interaction with expertise

Although previous authors have indicated that there may well be an interaction with the level of the learner,36 it is generally assumed that increasing expertise should be accompanied by increasing fidelity. However, this may not be the case at all. Two potential uses of HFS may lie in the teaching of an approach to rare and difficult problems, and in the teaching of the management of complex problems involving multiple health professional roles. Paradoxically, neither may require any sophistication of simulation. In the context of a rare problem, experts may regard the issue as a cognitive problem and simply ‘think’ their way to a solution. Their experience in the real setting is likely to be sufficiently extensive to make practice of the motor skills involved superfluous. In the context of a complex problem that requires input from multiple health professionals, the focus of activity is between professionals; the features of the simulator may be irrelevant.

Although the role played by each of these classes of variables is necessarily speculative, it does illustrate, as does the evidence reviewed in this paper, that the relationship between simulation fidelity and learning is not unidimensional and linear. It should come as no surprise, therefore, that the large investments required to acquire more advanced (higher-fidelity) simulators are not accompanied by commensurate increases in learning.

Sadly, this lesson was learned in aviation many years ago and we are now relearning it. As Salas et al. wrote with reference to aviation simulators:

‘In sum, more is not necessarily better. Generally, the level of simulation fidelity does not translate to learning. High-fidelity simulations… should be used as determined by training and task requirements, cost and learning objectives.’15

Contributors:  GN conceptualised and drafted the article, and finalised the submission. KD and LG contributed to the conceptualisation of the article and critiqued the drafts. All authors approved the final manuscript for submission.

Acknowledgments

  1. Top of page
  2. Abstract
  3. Introduction
  4. Theoretical framework
  5. Methods
  6. Results
  7. Training in critical care and crisis management skills
  8. Discussion
  9. Acknowledgments
  10. References

Acknowledgements:  the authors acknowledge the continuing support of the Medical Council of Canada and Canadian Institutes of Health Research (GRN Holds a Canada Research Chair).

Funding:  none.

Conflicts of interest:  none.

Ethical approval:  not applicable.

References

  1. Top of page
  2. Abstract
  3. Introduction
  4. Theoretical framework
  5. Methods
  6. Results
  7. Training in critical care and crisis management skills
  8. Discussion
  9. Acknowledgments
  10. References