Simulation as a tool to improve the safety of pre-hospital anaesthesia – a pilot study*
Presented in part at the Joint Annual Meeting of the National Association of Medical Simulators and the Clinical Skills Network, London; September 2008, and the International Forum on Quality and Safety in Health Care, Berlin; March 2009.
We conducted a pilot study of the effects of simulation as a tool for teaching doctor-paramedic teams to deliver pre-hospital anaesthesia safely. Participants undertook a course including 43 full immersion, high-fidelity simulations. Twenty videos taken from day 4 and days 9/10 of the course were reviewed by a panel of experienced pre-hospital practitioners. Participants’ performance at the beginning and the end of the course was compared. The total time from arrival to inflation of the tracheal tube cuff was longer on days 9/10 than on day 4 (mean (SD) 14 min 52 s (2 min 6 s) vs 11 min 28 s (1 min 54 s), respectively; p = 0.005), while the number of safety critical events per simulation were fewer (median (IQR [range]) 1.0 (0–1.8 [0–2]) vs 3.5 (1.5–4.8 [0–8], respectively; p = 0.011). Crew resource management behaviours also improved in later simulations. On a personal training needs analysis, participants reported increased confidence after the course.
A significant proportion of deaths from injury are considered to be preventable . Many of these result from failure to identify and treat life-threatening injuries promptly in the pre-hospital phase of care. The National Confidential Enquiry into Patient Outcome and Death  recently concluded that ‘the current structure of pre-hospital management is insufficient to meet the needs of the severely injured patient’ and that ‘the pre-hospital response... should include someone with the skill to secure the airway (including the use of rapid sequence intubation), and maintain adequate ventilation’. The recently published guidelines for pre-hospital anaesthesia from the Association of Anaesthetists of Great Britain and Ireland  emphasise that, although pre-hospital anaesthesia can be of benefit, it can result in unnecessary morbidity and mortality if performed by the inexperienced or poorly trained. The inclusion of pre-hospital emergency anaesthesia in the evolving subspecialty competence framework for Pre-hospital Emergency Medicine mandates dedicated training programmes. Access to appropriate patients for supervised training is, however, extremely limited.
Simulation-based education has been embraced by training programmes in anaesthesia, emergency medicine and surgery [4, 5]. A recent review of the literature concluded that whilst educationally effective, the rigor and quality of research in this field needs improvement . A variety of scientific and interpretive methods for assessing the effectiveness of simulator-based training have been reported [6–10]. These include the use of objective measures, such as time to intubation, number and duration of desaturation episodes, verbalisation of observations, and global assessments of technical and behavioural performance [11–15].
With these aspects in mind, we sought to measure the effectiveness of simulation within a training course for Pre-hospital Emergency Medicine. The course included multiple full immersion, high-fidelity simulations aimed at training physician-paramedic teams with appropriate backgrounds to deliver safe pre-hospital emergency anaesthesia and critical care. We sought to examine a specific element of the training and attempt to measure and quantify changes in performance.
All participants gave informed written consent for the use of training videos for the purposes of the study. The Local Research Ethics Committee confirmed that further ethical approval was not required.
Six paramedics and six physicians participated in a 12-day training course in preparation for joining a physician-paramedic pre-hospital emergency medical team . The course included 43 thirty-minute full immersion simulations involving all aspects of pre-hospital critical care. It also included group discussions, practical sessions, tutorials, lecture-based teaching sessions and a final assessment. Simulator based training was programmed throughout the course and was designed to provide an opportunity to practise knowledge and techniques learned on the course and consolidate prior learning.
The course commenced with a personal training needs analysis and a 2-day induction period in which all participants were familiarised with the capabilities and limitations of SimMan® simulators (Laerdal, Norway). The personal training needs analysis is a self-administered questionnaire that covers previous experience and confidence. It uses a three-point scale to record participants’ level of confidence across all elements of the competence framework for Pre-hospital Emergency Medicine (Table 1) .
Table 1. The Magpas Competence Framework for Pre-hospital emergency medicine . The Pre-hospital Emergency Anaesthesia theme contains units of related elements of underpinning knowledge, technical skill and non-technical skill.
| 1. Emergency medical systems||7.1 Principles of pre-hospital emergency anaesthesia|
| 2. The operational environment||7.2 Anaesthetic pharmacology|
| 3. Resuscitation and clinical care||7.3 The decision to anaesthetise|
| 4. Equipment and monitoring||7.4 Direct laryngoscopy|
| 5. Care of special groups||7.5 Rapid sequence intubation|
| 6. Analgesia and procedural sedation||7.6 The difficult airway|
| 7. Pre-hospital emergency anaesthesia||7.7 The failed airway|
| 8. Rescue and extrication||7.8 Maintenance of anaesthesia|
| 9. Retrieval and transport||7.9 Ventilatory support|
|10. Emergency preparedness and response||7.10 Pre-hospital emergency anaesthesia in clinical practice|
Simulation scenarios were based upon real incidents and took place in a range of re-created pre-hospital environments, including incident scenes and within an ambulance. To achieve full immersion, participants wore full personal protective equipment, operated in physician-paramedic teams and carried equipment normally available to the team. Participants were not acquainted before the course, team pairings changed regularly and all participants undertook all scenarios. Each simulation commenced with a situation report transmitted by radio. Participants then entered the simulated environment, which was enhanced by use of large format photographs, variable lighting conditions and pre-recorded background noise, and were met by paramedics or bystanders. Experienced pre-hospital physicians and paramedics operated the simulators, using pre-determined injury patterns and physiological status. Physiological changes were made according to the planned clinical course or in response to the teams’ interventions and actions. All simulations were video-recorded to facilitate feedback and clarify any recall discrepancies. After completion of the course the personal training needs analysis was repeated.
A panel of five experienced pre-hospital practitioners independently reviewed videos of the pre-anaesthetic, induction, laryngoscopy and intubation elements of 20 scenarios. The panel comprised one emergency physician, one anaesthetist and two paramedics who work on the regional pre-hospital emergency medical team, together with one independent anaesthetist who had not previously had any contact with the course or the team. Ten videos recorded on day 4 of the course and ten recorded on days 9 and 10 were reviewed in random order. The element of the scenario reviewed related to the interval between arrival of the team and inflation of the tracheal tube cuff. Each of the scenarios involved a patient with an unambiguous indication for pre-hospital emergency anaesthesia. None of the scenarios selected for review were designed to simulate difficult or failed laryngoscopy.
Reviewers were blinded as to when the simulation had occurred. Performance was scored in terms of three areas (timings, the occurrence of safety critical events and crew resource management (CRM) behaviours) using a simulator assessment tool (Appendix 1). This tool had been previously developed by a team of experienced pre-hospital simulation facilitators working within our training programmes. It allows facilitators to record, in a structured and reproducible way, whether appropriate knowledge, skills and behaviours are demonstrated during a simulation. Reviewers were instructed in the use of the tool and given the opportunity to practice on a pilot video before conducting the study.
Time zero was the team’s arrival and measured times were time to decision to anaesthetise, time to completion of pre-induction checks, and time to successful intubation (inflation of tracheal tube cuff). The panel of reviewers discussed any events that they considered to represent a clinical risk. These events were recorded in free text. ‘Safety critical events’ included actions or interventions that were harmful or had the potential to cause harm. Reviewers were asked to rate CRM behaviours individually (including situational awareness, leadership, followership, closed loop communication, use of standardised language, assertive communication, adaptive behaviour and workload management) using an eight-point Likert scale .
Statistical analysis was performed using spss 14.0. Mann–Whitney U-tests were performed to compare the CRM behaviour scores, intervals between mission-critical actions and number of occurrences of safety critical events in the simulations at the beginning and end of the course.
The mean time from arrival to inflation of the tracheal tube cuff increased by over 3 min in the later simulations, resulting from increases in both the interval between arrival and decision to intubate and the interval between decision to intubate and initiation of pre-induction checks (Table 2). The median inter-observer variance in timings was 2 s (0.3%).
Table 2. Key time intervals on day 4 and days 9/10 of a course in pre-hospital anaesthesia. Data are mean (SD).
|Arrival to decision to intubate; min:s||2 : 16 (0 : 37)||3 : 38 (0 : 55)||0.004|
|Decision to intubate to initiation of pre-induction checks; min:s||4 : 28 (1 : 33)||6 : 30 (1 : 43)||0.013|
|Initiation of pre-induction checks to tracheal tube cuff inflation; min:s||4 : 44 (0 : 37)||4 : 44 (1 : 14)||NS|
|Total time (arrival to tracheal tube cuff inflation); min:s||11 : 28 (1 : 54)||14 : 52 (2 : 06)||0.005|
The number of safety critical events per simulation fell in the later simulations compared with the early ones (median (IQR [range]) 1.0 (0–1.8 [0–2]) vs 3.5 (1.5–4.8 [0–8], respectively; p = 0.011). The most common safety critical events were failure to assess the airway in terms of likely difficulty of intubation, and failure to reassess the patient adequately before extending anaesthesia (Table 3).
Table 3. Most common safety critical events (actions or interventions considered to be harmful, or have the potential to cause harm), on day 4 and days 9/10 of a course in pre-hospital anaesthesia. Values are number (observed frequency).
|Failure to immobilise cervical spine||2 (20%)||0|| |
|Inappropriate airway management||2 (20%)||0|| |
|Failure to perform adequate airway risk evaluation||4 (40%)||3 (30%)|| |
|Inappropriate dose of chosen induction agent||2 (20%)||1 (10%)|| |
|Failure to communicate physiological compromise to team member||2 (20%)||0|| |
|Failure to assess physiology adequately before induction of anaesthesia||5 (50%)||2 (20%)|| |
|Failure to reassess physiology adequately before extending anaesthesia||5 (50%)||1 (10%)|| |
There was a trend towards higher CRM behaviour scores for the later simulations in all of the eight categories, reaching statistical significance for two (Table 4).
Table 4. Crew resource management behaviour scores on day 4 and days 9/10 of a course in pre-hospital anaesthesia (maximum score 8). Values are median (IQR [range]).
|Situational awareness||5.0 (4.8–5.3 [3.0–5.6])||5.6 (5.3–5.8 [4.8–6.2])||0.020|
|Leadership||5.3 (4.5–5.7 [2.8–5.8])||5.7 (5.4–5.8 [4.8–6.2])||NS|
|Followership||5.1 (4.0–5.8 [3.2–6.0])||5.4 (5.1–5.4 [4.0–5.8])||NS|
|Closed loop communication||5.1 (4.7–5.2 [3.8–5.8])||5.4 (5.0–5.6 [3.8–5.8])||NS|
|Standardised language||5.4 (4.9–5.6 [4.4–6.0])||5.6 (5.6–6.0 [5.2–6.2])||0.039|
|Assertive communication||5.4 (4.9–5.8 [2.8–6.0])||5.8 (5.3–6.2 [4.2–6.4])||NS|
|Adaptive behaviour||5.2 (4.8–5.6 [3.4–5.8])||5.6 (5.2–5.8 [4.6–6.0])||NS|
|Workload management||5.5 (5.2–5.8 [4.0–6.4])||5.6 (5.2–5.8 [4.3–6.0])||NS|
All participants reported increased confidence in the pre-hospital anaesthesia competencies after the course.
We would of course expect to see improvements in underpinning knowledge, technical/non-technical skills and confidence over the course of a 12-day focused training period, irrespective of the use of simulators. What we sought to do in this preliminary study was to examine a specific element of simulator training and attempt objectively to measure and quantify changes in performance. We were able to measure changes and, although there was an increase in overall time, we considered that from a safety and quality perspective, the changes represented improvement. We should be cautious, however, about attributing this improvement solely to the use of simulators. There are a number of reasons why improvement would be expected with repeated practice and exposure to the environment, equipment, colleagues and clinical problems within a supportive learning setting. Comparison with a control group who were exposed to all elements of the training course except the simulator would provide definitive evidence, but such a study would be difficult to conduct. Given that the opportunity for the practical application of technical and non-technical skills is key to effective training, there would be few volunteers (amongst trainers and participants) for a study group that involved no simulated patient practice.
An alternative might be to compare performance of participants with that of a control sample of individuals not exposed to the training course at all. However, the comparison would have to involve control subjects who were inexperienced in the provision of pre-hospital emergency anaesthesia. Our experience from the induction phase of our course (the first two days) is that the inexperienced consistently perform poorly in this simulated environment until they become familiar with the simulator and system. This was the main reason we elected to review videos from day 4 of the course.
The study has a number of other limitations. First, the sample size is small. Sample size was determined by the size of the course (this was a relatively large course) and the time commitment associated with this methodology. Second, achieving true blinding was difficult due to the involvement of some of the video assessors in the course. This influence was mitigated through the involvement of an external assessor. Third, we were unable to standardise all the scenarios fully. Within the constraints of the training course structure, we sought to standardise the element of simulation under scrutiny by only focusing on pre-anaesthetic assessment, laryngoscopy and intubation, in scenarios where laryngoscopy and tracheal intubation were indicated but not programmed to be difficult. Although later scenarios in the course tended to be more complex, this related to post-anaesthetic complications, in-transit emergencies or physiological deterioration related to a specific type of injury rather than the initial assessment, induction, laryngoscopy and intubation phases. Limiting the review to the point of tracheal cuff inflation, therefore, also helped standardise the scenarios and remove the influence of post-anaesthetic complications or planned physiological deterioration.
There is a need to provide effective simulation training for healthcare professionals involved in the pre-hospital management of critical injury – in terms of both the initial interventions and the management of unexpected complications. In true emergency situations the traditional apprenticeship models of training rarely facilitate learning, as the trainee is usually relegated to the role of assistant or observer. Simulation within the airline industry has enabled pilots to rehearse the management of important but rarely encountered events in a safe learning environment. The development of human simulators has provided similar learning opportunities for healthcare providers, allowing the dynamic re-creation of a task environment with real-time physiological responses and a need to demonstrate both psychomotor skills and behavioural attributes . Simulation training facilitates experiential learning and emphasises the value of rehearsal, repetition and reaction within a range of scenarios . However, to ensure transferability to the actual setting, the simulated environment must closely approximate the real-life context .
Reflecting the experience of the airline industry, we found an improvement in CRM over the duration of the course. This was demonstrated for all CRM behaviours; the failure of some of the parameters to reach statistical significance may be a consequence of the small sample size. Arguably, the improvement in CRM may be partially attributable to increased familiarity with other participants. Data demonstrating the direct efficacy of CRM training are difficult to obtain . Nevertheless, the importance of human interaction as a potential source of error has led to the aviation industry’s introducing legislation mandating CRM training . In terms of generalisability of these results, CRM behaviours are largely independent of clinical context, implying a high level of transferability .
A significant reduction in the total number of safety critical events was also observed. Those in early simulations mainly resulted from failure to pay adequate attention to the components of the primary survey. The corresponding increase in the interval between the decision to intubate and commencement of pre-induction checks in later simulations may reflect greater emphasis on pre-anaesthetic optimisation of the patient, equipment and surroundings. This increase may also reflect the aforementioned improvement in CRM, as team members engage in a more thorough discussion of the assessment and management options.
This pilot study suggests that simulator-based training offers a practical and effective method of supporting the teaching of pre-hospital emergency anaesthesia. Further studies using larger numbers would be required to confirm our findings and to explore the degree to which these skills are retained and translated into clinical practice. In addition, the methodology used in this study may offer a tool for prospective evaluation of simulator-based training .
This study was funded by the Emergency Medical Charity Magpas and a grant from the Wellcome Trust. We are grateful to all the paramedics and doctors who allowed their performance to be scrutinised in this study.
Simulation assessment tool used to score performance before and after a course in pre-hospital anaesthesia