Debriefing as Formative Assessment: Closing Performance Gaps in Medical Education

Authors

  • Jenny W. Rudolph PhD,

    1. From the Center for Medical Simulation (JWR, RS, DBR), Cambridge MA; the Department of Anesthesia and Critical Care, Massachusetts General Hospital (DBR, JWR, RS), Boston MA; Harvard Medical School (JWR, DBR, RS), Boston MA; KidSTAR Simulation Program, Northwestern University Feinberg School of Medicine, Department of Pediatrics, Division of Emergency Medicine, Children’s Memorial Hospital (WE), Chicago, IL.
    Search for more papers by this author
  • Robert Simon EdD,

    1. From the Center for Medical Simulation (JWR, RS, DBR), Cambridge MA; the Department of Anesthesia and Critical Care, Massachusetts General Hospital (DBR, JWR, RS), Boston MA; Harvard Medical School (JWR, DBR, RS), Boston MA; KidSTAR Simulation Program, Northwestern University Feinberg School of Medicine, Department of Pediatrics, Division of Emergency Medicine, Children’s Memorial Hospital (WE), Chicago, IL.
    Search for more papers by this author
  • Daniel B. Raemer PhD,

    1. From the Center for Medical Simulation (JWR, RS, DBR), Cambridge MA; the Department of Anesthesia and Critical Care, Massachusetts General Hospital (DBR, JWR, RS), Boston MA; Harvard Medical School (JWR, DBR, RS), Boston MA; KidSTAR Simulation Program, Northwestern University Feinberg School of Medicine, Department of Pediatrics, Division of Emergency Medicine, Children’s Memorial Hospital (WE), Chicago, IL.
    Search for more papers by this author
  • Walter J. Eppich MD, MEd

    1. From the Center for Medical Simulation (JWR, RS, DBR), Cambridge MA; the Department of Anesthesia and Critical Care, Massachusetts General Hospital (DBR, JWR, RS), Boston MA; Harvard Medical School (JWR, DBR, RS), Boston MA; KidSTAR Simulation Program, Northwestern University Feinberg School of Medicine, Department of Pediatrics, Division of Emergency Medicine, Children’s Memorial Hospital (WE), Chicago, IL.
    Search for more papers by this author

  • Presented at the 2008 Academic Emergency Medicine Consensus Conference, “The Science of Simulation in Healthcare: Defining and Developing Clinical Expertise,” Washington, DC, May 28, 2008.

Address for correspondence and reprints: Jenny W. Rudolph; e-mail: jwrudolph@partners.org.

Abstract

The authors present a four-step model of debriefing as formative assessment that blends evidence and theory from education research, the social and cognitive sciences, experience drawn from conducting over 3,000 debriefings, and teaching debriefing to approximately 1,000 clinicians worldwide. The steps are to: 1) note salient performance gaps related to predetermined objectives, 2) provide feedback describing the gap, 3) investigate the basis for the gap by exploring the frames and emotions contributing to the current performance level, and 4) help close the performance gap through discussion or targeted instruction about principles and skills relevant to performance. The authors propose that the model, designed for postsimulation debriefings, can also be applied to bedside teaching in the emergency department (ED) and other clinical settings.

Emergency medicine (EM) educators seek efficient strategies to help residents and medical students address and even master the complex clinical, social, and logistical challenges of practicing medicine in the emergency department (ED). Formative assessment, the process of providing individually tailored doses of feedback to students on their performance is a concrete, effective way to provide this help. In simulation-based education, postscenario debriefing is an ideal forum for formative assessment. To describe and illustrate debriefing as an effective approach for formative assessment, this article draws on the existing evidence in the education, social, and cognitive science literature, as well as the authors’ experience in over 3,000 debriefings and in developing a curriculum used to teach debriefing to approximately 1,000 clinicians worldwide.1

Debriefing and Formative Assessment

Debriefing as formative assessment is a highly interactive process in which skills and understanding are not simply dispassionately assessed by the instructor, but in which new insights are cocreated in a dialogue between instructor and students. Debriefing and formative assessment, although their origins are very different, serve many of the same purposes. Debriefing, or after-action review, originated in the military practice of reviewing a mission for lessons that might improve the next one.2–4 Formative assessment emerged in the classroom as a way to evaluate curricula or students and provide midcourse feedback and correction.5,6

Debriefing

Reflecting on one’s own clinical or professional practice is a crucial step in the experiential learning process. It helps learners develop and integrate insights from direct experience into later action.7,8 After participating in a simulated case, debriefing allows clinicians to reflect. Some of the important goals and processes of debriefing or after-action review are to help participants understand, analyze, and synthesize what they thought, felt, and did during the simulation to improve future performance in similar situations. Achieving these goals usually involves a series of steps such as naming and processing emotional reactions, analyzing the social and clinical aspects of the situation, generalizing to everyday experience, and importantly, shaping future action by lessons learned.2,4,9–13 Effective debriefers are neither harshly judgmental nor falsely “non judgmental”; they neither berate students nor sugar-coat or camouflage criticisms. Rather, they provide clear, honest critiques in a way that is respectful and curious about the student‘s perspectives.14–15

Formative Assessment

Sometimes known as assessment for learning, formative assessment is contrasted to summative assessment, often characterized as assessment of learning. Three reviews on assessment,16–18 as well as a range of other empirical work19–21 provide the following insights: summative assessment is relatively infrequent, usually involves grades or formal ratings, occurs at the end of the training period, and is associated with high stakes (such as advancing or not advancing to the next stage of training, being certified or not certified). If the stakes are high, the assessment is usually standardized to ensure valid and reliable results. Summative assessment provides implicit feedback on where the student stands and may prompt changes in the students’ knowledge or behavior, especially through the process of studying for the exam. Formative assessment, in contrast, is ideally conducted separate from grades or formal ratings, occurs throughout the training period, is relatively frequent, involves lower stakes such as feedback on subtasks of a profession or skill set, and is tailored to the individual learner. Formative assessment can be “convergent” testing if a specific objective is known or “divergent,” exploring what is known.21 A key feature of formative assessment in both classroom and experiential contexts is that it provides feedback to the student with the goal of improving current performance.16

What is “formative” about formative assessment? The term “formative evaluation” entered the educational literature in the late 1960s with the work of Bloom,5 who was interested in providing students with “feedback and correctives at each stage of the teaching-learning process,” and Scriven, who applied it to ongoing improvement of education curricula.6 Formative assessment “forms” trainees in two ways: 1) it shapes skills and knowledge through feedback, and 2) it helps develop professional identity through the social interaction of learning conversations. While both instructors and students often understand the main curriculum of improving clinical skills or teamwork, the “hidden curriculum” of assessment includes implicit feedback about how well the trainee is performing a new professional role, such as being a doctor.22 Summative assessments, such as certification or graduation from medical school, grant students a formal identity as a physician. Formative assessment in learning conversations such as debriefing allows students to enact provisional or aspirational identities as a doctor. The respect or disdain instructors convey to the trainees can help form or undermine the trainee’s identity as a valued clinical colleague.22–24

Debriefing as Formative Assessment: the Role of “Cognitive Detective.”  To understand the process of conducting a formative assessment through debriefing, key findings from cognitive science, social psychology, and anthropology about how people perceive reality are useful. People make sense of the external clinical and social environment through cognitive “frames,” which are internal images of external reality.25–28 Terms for these images are myriad: “frames of reference,”“schemata,”“mental models,” etc. People engage in “sense-making” about external reality in which they actively filter, create, and apply meaning to their environment using frames.29,30

Frames shape the actions people take. Clinical frames and social or interpersonal frames play crucial roles in medical decision-making. For example, an emergency physician (EP) facing a patient with decreasing oxygen saturation will take one set of actions if she frames the signs as due to an airway obstruction and another if her working diagnosis is a tension pneumothorax. A nurse who holds the frame that reporting an error will lead to punishment is likely to report errors at a very different rate than one who believes the report will be used to improve work processes.31

The heart of debriefing as formative evaluation is to investigate the frames that underlie a performance gap. (This approach is like convergent formative assessment in that it assesses whether the trainee can achieve predetermined objectives. It is like divergent formative assessment in that it explores deeply what the trainee thinks and how he or she came to produce certain behaviors in a scenario.21) Just as Sherlock Holmes used visible clues to uncover crimes, the debriefer works backward from an observed performance gap to discover what frames (assumptions, goals, knowledge base) drove the actions contributing to that gap (Figure 1).

Figure 1.

 Map of the performance gap. Frames are invisible but can be discovered through questioning; they drive trainee’s actions. Actions (including speech) are observable. The performance gap is the difference between the desired actions and the trainee’s actions during the scenario (the gap can be an increment or decrement).

Consider an EM resident serving as the team leader while caring for a simulated patient with a gunshot wound. She holds the frame, “Everyone on my team knows that a patient with a gunshot wound to the chest is at risk for a hemo- or pneumothorax.” She fails to state a management plan for the patient, even as her team members appear to pursue every task except for needle decompression or chest tube placement. The model in Figure 1 suggests that people’s actions, including those of the trauma team leader, are an inevitable result of how they frame the situation.

Mistakes are usually the result of “intendedly rational” actions.27,32 In the trauma team scenario, the EM resident calls for a chest tube but does not notice when no one repeats back her request. As the patient decompensates and eventually loses vital signs, other team members remain focused on relatively lower priority tasks: the tech cuts off the patient’s clothes, the surgeon calls for blood, and the nurse obtains intravenous access. The resident does not explicitly declare the crisis, its suspected cause, and an action plan (“I think we have a tension pneumothorax and we need to decompress the chest now.”). The resident believed that the team members knew that a hemo- or pneumothorax was the main risk, assumed that people were as concerned and vigilant about it as she was, and therefore thought that she didn’t need to explicitly state the need for a needle decompression or ask the surgeon to prepare to place a chest tube.

To conduct a formative assessment of the trainee’s assumptions and knowledge, the instructor uses inquiry to bring frames to the surface and then analyzes their impact on the trainee’s actions. Once frames have been clarified, the instructor helps the trainee craft new frames, such as “Just because the clinical situation is obvious to me doesn’t mean others see it the same way. I have to reveal my thinking, especially when I’m the team leader.” The instructor also works with the team to develop new, more effective actions for the future.

Debriefing as Formative Assessment: the steps

1. Creating a Context for Learning: How to Introduce Debriefing as Formative Assessment

Studies from the domains of education, psychotherapy, and team learning indicate that three factors help create a debriefing context friendly to the conversational probing needed for formative assessment: clarifying the debriefing process, creating psychological safety, and articulating assumptions that actively support the aspirational identities of participants as competent doctors. First, trainees need to know the rules of participation and what is expected of them. Introducing and explaining the concepts of debriefing increases both the amount and depth of participation.4 Setting up a simple learning contract—an agreement that the trainees will think deeply about their performance and that the instructor will provide feedback on it—improves adults’ collaboration and satisfaction with the learning process.33 Second, creating a psychologically safe “container” or learning environment enhances people’s willingness to examine and discuss their frames and actions critically and openly.34–35“Psychological safety” means that students perceive the environment as predictable and secure enough to describe and scrutinize their thoughts, motivations, and goals. This feeling of psychological safety is not the same as feeling comfortable. Rather, psychological safety allows participants to take interpersonal risks36 and permits instructors to explore difficult topics. Third, instructors create an environment conducive to learning when they validate the trainee’s tacit desire to be respected as an aspiring member of the profession.22–24 The instructor may do this by granting trainees the benefit of the doubt and assuming that they have the basic aptitudes, training, and ambition to succeed.19 Tenets of family psychotherapy suggest that an explicit statement by the faculty to this effect is useful but must be matched by a respectful and curious approach to students’ problems and successes during the simulation.37 At the Center for Medical Simulation, we achieve this by posting the following “basic assumption”: “We believe everyone participating in activities at the Center for Medical Simulation is intelligent, well-trained, cares about doing their best, and wants to improve.”

2. Have Objectives in Mind

Formative assessment in debriefing relies on predetermined learning objectives and/or learner-generated objectives in three ways: 1) objectives provide the desired performance level against which actual trainee performance is compared, 2) objectives allow for clear feedback that describes the gap (below or above) between actual and desired performance level. Such feedback, if specific and actionable, is one of the strongest predictors of improved performance:21,38,39 3) objectives guide the instructor to develop short (2- to 5-minute) didactic lectures describing the current evidence base or best practices.

Effective objectives are specific, will likely be observed during the scenario, and are relatively easy to assess. Ineffective objectives are vague, difficult to observe, and relatively hard to assess (see Table 1). Occasionally, student concerns will come in the form of complaints or attributions about other people in the scenario. These should not be dismissed, but rather recast to foster productive inquiry; complaints reveal what is important to students.40

Table 1.   Effective and Ineffective Objectives to Support Formative Assessment
Ineffective (Too Vague) ObjectiveEffective (Specific) Objective
Good team leadershipSomeone articulates clear clinical priorities to focus team’s effort
Effective clinical management of the caseTeam performs a needle decompression quickly and follows with chest tube placement if indicated
Trainee concern: “The surgeon was a jerk!”(Reframed by the instructor): Know ways to acknowledge and diffuse another provider’s frustration (in this case the surgeon) and refocus his or her attention

3. The Debriefing

Debriefings are rewarding and interesting and lead to higher levels of retention when trainees actively think about, analyze, and discuss what happened.41 Debriefing usually includes the following steps: a reactions phase in which trainees “blow off steam” and the instructor gets a first glimpse of what is most concerning to trainees, an analysis phase in which the instructor and trainees discuss and analyze trainees’ performance, and a summary phase in which trainees distill lessons learned for future performance. The analysis phase is the primary occasion for formative assessment, but the reactions phase is crucial to setting the stage.9–13

Reactions Phase.  The main goal of the reactions phase is to allow trainees to express their initial emotional reactions to the simulation; a secondary goal is for the instructor to provide facts underlying the simulation so that trainees are not confused about what happened. As part of a formative assessment, the reactions phase offers precious insights into what was most exciting or troubling for trainees and allows the instructor to focus on her own and learner-centered objectives. Following the tenets of adult learning theory, the instructor should weave one or two of these learner-centered topics into later conversation or address them directly.33,42 Expressions of regret or exhilaration are common and are linked to trainee’s aspirational identities as clinicians. Instructors can allay trainees’ worries by demonstrating respect for them, by sharing information about how they compared to other teams, and by showing sincere interest in trainees’ thoughts and analyses.

The Analysis Phase.  The fulcrum of the formative assessment in debriefing is the analysis phase, which includes the four steps outlined in Figure 2. The analysis phase also usually includes a discussion to generalize lessons learned to a trainee’s real-world context.13,43 In an analysis phase geared to formative assessment, instructors: 1) note salient performance gaps related to the predetermined objectives (these can be performance decrements or increments), 2) provide feedback on the gap by describing what they observed in a respectful but direct manner, 3) investigate the basis for this gap by exploring the frames (and sometimes emotions) that contributed to the current performance level, and 4) help close (or, when trainees perform above expectations, help the group learn from) the performance gap by discussing principles and skills relevant to performance in the case.

Figure 2.

 Steps of formative assessment in medical education.

Consider again the example of the trauma team trying to manage the patient with a gunshot wound. Table 2 and the following text illustrate each step of the formative evaluation process. In Step 1 of the formative assessment depicted in Table 2, the instructor observes a gap between desired and actual performance. The instructor’s objective (and desired level of performance) is that trainees master the communication skills needed to organize and focus a team in the often stressful setting of trauma resuscitations. The instructor would like to see the team leader, Dr. Andre Allen (a pseudonym), declare a crisis explicitly, vocalize possible causes, and state an action plan. The actual performance she observes includes none of these, although some important clinical tasks are carried out. Next, the instructor shares her observation with the team to execute Step 2 of the formative evaluation process, providing feedback on the performance gap (failure to declare a crisis, possible causes, and an action plan). For feedback to be helpful it must be accurate and timely.21,38,39 Note that in Step 2, depicted in Table 2, the instructor expresses her concern directly, respectfully (i.e., without sarcasm, meanness, or sugar-coating) and from her first-person perspective. She then immediately moves to Step 3a, investigating the basis for the gap by eliciting the trainee’s perspective to uncover his frames (Step 3a).

Table 2.   Illustration of Four Steps of Formative Assessment in Debriefing
STRUCTURE OF FORMATIVE ASSESSMENT IN DEBRIEFINGDEBRIEFING DIALOGUE
StepDebriefer’s ThinkingDebrieferTrainees
  1. ATLS = advanced trauma life support; GSW = gun shot wound.

1. Observe gap between desired and actual performance.Desired performance: mastery of communication strategies to organize and focus a team under stress, declare a crisis explicitly; vocalize possible causes; state an action plan; timely decompression of the chest.
Actual performance: no coordinated effort by the team and no timely decompression of the chest.
  
2. Provide feedback on the performance gap.I want to focus trainees’ attention on key objectives and establish clinical consequences. Clinical consequences show why lack of explicitly organizing the group matters.To team leader (Dr. Andre Allen): As the patient abruptly decompensated and then lost vital signs, Andre, I didn’t hear an explicit statement of what could be causing it (either a tension pneumothorax or massive blood loss) and what the team’s immediate priority should be. I am concerned that not stating a plan may have contributed to the delay in decompressing the chest. It looked to me like the team did not focus on that single high-priority action. 
3a. Investigate basis for the performance gap.I want to start the detective process of formative assessment; what frames drove Andre’s actions?So I was thinking an “out loud action plan” would have been helpful. I wonder what you think about this?Well, honestly, I thought everyone who has taken ATLS knows that the most likely injury to a patient with a GSW to the chest would be a pneumothorax or hemothorax from large-vessel injury, especially when I stated there were decreased breath sounds on the left side of the chest and asked for a chest tube. The rapid decompensation made a tension pneumothorax an obvious cause we could address right away.
3b. Investigate deeper into the basis for performance gap.It sounds like an “invisible driver” of Andre’s performance was that he believed the cause of the decompensation was obvious to other team members. To Andre, it seemed everyone would know what to do.
By sharing evidence that other team members were not on the same page, I may start the process of closing the performance gap by provoking or disconfirming Andre’s thinking on this point.
I hear you saying that a tension pneumo- or a hemothorax seemed like the obvious thing to focus on, yet as I watched the case, I saw team members working on many other tasks and not decompressing the chest. Let’s take a look at the video. (Turning to other team members after showing a short clip of video) What was on your mind? Tech: I figured Dr. Allen would take care of the pneumo, so I focused on cutting the clothes off.
Surgeon: Well I was worried about a hemothorax and blood loss, so I wanted blood in the room in case we needed it. I thought Andre would needle the chest and that I should focus on getting blood ready for transfusion before preparing the chest tube.
Nurse: I got busy hanging fluids and then was focused on the deteriorating vital signs. I forgot about the chest tube at that moment.
4a. Help close the gap through discussion and didactics.Through discussion: I want to highlight the contrast between Andre’s frame (“It’s obvious”) and each team-member’s focus. This is a way to start generating new frames and actions to improve performance next time.To other team members: I hear Andre and others saying the pneumo or hemo was on their minds, yet no one needled the chest or got the chest tube. What would it have taken to get you focused on these tasks?Nurse: It would have helped focus us on priorities to hear Andre say: “I think the patient has a tension pneumothorax and the first priority is we to needle the chest…” We all know about GSWs and tension pneumothorax, but in the heat of the moment it is really hard to process all the information and get things done!
4b. Help close the gap through discussion and didactics.Through discussion and establish:
? What the team knows about stress and cognitive processing
? How to establish a shared mental model
OK, you have some great ideas about why this situation was hard and suggestions about how to deal with it. A couple good ideas I heard were: 1) It’s hard to think in the heat of the moment and 2) state priorities out loud in situations like that so we’re all thinking along the same lines. Is that a fair summary?Yes, those things would have been good, but there’s another thing. Usually we get everything done smoothly and we don’t have to talk a lot. It was this situation with all the urgency and noise that made us not perform as well as usual.
4c. Help close the gap through discussion and didactics.Through didactics:
I’m going to share my previously rehearsed short “lecturette” of how to organize and direct a team under stress.
I’d like to say a little bit about what the literature tells us about stress and team performance. How does the event manager “defibrillate” a team and make sure the team grasps your priorities? Well, when people are under stress, you need to give clear, simple directions.
That is because when autonomic arousal increases to a certain level, cognitive efficiency decreases. People do not process at a very high level when stressed. You cannot assume they will put 2 + 2 together the same way you do. Another advantage of an explicitly stated plan is that it provides the basis for a shared mental model of the problem. A shared mental model helps people all focus on the same priorities.
 

When the instructor learns that the trainee thought it would be completely obvious to his colleagues that a pneumo- or hemothorax was the problem, she further investigates the performance gap (depicted in Step 3b of Table 2). The instructor hypothesizes that others were not focused on decompressing the chest (although they may well have been aware this was a top priority). She shows a video clip (where no one is doing anything related to decompressing the chest) and uses this to elicit other team-member frames and challenges the team leader’s frame that everyone knew what to do. Once the team and its leader realize that most members were focused on issues other than chest decompression, the team’s attention is turned toward closing the performance gap (Steps 4a and 4b of Table 2).

Since knowledge retention is enhanced by engagement, participating in a discussion rather than didactics is a more potent way for learners to develop prescriptions for what to do in the future. The instructor should attempt to elicit the group’s relevant knowledge and work with them to synthesize ideas for improving performance (Steps 4a and 4b of Table 2). In Step 4c, the instructor augments ideas generated by the group with evidence from the existing literature about stress and team behaviors. If knowledge deficits contributed to the performance gap, the instructor could provide brief didactics targeted to immediate learning needs. It is essential for instructors to design and pilot scenarios that link learning objectives with the expected actions in the simulation; they can also review relevant studies to be prepared with short didactic pieces like the one in Step 4c. Once the analysis phase is complete, the debriefing moves on to the summary phase.

The Summary Phase.  In accordance with Kolb’s theory of experiential learning,7 the purpose of the summary phase is to distill lessons learned from the debriefing into memorable rules-of-thumb or concepts that trainees can take with them to improve their practice. This step helps codify the insights developed in the formative assessment through the analysis phase. A simple and effective approach is to have trainees review the simulation and debriefing by asking, “What went well in that simulation? What things would you like to repeat in the future?” After allowing discussion, the next question might be, “Given another simulation like this, or a similar real situation, what would you do differently?” Health care trainees often are bashful about citing things they did well, so the instructor should be prepared to name and illustrate how the team was effective.

Conclusions

We have presented a model of formative assessment with an emphasis on feedback to be used in debriefing EM simulations. Effective debriefings shed light on the underlying drivers of performance, and with these drivers illuminated, instructors can target discussion and teaching to what is meaningful to learners. While the four-step approach to formative assessment is presented in the context of postsimulation debriefings, the approach can also be used in short ad hoc learning conversations immediately after a patient encounter in the ED to encourage reflection and promote deeper learning.

Ancillary