Direct Observation of Residents in the Emergency Department: A Structured Educational Program
Address for correspondence and reprints: Michele L. Dorfsman, MD; e-mail: email@example.com.
Objectives: The objective was to describe the implementation of a program of structured direct observation of emergency medicine (EM) residents during clinical shifts in the emergency department (ED).
Methods: The authors developed a program in which an observer spent 4 to 5 hours with each resident, without intervening in the clinical encounters. A structured data form was developed to document the resident’s performance in a number of defined clinical areas relevant to patient care and mastery of the core competencies. Individual strengths and weaknesses were noted, and the observer provided directed feedback at the end of the session.
Results: Over an 18-month period, 32 EM residents were observed during their ED shifts. The sessions not only provided specific information on individual residents’ performances, but also identified areas where the residency program curriculum could be enhanced and provided a means of assessing mastery of the core competencies. In addition, the program provided an opportunity to give detailed and timely directed feedback to residents. Both residents and attending staff found the sessions acceptable and useful.
Conclusions: Implementation of a structured direct observation program was feasible and well received and provided insight into the strengths and weaknesses of residents both individually and as a group.
Assessing resident performance poses many challenges. Most current efforts focus on subjective feedback and structured evaluations after each shift or rotation, but these are limited by recall bias. Direct real-time observation of resident performance could overcome this limitation and provide an opportunity for a structured assessment of the core competencies. It could also provide a more objective and more detailed view of resident performance.
The American Board of Medical Specialties has delineated six core competencies as essential to medical practice: patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice. These six competencies were subsequently endorsed by the Accreditation Council on Graduate Medical Education (ACGME) in 1999. In 2001, the ACGME mandated the integration of these competencies into residency program curricula and required that they be assessed by residency faculty. In 2005 at its academic assembly, the Council of Emergency Medicine Residency Directors (CORD-EM) discussed the integration of the six competencies into emergency medicine (EM) training programs.1 The group concluded that many of these competencies could be assessed through the use of objective structured clinical exams (OSCEs), standardized patients, simulation, or direct observation. OSCEs and standardized patients require considerable resources, and the use of simulation requires specialized equipment and expertise. Direct observation would be expected to provide an additional means of assessment of resident performance, and it has in fact been shown in preliminary work to be a useful modality.2
Direct observation is an attractive methodology for resident assessment, but depends critically upon the experience, background, and availability of faculty members to perform the observation. We therefore developed a program of structured direct observation that could be used in a reasonably uniform way by multiple faculty observers in our emergency department (ED). It could also potentially be adapted for use at other EM training programs.
We arranged to have resident performance observed by an attending physician who was not directly involved in patient care and was free from other clinical duties, allowing for a prolonged observation period that involved multiple patient encounters. In this way, we hoped to better identify specific strengths and weaknesses of individual residents and to provide residents with directed and timely feedback regarding these areas.
Development of the program
This reporting was reviewed by the Institutional Review Board of the University of Pittsburgh and was designated as “exempt.”
Before we initiated this formal faculty/resident “shadowing” program, direct observation had been performed in our ED informally, but only sporadically. We wished to have a means of identifying areas for improvement or remediation for each individual resident. We also hoped to strengthen the residency program’s overall assessment of the six competencies. The shadowing program was not intended to enhance usual bedside teaching, which was still provided by the ED attending caring for the patient. The faculty observer was to spend 4 to 5 hours with a single resident during protected nonclinical time. Without the distraction of the usual clinical responsibilities of a regular shift, the observer would thus have an uninterrupted period in which to observe all of the resident’s activities in the ED.
Our facility is an academic, urban, adult tertiary care referral hospital with a volume of over 50,000 ED patients per year. The ED is staffed by 25 attending physicians, with single coverage for 7 hours a day, double coverage for 9 hours a day, and triple coverage for 8 hours a day. There are 9 to 12 house staff working daily, of whom 4–5 are EM residents. Residents typically see 1.0–1.5 patients per hour. Twenty-four hour consultation is available in all major specialties. Our facility is a Level 1 trauma center and has an inpatient toxicology service staffed 24 hours a day by EM faculty who are board-certified toxicologists.
The faculty observer was provided with protected nonclinical time to perform the observation shifts. The shifts were scheduled well in advance to ensure that the resident being shadowed did not have medical student teaching responsibilities that day.
We adapted the CORD-EM Standardized Direct Observation Assessment Tool (SDOT)3 for use in the documentation of a resident’s multiple patient encounters during a single observation period. This instrument has not been validated, but it appears to have good interrater reliability and was felt to be appropriate for our purposes of providing resident feedback and identifying strengths and weaknesses. For each listed skill or action, our data form identifies one to three of the corresponding competencies (Table 1) and provides an area to assess and record performance in each category. There is additional space in each section for individualized comments, and a final space is designated for general comments. This form was designed to be discussed with the resident during the postobservation session.
Direct Observation Data Collection Form
| ||History/Data Gathering|| || || || || |
|IC/P||Introduces self|| || || || || |
|IC/P||Establishes rapport|| || || || || |
|IC/P||Demeanor/bedside manner|| || || || || |
|IC/P||Communicates with patient in layman’s terms|| || || || || |
|PC||Gathers data from all available sources, systematic|| || || || || |
|PC||Reviews nursing notes|| || || || || |
|PC||Documentation notes while gathering data|| || || || || |
|IC/P||Respectful of patient confidentiality and privacy|| || || || || |
| ||Physical Examination|| || || || || |
|PC||Complaint-oriented physical examination and appropriate general exam|| || || || || |
|PC||Handwashing|| || || || || |
|PC||HEENT|| || || || || |
|PC||Eye|| || || || || |
|PC||Neck|| || || || || |
|PC||Lungs|| || || || || |
|PC||Cardiac|| || || || || |
|PC||Abdomen|| || || || || |
|PC||Musculoskeletal|| || || || || |
|PC||Neurologic|| || || || || |
|PC||Cranial nerves|| || || || || |
|PC||Cerebellum|| || || || || |
|PC||Skin (including expose patient)|| || || || || |
|PC||Vascular|| || || || || |
|PC||Pelvic/GU/rectal|| || || || || |
| ||Synthesis/DDX|| || || || || |
|MK/PBL||Explains pathologic basis for management when asked (prompt)|| || || || |
|MK/PBL||Knows where to look for more information|| || || || || |
|MK/PBL/IC||Attending presentation appropriate to condition/complexity|| || || || || |
|MK||Discuss appropriate differential diagnosis|| || || || || |
|MK/PBL||Evaluation of need for testing|| || || || || |
|MK/PC||Discuss and develop appropriate treatment plan|| || || || || |
|MK/PC||Discuss appropriate disposition|| || || || || |
| ||Management|| || || || || |
|MK/PC||Care appropriate for urgency, critical vs. noncritical|| || || || || |
|MK/PC||Procedural skills|| || || || || |
|MK/PC||Consent, risks, benefits|| || || || || |
|IC/PC/P||Communication with ancillary staff|| || || || || |
|IC/PC/P||Communication with consultants, colleagues|| || || || || |
|P||Conflict resolution/patient satisfaction|| || || || || |
|IC||Update to family, patient|| || || || || |
|PC||Written orders for labs/therapeutics|| || || || || |
|PC||Prioritization of initial patient evaluation, acuity, time|| || || || || |
|PC||Prioritization of reevaluation of patients|| || || || || |
|SBP||Plans patient work-up in the context of health care system limitations (staffing, consultants, testing availability)|| || || || || |
|SBP||Assessment of patient’s social constraints/needs|| || || || || |
|PC||Follow-up on diagnostics|| || || || || |
|MK||Decisions based on data/interpretation of diagnostics|| || || || || |
|PC||Documentation in a timely fashion|| || || || || |
|PC||Documentation at an appropriate time|| || || || || |
| ||Disposition|| || || || || |
|IC/PC||Discuss disposition with patient/family|| || || || || |
|PC||Carry out disposition plan in a timely fashion, notify attending|| || || || || |
|SBP||Appropriate follow-up plan (social, financial)|| || || || || |
We chose to initiate the program by observing second-year EM residents, because they are already quite familiar with managing patients in the ED setting, but are still at a stage where they can significantly improve their clinical and organizational skills. We envisioned expanding the program to include our first- and third-year residents, as well, to allow residents to have multiple shadowing shifts throughout their training.
We introduced the program to our faculty at our departmental faculty meeting and sent e-mails to those who were not present at the meeting. We explained to the faculty that the observer would spend time observing all of the resident’s activities, including the presentations to the attending physician working that day. The observer would not intervene in the patient care encounter and would not be managing the patient. We also wanted faculty members to know that the observer did not plan to intervene if the resident provided history or other data that were inaccurate or based upon misinterpretation. We asked the faculty to precept the residents just as they would if the observer were not present.
We prepared the residents as a group for this experience with an introductory e-mail explaining that the direct observation program was being introduced as a routine part of the training program, so that there would be no misunderstanding and no resident would feel singled out for attention. A few days prior to each observation shift, we e-mailed a reminder to the resident who was to be observed, with a few more procedural details and confirmation of the time the session would take place.
Each observation session was anticipated to include resident encounters with several patients, depending upon patient complexity and time constraints. We expected each session to last several hours, allowing for multiple opportunities for the residents to perform clinical tasks so that the observer could see how they performed in response to different clinical problems and varied levels of patient acuity. The observer planned on following the resident from patient to patient, watching him or her perform histories and physical exams, communicate with the patient and family, gather data from other sources (such as nursing notes, electronic medical records, family members, nurses, and paramedics as appropriate), present cases to the ED attending, order labs and other diagnostic tests, perform procedures, and complete documentation and patient disposition. During a patient encounter the observer would be introduced to the patient by name without further explanation, allowing the resident to proceed as he or she would normally. If the patient had any question about why the observer was present, the observer was to explain that the resident was spending time that day with a faculty observer as a routine part of our training program.
Interpretation, Feedback, and Documentation
The observer was to ask the residents to verbalize their thoughts regarding differential diagnosis and decision-making, but was not to comment on the evaluation and management plans as they were developed by the resident or the ED attending. The observer intentionally did not plan on serving as the ED attending managing the case, because observation of resident presentations could provide insight into the resident’s understanding of the case and allow the observer to see how the resident communicated with the ED attending. After observing several patient encounters without providing comments, the observer was to provide some immediate feedback regarding efficiency tools or decision-making, but specifically avoid giving feedback on data gathering, bedside manner, professional communications, or physical examination until the end of the observation shift. If an ethical issue arose, the observer would intervene in the patient–resident interaction, but only when it was absolutely necessary.
The observer would compile a data sheet on each resident during the patient encounters, and at the end of the 4- to 5-hour observation session would spend 15–30 minutes in a private location with the resident, detailing observations made throughout the session and providing feedback. The resident would then be given the opportunity to provide feedback on the experience and to ask questions. The data from each observation session were to be compiled in an electronic form to be shared with the residency program director and associate program director.
Over an 18 month period, 32 EM residents were observed during their ED shifts. The observer spent 4 to 5 hours with each resident, and four to seven patient encounters were observed during a session. Most of the residents observed were second-year EM residents. Three of the residents were observed during the latter part of their first year, and 1 was observed at the beginning of the third year because of scheduling constraints.
Our direct observation program was well accepted by faculty and residents alike. The residents almost universally said they had found the experience both educational and enjoyable and inquired about doing some additional sessions if time was available.
The observers were able to complete the data forms (Table 1) while still being able to pay attention to the clinical encounters. A summary of the data from 25 of the observation sessions is provided in Table 2. The data form was revised after the first seven sessions were completed, explaining why Table 2 includes only 25 of the 32 residents observed. Three different observers have used the data collection form and found it easy to use and complete.
Summative Data from 25 Direct Observation Sessions
| Introduces self||16||7||2|| |
| Establishes rapport||17||5||3|| |
| Demeanor/bedside manner||16||5||4|| |
| Communicates with patient in layman’s terms||8||12||5|| |
| Gathers data from all available sources, systematic||12||5||8|| |
| Reviews nursing notes||16||6||3|| |
| Documentation notes while gathering data||14||8||3|| |
| Respectful of patient confidentiality and privacy||7||14||4|| |
| Complaint oriented physical examination and appropriate general exam||8||10||7|| |
| Handwashing||4||2||19|| |
| HEENT||5||14||6|| |
| Eye||5||13||3|| 4|
| Neck||2||12||5|| 6|
| Lungs||11||13||1|| |
| Cardiac||9||16||0|| |
| Abdomen||10||10||5|| |
| Musculoskeletal||6||14||5|| |
| Neurologic||3||7||15|| |
| Cranial nerves||1||5||13|| 6|
| Cerebellum||0||4||12|| 9|
| Skin (including expose patient)||6||13||6|| |
| Vascular||4||11||4|| 6|
| Explains pathologic basis for management when asked (observer prompt)||6||15||2|| 2|
| Knows where to look for more information||4||14||1|| 6|
| Attending presentation appropriate to condition/complexity||7||11||7|| |
| Discuss appropriate differential diagnosis||6||16||3|| |
| Evaluation of need for testing||7||16||2|| |
| Discuss and develop appropriate treatment plan||8||14||3|| |
| Discuss appropriate disposition||8||16||1|| |
| Care appropriate for urgency, critical vs. noncritical||5||17||2|| 1|
| Procedural skills||3||4||0||18|
| Consent, risks, benefits||1||0||2||22|
| Communication with ancillary staff||11||9||5|| |
| Communication with consultants, colleagues||6||10||2|| 7|
| Conflict resolution/patient satisfaction||3||13||1|| 8|
| Update to family, patient||9||15||1|| |
| Written orders for labs/therapeutics||6||17||2|| |
| Prioritization of initial patient evaluation, acuity, time||6||13||4|| 2|
| Prioritization of reevaluation of patients||5||14||6|| |
| Plans patient workup in the context of health care system limitations (staffing, consultants, testing availability)||3||21||1|| |
| Assessment of patient’s social constraints/needs||3||17||3|| 2|
| Follow-up on diagnostics||6||14||5|| |
| Decisions based on data/interpretation of diagnostics||7||16||1|| 1|
| Documentation in a timely fashion||2||12||7|| 4|
| Documentation at an appropriate time||2||14||5|| 4|
| Discuss disposition with patient/family||12||12||1|| |
| Carry out disposition plan in a timely fashion, notify attending||8||16||1|| |
| Appropriate follow-up plan (social, financial)||3||17||0|| 5|
Ideal Observation Shift
Sessions of 4 to 5 hours provided enough time to observe multiple patient encounters, seeing many of these through to disposition, as well as time for a 15- to 30-minute period for feedback and debriefing at the end of the session. The ideal shift for these sessions seemed to be midday or afternoon-evening, when there tended to be higher patient volumes and little potential downtime for the residents. Residents could thus choose which patient to see (we do not have assigned rooms at our program) based upon acuity and waiting times and could care for several patients at a time, providing an opportunity for assessment of multitasking skills. The sessions worked best when they started at the beginning of the resident’s shift, providing the opportunity to evaluate multiple patients together from start to finish, without the resident already being involved in other cases at the start of the observation.
Ideal Patient Type for Observation
There was no single type of patient that seemed to be ideal for the resident to care for while being observed. However, it was sometimes difficult to maintain the role as an observer when the patient was critically ill, as the attending on duty would often ask the observer to offer an opinion on the case. This tended to occur more often when the attending was a junior faculty member. In addition, the resident was less able to operate independently when seeing a critically ill patient. However, seeing both low- and high-acuity patients provided an opportunity to watch residents perform procedures, provide discharge instructions to some patients, arrange for admission, speak to admitting physicians on the phone, and interact with consultants. Evaluating patients with severe dementia or altered mental status provided an opportunity to evaluate residents in their ability to gather data from sources other than the patients themselves.
Identification of Areas of Strength and Need for Improvement
We were able to identify individual strengths and weaknesses during the observation sessions. A number of the residents have reported that their efficiency has improved and that the quality of their history-taking and physical exam skills has improved as a result of efforts they undertook in these areas after receiving the observer’s feedback. We were able to observe concrete examples of individual difficulties involving communication and interpersonal skills that had previously been difficult or impossible to identify adequately. For example, some residents presented a detailed diagnostic and management plan to the patient before presenting the case to the attending. This led to certain expectations on the part of the patient, who then had to be informed when the attending disagreed with the resident’s plan.
Histories and Physical Examinations
Among the core competencies that could be effectively assessed in these sessions were patient care and interpersonal communications. Regarding history-taking skills, individual residents exhibited marked differences in their abilities to obtain information from patients (Table 2). The specific measures that were assessed included review of the nursing notes; use of the electronic medical record that was available on each patient; data gathering from family members, transfer records, or emergency medical services providers; use of layman’s terms when communicating with patients and family members; and use of jotted notes for later review and to assist in final documentation, which at our institution consists of telephone dictation.
The components of the musculoskeletal, neurologic, and abdominal parts of the physical examination varied quite a bit from resident to resident (Table 2). There were differences among residents in the detail with which a physical exam was performed, depending in part on each patient’s clinical problem. Some residents performed a full physical exam on every patient, including the low-acuity ones. Others performed a focused, complaint-specific physical examination. Some residents had a disorganized and less-than-systematic way of performing a neurologic examination. Most residents tended not to have a routine way of testing the cranial nerves. Some residents would not examine patients unless they had been completely undressed, while others would examine patients fully clothed. Several of the residents did not expose the abdomen when examining the patient, but palpated through the patient’s clothing or gown.
In assessing the professionalism and interpersonal communication competencies, we observed that residents, in general, approached patients in a consistent manner, regardless of differences in patient personality and acuity of illness. The specific behaviors observed were how the residents introduced themselves; establishment of rapport; position in the room; respect for patient confidentiality and privacy; demeanor and bedside manner; the use of layman’s terms; providing the opportunity for questions; and behaviors such as cutting the patient off, allowing the patient to provide the chief complaint and history of present illness, and jumping to conclusions. Eye contact and response to nonverbal behaviors were also noted.
In their presentations to attendings, residents varied in terms of length of presentation, amount of detail, and the extent of plans for further evaluation and treatment. Some of this seemed to be attending-dependent and might have been due to the resident’s previous experiences with individual attendings.
Observing the residents’ presentations to the attendings allowed us to assess their medical knowledge, problem-based learning, systems-based practice, and again their interpersonal communications and professionalism. When prompted by the observer, the residents were consistent in their abilities to synthesize an adequate differential diagnosis. Most residents were easily able to explain the rationale for testing and develop a disposition plan. They varied in the length and complexity of their presentations to attendings. A few individuals had trouble with one or two areas in the synthesis/differential diagnosis category, but none were consistently deficient.
Patient care, interpersonal communications, and professionalism were assessed by observing the resident implementing the management plan. Individual efficiency skills were varied. Some residents were able to streamline their activities, such as checking on the status of labs and x-rays on multiple patients at the same time or reevaluating patients in the same area of the ED at the same time. Others made several (often lengthy) journeys back and forth to different areas of the ED over and over again because they focused more on single tasks. Some residents were very disposition-focused and efficiently managed both admissions and discharges. Many were not as efficient, and occasionally the attending completed disposition tasks before the resident did. For several residents, we were able to identify focused areas for improvement, such as following up on diagnostics and periodically updating patients and family members.
The medical knowledge competency was also assessed during the presentation to the attending physician on duty, as well as when the resident interpreted the diagnostic tests. As a group the residents were able to independently develop their own plans and interpret their own diagnostic tests.
At our institution documentation is performed by telephone dictation. Only two of the residents we observed incorporated documentation into their routine without outside prompting. Many were more conscientious about it when prompted, but most admitted that they were more likely to continue picking up new patients than to stop and complete documentation after disposition was determined. This resulted in residents staying in the ED beyond their scheduled shifts. We helped the residents identify ideal times for dictation so they could complete documentation in a timely fashion and avoid staying late after shifts. Interestingly, some residents mentioned that they did not want to “waste” the observer’s time by documenting while being observed, and therefore they did not spontaneously begin this task until prompted.
The feedback provided to each resident was based upon the observations made throughout the session. We discussed subtleties in their tone of voice, nonverbal behavior, and bedside manner, as well as the amount of detail in their physical exams, their efficiency, and how many times throughout the session they had introduced themselves or washed their hands. The residents found this helpful and were grateful to have had this extended period in which an attending’s attention was focused only on them. Many residents asked if they could have additional shadowing shifts scheduled in the future because they found this direct, immediate feedback helpful.
Reactions to the Program from Patients and Family Members
Although we did not measure this specifically, the program seemed to be well accepted. There were surprisingly few questions from patients and family members regarding why the observer was present in the room with the resident. We received no complaints regarding this program from patients or family members.
Reactions to the Program from Faculty
Some faculty members were initially uneasy having an observer present while developing a plan with the resident. The presence of an observer did lead to discussions between faculty members about individual differences in practice patterns. Some of the faculty mentioned that being observed reminded them to identify “teachable moments” and that they felt as if they taught more when this was the case. There were a few situations in which the observer had concerns that the resident had missed important historical or physical exam findings or was following a diagnostic pathway that did not seem warranted. The observer in these cases found it extremely difficult not to step in and had some concerns that there would be patient care issues without intervention by the observer, but the attendings in every case were able to identify the missed findings after hearing the story or sometimes after seeing the patient themselves.
There were a few instances in which the observer needed to step out of the observer role. On one occasion the nurses called the observer to secure the airway of a trauma patient when the attendings on duty were occupied with other critically ill patients. After more than a year’s exposure to the direct observation program, however, faculty, nursing, and administration have become more accustomed to the observer’s role during these observation shifts and this has not been a continuing issue. On another occasion, the observer was forced to step in and involve the ED attending during one patient encounter when the resident and a family member began arguing about the patient’s treatment plan. The observer was also forced to step in during an encounter when the patient became offended by a resident’s comments because of a misunderstanding. In each of these cases, the situations were defused by the observer or the ED attending caring for the patient, and we were able to continue the direct observation shift without further difficulty.
The hectic and busy environment in the ED often precludes extended observation of each resident by faculty members. It is easy to see how bedside teaching and resident assessment could suffer under these conditions.4 An individual teaching physician is pulled in many different directions. There are obligations to teach medical students and residents, provide patient care, accept referrals, provide direct medical oversight to emergency medical services, and respond to the concerns of nurses and other staff members, yet there is great benefit in being able to watch a resident uninterrupted during a patient care encounter. Direct observation of resident performance is a unique way to assess residents both individually and as a group as well as to identify areas of strength and areas for improvement. Previous direct observation programs have found it useful to provide the observer with protected teaching time, combining observation with teaching.2,5,6 We designed our program in a similar fashion to provide the observer with protected time to focus solely on the resident being assessed, but the primary goal in this case was observation and feedback rather than didactic teaching.
Although many educators may on occasion be able to successfully observe residents perform their histories and physical exams, there is an advantage to observing a resident from the very beginning to the end of a physician–patient interaction, watching the resident present that patient to a teaching physician, and observing the resident manage multiple patient encounters. Nuances of tone and body language may engage a patient or turn the patient against the resident. Patients may not understand the reasoning behind a plan of care or may not understand the details and importance of their discharge instructions. The resident may be in touch with the nonverbal or verbal cues given by the patient or may simply overlook them, leading to lapses in patient–physician communication. By listening to the resident present the case to another physician, misinterpretations can be identified and the resident’s interpretation of the history and physical examination findings can be assessed. When they are speaking to consultants or admitting physicians, residents may not be able to provide a cogent account of the reasoning behind the decisions made. Furthermore, a resident may not yet have developed a systematic way to evaluate and treat multiple patients simultaneously. Ideally, a teaching physician would best be able to appreciate these behaviors by observing a resident for an extended period of time without interacting or interfering in the encounters.
Emergency physicians at other institutions have developed programs designed to combine bedside teaching, didactics, and clinical evaluation exercises by an attending physician who is free of clinical duties.2,6 Our goal was not to provide additional bedside teaching during this program, but to focus on observing all of the EM residents’ clinical activities during a typical shift. This entails observation of the resident’s real-time performance in the ED, from the initial resident–patient encounter through the multiple steps in evaluation and management to the final patient disposition. Ideally, multiple patient encounters are observed during the session.
Graduate medical education programs are currently expected to provide evidence of resident knowledge of the core competencies and to incorporate competency-based assessment into resident evaluations. EM is uniquely suited to be a leader in the development of methods of assessment and program improvement because of our high volume of acute patient interactions.7 Incorporation of a direct observation program is one of the only ways to qualitatively assess performance in the competencies of professionalism and interpersonal communication skills while simultaneously measuring efficiency and multitasking skills. Observation in real time during a shift provided us with the opportunity to assess more than just medical knowledge or history and physical examination skills. Observing presentations and interactions with consultants, admitting physicians, and ED staff provided an additional method for assessing professionalism and communication skills, which is difficult to do in a standardized patient or simulation environment.
Direct observation programs have been described previously in the EM literature.2,5,6,8 These programs have focused on specific aspects of resident performance, such as interpersonal relationships5 or death notification skills,8 or have combined observation with didactic and bedside teaching.2,6 The program described here sought to incorporate some of the successful features of these programs while concentrating on the assessment of performance and incorporating assessment of the core competencies, with less focus on bedside teaching. The unique aspect of this program is that residents are observed continuously over multiple patient encounters in their usual work setting without interference and then receive individualized feedback by an attending physician who is not occupied by other clinical responsibilities.
This program allowed us to identify strengths and weaknesses in individual residents during the observation sessions and allowed us to provide timely and specific feedback. We were able to identify concrete areas in which each resident could focus his or her energy to improve performance. The program has been perceived by the residents as extremely helpful and nonthreatening. Residents have incorporated our suggestions into practice, and many have asked us to schedule more shifts like this with them. In addition, this program provides our residency program with an additional means to assess the core competencies. We are able to assess all of the competencies simply by watching a resident work during a clinical shift. We have also been able to make use of these observations. For example, based upon some of our observations, we have instituted individualized remediation plans for selected residents.
We developed this program in a way that would be adaptable for use by multiple observers and potentially by other residency programs (depending on faculty availability and funding, however). Providing the observer with protected nonclinical time for observation allows for an in-depth observation session. At the residents’ request, we have begun to do additional shifts with interns near the end of their first year, in the hope that we will be able to provide them with more opportunities to be directly observed throughout their training. Several of the residents have suggested adding shifts in which they shadow an attending, observing how attendings are able to efficiently manage their time.
Direct observation performed in this manner is inherently subjective and is subject to the biases of each observer. Ideally, multiple observers would have a similar opportunity to observe a single resident, and it would be important to determine how much interobserver agreement exists when the observation is performed in this manner.
The residents’ behaviors are subject to the Hawthorne effect (the phenomenon of behavior changing simply by being observed). However, during these sessions we were able to identify several encounters in which interpersonal relations were an issue, despite the Hawthorne effect. We believe that by spending an extended period of time being observed, the residents gradually got used to the idea and performed as they would if the observer were not there.
We did not ask the residents or faculty to perform anonymous before and after surveys of their feelings regarding the shadowing program. Perhaps we could have gained more insight from both residents and faculty had we done this from the outset, rather than collecting comments after the sessions that were not anonymous. We also did not obtain any quantitative data on performance before and after the sessions, such as performance on evaluation of a standardized patient or performance on a written test.
There are advantages and disadvantages to having a single observer engage in all of the shadowing sessions. We were able to better assess differences between residents and to note trends, because a single observer performed all of the sessions. Shadowing sessions are very time-consuming, however, and the observations are obviously subject to biases in terms of education and clinical practice. Further studies of interrater reliability will be helpful in establishing the reproducibility of these findings.
A direct ED observation program of EM residents is feasible and accepted and provides insight into the strengths and weaknesses of our residents both individually and as a group. Arranging for the observer to have protected time to be free of other responsibilities, including clinical care, allows for a more in-depth assessment of resident performance than is otherwise possible. Direct observation also provides a means to assess the core competencies in real time without using standardized patients or an artificial environment. Finally, it provides residents with specific and immediate feedback and allows them to immediately incorporate that feedback into practice.
The authors thank Donald M. Yealy for his support in preparation and review of the manuscript.