Academic Emergency Medicine 2011; 18:S71–S78 © 2011 by the Society for Academic Emergency Medicine
Objectives: Many factors affect the clinical training experience of emergency medicine (EM) residents, and length of training currently serves as a proxy for clinical experience. Very few studies have been published that provide quantitative information about clinical experience. The goals of this study were to determine the numbers of clinical encounters for each resident in emergency department (ED) rotations during training in a 3-year program, to characterize these encounters by patient acuity and age, to determine the numbers of encounters for selected clinical disorders, and to assess the variation in clinical experience between residents.
Methods: This was a retrospective analysis of the ED clinical and administrative databases at two hospitals that provide EM training for a southeastern U.S. EM residency program. Data were gathered for three complete cohorts of residents, with entering years of 2003, 2004, and 2005, so the total study period was 2003–2008. ED clinical encounter information included hospital training site (tertiary or community), postgraduate year (PGY) of the resident, patient triage acuity reflected by the Emergency Severity Index (ESI); patient International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnostic code; and patient age group.
Results: There were 25 residents with 120,240 total ED clinical encounters from 2003 to 2008. The median number of ED clinical encounters for a resident during his or her training was 4,836 (range = 3,831 to 5,780), based on a maximum of an 80-hour work week, and 24 or 25 four-week blocks of EM rotations. Overall, clinical encounters increased by 30% from PGY 1 to PGY 2, and another 14% from PGY 2 to PGY 3. There was 30% to 60% variation in clinical encounters between individual residents. Variability was most prominent in the care of children and in the care of time-sensitive critical illness. Resident encounters with lower-acuity problems during training were much less than the anticipated lower-acuity burden during practice. Additionally, residents did not encounter some high-risk conditions clinically during the study period.
Conclusions: Methods should be developed to decrease resident variance in both numbers and types of clinical encounters and to provide curriculum supplementation for individuals and for the entire residency cohort in areas that are important for the clinical practice of EM, but that are rare or not encountered during residency training.
Emergency medicine (EM) resident training is a focus of national and international discussion.1–3 Many factors can affect the clinical experience of EM residents, such as emergency department (ED) census, trauma types and volume, basic ED characteristics (tertiary care, inner city, community, rural), overcrowding, duty hour restrictions, and length of training.4–6 The Residency Review Committee for Emergency Medicine (RRC-EM) sets training standards for some of these variables, but none of the variables measure the individual resident’s experience. Length of training and ED rotations currently serve as a proxy for clinical experience. Very few studies have been published that provide quantitative information about the ED clinical experience of individual EM residents during training.7–9
The objective of this study was to quantify individual and collective EM resident clinical ED experience during all ED rotations in an RRC-approved residency program with two participating training hospitals, over 3 years of resident training. The goals of the study were: 1) to characterize the resident clinical experience in the ED with regard to numbers of clinical encounters and encounters based upon patient acuity, sex, and age; 2) to determine the numbers of ED encounters over 3 years for selected disorders identified by the authors; and 3) to assess the variation in clinical experience between residents.
This was a retrospective analysis of the ED clinical and administrative databases at one tertiary hospital and one community hospital from 2003 to 2008. The study was exempted by the institutional review boards of both hospitals. Since all residents had graduated from the program at the time the study began (2009), and since the data had no effect upon any resident’s training evaluations, neither institutional review board requested resident consent.
Study Setting and Population
This southeastern U.S. program uses two teaching hospitals to provide 3 years of EM residency training. The residency program is fully accredited by the Accreditation Council for Graduate Medical Education (ACGME) RRC-EM. The same residency program director was in place for the entire study period.
Tertiary Hospital. During the study period, the tertiary hospital was an academic medical center with a Level I trauma center and stroke center, with the full spectrum of residency and fellowship training programs. EM faculty have academic appointments in the affiliated medical school. Most patients in the tertiary hospital ED are evaluated primarily by EM and rotating house officers or nurse practitioners (NPs), with just a small percentage of patients evaluated and treated only by the supervising EM attending physician. NPs provide a large proportion of the lower-acuity care. The yearly adult ED census at the tertiary hospital was approximately 65,000 ED visits during the study period. The small pediatric ED did not participate in EM resident training. Between the hours of 11 pm and 7 am, children 15 years and under were triaged to the main ED and were evaluated primarily by pediatric residents under the supervision of EM faculty. Major pediatric trauma patients, however, were always triaged into the main ED and were cared for by EM residents and the supervising ED attending physician.
Community Hospital. The community teaching hospital was a Level II trauma center until 2005 and a Level I trauma center in 2006, with a large pediatric ED staffed by pediatric emergency physicians (EPs). The community hospital has no independent residency training programs, but provides training for the tertiary hospital’s programs in EM, internal medicine, general surgery, orthopedics, obstetrics and gynecology, and otolaryngology. The ED is staffed by a private, democratic group, some of whom have clinical appointments in the medical school. Most patients are seen primarily by the ED attending physician or physician assistants (PAs). There were no residents other than EM residents in the adult ED and only a few rotating pediatric residents in the pediatric ED. ED attendings and PAs provide lower-acuity care. The yearly census was 75,000 adults and 45,000 children. The vast majority of pediatric EM clinical experience was provided in the community hospital’s pediatric ED.
Study Subjects. Twenty-five EM residents were the study subjects. Data were gathered for three complete cohorts of residents, with entering years of 2003, 2004, and 2005, so the total study period was 2003–2008. Cohort 1 had 24 months of EM block rotations (6, 7, and 11 for each postgraduate year [PGY], respectively); Cohort 2 (7, 7, and 11 for each PGY, respectively) and Cohort 3 (7, 8, and 10 for each PGY, respectively) had 25 months of EM block rotations. EM rotations were in 4-week blocks. A work week maximum of 80 hours was in force for the entire study period. All residents in the three cohorts have passed both written and oral American Board of Emergency Medicine (ABEM) certification examinations.
All data elements were standardized to allow for analysis and comparisons. Both hospitals used the Emergency Severity Index (ESI) for triage acuity, and the same training group educated the triage nurses at both hospitals. ESI level was used as a proxy for patient acuity, since International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM), codes do not allow for that concept. Both hospitals used electronic ED medical records. As part of the electronic documentation process, residents were required to enter their names and the responsible ED attending’s names for every clinical encounter.
A resident clinical encounter was any encounter in which a resident electronically documented participation, either by initial patient evaluation or by handover at shift change. Consequently, the number of ED encounters exceeds the total number of patients treated by residents. Clinical conditions encountered by each resident were identified by extracting the first four ICD-9-CM diagnosis codes, for each patient encounter. The tertiary hospital captures up to 15 ICD-9-CM diagnosis codes, and the community hospital captures up to four ICD9-CM diagnosis codes per encounter. ED diagnosis codes were determined by hospital coders at the tertiary hospital and by the ED information system software (Allscripts ED, Chicago, IL) based on the text diagnoses entered by the care providers at the community hospital. An arbitrary list of clinical encounters of interest was selected by three EM educators, by reviewing the ICD-9-CM and collapsing codes into clinically coherent groups.
Data elements extracted for each patient encounter were patient age, sex, ESI triage acuity level, first four ICD-9-CM diagnosis codes per encounter, the individual resident, the resident’s PGY of training, and training site (tertiary vs. community). Each diagnosis was separately tabulated in the diagnosis count, and where appropriate, diagnoses were aggregated into clinically coherent groups. Each resident was tracked by his or her year of postgraduate training (1–3) and hospital. Data are presented as medians with ranges. All analyses were performed using SAS statistical software (Version 9.2, SAS Institute, Cary NC).
Number of Clinical Encounters
Twenty-five residents had 120,240 clinical encounters from 2003 to 2008. The median number of ED clinical encounters for a resident during his or her training was 4,836 (range = 3,831 to 5,780; Figure 1). Overall, clinical encounters increased by 30% from PGY 1 to PGY 2 and another 14% from PGY 2 to PGY 3. There was 30% to 60% variation in clinical encounters between individual residents.
The difference over 3 years of training between the most productive and least productive resident was 1,949 clinical encounters, representing an increase of 1.5 times the number of patient contacts for the most productive resident compared to the least productive resident. The actual numbers of clinical encounters by resident and by year of training for the entire resident cohort are represented in Figure 2. In PGY 1, the range of ED patient encounters per resident was 720 to 1,903 (median = 1,249), with a difference in productivity from the least to most productive PGY 1 of 64%. In PGY 2, the range was 1,140 to1,978 (median = 1,661), with a difference in productivity from the least to the most productive PGY 2 of 42%. In PGY 3, the range was 1,526 to 2,259 per resident (median = 1,942), with 15 of the 25 PGY 3 residents documenting more than 2,000 clinical encounters. The difference in productivity from the least to most productive PGY 3 was 32%.
Variation in clinical encounters from year to year by each resident is not accounted for by ED rotation assignments, since the rotation assignments were stable for each cohort of residents. While the general number of clinical encounters increases with each year of training, the less clinically productive residents remain so, and the highly productive ones continue to sustain a high level of activity throughout their residency (Figure 2).
Clinical Encounters by Hospital
Resident productivity ranking remained about the same at either hospital (data available but not shown). The difference was consistent across the two institutions. Fewer patients were seen at the community hospital compared to the tertiary hospital because shifts were shorter, less clinical ED time is spent at the community hospital, and patients at the community hospital were less likely to have care transferred from resident to resident.
Based upon triage ESI, total resident encounters by acuity are presented in Figure 3. ESI acuity levels were similar in both hospitals (Table 1). If ESI Levels 1 and 2 are summed to indicate the highest acuities, then 32% of resident encounters were critical, and 3% of those were ESI Level 1. Individual resident encounters with ESI Level 1 ranged from 100 to 162 per resident over 3 years, with a difference of 62% in encounters with patients requiring immediate life-saving intervention between the highest and lowest productive residents. Overall, resident clinical encounters are weighted to ESI Level 3, for about 50% of encounters. About 18% of resident clinical encounters are lower acuity (ESI 4 or 5), compared to ABEM test content of 27% and compared to the ESI Level 4 or 5 patient mix of 37% to 40% at both hospitals (Table 1).
|ESI Level||Resident Encounters (%)||Tertiary Hospital (%)||Community Hospital (%)||ABEM Exam Content (%)|
|5||2.0||7.7||5.0||Lower acuity 27%|
Age Distribution of Clinical Encounters
The vast majority of children younger than 12 years of age were encountered at the community hospital, since that hospital has the large pediatric ED. All residents were assigned similar clinical blocks of time in the pediatric ED. The range of clinical encounters with children younger than 12 years was 455 to 846 (median = 636). The difference between the least and most productive residents, in terms of encounters with children under 12 years, was 46%. For children younger than 1 year old, the range was 158 to 314 encounters (median = 221). The difference between the least and most productive residents, in terms of encounters with children under 1 year, was 49%.
Types of Clinical Encounters
The most common disorders encountered by the resident cohorts in both hospitals, as identified by any one of the first four ICD-9-CM diagnosis codes recorded, are listed in Table 2. Common pediatric disorders are represented by experience at the community hospital. The list of common disorders is as expected, except for the proportion of psychiatric and substance abuse disorders (21.8% at the tertiary hospital and 6.5% at the community hospital). The tertiary hospital has psychiatric inpatient beds for adults, children, and adolescents, and the community hospital does not have psychiatric inpatient beds.
|Tertiary Hospital||%||Community Hospital (Includes Children)||%|
|Psychiatric disorders||16.0||Chest pain/ischemic heart disease||8.3|
|Chest pain/ischemic heart disease||11.4||Psychiatric disorders||4.8|
|Fluid, electrolyte, acid–base disorders||6.1||Fluid, electrolyte, acid–base disorders||3.4|
|Substance abuse/intoxication/withdrawal||5.8||Motor vehicle collision||3.1|
|Cardiac dysrhythmias||4.9||Ear emergencies||3.0|
|Heart failure/pulmonary edema||4.4||Migraine/headache||2.7|
|Motor vehicle collision||2.5||Substance abuse/intoxication/withdrawal||1.7|
|Adult mixed endocrine disorders||2.1||Cellulitis/skin abscess||1.5|
|Stroke||1.6||Heart failure/pulmonary edema||1.3|
|Hand/finger laceration||1.3||GI hemorrhage||0.8|
A number of selected disorders were identified by the authors as important for resident training (data available but not shown), and these were identified by capturing any one of the first four ICD-9-CM diagnosis codes. Examples of some conditions that were not encountered by residents over the 3 years of experience are listed in Table 3.
|Tertiary Hospital||Community Hospital (Includes Children)||Conditions Missing in Both|
|Drug dystonias||Drug dystonias||Drug dystonias|
|Cardiac drug toxicity||Adult sexual assault||Cardiac drug toxicity|
|Carbon monoxide poisoning||Arterial embolism/thrombosis||Drowning|
|Frostbite||Bacterial or Neisseria meningitis||Jaw dislocation|
|Hypothermia||Bartholin gland abscess||Malaria|
|Jaw dislocation||Cardiac drug toxicity||Mastitis|
|Malaria||Carbon monoxide poisoning||Pediatric sexual assault|
|Mastitis||Compartment syndrome of limb||Perforated ulcer|
|Patella dislocation||Drowning||Pesticide poisoning|
|Pediatric sexual assault||Frostbite||Rectal prolapse|
|Perforated ulcer||Gunshot wounds||Testicular torsion|
|Pesticide poisoning||Jaw dislocation|
|Testicular torsion||Malignant hypertension|
|Toxic skin erythemas||Mastitis|
|Pediatric metabolic disorders|
|Pediatric sexual assault|
An ACGME-sponsored EM consensus workgroup identified a number of measurable outcomes for EM resident training, based on the six core competencies.10 Measurable outcomes were employer surveys, patient satisfaction, relative value units, throughput times, Centers for Medicare and Medicaid Services quality measures, 360° evaluations, patients/hour, use of tests and consultants, diagnosis, specific treatment, length of stay, decision to admit time, complication rate, and return visits to the ED.11 The numbers and types of patient contacts during training were not addressed. The European Curriculum for EM does require “benchmarks” to “orient the trainee” without providing more detail about any potential quantitative benchmarks.2
Currently in the United States, EM training consists of three or four years spent in a residency program with clearly delineated clinical rotations. We found that the actual patient encounters during those years vary tremendously between residents, even when their clinical rotations were nearly identical. Variance in clinical encounters was also found in resuscitations and pediatric care. Furthermore, in our cohorts, those residents who tended to see fewer patients, continued to see fewer patients throughout residency regardless of hospital assignment. This culminated in a greater than 33% difference in number of ED clinical encounters between the most productive and least productive residents. Stated in another way, in a 3-year program, the maximal variation corresponds to roughly 1 year of clinical EM training on ED rotations. Berk et al.12 found that postresidency clinical practice experience of 1.5 years or more was associated with a lower patient care error rate than experience of under 1.5 years. So, the variance in clinical encounters during residency may be related to clinical competency.
One goal of the ACGME Milestones Project13 is the development of optimal clinical workloads specified by each RRC for its respective specialty. In this context, a milestone is a measurable outcome of one of the ACGME competency domains that reflects an accomplishment of a resident at a particular point in time.14 Our study suggests that goal- or milestone-directed EM training should move toward developing methods to track resident clinical encounters by PGY of training and develop clinical benchmarks for each year. Patient care benchmarks could include both the total number and the types of patients cared for during each academic year, as well as specific clinical conditions or diagnoses felt to be fundamental for EM training. Such a concept may become more important as duty hour restrictions and the use of midlevel ED providers further affect the training experience. To facilitate tracking of patient encounters, electronic systems should be developed that allow for regular reporting on the types, ages, and acuities of patients seen by each resident. This would enable program directors to guide residents toward patient selection that will develop needed competencies.
We also discovered many specific areas of extremely limited patient encounters during resident training, including conditions such as extremity compartment syndrome, sexual assault, pediatric resuscitation, and poisoning. While some of these shortcomings may be training program specific, monitoring the types of resident clinical encounters allows the program director to further tailor the didactic and interactive components of the curriculum to fill in gaps. For example, simulation is useful for teaching high-acuity, low-frequency situations such as pediatric resuscitation.15 While many centers may already be using simulation to teach resuscitation and other high-acuity events, this study suggests an important need for interactive educational experiences to teach additional skills or concepts that have low or no frequency during training.
The clinical experience was somewhat different in the two hospitals. The tertiary hospital is a center for disorders such as vascular intervention, acquired immunodeficiency syndrome, chronic neurologic disorders such as myasthenia gravis, and pediatric metabolic disorders. The tertiary hospital has the full spectrum of psychiatric inpatient services, possibly accounting for the large proportion of psychobehavioral disorders. The community hospital provides the residents with exposure to the private practice environment, and its large children’s ED provides pediatric EM experience. For those programs that use multiple training hospitals, identifying the types of patient encounters in each hospital may also help direct the residency curriculum.
The overall frequency of psychiatric and substance abuse disorders encountered by our residents during training is much greater than the amount of educational time usually devoted to these topics. The ABEM examination content for psychiatric disorders is 3%.16 If the frequency of these disorders is actually also higher in other training programs or in practice than usually assumed, further education in these areas may be necessary during residency.
In this study, only 18% of resident encounters included ESI Levels 4 or 5. According to national data, 30% of ED patients present with semiurgent or nonurgent conditions,17 and roughly 40% of patients in the two participating hospitals in this study were triaged as ESI Level 4 or 5. The care of less acute complaints is a cornerstone of EM practice.18 Assuming that our training model is similar to others, an alarm should be raised because the care of lower-acuity conditions may be a training deficit. As the use of NPs and PAs in EDs increases,19,20 resident exposure to lower-acuity patients likely decreases. Residents must also be afforded the opportunity to supervise and collaborate with midlevel providers.21–23 Ensuring adequate exposure to low-acuity conditions should be a priority for program directors and should affect the staffing plans for EDs with EM residency programs.
This study assessed resident productivity in terms of the number of clinical ED encounters at a single southeastern U.S. residency program. The conclusions may not be generalizable to other programs. Off-service rotations could have rounded out experience with some conditions that were missed in the ED.
Block EM rotations were constructed for each year for each cohort. There could have been minor variance based upon personal resident needs.
Data were not assembled for the number of shifts or for patients seen per shift. Shifts varied in length and number at both hospitals, depending on service needs, resident work hours, and accommodations for weekend time off. ED block rotations were totaled at both hospitals because there was more clinical time at the tertiary hospital than at the community hospital. The goal was to assess the totality of resident experience over three years, and summary data were the best reflection of this.
We did not assess the effect of the different training environments afforded by the two hospitals. The tertiary hospital is a typical resident-managed institution, while the community hospital is primarily private physician-based.
It was not possible to validate the precision and accuracy of ICD-9-CM diagnosis codes for either hospital. However, the most common conditions seen by residents in each hospital were quite similar to statewide North Carolina ED data.24
Data capture and analysis hinged on the need for each resident to electronically “sign up” and document each patient encounter, whether for the majority of ED care, for shift changeover, or for a procedure. Residents may have assisted in the care of some patients without documenting their involvement. The electronic sign-up processes are a bit different in the two institutions, but resident productivity is consistent across both (data available but not shown). The same work habits are reinforced at both institutions.
Resident productivity data could be affected by both initial sign-up for patient care, as well as sign-up for patient handover. While resident productivity could also have been measured by counting only the first resident to sign up for the patient, at the tertiary hospital, patient handover requires a review of all laboratory and imaging results, and discussion with the patient and attending physician, before disposition. The authors thus feel that all residents who sign up participate in legitimate patient encounters. At the community hospital, patient signout is typically to an ED attending and not another resident.
Because there is no ICD-9-CM code specifically for conditions important for EM training such as “resuscitation,”“multisystem trauma,” or “red” or “yellow” trauma codes, the ESI level was used as a proxy for immediate life and limb threats. While nurse training for ESI levels was comparable at both institutions, critical conditions could have been either under- or overtriaged. If the patient’s condition deteriorated during the ED stay, that would not be represented by the initial ESI level.
It was not possible to extract diagnosis codes of conditions that were not seen at either hospital by the residents. Still, a method to measure clinical training omissions is needed to complement the lack of experience by providing simulation exercises, seminars, workshops, or lectures.
The median number of clinical ED encounters for three cohorts of EM resident classes (2003 to 2008), based on a maximum 80-hour work week and 24 or 25 four-week blocks of EM rotations over 3 years of training, was 4,836. The actual numbers of patient encounters during those years vary tremendously between residents, even when the clinical rotations are nearly identical.
While the general number of clinical encounters increases with each year of training, the less clinically productive residents remain so, and the highly productive ones continue to sustain that level of activity throughout the residency.
There is substantial variation in clinical experience between residents in the same cohort, and in different cohorts, ranging from about a 30% to a 60% difference, depending on the variable measured. Stated in another way, in a 3-year program, the maximal variation corresponds roughly to 1 year of clinical EM training on ED rotations. Variability was highest during the PGY 1 year, and narrowed in the PGY 2 and PGY 3 years, but was still considerable. Variability was especially notable in total numbers of clinical encounters, in the care of children, and in the care of critical time-sensitive illness. Some life-threatening or high-risk conditions may never be encountered clinically during 3 years of residency training. Resident encounters with lower acuity problems during training are much less frequent than the anticipated lower-acuity burden during practice, and resident clinical experience is weighted to more critical and emergent conditions.
Methods should be developed to decrease resident variance in total clinical encounters. Curriculum supplementation should be provided for individuals and for the entire residency cohort, to decrease variance in clinical areas that are rare or not encountered during residency training, but that are important for the clinical practice of EM.
The authors acknowledge Cherri Hobgood, MD, for her assistance in the early phase of methodology development and in the selection of types of clinical encounters analyzed.