“Oh! She Doesn't Speak English!” Assessing Resident Competence in Managing Linguistic and Cultural Barriers
The authors do not have any conflicts of interest to declare.
Presented in part at the Annual Meeting of the Society of General Internal Medicine, May 10, 2004. Funded by HRSA, Bureau of Health Professionals.
Address correspondence and requests for reprints to Dr. Zabar: First Avenue, Old Bellevue, D401B, New York, NY 10016 (e-mail: email@example.com).
BACKGROUND: Residents must master complex skills to care for culturally and linguistically diverse patients.
METHODS: As part of an annual 10-station, standardized patient (SP) examination, medical residents interacted with a 50-year-old reserved, Bengali-speaking woman (SP) with a positive fecal occult blood accompanied by her bilingual brother (standardized interpreter (SI)). While the resident addressed the need for a colonoscopy, the SI did not translate word for word unless directed to, questioned medical terms, and was reluctant to tell the SP frightening information. The SP/SI, faculty observers, and the resident assessed the performance.
RESULTS: Seventy-six residents participated. Mean faculty ratings (9-point scale) were as follows: overall 6.0, communication 6.0, knowledge 6.3. Mean SP/SI ratings (3.1, range 1.9 to 3.9) correlated with faculty ratings (overall r=.719, communication r=.639, knowledge r=.457, all P<.01). Internal reliability as measured by Cronbach's α coefficients for the 20 item instrument was 0.91. Poor performance on this station was associated with poor performance on other stations. Eighty-nine percent of residents stated that the educational value was moderate to high.
CONCLUSION: We reliably assessed residents communication skills conducting a common clincal task across a significant language barrier. This medical education innovation provides the first steps to measuring interpreter facilitated skills in residency training.
Despite the ever expanding diversity of North America's population, most U.S. and Canadian medical schools do not employ effective methods of training and evaluation to ensure culturally competent care.1–4 In New York City, 22% of the population has limited English proficiency (LEP).5 United States teaching hospitals report that up to 27% of patients require an interpreter for optimal care.6 Effective and efficient use of interpreters, eliciting health belief models, and awareness of culturally determined biases and barriers are essential competencies medical learners must master.7
Language barriers hinder adequate establishment of rapport and information gathering and therefore can negatively impact quality of care.8 Title VI of the US Civil Rights Act of 1964 required that all federally funded health facilities provide access to interpreters.9 Although this act was federally mandated by Executive Order 13166 in 2000 and 2002, insufficient funds and lack of trained interpreters have prevented this goal from being realized.10 Unfortunately, house officers may be forced to rely on ad hoc, untrained interpreters such as family members and staff who are likely to commit errors in interpretation such as omission (message completely or partially deleted by interpreter), addition (information not expressed by the patient), condensation (response is simplified and paraphrased) and substitution (replacement of 1 concept with another).11,12 Although not ideal, in these situations, the ability to “professionalize” an ad hoc interpreter into a powerful partner in the patient's care becomes an essential skill of clinical practice. Training health care providers in the effective use of medical interpreters has the potential to improve accuracy of communication, quality of clinical care, and health outcomes for LEP patients.13
Culturally appropriate care requires multiple competencies that are infrequently taught or assessed.14 This paper reports on the development, implementation, and evaluation of a performance-based assessment of residents' ability to conduct a medical encounter using an ad hoc standardized interpreter (SI) and a standardized patient (SP), as 1 of 10 stations in an objective structured clinical examination (OSCE) for medical residents covering a broad range of primary care competencies.15
Objective structured clinical examinations, introduced in 1975, have gained popularity as both a teaching and evaluation method in medical schools and residency programs.16,17 In 2003, during our annual 10-station clinical skills OSCE, residents had 10 minutes to perform a specific task. Faculty observers, the SPs and the residents themselves independently completed distinct rating forms immediately following each encounter. Subsequently residents received 5 minutes of feedback from faculty and SPs. Following the 3-hour experience, residents and faculty debriefed all 10 scenarios and received relevant readings.
INTERPRETER STATION DEVELOPMENT
Identifying Residents' Competencies
We defined our interviewing competencies and case-specific interpreter skills through a literature review,18,19 consultation with the categorical and primary care medical residency directors and the director of the NYU Center for Immigrant Health. The competencies assessed included data gathering, rapport building, patient education, and skills needed to work with an interpreter. The case-specific skills included providing explicit instructions to a family member on how to interpret, maintaining rapport with the patient and maintaining a respectful attitude towards the interpreter.
We developed a scenario involving a 50-year-old reticent, Bengali-speaking woman and her bilingual brother recalled to clinic for a positive fecal occult blood test (FOBT). The residents' task was to review the lab results and make an appropriate plan by communicating through the brother interpreter. The case was designed to assess residents' skills in both using an interpreter and educating their patient about the meaning of an abnormal FOBT and the need for colonoscopy. The interpreter was programed to challenge the resident by making errors typical of an untrained family member serving as an interpreter, such as questioning medical terms (e.g., “do you mean a screen like a window” when the phrase “screening test” was used), not interpreting word for word unless specifically instructed to do so at least twice, shying away from sharing frightening information with his sister unless gently encouraged, and inquiring about his own health. The case was written by faculty and reviewed and practiced with chief residents. We chose a Bengali-speaking patient to eliminate the possibility that any of our residents could conduct the interview without an interpreter.
Recruitment and Training of SP, SI, and Faculty Observers
We recruited a professional Bengali interpreter (SI) and his wife (SP), both bilingual, and trained them for 2 hours with scripts and role-play to standardize their case portrayals and resident ratings. We used the same SI and SP for all 76 residents. Faculty observers practiced rating video tapes together to enhance reliability.
Assessment of Residents' Competency
The 20 item OSCE checklist for the interpreter station is included in Table 1. The faculty used it to assess the individual skills and 3 global assessments of performance (overall performance, communication skills, and fund of knowledge demonstrated). The SPs and SIs rated global satisfaction with the encounter on a 4-point scale. Residents evaluated their own performance on a single item using the same 9-point scale as faculty (see Table 1). Immediately following all stations, residents and faculty completed a debriefing questionnaire that addressed case difficulty and educational value of the experience.
Table 1. Three Evaluation Instruments Measuring Interpreter Station as Compared to Other 9 Stations (N=76)
|1. Faculty evaluation of residents' competence in using an interpreter (20 item checklist covering 4 specific competencies + global performance)|
| Data gathering (4-point scale; 1=not done, 4=done excellently)||3.2 (0.6)||3.1 (0.4)|
| Chooses appropriate type of questions||2.0 to 4.0||2.0 to 4.0|
| Facilitates patient to tell own story|| || |
| Uses jargon free language|| || |
| Asks questions 1 at a time|| || |
| Summarizes history/checks for accuracy|| || |
| Rapport building (4-point scale; 1=not done, 4=done excellently)||3.2 (0.5)||3.1 (0.3)|
| Communicates non judgmental respectful attitude||2.2 to 4.0||2.0 to 4.0|
| Exhibits appropriate non verbal behavior|| || |
| Recognizes and names emotion|| || |
| Responds to emotion with PEARLS or nonverbally|| || |
| Patient education (4-point scale; 1=not done, 4=done excellently)||3.0 (0.7)||3.0 (0.3)|
| Asked questions to see what patient understood||1.5 to 4.0||2.0 to 4.0|
| Provided clear explanations/information|| || |
| Collaborated with patient in identifying possible next steps/plan|| || |
| Encourages questions|| || |
| Knowledge base demonstrated (4-point scale; 1=not done, 4=done excellently)||3.1 (0.5)||2.9 (0.4)|
| Advises colonoscopy for patient as a follow-up test||2.0 to 4.0||2.0 to 4.0|
| Makes clear statements about translating process (e.g., word for word, used first person, specified position)|| || |
| Maintains eye contact with the patient not the interpreter|| || |
| Maintains respectful attitude towards interpreter|| || |
| Global performance (9-point scales; 1=needs improvement, 9=done excellently)|| || |
| Overall performance||6.0 (1.7) 2 to 9||6.1 (0.9) 3 to 8|
| Overall communication skills||6.0 (1.7) 2 to 9||6.2 (0.9) 3 to 8|
| Overall case-specific knowledge||6.2 (1.6) 2 to 9||6.1 (0.9) 3 to 9|
|2. Resident self-assessment for overall case performance (1 item, 9-point scale; 1=needs improvement, 9=done excellently)|
| ||4.3 (1.4)||4.3 (1.5)|
|“Please rate your assessment of overall case performance.”||3.0 to 9.0||3.0 to 9.0|
|3. SI/SP Assessment of Satisfaction Competence (1 item, 4-point scale; 1=not satisfied, 4=very satisfied)|
| ||3.1 (0.4)||3.1 (0.3)|
|“Please rate your satisfaction with this doctor.”||1.0 to 4.0||1.0 to 4.0|
Seventy-six medical residents (8 postgraduate year 1 (PGY1), 60 PGY2, and 8 PGY3) participated in the 2003 OSCE. There were 3 faculty observers.
Scale and Rater Reliability and Validity
Internal reliability as measured by Cronbach's α coefficients for the 20-item instrument was 0.91. Inter-rater reliability for the faculty checklist overall performance was significant (Spearman's ρ=0.646 P<.001) and the κ for the same measure was .484 for each of 2 pairs of raters. The inter-rater reliability for overall communication and knowledge was moderate (ρ=0.403 P<.09 and ρ=0.371 P<.06). Correlations among ratings of SI/SP and faculty ratings on overall performance (Pearson's r=.719, P<.01), overall communication and knowledge evaluations (Pearson's r=.639, P<.01, r=.457 P<.001) were conducted as a measurement of convergent validity.
Performance on Interpreter Case Compares with Other Cases. Table 1 shows overall resident performance on the interpreter case had a similar mean and range to performance on all other cases in the OSCE. Poor resident performance on the interpreter case as categorized by SP/SI ratings of ≥2 out of 4 was associated with poorer performance on other cases as judged by the faculty overall assessment (5.4 vs 6.2, P<.003). This identifies a group of 15 residents who performed poorly across the other 9 cases based on faculty overall assessment.
Interpreter Stations are Realistic and Educational. On the post-OSCE survey, 94% of residents reported some or much prior experience with the clinical challenge presented in this scenario, 92% reported the degree of difficulty was “just right” and 89% indicated educational value was moderate to high. All faculty thought it was worthwhile, because it was an opportunity to give feedback on skills rarely observed.
One year later, we surveyed a convenience sample of the original participants (29/76, 38%) about their previous experience. On a 5-point scale (1=strongly agree to 5=strongly disagree), residents reported that the interpreter case was a useful educational experience (mean 2.3), it helped them use interpreters more effectively (mean 2.2) and they would recommend it for all residents (mean 2.1).
The 3 faculty observers reported that the most common resident mistakes included spending more time explaining the normal results than the abnormal ones, using jargon (e.g., “Your screening test is positive”), talking louder to make things clearer and using frightening descriptions of how a colonoscopy is done (e.g., “A camera on the end of a long tube will be put inside”). In addition, there was a range of performance on skills specific to using an interpreter, such as talking to the patient, not the interpreter; instructing the interpreter and arranging seating positions (see Table 1).
We developed and implemented a practical, reliable, and realistic performance-based assessment of relevant resident skills that are important components of cultural competency and systems-based practice. Physicians with prior training in using an interpreter report increased use of professional interpreters and greater satisfaction with the medical care provided to LEP patients, when compared with physicians with no prior training.20 This case, perceived by the residents as one of the most challenging, has the potential to be a lasting educational experience for residents and improve their willingness to provide care cross language care.
The need to train physicians to effectively work with interpreters has been fuelled by the rapid increase in limited English proficient patients and a federal mandate to enforce the Civil Rights Act of 1964. The standard for medical interpreting is simultaneous interpretation by a professional interpreter. The ideal professional interpreter promotes accuracy of communication by translating both physician and patient statements word for word and supports development of a doctor-patient relationship by becoming as “invisible” as possible. Real-world practice, unfortunately, presents multiple situations in which professional interpreters are not available, such as disasters, limited resource clinical settings or an uncommonly spoken language. In these situations, physicians can instruct nonprofessional interpreters to become more “professional.” Necessary skills include striving for accuracy (no adding, omitting, or substituting); speaking in the first and second person (“I and you”); positioning themselves behind the patient; maintaining eye contact with the doctor, not the patient; and employing clarification or back translation when needed.21 Residents can be trained to follow these guidelines and optimize the accuracy of interpreters, but this training is not a common feature of residency curricula, and there are few standardized methods to evaluate this competency.7,22
This faculty-observed, performance-based assessment interpreter station offers a number of teaching opportunities beyond just assessment of the skills needed to effectively work with an interpreter. During the station, residents were also required to demonstrate their competence in patient education and informed consent involving a common but complex preventive intervention. Specific skills included using a conversational, ask-tell-ask approach and using understandable language to describe a fairly complex procedure (colonoscopy). Additionally, reviewing videotapes of this station as a group allowed residents to reflect on their attitudes regarding working with interpreters. Aggregate data of resident performance provided programmatic feedback for the current curriculum.
This study has some limitations. The SP literature is clear about the problem of case specificity and the need for multiple stations to reliably assess a particular skill.23 Specifically, the same resident who performs well in the Bengali colon cancer case might do poorly on a Chinese smoking cessation case. In fact, the correlation between performance on the communication skills in this case and others might indicate that we are measuring general communication skills, not something specific about intercultural communication. In addition, performing an SP assessment may not truly reflect the abilities of residents to conduct such interactions in real life. Furthermore, this is just 1 program's experience. In light of these limitations we consider this the first step toward establishing valid measurements of residents' ability to use interpreters.
Ultimately, cultural competency curricula that emphasize appropriate skills and attitudes to work with lay and trained interpreters to effectively conduct an interpreted medical interview must be a standard part of all physicians' training. Reliable and valid OSCE assessments can provide outcome measures with which to test the effectiveness of these critical clinical skills and act as both teaching and evaluation tools that may improve confidence in use of interpreters. Further research is needed to establish the impact of cultural competency curricula on patient satisfaction and patient outcomes.