Context and setting The Accreditation Council for Graduate Medical Education (ACGME) mandates US residency programmes to establish competency-based assessments of professionalism. Although ACGME has suggested some means of evaluating professional behaviour, no single best method allowing for the assessment of professionalism in actual clinical settings has been identified.
Why the idea was necessary Standardised patients (SPs) are among ACGME’s recommended approaches for assessing residents’ competencies in professionalism. Although SPs have been used in medical education, their use in evaluating residents’ professionalism has been limited. Prior reports suggest that doctors behave differently in examination settings than in real practice. Thus, it is important to assess doctors’ authentic behaviour. This can be achieved by using unannounced SPs. We have previously established the reliability of adolescent SPs in assessing residents’ professionalism in announced, faculty-observed SP exercises. We decided to determine if residents would display similar levels of professionalism in announced versus unannounced SP encounters.
What was done We conducted a prospective repeat-measures study in a major paediatric residency programme. Twelve adolescent SPs were trained to assess elements of professionalism and rate residents using a Likert-type scale (1 = strongly disagree, 5 = strongly agree). A cohort of 16 residents was subjected at random to the following 3 SP exercises:
- 1announced SP encounters observed by a faculty member;
- 2announced SP encounters unobserved by a faculty member, and
- 3unannounced SP encounters.
Announced and observed SP−resident encounters were directly observed by 2 trained faculty members. For the unannounced exercises, we inserted SPs into residents’ regularly scheduled ambulatory clinic sessions. Unannounced SPs were presented to the supervising attending doctors as regular patients. Standardised patients rated residents’ professionalism immediately following each exercise. Faculty rated residents’ professionalism following the observed encounter using the same checklist as SPs. Faculty ratings validated SP ratings on the observed encounter, as we have previously reported. We used multivariate analysis of variance (manova) for repeat measures to test mean differences in continuous variables and Mann–Whitney U-rank order sum to test ordinal rank differences.
Evaluation of results and impact Overall, 60 resident−SP encounters were analysed (18 announced/observed, 18 announced/unobserved, and 24 unannounced). Residents consistently demonstrated lower levels of professionalism in the unannounced SP encounters. Examples of statements from the checklists comparing ratings of resident performance in 3 SP exercises are as follows: ‘The doctor showed warmth and compassion’ (announced/observed = 4.5, announced/unobserved = 4.6, unannounced = 3.7; P = 0.004); ‘The doctor encouraged me to ask questions’ (announced/observed = 4.4, announced/unobserved = 4.3, unannounced = 3.2; P = 0.001); ‘I would talk to this doctor if something was troubling me’ (announced/observed = 4.3, announced/unobserved = 4.2, unannounced = 2.9; P < 0.001), and ‘The doctor asked for my opinion when making decisions about my treatment’ (announced/observed = 4.0, announced/unobserved = 4.1, unannounced = 3.0; P = 0.002). In this study, residents displayed lower levels of professionalism in unannounced compared with announced SP encounters. The use of unannounced SPs allowed us to assess residents’ professionalism in real clinical practice.