Adjusting Our Lens: Can Developmental Differences in Diagnostic Reasoning Be Harnessed to Improve Health Professional and Trainee Assessment?

Authors

  • Jonathan S. Ilgen MD, MCR,

    1. From the Division of Emergency Medicine, University of Washington School of Medicine (JSI), Seattle, WA; the Department of Medical Informatics and Clinical Epidemiology (JLB, RAL), the Department of Emergency Medicine (LMY, RF, RAL), and the Department of Public Health and Preventive Medicine (RAL), Oregon Health & Science University, Portland, OR; and the Centre for Health Education Scholarship, University of British Columbia (KE), Vancouver, British Columbia, Canada.
    Search for more papers by this author
  • Judith L. Bowen MD,

    1. From the Division of Emergency Medicine, University of Washington School of Medicine (JSI), Seattle, WA; the Department of Medical Informatics and Clinical Epidemiology (JLB, RAL), the Department of Emergency Medicine (LMY, RF, RAL), and the Department of Public Health and Preventive Medicine (RAL), Oregon Health & Science University, Portland, OR; and the Centre for Health Education Scholarship, University of British Columbia (KE), Vancouver, British Columbia, Canada.
    Search for more papers by this author
  • Lalena M. Yarris MD, MCR,

    1. From the Division of Emergency Medicine, University of Washington School of Medicine (JSI), Seattle, WA; the Department of Medical Informatics and Clinical Epidemiology (JLB, RAL), the Department of Emergency Medicine (LMY, RF, RAL), and the Department of Public Health and Preventive Medicine (RAL), Oregon Health & Science University, Portland, OR; and the Centre for Health Education Scholarship, University of British Columbia (KE), Vancouver, British Columbia, Canada.
    Search for more papers by this author
  • Rongwei Fu PhD,

    1. From the Division of Emergency Medicine, University of Washington School of Medicine (JSI), Seattle, WA; the Department of Medical Informatics and Clinical Epidemiology (JLB, RAL), the Department of Emergency Medicine (LMY, RF, RAL), and the Department of Public Health and Preventive Medicine (RAL), Oregon Health & Science University, Portland, OR; and the Centre for Health Education Scholarship, University of British Columbia (KE), Vancouver, British Columbia, Canada.
    Search for more papers by this author
  • Robert A. Lowe MD, MPH,

    1. From the Division of Emergency Medicine, University of Washington School of Medicine (JSI), Seattle, WA; the Department of Medical Informatics and Clinical Epidemiology (JLB, RAL), the Department of Emergency Medicine (LMY, RF, RAL), and the Department of Public Health and Preventive Medicine (RAL), Oregon Health & Science University, Portland, OR; and the Centre for Health Education Scholarship, University of British Columbia (KE), Vancouver, British Columbia, Canada.
    Search for more papers by this author
  • Kevin Eva PhD

    1. From the Division of Emergency Medicine, University of Washington School of Medicine (JSI), Seattle, WA; the Department of Medical Informatics and Clinical Epidemiology (JLB, RAL), the Department of Emergency Medicine (LMY, RF, RAL), and the Department of Public Health and Preventive Medicine (RAL), Oregon Health & Science University, Portland, OR; and the Centre for Health Education Scholarship, University of British Columbia (KE), Vancouver, British Columbia, Canada.
    Search for more papers by this author

  • Presented at the 17th Annual National Research Service Award (NRSA) Trainees Research Conference, Seattle, WA, June 2011.

  • This publication was made possible with support from the Oregon Clinical and Translational Research Institute (OCTRI), grant UL1 RR024140 from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH), and NIH Roadmap for Medical Research. The authors have no conflicts of interest to report.

  • Supervising Editor: John H. Burton, MD.

Address for correspondence and reprints: Jonathan S. Ilgen, MD, MCR; e-mail: ilgen@u.washington.edu.

Abstract

Academic Emergency Medicine 2011; 18:S79–S86 © 2011 by the Society for Academic Emergency Medicine

Abstract

Objectives:  Research in cognition has yielded considerable understanding of the diagnostic reasoning process and its evolution during clinical training. This study sought to determine whether or not this literature could be used to improve the assessment of trainees’ diagnostic skill by manipulating testing conditions that encourage different modes of reasoning.

Methods:  The authors developed an online, vignette-based instrument with two sets of testing instructions. The “first impression” condition encouraged nonanalytic responses while the “directed search” condition prompted structured analytic responses. Subjects encountered six cases under the first impression condition and then six cases under the directed search condition. Each condition had three straightforward (simple) and three ambiguous (complex) cases. Subjects were stratified by clinical experience: novice (third- and fourth-year medical students), intermediate (postgraduate year [PGY] 1 and 2 residents), and experienced (PGY 3 residents and faculty). Two investigators scored the exams independently. Mean diagnostic accuracies were calculated for each group. Differences in diagnostic accuracy and reliability of the examination as a function of the predictor variables were assessed.

Results:  The examination was completed by 115 subjects. Diagnostic accuracy was significantly associated with the independent variables of case complexity, clinical experience, and testing condition. Overall, mean diagnostic accuracy and the extent to which the test consistently discriminated between subjects (i.e., yielded reliable scores) was higher when participants were given directed search instructions than when they were given first impression instructions. In addition, the pattern of reliability was found to depend on experience: simple cases offered the best reliability for discriminating between novices, complex cases offered the best reliability for discriminating between intermediate residents, and neither type of case discriminated well between experienced practitioners.

Conclusions:  These results yield concrete guidance regarding test construction for the purpose of diagnostic skill assessment. The instruction strategy and complexity of cases selected should depend on the experience level and breadth of experience of the subjects one is attempting to assess.

Ancillary