SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Summary and Recommendations
  7. References
  8. Supporting Information

Professionalism is one of the six Accreditation Council on Graduate Medical Education (ACGME) core competencies on which emergency medicine (EM) residents are assessed. However, very few assessment tools exist that have been rigorously evaluated in this population. One goal of the 2012 Academic Emergency Medicine consensus conference on education research in EM was to develop a research agenda for testing and developing tools to assess professionalism in EM residents. A literature review was performed to identify existing assessment tools. Recommendations on future research directions were presented at the consensus conference, and an agenda was developed.

In 1999 the Accreditation Council on Graduate Medical Education (ACGME) shifted to an outcomes-based assessment model with emphasis on six core competencies, one of which is professionalism. The ACGME requires these competencies be systemically assessed for purposes of both formative and summative assessment.[1] The ACGME lists several characteristics of the professionalism competency that residents are expected to demonstrate, including compassion, integrity, respect, altruism, accountability, and sensitivity.[2] Other international medical education governing bodies have variably delineated the elements of professionalism. The Royal College of Physicians and Surgeons of Canada organized the elements of physician competency into the CanMEDS framework, in which the domain of professionalism contains 17 elements.[3] To objectively assess professional behavior, however, requires an operational definition consisting of observable behaviors rather than conceptual elements. In 2008, the Council of Emergency Medicine Residency Directors (CORD) Workgroup on Outcomes Assessment published a list of behaviors specific to the assessment of professionalism in emergency medicine (EM) residents.[4] The ACGME recently introduced the Next Accreditation System (NAS).[5] The NAS includes specialty-specific milestones within each competency that residents must meet. EM will be implementing the NAS in 2013 and has recently published its list of milestones, two of which are for professionalism[6] (Data Supplement S1, available as supporting information in the online version of this paper).

The 2012 Academic Emergency Medicine consensus conference “Education Research in Emergency Medicine: Opportunities, Challenges, and Strategies for Success” had as one of its goals to develop and advance a consensus-based research agenda on the assessment of learner competency in the six ACGME core competencies. The objective of this article is to summarize those findings and recommendations as they apply specifically to the assessment of professionalism.

Methods

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Summary and Recommendations
  7. References
  8. Supporting Information

Prior to the consensus conference, a comprehensive review of the published literature on professionalism assessment was performed. The Medline database was searched using the terms “professionalism” and “humanism” independently combined with the term “assessment,” resulting in 464 and 211 references, respectively. Combining “professionalism” and “evaluation” identified 509 references. Each of these had their abstract and/or full text reviewed by one or more of the authors to determine relevance. All relevant articles had their bibliographies reviewed for any additional previously unidentified references. Additionally the MedEdPORTAL database (https://www.mededportal.org) was searched using the term “professionalism,” which resulted in 216 results, but only six remained when the term “assessment” was added to the search strategy. Presenting a comprehensive systematic review of this topic is beyond the scope of this article, but the reader is directed to the reviews by Lynch et al.[7] and Veloski et al.[8] The summaries cited in the article by Veloski et al. can be obtained directly from ABIM (info@abimfoundation.org). The results of our review were then discussed among educational experts at a breakout session of the consensus conference. Their input was incorporated into the final recommendations and agenda.

Results

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Summary and Recommendations
  7. References
  8. Supporting Information

Identified assessment tools were classified into several categories. These include: ethical knowledge and moral reasoning tests; direct observation assessment tools; ratings and survey-based assessment tools (this category includes global rating instruments and multisource feedback [MSF] tools); critical incident reporting systems; portfolios and narratives; and simulated encounter observations, including objective structured clinical examination (OSCE), unannounced standardized patients (SP), and mannequin-based simulation.

Ethics Knowledge and Moral Reasoning Tests

There have been several instruments developed to assess ethics knowledge and moral reasoning using selected response questions, modified essay questions, and simulation-based cases.[9-12] The most widely studied tool for the assessment of moral reasoning is the Defining Issues Test (DIT). The DIT is a multiple-choice test that is easy to administer and score, so has good feasibility. It has been extensively used, demonstrating good acceptability, and has good reliability and validity.[13] The DIT was revised and is now available as the DIT2, which was found to have better reliability and validity than the DIT.[14]

While the reliability, validity, and feasibility of the DIT are good, limitations and questions still remain for these types of instruments. Some studies question how well these types of ethics knowledge and reasoning assessments correlate with actual professional behavior.[15] The DIT has been studied in other specialties,[16] but there are no published studies evaluating these types of tools in EM resident populations.

Direct Observation Assessment Tools

Direct observation tools, for purposes of our discussion, refer to instruments used to assess learner performance in the workplace during real clinical encounters. It is argued that actual clinical encounters are necessary to fully capture the complexity of context in ethical conflicts.[17]

The Professionalism Mini-Evaluation Exercise (P-MEX) is a professionalism-specific direct observation tool. It demonstrated good reliability and both content and construct validity when evaluated in Canadian medical students.[18] Most recently, the Standardized Direct Observation Assessment Tool (SDOT) has been described by Shayne et al.[19] The tool is developed for EM residents, and while not professionalism-specific, it does incorporate several elements of professional behavior, including a global rating for that competency. In a simulated resident–patient setting, it showed good inter-rater agreement but validation was not assessed. The SDOT has also shown good reliability in real resident–patient encounters.[20]

The need for an EM-specific validated direct observation professionalism-specific assessment tool remains. The P-MEX may be adaptable for this purpose, but evaluation of its reliability and validity must be undertaken with EM residents. The instrument is brief and appears to have good feasibility, although consideration of the barriers to implementing a program for its use should also be explored. The SDOT, while not professionalism-specific, has demonstrated good reliability in assessing EM residents but validity has yet to be assessed. These tools have potential for educational impact as they can help provide timely, formative feedback and evaluative assessment.

Ratings- and Survey-based Assessment Tools

Ratings and surveys can be used to assess professionalism. They allow for evaluation of observed behaviors in both direct and indirect patient care activities, as well as in other settings such as educational activities.

Global rating forms or summary rating forms are fairly common tools used in the evaluation of EM residents' professionalism,[21] often combining the assessment of multiple core competencies into a single form. Because the core competencies are each unique in their constructs, they may not be accurately assessed with a single instrument.[22] Reisdorff et al.[23] attempted to validate a global rating tool for EM residents. Principal component analysis revealed professionalism to be a complex multidimensional phenomenon to measure, and as such, comprehensive global rating tools may not be ideal tools for assessing professionalism in EM residents.

When two or more rater groups are combined it is termed MSF, or 360-degree evaluation. Raters may include self, peers, nurses, faculty, and patients. While some MSF instruments may attempt to rate multiple core competencies simultaneously, MSF seems best suited to the assessment of professionalism and interpersonal and communication skills.[24] The main advantage of MSF is that it allows for multiple raters from multiple vantage points to provide feedback on observed behaviors. A review by Lockyer[25] found MSF to have good acceptability and potential for educational impact. More robust evidence of educational impact of MSF on communication skills and professional behavior was demonstrated in pediatric residents.[26]

Several MSF instruments have been studied in various populations. Tsugawa et al.[27] modified the P-MEX into an MSF tool in a multispecialty study of Japanese residents and fellows. The P-MEX demonstrated good validity and overall reliability, although the peer rater subgroup had poor reliability. MedEdPORTAL contains a professionalism-specific MSF tool used to assess medical students; however, the reliability and validity are not reported.[28] The National Board of Medical Examiners has developed an MSF tool based on observable professional behaviors delineated in their Assessment of Professional Behaviors program.[29] This tool was developed as part of a proprietary longitudinal professionalism evaluation program. The reliability and validity data for the tool is still being compiled (M. Richmond, personal communication, January 19, 2012), but a description of feasibility and implementation issues for this MSF program has been published.[30] Another MSF instrument was developed by the College of Physicians and Surgeons of Alberta's Physician Achievement Program (CPSA PAP) for physicians who provide episodic care, including emergency physicians (EPs). It is not professionalism-specific but aspects of professionalism make up a large part of the assessment. Lockyer et al.[31] evaluated the tool in a population of practicing EPs. Their protocol required ratings from eight coworkers, eight colleagues, and 25 patients who were selected by the physician. It was found to be a feasible, reliable, and valid tool. Garra et al.[32] studied an MSF tool, the EM-Humanism Scale, which evaluated U.S. EM residents' professionalism and interpersonal and communication skills. The instrument had good reliability for nurses and faculty, but only fair reliability for patient ratings. The study was not designed to assess validity.

There are some general concerns with all of these types of ratings instruments. Their susceptibility to systematic bias (e.g., halo/horns effect, severity/leniency bias, and range restriction) is an issue.[33] Validity concerns, especially when physicians choose their own assessors, have also been raised.[34] A structured approach to development of the evaluation items is important to assure validity.[25] Rater training is important as raters may not consistently assess residents solely on observed behaviors.[35] The number of evaluations per rater category needed to optimize reliability merits further study as well.

Critical Incident Reporting Systems

Incident reports, comment cards, and other similar mechanisms for reporting lapses in professionalism have been developed. These types of tools, when implemented systematically, allow for the longitudinal collection of snapshot assessments of professional behavior. Some systems allow for these tools to be used not only to identify unprofessional behavior, but also to commend exemplary professional behavior. The system should be transparent, non-anonymous, behavior-focused, and allow for formative feedback and remediation (unless so egregious that it triggers mandated disciplinary action).[36] In a survey of EM residency directors, Sullivan et al.[21] report that the majority of professional lapses are discovered through informal “curbside” complaints of faculty or nurses, and less than 8% come from ED evaluations. The superiority of face-to-face communication in revealing concerns in learners' professionalism is supported by other researchers.[37] Critical incident reporting systems may help overcome some of those barriers.

The Physicianship Evaluation Form was developed for the reporting of unprofessional behavior of medical students by course and clerkship directors. The form has been indirectly validated in a 2004 study by Papadakis et al.,[38] who found an increased likelihood of state medical board disciplinary action if a student had negative excerpts in his or her medical school records significant enough to have warranted submission of a Physicianship Evaluation Form (odds ratio = 2.15). The American Board of Internal Medicine has developed comment cards that have been used for assessment of professionalism in internal medicine residents, but we have not been able to identify any studies evaluating the reliability or validity of this tool.[39] We have also been unable to identify any literature documenting the use and evaluation of this type of tool in EM residents.

One weakness of these systems is that they tend to focus on the extremes of behavior and therefore are not ideal tools for assessing all levels of professionalism. Another concern is that physicians have demonstrated difficulty in identifying issues with and providing feedback on professionalism, making faculty education and training an important determinant of success for these systems.[40]

Portfolios and Narratives

A portfolio can take many forms, but in general terms, comprises the work that a student feels demonstrates his or her progress and success in a given area and can include essays, presentations, patient notes, letters, and any other evidence the student finds important to demonstrate his or her achievement.[41, 42] Portfolios have been utilized and assessed in EM residency programs as well as in other disciplines and health professions.[43]

Narratives have been employed in the teaching and assessment of professionalism.[44-46] A workshop for EM residents was created to teach the concepts of altruism, excellence, and duty and to incorporate those values into future job performance. Residents were subsequently assessed on the timely completion of residency-related duties, with most participants demonstrating improvement.[47] In evaluating the ability of fourth-year medical students in an EM clerkship to reflect on and interpret aspects of professionalism during an isolated event, Baernstein and Fryer-Edwards[48] utilized the critical incident report (CIR), a brief written account of an event that is judged meaningful to the learner. In that study, the CIR was found to be inferior to a one-on-one interview with a faculty member in the learner's ability to identify and explore issues of professionalism arising in the chosen event.

An assessment of these tools is made difficult by the diversity of forms they take and by the complexities of converting narrative data into quantitative metrics. Tochel et al.[43] performed a systematic review of all articles assessing portfolios for postgraduate assessment and education. She found few studies with high enough quality design to make generalized statements about the tool's effectiveness. The reliability of portfolios for assessment has been shown to be improved by increasing the number of reviewers, training those reviewers, and using the portfolio as one component among other metrics to assess competence. There is a theoretical, though inconclusively proven, utility of portfolios for formative assessment.

Other concerns about portfolios exist as well. Some authors argue that portfolios are insufficient to assess professionalism because they are limited by the scope of the material submitted by the learner.[49] There is not convincing evidence that portfolios do promote reflection, and some groups have expressed decreased willingness to share reflections when concerned that the information could be used against them.[43] One residency program in psychiatry compiled a portfolio of documents created during patient care to assess residents (though not explicitly professionalism), which decreases the burden on residents to create new works to contribute.[42] This may not translate to EM residencies because the patient care documents created do not typically lend themselves to robust assessments of professionalism.

Simulation

Simulation in all its forms has been increasingly used in medical education. The evidence supporting its use in professionalism, though, is lacking.

Standardized patients have been employed as both patient and rater to assess professionalism. Zabar and colleagues[50] used unannounced SPs in a real clinical setting to evaluate EM residents' communication and professionalism competence. They determined that this approach was feasible and noted no difference in scores from residents who identified the SP during the encounter versus those who did not.

Professionalism has also been assessed with OSCEs by multiple researchers. Many have reported success in multiple populations: in medical students[51] and residents, both in EM[52, 53] and in other specialties,[54] although rigorous reliability and validity analysis is lacking. Finally, one group evaluated professionalism in EM residents using high-fidelity simulation during a crisis resource management course with an unvalidated checklist and was able to demonstrate construct validity using their tool for the specific scenarios they created.[55]

While OSCEs have gained acceptance for their reliability and validity in measuring many of the competencies, there are scant data on how effective they are at assessing professionalism.[56] In particular, concerns remain regarding inter-rater reliability, interstation consistency, and construct validity depending on the scenarios created.[51, 54, 57, 58] Implementation of simulation for assessment is also limited by the amount of money and faculty time required.

Discussion

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Summary and Recommendations
  7. References
  8. Supporting Information

We have identified a multitude of tools used in the assessment of professionalism. When evaluating the utility of an assessment tool it is helpful to consider five elements: reliability, validity, educational impact, acceptability, and cost (feasibility).[59] Unfortunately, none of the identified instruments have been evaluated rigorously enough to define all five of these factors in an EM residency setting (Table 1). While reliability and validity may be generalizable if staying within similar populations, other factors such as acceptability and feasibility may be residency program dependent. The potential for educational impact was often implicit but rarely explicitly evaluated in the studies reviewed. None of the assessment tools we have identified have as yet been evaluated rigorously enough in EM residents to confidently recommend their use for summative evaluation purposes.

Table 1. Potential Professionalism Assessment Toolbox for EM Residents
ToolReliabilityValidityFeasibilityStudied in EMReference
  1. CIRs = critical incident reports; MSF = multisource feedback; NA = not applicable; OSCE = objective structured clinical exam; SP = standardized patient.

Ethical/moral reasoning tests+++Rest et al.[14]
Direct observation++/–+Shayne et al.[19]
 +++/–Cruess et al.[18]
MSF+++Garra et al.[32]
 ++++Lockyer et al.[31]
CIRsNA++Papadakis et al.[38]
Portfolios+/–+Baernstein et al.[48]
Simulation     
High-fidelity++/–+Gisondi et al.[55]
SP+/–+Zabar et al.[50]
OSCE+/–+Wallenstein et al.[56]

While only a minority of the instruments have been evaluated in EM residency settings, many still may be appropriate for them. However, professionalism is culture- and context-specific, so tools from other environments should be reevaluated in U.S. EM settings prior to their adoption.[60]

As professionalism is a complex construct, it is recommended that it be assessed using a variety of tools so that all of its important elements are included.[49, 60-62] This multimethod assessment concept is called triangulation, and it is useful when attempting to assess complex human behavior.[63] Wilkinson et al.[49] developed a framework for assessing professionalism that involved deconstructing professionalism into themes and subthemes. He identified gaps in seven subthemes, six of which fell under the theme of commitment to autonomous maintenance and continuous improvement of competence in self, others, and systems. Many of these subthemes may be assessable using well-designed qualitative assessment tools such as portfolios. At this point, however, those types of tools still need to be developed for EM residents.

Summary and Recommendations

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Summary and Recommendations
  7. References
  8. Supporting Information

Professionalism is a complex construct that defies comprehensive assessment with a single instrument. The assessment of professionalism should be programmatic, multimodal, and longitudinal. Qualitative assessments will probably need to be included to avoid gaps in assessing the myriad of elements that comprise the construct of professionalism. The consensus process has led us to develop the following research agenda:

  • 1.
    Because multiple experts advocate for a multimodal approach to professionalism assessment, develop and evaluate triangulation strategies that combine accuracy with efficiency.
    1. Determine if certain modalities or tools have sufficient psychometric rigor to allow for summative assessment.
  • 2.
    Evaluate whether non-EM tools (e.g., DIT2, P-MEX) can produce data that are valid and reliable reflections of professionalism in EM residents.
  • 3.
    Evaluate qualitative tools, such as portfolios, for EM professionalism assessment.
    1. Conduct a needs assessment of current practices among EM residencies.
    2. Determine the essential elements of a portfolio so that it provides a meaningful representation of an EM resident's professional development.
    3. Develop and evaluate portfolio assessment rubrics.
  • 4.
    Evaluate existing MSF instruments.
    1. Determine the number of evaluations needed to achieve adequate reliability, especially for patient evaluations.
    2. Confirm the validity of these instruments in EM residents.
    3. Evaluate strategies, such as rater training, to minimize biases that may compromise the validity of MSF assessments.
  • 5.
    Direct observation may be an ideal method to assess professionalism; however, certain issues merit further investigation.
    1. Evaluate implementation strategies that improve feasibility.
    2. Evaluate the SDOT's validity specifically for professionalism.
    3. Create and validate a direct observation tool incorporating the ACGME NAS EM Milestones.
  • 6.
    Evaluate the role of simulation in the assessment of professionalism, especially considering the importance of context in professional behaviors.
    1. Explore which, if any, aspects of professionalism are best suited to evaluation with simulation.
  • 7.
    Explore in what format the critical professional lapse should best be reported.
    1. Evaluate the utility of existing critical incident reporting tools.
    2. Determine the role of these reports in formative and summative assessment.

References

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Summary and Recommendations
  7. References
  8. Supporting Information

Supporting Information

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Summary and Recommendations
  7. References
  8. Supporting Information
FilenameFormatSizeDescription
acem12031-sup-0001-DataS1.docWord document35KData S1. National Accreditation System5 Milestones: Professionalism.The document is in PDF format.

Please note: Wiley Blackwell is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.