Peer assessment of professionalism attributes

Peer assessment can support the development of professionalism by providing feedback that enables learners to reflect on their professional behavioural attributes.

they valued the peer assessment and feedback process. Although the assessment was formative and confidential, students were reluctant to provide negative comments about their peers. 'Disengaged', 'aloof' and 'argumentative' were the most likely negative adjectives that indicated students with low-level professionalism concerns.
Implications: Future development will focus on introducing students who can act as peer champions for the process and repeating the peer assessment over time to identify the change in professionalism development.

| INTRODUCTION
Peers are in a unique position to assess and provide feedback on their peers' professional behaviour because they have opportunities to see each other more closely compared with their tutors and supervisors. 1,2 However, students may struggle to report honestly, in case they cause upset to their peers; such anxieties can be minimised through an appropriately informed assessment system 3 and ensuring that peer assessment remains formative. 4 During the pre-clinical years, peer assessment for the development of professionalism has been successfully implemented by many medical schools, often by assessing and providing feedback on limited observable behaviours such as punctuality. 4,5 A more meaningful approach is where the focus is on the attributes that drive these observable behaviours, such as conscientiousness. [6][7][8] However, currently, there is a lack of tools that focus on behavioural attributes within undergraduate medical education to support the development of professionalism. Therefore, we developed, implemented and evaluated a new and innovative approach for online multiple peer assessment during the clinical years that focuses on professional behaviour attributes.
A more meaningful approach to peer assessment of professional behaviour is where the focus is on the attributes that drive observable behaviours.

| APPROACH AND EVALUATION
The study was conducted within an integrated undergraduate medi-

| Phase 1: Development of the peer assessment tool
A literature review of tools for peer assessment of professionalism in medicine did not identify an appropriate tool. However, Burns et al. 9 conducted a study with preclinical students in which peer assessors were asked to anonymously choose at least three adjectives presented in a list describing the student they were assessing. This study also found an association between students given a small number of negative adjectives and future concerns about their professionalism.
A working group of staff and potential medical student users developed an online tool for assessment and feedback by generating a list of adjectives about professionalism, and these were matched to Burns et al. 9 's list. Through group discussions, until consensus was reached, a final list of adjectives was produced, 10 grouped into four domains (integrity, conscientiousness, agreeableness and resilience) and contained an equal mix of positive and negative adjectives (see Figure 1). These domains, which represent professional behavioural attributes, are core to medical professionalism and good medical practice. 8, 11 The peer assessors' adjective-based ratings were collated through an online form as a word cloud. Larger font sizes indicated more assessors selecting that particular adjective. Assessors were encouraged to provide free-text comments that justify their selection of the adjectives.
The peer assessors' adjectivebased ratings were collated through an online form as a word cloud.

| Phase 2: Implementation of peer assessment
In their first year of study, the students have an interactive lecture on how to provide feedback to others. Peer assessment was implemented with all students (n = 291) towards the end of the first year of their clinical course (3rd year of the programme), having completed a selfassessment with the same tool 4 months beforehand. It was explained to students that the assessment was intended to provide confidential and anonymous feedback for reflection and development and would not be used to make progress decisions. It was emphasised that if students had significant concerns about a peer, this must be raised through the reporting process in place and not through the peer assessment. During a 6-week window, students selected and received feedback from up to 12 peer assessors, who could be nominated from within any phase of the programme to allow a broad selection and to prevent an overload of students within the clinical year group being assessed. Assessors were instructed to choose at least two words from each domain and as many words as they felt appropriate.
Detailed guidance was provided in addition to a named contact in case there were any concerns. The results were checked by staff before it was released, and students given one or more negative adjectives were offered the opportunity to discuss their feedback before it was released. Students who had very few ratings (less than six raters) were also invited to discuss their experiences and reflections. All students were also encouraged to discuss their feedback with a staff member.

| Phase 3: Mixed-methods evaluation
There were four purposes as informed by Kirkpatrick's framework 12 and described below. Students understood the profiles were formative only. All interviews were audio-recorded, transcribed and thematically analysed using a coding matrix by PVS, which was validated by FO. 13 Two main themes were identified and supported by example quotes in Box 1:

Experience of receiving feedback
Students receiving their peer assessment made them realise the exercise was worthwhile and that peers had insights that others do not have, although they would like to know their peer assessor. (Quote 1) Most students considered that feedback with negative adjectives was important, even if it appeared harsh. (Quote 2)

Experiences of giving feedback
A number of students expressed they will never express their negative view about their friends. (Quote 3)

Feedback and student profiles
All students had a peer assessment, with 11.6% of the students receiving ratings from six or lesser peers, 21.9% from seven to nine peers and 66.4% from 10 or more peers.

Identification of concerns
We evaluated the number of adjectives in the word cloud with the feedback provided as free text comments through thematic analysis, as in component 1. 14 Students with a positive profile and more than 10 raters were more likely to receive comments that were lengthy and supported by examples, including descriptions of personal qualities relevant to becoming a good doctor (Table 1). However, when this profile is made up of six assessors or less, the feedback was likely to be shorter, generic and also less likely to focus on how the student was in the workplace. For profiles that contained at least one negative adjective, the feedback was unlikely to focus on how the student was with patients but on their relationship with the peer. Only 15/41 students were given text feedback here.
Students with a positive profile and more than 10 raters were more likely to receive comments that were lengthy and supported by examples.

Identifying early professionalism concerns
To identify adjectives that were more likely to indicate lowlevel professionalism concerns, we identified 10 participants who were already on the school's progression database with three or more concerns, such as poor engagement during placements. Their peer-assessment profiles were analysed through an odd ratio calculation in SPSS 25 against the rest of the year group (Table 2) to obtain a 'measure of association' that quantifies the relationship between a word being present in a profile only when there is a concern.
The comparison group of 10 students had an average of 7-9 raters; no one in the comparison group received less than 6 ratings.
However, a stepwise maximum likelihood logistic regression was also conducted to identify the most popular positive and non-positive adjectives that are likely to be chosen when there is a concern, a measure in addition to odd ratios. The predicted model is À6.53

| IMPLICATIONS
The students found the peer assessment process valuable, enabling them to appreciate the perspective of another person, and engagement with the process was high, with the whole year group participat- ing. Feedback provided about professional concerns was often without explanation, even when there was the assurance of confidentiality and the process was formative. The most likely selected adjectives were disengaged, aloof and argumentative if they had professionalism concerns about another student, and the selection of manipulative, prejudiced and unstable was avoided. The only change planned for the assessment tool is to replace the adjective 'prejudice' with 'biased' as it was not chosen by anyone.
The most likely selected adjectives were disengaged, aloof and argumentative if they had professionalism concerns about another student.
Within Kirkpatrick's 12 framework, the data on feedback and student perceptions indicate the peer assessment enables students to T A B L E 1 Illustrative quotations from free-text feedback.
Positive profile <10 raters 'X is a very hardworking and conscientious student. She forms good relationships with her peers and staff in the medical school. X is very approachable and I believe that I could turn to her if I had any problems or needed help understanding something. She is reliable and punctual and has a lot of respect for patients we work with. X also engages with a lot of activities outside of the medical curriculum such as running and cycling, she has a keen interest in nutrition which she is hoping to intercalate in next year. X is a very well rounded and kind person and I enjoy working and studying alongside her.' 6 assessors or less 'Very easy going and friendly to everyone on placement. Has a nice manner with patients'.
Profiles with at least one negative adjective Encouraging students to constructively contribute to their peers' professional development is a complex change management process.
Future development will focus on introducing peer champions for the process, especially to demonstrate how to effectively provide freetext feedback in addition to the word cloud and to support students in using that feedback information for their development. We also plan to implement additional opportunities within the curriculum for students to develop their confidence in providing feedback to peers.
An important aspect will be to provide examples of feedback so students understand what excellent feedback looks like, especially when choosing negative adjectives for the word cloud. Developing students' skills in providing meaningful feedback that helps their peers' professional development is also relevant for their future postgraduate practice. 15 Repeating the peer assessment over time is important to identify the change in professionalism development, and we plan to conduct long-term follow-up, including research on the development of skills in providing peer feedback.