A quantitative analysis of the perceived barriers to implementation of multisource feedback in surgical training

The adoption and integration of work‐based assessments by surgical units and training programs continues to increase, it is important to identify challenges in their implementation. The authors evaluated the barriers involved in the deployment of a supervisor assessment tool in Australia.


Introduction
Multisource feedback is a method of workplace-based assessment that was first introduced to the medical profession in the 1990 s in United States and Canada. 1,2 It has since been used across various specialities in numerous countries and has demonstrated to be a feasible method to evaluate physician performance in both technical, and non-technical skills. 3 Specialities that have begun adopting the multisource feedback instrument include, but not limited, to emergency medicine, internal medicine, paediatrics, psychiatry and radiology. [4][5][6][7][8][9] Implementation of a multisource feedback instrument does not have to be done in a single step. Due to the flexibility of the method, the instrument can be introduced to supervisors first, prior to integrating junior colleagues, nursing staff, allied health and patients to achieve the full comprehensive 360 evaluation. The authors have previously designed and published a novel instrument that was designed to incorporate the Royal Australian College of Surgeons (RACS) original nine competencies utilizing the JDocs framework. 10 It is generally accepted that clinical assessment instruments are superior when the rubric is behaviourally anchored and referencing specific behaviour patterns, rather than generic descriptors of values and principles. [11][12][13][14] In this pilot study, behaviourally anchored questions were used to maximize relevance and accuracy of the RACS competencies to the daily clinical practice of trainees.
As the adoption and integration of work-based assessments by surgical units and training programs continues to increase, it is important to identify challenges in their implementation and solutions to optimize engagement and acceptance. The barriers to multisource feedback have been researched in non-surgical fields but there remains a paucity of studies looking specifically in the surgical setting. 15,16 Barriers, such as those relating to infrastructure, administrative and limited resources are present. Other barriers such as acceptability, bias in the selection of assessors and lack of resources required for follow-up are important considerations. It is not clear whether there may be differing levels of understanding, experience and acceptance by trainees and surgical units, and evaluation of these instruments need to be specific to the surgical culture. 5,17,18 Although instruments are available from international sources, there is a paucity of suitable and validated instrument for the Australian surgical training context, particularly those in the prevocational stage of their training. 15,16,19,20 The original pilot study by the authors has introduced a novel instrument for the Australian surgical training context. 10 We evaluate the potential barriers involved in its deployment, and assess the relative importance of these factors specific to our instrument in the Australian surgical training climate to allow future adjustments, optimisation of participation rates, engagement and improvement in acceptance and uptake as a step towards expanding it into a multisource assessment.

Methods
In the original study we piloted the use of a novel instrument for assessment of technical and non-technical skills in surgical trainees. The novel instrument was designed based on the RACS competencies utilizing the JDocs framework and covers 48-items across nine surgical competencies. The instrument was presented as a formative assessment for research purposes and was aligned to the surgical competencies that was delivered in April 2019 to trainees working in the ACT and South-East NSW health network. Trainees that participated included prevocational surgical registrars, registrars on a vocational program (the RACS Surgical Education and Training (SET)), and post-specialty examination 'fellows'. Participating trainees nominated two supervisors to provide ratings utilizing an analogous modified version of the instrument to suit the supervisor perspective. 10 The original instrument used in the pilot study is included as supplementary data for reference.
In this follow-up study to our original pilot, a list of variables that impact multisource feedback assessments was created based on existing literature from international, and non-surgical contexts. 17,[21][22][23] The factors thought to be relevant to the provision and reception of accurate evaluations in the Australian surgical context were reviewed by four independent senior surgical and medical educators in the ACT Health and included in the final questionnaire with 15 items. Questions designed were single-step, delivered in a balance of affirmative, negative and neutral with no leading questions. 24 The follow-up questionnaire (Appendix 1) was delivered in September 2019 to all original participants (34 trainees and 25 supervisors) of the original pilot study. Participants then rated each item on a 5-point Likert scale. All collected data was deidentified and the instrument was accessible for 6 weeks on SurveyMonkey (Momentive Inc., San Mateo, CA, USA) for participants to complete. The average time it took a trainee and supervisor to complete the original pilot assessment along was reviewed.

Results
During the survey period, 23 of 25 (92%) supervisors and 31 of 34 (91.1%) trainees completed the questionnaire. The average ratings on the perceived impact that each factor has on the accurate evaluation of trainees along with the Student's t-test is presented in Table 1 and Figure 1. The results show that both trainees and supervisors felt the assessment form to be easy to access and navigate, agreed that a greater number of evaluations from both junior and senior colleagues would be welcomed, and that this instrument was a good system to provide feedback with. However, face-to-face discussions for feedback was preferred and finding time to complete the assessment was difficult.
The results also show that there was a statistically significant difference in the perspectives surrounding trainee-supervisor relationships, how a trainee manages negative feedback, the consequences of poor scores and the perceived importance of anonymity and input from allied health and nursing staff.

Discussion
A barrier to implementation of any work-based assessment instrument is accessibility. The ease of access afforded by the electronic delivery method and cross platform interface could have contributed to the good response rate. In our original pilot study of the novel instrument, 60% of trainees and 98% of supervisors responded. 10 This was reflected in the scores provided by both supervisors and trainees ( Table 1). The electronic delivery method and formatting of our pilot study was able to overcome this barrier. 10,22 In the original pilot study of the novel instrument, trainees on average took 14 min 36 s to complete the assessment, while supervisors took 26 min 56 s (84% longer). 10 Given the time commitment needed to complete evaluations, it is important to consider whether protected time should be provided to trainees and supervisors when this instrument is no longer research and formally used as part of an assessment program in surgical units. Future iterations of the assessment could explore whether it is possible to further streamline some questions in each competency.
The intention of feedback is to allow an opportunity for trainees to reflect and identify their weaknesses in order to improve upon them. A concern with providing this feedback as a supervisor is whether it may upset the supervisor-trainee relationship, particularly when negative feedback is provided. 9,21 Supervisors felt that if they provided trainees with ratings that were too negative, there may be consequences for both parties -this is concerning (Fig. 1). The consequences could include formal remediation for trainees, extra mentoring sessions with supervisors and the plethora of paperwork associated with it. However, the ratings from trainees suggest that they do not perceive this as an issue (Table 1). This difference in perspective provides reassurance for supervisors that perhaps their concern does not translate into reality. Having an instrument that de-identifies the feedback further alleviates supervisor concern. With further expansion in the use of this instrument in the future, the benefits of anonymity to provide accurate and honest feedback may become more apparent.
One common goal among any assessment and evaluation of a trainee is to identify candidates that are performing below the expected standard; to allow an opportunity to implement strategies and help the trainee improve their weaknesses. This goal of assessment conflicts with the potential reluctance to provide honest ratings, and originates from a far more complex social dynamic around supervisor attitudes on multisource feedback. 5,9,17,21 A broader implementation of this instrument with more evaluators would improve the identification of struggling trainees by normalizing the distribution of all trainee scores and identifying outliers rather than being fooled by raw and averaged scores that are seemingly good with 'all fours and fives'; where in actual fact scoring a four may be well below the average trainee.
Supervisors and trainees both felt that this instrument was a valuable method of providing and receiving feedback (Fig. 1). The more conventional approach of providing feedback on the job, or as a formal face-to-face session still plays an important role. Interestingly, the positive outlook on both modalities involved creates a duality, rather than a dichotomy. Trainee perspectives appear to be far more accepting of senior colleague evaluation and feedback compared to other potential parties such as allied health and nurses and this is consistent with existing literature from other fields (Fig. 1). 5,17,21,23 Ultimately, surgical education retains many elements of an apprenticeship model and the ingrained feeling that non-surgeons should not be evaluating surgeons. 5,17 True to some extent, an obvious example would be the evaluation of surgical technique. The ideal approach would be integrating both face-toface feedback sessions and evaluation questionnaires into surgical training rather than trying to identify and selecting a single superior method for use.
This study into the perceived barriers of a work-based assessment in the format of an evaluation form highlights the areas needing improvement. We recognize our pilot study is a deployed as a supervisor feedback assessment with only two supervisor evaluations per trainee and not as a complete multisource feedback assessment. The lack of other colleague evaluations in our pilot means we are unable to draw conclusions on how a multisource format may differ, but it still provides us with a direction for future iterations. An advantage of this instrument is that it has been designed to be easily scalable into a comprehensive multisource feedback evaluation tool. The flexibility of our instrument allows modifications to fit the evaluator's role appropriately and it is possible to adapt the current two versions (trainee version, and supervisor version) to better suit junior colleagues, allied health or patients in order to enable a complete multisource feedback assessment. For example, an appropriate change to this instrument if it were to be provided to a physiotherapist who often works with the trainee in the outpatient clinic, would be the exclusion of technical skills. Assessment would focus more on communication skills, professionalism or other non-technical skills. The expansion and implementation of this instrument could be performed in several steps to allow participants to familiarize themselves with the assessment and improve acceptance by trainees, supervisors, surgical educators and other involved parties. 6,17,21,25,26 A limitation in our study lies in our approach where we have gathered quantitative data centred around known barriers described in the literature from other non-surgical specialities. A qualitative study specific to surgery may be warranted to identify if there are any additional barriers unique to the surgical environment. Another consideration is the good response rate in the original pilot study, which could have confounding factors that contributed to, such as the authors having a local presence.
The multisource feedback approach has the benefit of gaining additional insight from the perspectives of other members of the multidisciplinary team, and even patients. With continued exposure of this approach, acceptance would likely improve over time as trainees become increasingly sensitized to this method. Further investigation into how to best incorporate nursing, allied health staff and patient perspectives would be valuable. When the instrument is delivered, emphasis needs to be placed on the fact that feedback will be in conjunction traditional methods and this instrument is not a complete replacement.

Conclusion
The findings of this study demonstrate that the assessment form is well received by trainees and supervisors with numerous factors being rated positively. There is disparity between trainee and supervisor views on the presence of different barriers. The results provide reassurance that the perception of supervisors who felt that negative ratings may affect supervisor-trainee relationships is unfounded. The results also demonstrate the acceptability of a digital format in the distribution of self-assessment and supervisor assessment forms. This method of assessment should provide future trainees an additional source of feedback, and an opportunity for self-reflection during their training.