- Top of page
Medical Education 2012: 46:604–612
Objectives Feedback in workplace-based clinical settings often relies on expert trainers’ judgements of directly observed trainee performance. There is ample literature on effective feedback, but in practice trainees in workplace-based training are not regularly observed. We aimed to examine external conditions that impact feedback in observational workplace-based assessment (WBA).
Methods Interviews were conducted and the resulting data analysed using a qualitative, phenomenological approach. Between October 2009 and January 2010, we interviewed 22 postgraduate general practice trainees at two institutions in the Netherlands. Three researchers analysed the transcripts of the interviews.
Results A three-step scheme emerged from the data. Feedback as part of WBA is of greater benefit to trainees if: (i) observation and feedback are planned by the trainee and trainer; (ii) the content and delivery of the feedback are adequate, and (iii) the trainee uses the feedback to guide his or her learning by linking it to learning goals. Negative emotions reported by almost all trainees in relation to observation and feedback led to different responses. Some trainees avoided observation, whereas others overcame their apprehension and actively sought observation and feedback. Active trainers were able to help trainees overcome their fears. Four types of trainer–trainee pairs were distinguished according to their engagement in observation and feedback. External requirements set by training institutions may stimulate inactive trainers and trainees.
Conclusions In line with the literature, our results emphasise the importance of the content of feedback and the way it is provided, as well as the importance of its incorporation in trainees’ learning. Moreover, we highlight the step before the actual feedback itself. The way arrangements for feedback are made appears to be important to feedback in formative WBA. Finally, we outline several factors that influence the success or failure of feedback but precede the process of observation and feedback.
- Top of page
Feedback on trainees’ performance in workplace-based clinical settings generally reflects the judgement of experts who have observed the performance.1 The intended effect of this feedback is to help trainees learn and improve their performance and is based on the assumption that feedback creates awareness of shortcomings and thereby motivates learners to improve or change.2 This study focuses on formal feedback that is based on the observation of a trainee conducting a consultation with a single patient. Although frequent observations also provide information about performance for the overall assessment of trainees, we focus in this study on the formative aspect of feedback, whereby trainees purposefully invite and receive feedback in order to learn and improve from it. Many articles have reported on when and how feedback is likely to be most effective,1–4 but there are strong concerns that trainees only infrequently receive feedback based on direct observation of a patient encounter.5–7 We will first present an overview of the literature that is relevant to the scope of the study and will then examine practice in the clinical setting.
Most of the studies on feedback on performance have focused upon factors that influence the use of feedback and strategies for delivering it.3,4 In a review, Shute showed that the actual use of feedback in different settings depends on motivation (the trainee needs the feedback), opportunity (feedback is given in time to be used by the trainee) and means (the trainee is able and willing to use the feedback).3 Shute also provides comprehensive lists of guidelines to enhance learning.3 Some of the implications of the guidelines are that feedback should be given in response to a problem or task, may prevent or correct errors and misconceptions, should preferably be written and should promote a specific learning goal. It should also be objective, which means that effective feedback will allow for a comparison of performance with an established standard of performance. In a review of the literature on the impact of feedback on learning and achievement, Hattie and Timperley4 reported that effective feedback should focus on the task, process and (self-)regulation. Feedback relating to the personal level is rarely effective because trainees are very strongly focused on avoiding risk and failure. In their model of feedback, Hattie and Timperley also emphasised goal orientation exemplified by three questions that a trainee should be able to answer if feedback is to be effective: Where am I going? How am I going? Where do I go next?4 The model gives a good overview of how feedback might enhance learning. Both reviews also showed that the type and delivery of feedback influence its effectiveness and that it may even have a negative impact.3,4
A review of feedback within the medical domain, by Archer,8 indicated that for feedback to be relevant and effective, it should be specific and not exclusively trainer-driven. It should be part of a two-way process and trainees should be given the opportunity to reflect on their actions. In Archer’s model of effective feedback, feedback is not a series of unrelated events, but a sequential process linked to personal goals that includes self-monitoring (reflection on action) that is supported by external feedback.8 In a study of multi-source feedback for practising doctors, Sargeant et al.9 found that acceptance and usage of feedback by individual doctors depended on the nature of the feedback. Positive feedback appeared to be easily assimilated by recipients, whereas negative feedback was first appraised for its credibility based on its process (feedback based on observed performance), source, content and specificity, as well as on its congruence with feedback from other sources. Watling and Lingard also concluded that participants’ perceptions of an evaluation process profoundly affected the usefulness of the evaluation and the extent to which it achieved its goal.1 Similarly, Eva et al.10 found that the self-perceptions of recipients of feedback, relating to their confidence, experience and fear of not being sufficiently knowledgeable, impacted on their interpretation and uptake of feedback.
According to Norcini and Burch,5 there are indications that trainees are observed only rarely and that faculty staff play a critical role in the successful implementation of formative assessment.5,6 By contrast, in their review, Miller and Archer2 pointed to evidence that formative feedback in the workplace is highly appreciated by users, who (subjectively) reported positive educational impact. How is this positive subjective response of users reconciled with Norcini and Burch’s claim5 that observation and feedback happen only too rarely?
In summary, with respect to formal feedback based on the observation of a trainee during a consultation with a single patient, we know that formative feedback in workplace-based clinical settings relies on the judgement of expert trainers and that the literature has provided models, lists and guidelines indicating when feedback is likely to be effective and how it is best provided. However, there seems to be a discrepancy between the evidence in the literature and what we see in day-to-day practice, in which the application of the evidence with respect to feedback falls short of what we would expect. To clarify this issue, we conducted a qualitative study in which we explored the lived experiences and perceptions of individual trainees by addressing the following research questions:
1 How is feedback, based on observation of a trainee performing a consultation with a single patient, carried out in the workplace-based clinical setting?
2 Which aspects influence the feedback process in the workplace-based clinical setting?
- Top of page
Three steps appeared to be necessary for feedback to have a beneficial effect during single-encounter assessments. The first step concerned arrangements for observation and feedback made by trainer and trainee together. The second step related to the content and delivery of the feedback. The third step concerned the incorporation of the feedback in the learning process and required the trainee to accept the feedback, reflect on it in relation to his or her learning goals, and use it to plan some kind of action to pursue these learning goals.
We combined these three steps into a scheme (Fig. 1). We will discuss the three steps consecutively and then answer the second research question by showing which aspects influenced the extent to which the steps of the scheme were actually used. Finally, we will discuss the effects of the different approaches recommended by the two institutions.
Step 1: organisation of observation and feedback
The frequency of observation of consultations varied considerably and ranged from twice weekly to not at all. Remarkably, trainees who reported a high observation rate also reported having made agreements with their trainers on clearly defined training routines at the beginning of the training year:
‘We have an arrangement that we just, say during consultation hours, that she observes me every Thursday: my last two consultations on Thursday afternoons.’ (Trainee 16)
Trainers and trainees who had set clear routines for observation also scheduled time for feedback immediately following direct observation or in the same week in the case of videotape observation. These routines appeared to promote the effective completion of the first step.
Trainees spontaneously mentioned observation during home visits, and weekend duty or night shifts. These occasions created natural moments for observation because trainers and trainees were generally working together. Trainees remarked, however, that the content of these encounters and, consequently, of the feedback differed from that of ‘regular practice’. This meant that these natural moments for observation could not replace observation and feedback on ‘regular consultations’.
Step 2: content and delivery of feedback
The analysis showed that most of the feedback related to communication. Even with the use of the mini-CEX instrument, which specifically invited feedback on three competencies (medical expertise, communication and professionalism), communication was the predominant topic of feedback.
There was considerable variety in the delivery of feedback. At one end of the spectrum were very intensive sessions in which specific feedback was given and trainees reflected on their performance and considered further action to improve it. One trainee described this as:
‘We frequently stop it [the video]. And then we watch… mostly he first asks me what I think of it, and what I think I could have done differently, or should have done differently, or could have done differently […]. And the effect is that we frequently role-play in between.’ (Trainee 4)
At the other end of the spectrum lay superficial, non-specific feedback in which no attention was paid to reflection or further action. One trainee reported:
‘My GP-trainer […] said twice: “Yes […] I have nothing to do right now’; two or three times: “I can sit in with you.” And at the end of the consultation: “Rather fine, I would not have done it differently, seems accurate.”’ (Trainee 14)
Step 3: incorporation of feedback into the learning process
Some trainees proceeded from the first and second steps to the incorporation of the feedback into their learning. These trainees were aware of the relationship of feedback with their learning goals, their portfolio and the results of other assessments:
‘We used it because in my portfolio, due to, yeah to find out how you explore the “request for help”, and we did that with a mini-CEX.’ (Trainee 20)
Trainees used observation and feedback to judge their progress with regard to learning goals formulated earlier, and they also used feedback to formulate new learning goals. When a trainee had completed all three steps, the cycle was able to start again when observation was planned to elicit further feedback on the same learning goals or on newly formulated goals. This process is represented by the large arrow pointing from Step 3 back to Step 1 in Fig. 1.
Factors influencing the process of receiving feedback
Not all trainees used all three steps: some failed to take the first step or to proceed to the second or third steps. To answer our second research question, we investigated the factors influencing this process.
Attitudes towards observation and feedback
The analysis of the interviews showed that trainees’ reactions to observation were primarily emotional. They talked spontaneously about feeling apprehensive about observation, saying that they felt they behaved differently when they were being observed. They also said they did not like being observed. Both videotaped and direct observation were considered to interfere with normal practice:
‘It is intuitive, if someone watches you, I get nervous. It is not that my GP-trainer is doing, or not doing, something, but generally…’ (Trainee 10)
Although almost all trainees mentioned these emotions, two distinct patterns of responding to them emerged. Some trainees wanted to be observed and actively sought feedback, despite their fears. They acknowledged that they could benefit from feedback and that it could help them to improve their performance. Other trainees, however, allowed their fears to prevail and were reluctant to make arrangements for observation and feedback. As observation and feedback were not subject to external control (other than being strongly recommended), these trainees were able to avoid them.
Avoidance behaviour could be counteracted by an active GP-trainer. Based on the data from the interviews, two groups of trainers were identified: one set of trainers took initiatives to ensure that observation and feedback took place, whereas the other set refrained from observation and feedback:
‘My trainer said: “[…] shouldn’t we do something like a mini-CEX?” I said: “Yes.” And then he said: “With my last trainee we used the walk-in surgery...”’ (Trainee 1)
‘I think it [observation] is useful, but also scary, I think that’s the reason I am not inclined to mention it to my trainer. But if you just do it, then it gets easier […] and then it is very helpful.’ (Trainee 1)
‘I think it is just laziness that I don’t do it. And I notice that my GP-trainer does not ask for it either. And of course that is a little childish [of me].’ (Trainee 5)
Most trainers who actively observed their trainees also invited their trainees to observe them in order to provide a learning experience for the trainee. Based on the attitudes of trainees and trainers towards observation, four groups of trainee–trainer pairs were distinguished (Table 1).
Table 1. Types of trainer–trainee pairs based on responses to negative emotions relating to feedback
| ||GP-trainer shows active behaviour in relation to giving feedback||GP-trainer does not show active behaviour in relation to giving feedback|
|Trainee shows active feedback-seeking behaviour||Frequent observation and feedback||GP-trainer does not take the initiative in giving feedback When the trainee asks for feedback, it may or may not be provided|
|Trainee does not show active feedback-seeking behaviour||Trainee does not seek observation and feedback, but they take place because the trainer takes action||Observation and feedback take place only if they are externally required (by the department)|
Recommendations from the training institution
In addition to trainee and trainer attitudes, the recommendations of the training institution influenced the occurrence of observation and feedback. The institutional recommendations (direct observation using the mini-CEX format at least three times in 3 months [Nijmegen]; videotaped observation with a feedback session [Maastricht]) influenced the feedback process in various ways. Some trainees reported that they were observed because the institution required it, whereas other trainees said they would organise more observations if a minimum number was required and closely monitored by the institution. By contrast, trainer–trainee pairs who showed a positive attitude towards observation and feedback regarded the imposition of strict requirements as excessive regulation:
‘I think it depends on the relationship with the trainer. If things are going well, he [the trainer] knows how you work and how you are doing. But if things aren’t going well, then you can use it and say: Hey, we need to do this [the mini-CEX].’ (Trainee 15)
Differences among trainees
We interviewed trainees from two different institutions, each of which recommended a different approach to formative feedback on observed consultations. Nevertheless, in practice, trainees in both institutions used both direct and videotaped observation. The mini-CEX instrument was used only in Nijmegen, where videotaped observation was also used. In Maastricht, video-based assessment is standard, but some trainees were also observed directly. The main reasons for selecting one of the observation methods referred to the preferences of the trainee or trainer and the method’s practical feasibility in the general practice. We found no striking differences between Nijmegen and Maastricht. Trainees differed in their appreciation of the two methods, but selected the method they preferred. Appreciation depended on the degree of apprehension of the trainee and the personality of the GP-trainer. No differences emerged between trainees of different genders or years of education. Trainees in each category (institution, gender, year of education) showed no differences in whether they completed the three steps or failed to take the first, second or third step.
- Top of page
Although the literature on feedback has mainly focused on the content and delivery of feedback3,4,9 and on the recommendation that feedback should promote a learning goal3,4,8 (the second and third steps of our scheme), our results underscore the importance of the step before the actual provision of feedback in the workplace setting. Deliberate planning of observation and feedback appears to be essential. The feedback literature explains how feedback can be effective, based on the assumption that feedback takes place, but Norcini and Burch5 and the present results show that this assumption may be rather unrealistic.
Although the second step in our scheme has been described earlier, our findings confirm that it is an important aspect of the feedback process and that its occurrence should not be taken for granted; some trainees take the first step of the scheme, but receive no effective feedback at all or not in an appropriate manner. With respect to the third step, Archer8 stated that feedback should not comprise a series of unrelated events, but should be incorporated into the overall learning process by relating it to learning goals and plans for improvement. In line with this, Hattie and Timperley4 highlighted the need to resolve the question of ‘where to next’ in order to make feedback effective. In addition, Shute3 showed that formative feedback should promote an orientation based on learning goals. We think that feedback based on the observation of a consultation in formative workplace-based assessment (WBA) should not be an isolated event, but should represent the starting point of a continuing learning process. It can also be used to reflect on an ongoing learning goal.
Although feedback is a concept with many facets, the three-step scheme we propose can be useful in pinpointing exactly where things go wrong for trainees and thereby may prevent failure. Our second research question indicated our wish to further examine factors influencing the successful completion of the steps.
Apprehension about being observed and receiving feedback proved to have a powerful negative effect on feedback. The role of fear has also been described in the literature.14 We found, however, that apprehension can be overcome when trainees are motivated to actively seek feedback. Teunissen et al.15 found a similar pattern, showing that trainees are not merely passive recipients of feedback, but differ in their ways of seeking or avoiding it. Our results showed that not only trainee motivation, but also the role of the trainer can help to overcome avoidance patterns induced by negative emotions. Future research should further examine this area, particularly in light of the substantial added value of trainer-led initiatives shown in this study. A trainer who actively promotes observation and feedback may be able to counteract trainees’ avoidance behaviour with regard to feedback.
External regulation by the institutions also influenced the feedback process. The literature shows, however, that simply introducing a tool does not suffice to ensure good feedback that ties in with personal goals16 (E.A.M. Pelgrim, A.W.M. Kramer, H.G.A. Mokkink and C.P.M. van der Vleuten, unpublished data, 2011).
Our findings show that it remains important for trainers and trainees to actively pursue observation and feedback. Nevertheless, it may be advisable for institutions to enforce recommendations more strictly, given that our results indicate that it is quite easy for trainees and trainers not to comply with institutional recommendations at present. This suggests that the implementation of requirements for observation and feedback is a prerequisite for the provision of feedback in formative WBA. Possible ways to enhance implementation might include the provision of instructions and training for trainees and trainers, stipulating a mandatory frequency of observation and feedback, and conducting a quality review.
The factors described here underscore the complex relationships between the feedback receiver, the feedback giver, the training institution and probably other environmental influences as well. Our explorative qualitative study has resulted in a framework of important elements that should be taken into account with regard to feedback on observed consultations in daily practice, but we do not claim that it offers comprehensive coverage of all possible influencing factors. More research is required to further investigate variables, the impact of variables and their interactions. In addition, we focused on the formative purpose of observation and feedback, but this is not as absolute as outlined. The process is not strictly formative because when it is used frequently, the outcomes can be used by trainers as input for summative purposes.
The prominence of communication as a topic of feedback may be attributed to the strong emphasis placed on communication skills in general practice training. Another explanation may be that summative performance assessment of communication skills during general practice training in the Netherlands is based on videotaped observations and several trainees use this method to obtain formative feedback.
There are some limitations to our study. The fact that participants were self-selected may have introduced bias. We do not know if participating trainees differed from non-participating trainees in their ways of seeking and organising observation and feedback. However, our results show wide variations in trainee experiences and ideas about observation and feedback. Furthermore, the transferability of the present findings to other medical specialties and work-based settings may be limited because the study was confined to general practice trainees.
An important finding, which can be interpreted as contributing to the theory on feedback, concerns the strong impact of organisation and arrangements made for observations and feedback. Our results show that this crucial first step in the feedback process in formative WBA is influenced by the individual characteristics of both trainer and trainee, as well as by external regulations imposed by the training institution. We therefore recommend that institutions set a mandatory minimum number of observation and feedback sessions to be completed, supported by a mini-CEX instrument to direct feedback content and help trainees to connect the feedback with their learning goals. Moreover, institutions should train their trainers to make better use of observation as a teaching aid. Training could be structured around the three steps identified in this study. We also recommend further research into trainer behaviour in the provision of feedback in order to investigate how trainers perceive their role in the feedback process and how trainees and trainers influence one another’s feedback-giving and feedback-seeking behaviours.
Contributors: EAMP is the principle author of the article and contributed to the study design, interviewing, analysis of interviews, interpretation of interview data, and the writing and revision of the article. AWMK contributed to the study design, the first pilot interview and the revision of interview questions; AWMK also observed one interview and contributed to the analysis of interviews, the interpretation of interview data and to the revision of the article. HGAM contributed to the study design, the formulation of interview questions, analysis of the interviews, interpretation of data and the revision of the article. CPMvdV contributed to the study design, the interpretation of interview data, and the revision of the article. All authors approved the final manuscript for submission.
Acknowledgements: the authors thank Elke Butterbrod, Greetje Sagasser, Department of Primary Care and Community Care, Radboud University Nijmegen Medical Centre and Mereke Gorsira, Department of Educational Development and Research, Faculty of Health, Medicine, and Life Sciences, Maastricht University, for their help in preparing this manuscript, and all participating trainees for the time they invested in the study and the stories they related.