- Top of page
Context Many studies have examined how educational innovations in postgraduate medical education (PGME) impact on teaching and learning, but little is known about effects in the clinical workplace outside the strictly education-related domain. Insights into the full scope of effects may facilitate the implementation and acceptance of innovations because expectations can be made more realistic, and difficulties and pitfalls anticipated. Using workplace-based assessment (WBA) as a reference case, this study aimed to determine which types of effect are perceived by users of innovations in PGME.
Methods Focusing on WBA as a recent instance of innovation in PGME, we conducted semi-structured interviews to explore perceptions of the effects of WBA in a purposive sample of Dutch trainees and (lead) consultants in surgical and non-surgical specialties. Interviews conducted in 2011 with 17 participants were analysed thematically using template analysis. To support the exploration of effects outside the domain of education, the study design was informed by theory on the diffusion of innovations.
Results Six domains of effects of WBA were identified: sentiments (affinity with the innovation and emotions); dealing with the innovation; specialty training; teaching and learning; workload and tasks, and patient care. Users’ affinity with WBA partly determined its effects on teaching and learning. Organisational support and the match between the innovation and routine practice were considered important to minimise additional workload and ensure that WBA was used for relevant rather than easily assessable training activities. Dealing with WBA stimulated attention for specialty training and placed specialty training on the agenda of clinical departments.
Conclusions These outcomes are in line with theoretical notions regarding innovations in general and may be helpful in the implementation of other innovations in PGME. Given the substantial effects of innovations outside the strictly education-related domain, individuals designing and implementing innovations should consider all potential effects, including those identified in this study.
- Top of page
It is widely recognised that, in addition to hard work and tenacity, the success of innovations depends on the early identification of potential pitfalls and opportunities.1,2 This notion has driven some studies of innovations in medical training programmes.3,4 In the domains of business, psychology and sociology, innovation research has focused on the diffusion and implementation of innovations,5,6 but very few, if any, empirical studies have addressed effects that emerge only after the implementation of an innovation, but may hold the key to its lasting success.7 In medical education research, the extent to which desired educational effects are achieved has been the subject of most studies of innovations, including those on recent innovations in postgraduate medical education (PGME), such as simulation-based education, workplace-based assessment (WBA) and portfolios.8–11 In addition to their intended impact, however, innovations may affect other areas of day-to-day practice.7 It is not yet clear what types of effects innovations in the field of medical education can bring about. Therefore, it is as yet unclear which areas of day-to-day practice might be affected by an innovation and should be considered when designing and implementing innovations in PGME.
The principal established theory about the diffusion of innovations is based in sociology. Its founder Rogers considered that innovations are diffused through communication among members of the social system affected by them.2 He proposed five dimensions of effects of innovations: (i) desirable versus undesirable effects; (ii) direct versus indirect effects; (iii) anticipated versus unanticipated effects; (iv) effects on adopters versus effects on rejecters of an innovation, and (v) effects that increase or decrease equality among people. Rogers2 also posited that innovations have a form, function and social meaning, which may be perceived differently by the developers and adopters of an innovation. Although it can trigger unanticipated effects, the social meaning of an innovation is particularly prone to being overlooked by developers.7
Although it provides a starting point for notions about effects of innovations, this theory does not point out the domains of day-to-day practice that might be affected by an innovation. To study the range of effects in the domain of PGME, we explored how users of one innovation perceived its effects in day-to-day clinical and educational practice.
We studied WBA, which is currently in various stages of implementation in many PGME programmes worldwide. Numerous studies of its educational impact12–15 and of instruments such as the mini-clinical evaluation exercise (mini-CEX)16 (for the assessment of clinical and generic competencies) and the objective structured assessment of technical skills (OSATS)17 (for the assessment of technical and procedural skills) have been conducted. The effects of WBA on learning, teaching, supervision, trainees’ clinical confidence, and trainees’ and assessors’ attitudes towards the concept have been studied,14,18,19 and concerns regarding its appropriate use have prompted further research.20 Some of these study groups have also reported unintended effects of the innovation. However, these noted unintended effects remain mostly within the education arena (e.g. stimulation of structure in training activities,14 inducing stress,15 improvement of junior residents’ skills after training seniors21). This is not surprising because these studies were not conducted with the intention of exploring all types of effects of innovations, including those beyond the area of education.
The addition of insights into effects in all areas of day-to-day practice may paint a more realistic picture of educational innovations and their effects, which may facilitate their implementation and adoption and enhance their effectiveness. Therefore, we sought to establish what types of effect of WBA are perceived by consultants and trainees in the course of using WBA in the clinical workplace.
- Top of page
The total of 17 participants included seven trainees and 10 consultants (including four lead consultants), and represented four different specialties, eight different departments and five hospitals (Table 1).
Table 1. Characteristics of participants
| ||Consultants, n||Trainees, n||Total, n|
|Internal medicine|| 2|| 2||4|
|Paediatrics|| 3|| 2||5|
|Obstetrics and gynaecology|| 3|| 1||4|
|Surgery|| 2|| 2||4|
|Total||10 (7 men)||7 (5 men)||17|
|Age, years, mean (range)||48 (35–61)||36 (30–44)|| |
|Time in present position, years, mean (range)||12 (1–27)|| 3.5 (2–6)|| |
We consecutively present the six different, albeit interrelated, domains of effects of WBA that resulted from the analysis of participants’ reports: sentiments; dealing with the innovation; specialty training; teaching and learning; workload and tasks, and patient care. These are illustrated by examples and quotations from the interviews. No differences were found between consultants and trainees, or among specialties. In fact, different participants made complementary contributions to the range of effects in each domain.
Participants expressed sentiments that related to their affinity with WBA.
Some participants expressed a clear understanding of the ideas underpinning WBA, such as direct observation and the documentation of focused feedback to promote learning, and felt the innovation made sense and was appropriate:
‘It makes you notice things at an earlier stage, which enables you to correct things and also, yes, make you provide a more nuanced training.’ (Consultant 4)
As the objectives of WBA and its place in the training programme were consistent with or complementary to participants’ natural approach to education, these participants seemed to incorporate WBA instruments into their work routines and did not feel constrained by the mandatory use of standard assessment forms. Rather, these participants indicated that they customised their use of the forms to match different situations in the workplace.
Other participants said they understood the concepts behind WBA and subscribed to its goals, but found the instruments quite unfamiliar and not compatible with their customary approaches to feedback and assessment. They mentioned that although they experienced WBA as quite demanding, they incorporated it into their teaching because they considered it to be worth the effort.
Participants who did not really understand the objectives of WBA said that they adhered to their customary approaches and used the WBA formats as obligatory add-ons. They perceived WBA as a formal exercise with little educational value and experienced it as a burden:
‘In my opinion, you learn this profession by doing. It’s a craft, we shouldn’t complicate things: you need to see a lot and do a lot. Feedback follows naturally. I can’t squeeze everything into forms.’ (Consultant 1)
Affinity with WBA was recognised in analysis as extending beyond a characteristic of individual participants. Individual affinity and sentiments in the training groups seemed to be interrelated. Training groups, representing social systems in which the lead consultant in each played a prominent role, appeared to develop a shared attitude towards WBA. In addition, individual and group affinity with WBA impacted on other effects of working with WBA.
Participants’ emotions on the topic seemed to be mostly related to positive or negative experiences of WBA and its perceived value. Affinity with WBA was related to positive emotions, such as satisfaction with effective teaching and learning, pleasure from a conversation about the specialty and satisfaction with the good organisation of training in the department:
‘I like it [OSATS]. I liked it as a trainee, and now as a consultant I still like it. [...] it’s good to talk through the procedure together beforehand.’ (Consultant 5)
Negative emotions related mainly to the experience of an imbalance between the burden of conducting regular mandatory assessments using standardised instruments and the perceived (low) value of the assessments. Frustration and irritation were expressed mainly, but not exclusively, by participants who had little affinity with WBA. Most consultants mentioned emotions such as irritation or guilt when trainees asked for assessment at moments of high time pressure. Trainees reported feeling uneasy about asking a clearly reluctant consultant to assess them, and tense when being observed or receiving feedback. Some consultants were apprehensive when they had to give difficult feedback.
Dealing with the innovation
Participants reported customising WBA to fit their personal preferences. Their experiences with this innovation shaped their expectations and anticipation of any future innovations.
Shaping the use of WBA
Acknowledging that WBA was an innovation, participants assumed it would take time and practice to achieve optimal results:
‘I think that at first people thought: “Oh my, another load!” [...] But not anymore, I think. Because by now everyone knows that it actually doesn’t take much time, and that it does add value.’ (Trainee 2)
Individually and in group interactions, participants deliberated about the acceptance and practical implementation of WBA and the experiences of other groups. Consultants indicated that they adapted their usage of WBA to fit conditions in the workplace. One gynaecologist reported that if she did not do an OSATS immediately after a laparoscopic procedure, she would watch the video of the procedure later with the trainee in order to give concrete feedback. Participants revealed that WBA was mainly used for elements of training that were considered to be core components of the specialty in question. Most surgical participants, for example, considered the relevance of assessing trainee performance in an out-patient clinic to be low if the trainee had already mastered the technical skills required, such as suturing and physical examination:
‘And actually, yes, it is just expected of you that you’re capable of doing that [out-patient consultations].’ (Trainee 6)
Anticipating future innovations
Attitudes to future innovations appeared to be shaped by participants’ experiences with the current innovation. Participants who felt their group was successfully managing the use of WBA and understood and valued its contribution to training voiced no explicit misgivings at the prospect of further educational innovations. Participants who had experienced significant difficulties with the implementation of WBA, however, were more likely to express strong aversion to this prospect. To most participants, WBA was only one among many innovations in PGME, the rationales for which were not always clear to them. A frequently mentioned barrier to acceptance of innovations was the perceived lack of scientific evidence in support of their value:
‘With this [WBA] as well, I think you should do much more research [...] instead of changing things without reviewing them.’ (Consultant 9)
Consultants and trainees voiced increased interest in matters relating to specialty training, which seemed to be related to the introduction of WBA.
Specialty training as an area of interest
In analysis, the researchers noticed that the implementation of WBA drew attention to education and training. Trainees and consultants engaged in formal and informal conversations about ways to accommodate WBA in their work. The introduction of WBA made specialty training a topic of discussion in departments. Some participants noted that this type of attention promoted recognition of PGME as an area of interest in its own right:
‘I think there is increasing awareness that, if you really want to learn something during training, that you really have to be in charge. I think that’s what’s going on.’ (Trainee 2)
Shaping specialty training
The growing recognition of PGME as a field of interest stimulated training groups to discuss education, compare their activities with those of other training groups and consider matters of consent for content and activities. Individual trainers and trainees and training groups as social units appeared to be customising training activities to fit within their practice routines. Several participants, for example, pointed to an emerging shared value in their training group: ‘Good training involves WBA’, ‘Good WBA depends on good feedback’, ‘Good feedback depends on the application of the Pendleton rules, so if you want good training you should always use the Pendleton rules for feedback’.
Teaching and learning
Workplace-based assessment was regarded by the participants as both a stimulus and a hindrance to teaching and learning. It was said to stimulate the learning of trainees by promoting higher quality and more frequent feedback, and the WBA instruments, the mini-CEX form in particular, were considered to stimulate consultants to give competency-oriented and specific feedback. The provision of this type of feedback was generally considered to require practice and training:
‘A standard structure, that’s the essence. So everyone has the mini-CEX’s structure in mind, that you have to focus on specific competencies [...] Maybe another structure would be as good or even better, I don’t know, but a standard structure that you can look up, on the computer, that’s important.’ (Trainee 1)
Consultants and trainees said that writing down comments when discussing an observed activity stimulated precision and comprehension of feedback. They also mentioned that WBA instruments or structure seemed to encourage consultants to report poor performance, something they might have avoided previously. Our analysis of the data showed these effects to be strongly associated with an affinity for WBA. Those with less affinity usually expressed an inability to fit their comments into the prescribed structure of the instruments and consequently did not give frequent feedback. Some of them felt that feedback was deteriorating because of the compulsory use of WBA instruments:
‘I think they’re annoying forms to fill out. And sometimes that makes me think: let’s just skip it this time.’ (Consultant 1)
The structure imposed by WBA was considered by most participants to generate more frequent observation-based feedback and more feedback on inadequate performance:
‘Afterwards, they always tell you what went well and then also what went badly. Always. It’s never the case anymore that you only hear what you did well. That [has] absolutely changed in the past 2 years.’ (Trainee 2)
Because trainees whose performance was generally adequate or even exceptional received more feedback in the new system, some participants felt that trainees made more progress from an earlier stage of training:
‘They get feedback more quickly and can correct themselves when they do something wrong.’ (Consultant 4)
If working properly, electronic recording of WBA was considered to afford a good overview of trainees’ strengths, weaknesses and areas requiring improvement, but when programs were slow, crashes frequent or computers not readily available, electronic recording was experienced by participants as a hindrance to assessment.
According to some participants, the recording of focused feedback on all competencies helped trainees and (lead) consultants to pinpoint strengths and weaknesses, and to formulate focus points for training.
A good overview of their performance boosted the self-confidence of some trainees. Moreover, the availability of solid information from the review of assessments to guide progress interviews increased the value of these interviews in the eyes of lead consultants and trainees.
Focus of educational attention
In analysis, the mandatory nature of WBA was recognised as affecting participants’ focus on trainees’ activities. Trainees and consultants reported that they actively looked for opportunities in the workplace to ‘get one [an assessment] done’. They experienced that WBA was easier to arrange for some activities than for others, leading to practical considerations rather than educational relevance tending to direct educational attention. More specifically, tasks not related to patient encounters, such as handovers and presentations, were cited as opportunities for WBA because they involved the simultaneous presence of trainees and consultants, and were not subject to strict time constraints:
‘So routinely, after a nightshift, you get a mini-CEX for your patient handover, how it went.’ (Trainee 3)
By contrast, it took considerable effort to arrange the observation and discussion of patient contacts that were routinely performed by trainees on their own such as out-patient consultations. This logistical challenge discouraged the frequent assessment of trainees’ tasks in patient care:
‘I just don’t know how to arrange that, if I’m not there together with a surgeon already. Very often, that’s just not the case.’ (Trainee 6)
In a similar vein, participants from surgical specialties reported that mainly logistical considerations led to OSATS being conducted more easily and more frequently than mini-CEXs. This appeared to be strongly influenced by the culture in the training group: groups that considered surgical skills to represent the core components of their specialty were less willing to arrange for the assessment of other activities, as a result of which the mini-CEX was largely ignored and reduced to a mere checklist exercise.
Workload and tasks
Workplace-based assessment influenced users’ tasks and responsibilities and the workload they experienced.
All participants regarded conducting WBA as an extra task, but views of the related workload differed. Those with an affinity for WBA experienced less workload and considered tasks easier to perform or well worth the extra effort. This also applied for participants from departments in which clinical work was organised to create natural opportunities and time for assessment:
‘There has to be an opportunity during supervision. If there is enough time or at least set moments for supervision, than you can ask them: “Could you please fill out a mini-CEX?”’ (Trainee 5)
Both trainees and supervisors expressed a preference to be relieved of the workload created by their joint responsibility for WBA, with each preferring arrangements to be made by the other party.
Some trainees experienced a WBA-related reduction of workload attributable to the insight it afforded into their performance and learning goals.
Certain informal practices in relation to the allocation of WBA tasks in training groups were mentioned. Lead consultants were reported by everyone as conducting many assessments in order to set a good example, presumably because they feel responsible ensuring that WBA is implemented successfully. Trainees sought assessment mainly from lead consultants or from consultants they knew to be in favour of WBA who appeared to be least bothered by these requests and to give the best feedback:
‘Some consultants […] are more inclined to sit down at the computer and take time to discuss it. Well, and the lead consultant himself is also mini-CEX-minded. For the rest, it differs per consultant.’ (Trainee 3)
Balancing care and training
Participants’ struggles to balance patient care and teaching usually turned out unfavourably for training activities, which participants confessed to skipping or shortening. Participants with a strong commitment to training expressed regret over missed training opportunities, which again might increase their workload.
Supporters of WBA felt it helped trainees to provide better care at an earlier stage in their training. They saw improvements in all competencies and skills. Some noted that WBA also met current societal demands by its focus on generic competencies, particularly patient-centred communication, the only aspect of patient care that participants considered likely to benefit from WBA in the long run. Consultants and trainees reported experiences in which observation caused trainees to perform more awkwardly or more correctly than usual. More experienced trainees said these effects diminished as they became used to observation. As a result of more frequent observation, patients were confronted more often with the presence of an extra doctor:
‘It feels unnatural, you’re not used to it; logistics-wise it’s often inconvenient. And patients, they automatically start to talk to the person with most grey hair.’ (Trainee 4)
Participants speculated that effects on patients ranged from the causing of confusion to the provision of reassurance.
Acknowledgements: the authors thank Mereke Gorsira, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands, for editing the manuscript for submission.