SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. References

Context  Many studies have examined how educational innovations in postgraduate medical education (PGME) impact on teaching and learning, but little is known about effects in the clinical workplace outside the strictly education-related domain. Insights into the full scope of effects may facilitate the implementation and acceptance of innovations because expectations can be made more realistic, and difficulties and pitfalls anticipated. Using workplace-based assessment (WBA) as a reference case, this study aimed to determine which types of effect are perceived by users of innovations in PGME.

Methods  Focusing on WBA as a recent instance of innovation in PGME, we conducted semi-structured interviews to explore perceptions of the effects of WBA in a purposive sample of Dutch trainees and (lead) consultants in surgical and non-surgical specialties. Interviews conducted in 2011 with 17 participants were analysed thematically using template analysis. To support the exploration of effects outside the domain of education, the study design was informed by theory on the diffusion of innovations.

Results  Six domains of effects of WBA were identified: sentiments (affinity with the innovation and emotions); dealing with the innovation; specialty training; teaching and learning; workload and tasks, and patient care. Users’ affinity with WBA partly determined its effects on teaching and learning. Organisational support and the match between the innovation and routine practice were considered important to minimise additional workload and ensure that WBA was used for relevant rather than easily assessable training activities. Dealing with WBA stimulated attention for specialty training and placed specialty training on the agenda of clinical departments.

Conclusions  These outcomes are in line with theoretical notions regarding innovations in general and may be helpful in the implementation of other innovations in PGME. Given the substantial effects of innovations outside the strictly education-related domain, individuals designing and implementing innovations should consider all potential effects, including those identified in this study.


Introduction

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. References

It is widely recognised that, in addition to hard work and tenacity, the success of innovations depends on the early identification of potential pitfalls and opportunities.1,2 This notion has driven some studies of innovations in medical training programmes.3,4 In the domains of business, psychology and sociology, innovation research has focused on the diffusion and implementation of innovations,5,6 but very few, if any, empirical studies have addressed effects that emerge only after the implementation of an innovation, but may hold the key to its lasting success.7 In medical education research, the extent to which desired educational effects are achieved has been the subject of most studies of innovations, including those on recent innovations in postgraduate medical education (PGME), such as simulation-based education, workplace-based assessment (WBA) and portfolios.8–11 In addition to their intended impact, however, innovations may affect other areas of day-to-day practice.7 It is not yet clear what types of effects innovations in the field of medical education can bring about. Therefore, it is as yet unclear which areas of day-to-day practice might be affected by an innovation and should be considered when designing and implementing innovations in PGME.

The principal established theory about the diffusion of innovations is based in sociology. Its founder Rogers considered that innovations are diffused through communication among members of the social system affected by them.2 He proposed five dimensions of effects of innovations: (i) desirable versus undesirable effects; (ii) direct versus indirect effects; (iii) anticipated versus unanticipated effects; (iv) effects on adopters versus effects on rejecters of an innovation, and (v) effects that increase or decrease equality among people. Rogers2 also posited that innovations have a form, function and social meaning, which may be perceived differently by the developers and adopters of an innovation. Although it can trigger unanticipated effects, the social meaning of an innovation is particularly prone to being overlooked by developers.7

Although it provides a starting point for notions about effects of innovations, this theory does not point out the domains of day-to-day practice that might be affected by an innovation. To study the range of effects in the domain of PGME, we explored how users of one innovation perceived its effects in day-to-day clinical and educational practice.

We studied WBA, which is currently in various stages of implementation in many PGME programmes worldwide. Numerous studies of its educational impact12–15 and of instruments such as the mini-clinical evaluation exercise (mini-CEX)16 (for the assessment of clinical and generic competencies) and the objective structured assessment of technical skills (OSATS)17 (for the assessment of technical and procedural skills) have been conducted. The effects of WBA on learning, teaching, supervision, trainees’ clinical confidence, and trainees’ and assessors’ attitudes towards the concept have been studied,14,18,19 and concerns regarding its appropriate use have prompted further research.20 Some of these study groups have also reported unintended effects of the innovation. However, these noted unintended effects remain mostly within the education arena (e.g. stimulation of structure in training activities,14 inducing stress,15 improvement of junior residents’ skills after training seniors21). This is not surprising because these studies were not conducted with the intention of exploring all types of effects of innovations, including those beyond the area of education.

The addition of insights into effects in all areas of day-to-day practice may paint a more realistic picture of educational innovations and their effects, which may facilitate their implementation and adoption and enhance their effectiveness. Therefore, we sought to establish what types of effect of WBA are perceived by consultants and trainees in the course of using WBA in the clinical workplace.

Methods

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. References

Setting

The study was performed in the Netherlands, where national guidelines for competency-based PGME came into effect in 2011. Specialty training is delivered by hospital departments, some of which began to use WBA instruments prior to 2011. All consultants in a department are expected to contribute to training, and trainees are expected to actively engage in their learning by reflecting, seeking feedback and documenting their progress, usually in an electronic portfolio. The ‘lead consultant’ in the department has overall responsibility for the programme. The guidelines require two to four annual progress interviews with each trainee, guided by WBA data in the trainee’s portfolio. Commonly used WBA instruments include the mini-CEX, the OSATS and multi-source feedback. All members of the ‘training group’, consisting of all consultants and trainees in a department, are expected to contribute to training, both individually and as a team.

Design

The research team consisted of medical doctors and educationalists with ample experience in medical education. Our epistemology was constructivist: we assumed that knowledge about the phenomenon at hand is constructed in dialogue between the researcher and participant, and therefore diverse interpretations of reality might arise, depending on the individuals involved.22 Accordingly, we performed this study with a phenomenological approach, aiming to gain insight into participants’ own experiences and perceptions, and by interpreting these accounts to identify some commonalities in these perceptions.23 Given the paucity of research into non-education-related effects of educational innovations in PGME, we conducted an exploratory qualitative study using a design informed by Rogers’2 diffusion of innovations theory. We conducted and analysed individual, face-to-face, semi-structured interviews with trainees and consultants guided by theoretical concepts concerning the diffusion, implementation and dimensions of effects of innovations.2,6 Individual interviews as opposed to group interviews were expected to elicit more details about personal experiences.24

Participants and procedure

With the aim of identifying a variety of effects, we purposively sampled trainees and (lead) consultants from different hospitals, from surgical (obstetrics and gynaecology, and surgery) and non-surgical (internal medicine and paediatrics) specialties and from specialties with differing degrees of experience of WBA. (Workplace-based assessment was introduced earlier in obstetrics and gynaecology, and paediatrics than in surgery and internal medicine25.) To ensure an equal distribution of different WBA users in the sample, we aimed to include per specialty at least two trainees and two consultants (but only one lead consultant). To explore interactions between users at departmental level, we aimed to recruit at least two participants from each department.

E-mail addresses of trainees and consultants from 10 departments (two internal medicine, two paediatrics, three obstetrics and gynaecology, and three surgical departments, ranging in size from five trainees and nine consultants to 80 trainees and 75 consultants) in six different hospitals were obtained from departmental secretaries. All of these trainees and consultants were sent an invitation to participate by individual e-mail. Because only one of the 11 participants in the first 2 months came from a surgical department, we sent a second e-mail to the (trainee) surgeons in one surgical department at that time. Of the 32 potential participants who responded to our invitation, 28 agreed to participate and four declined for time constraint-related or unspecified reasons. An individual appointment was made with each participant for an interview in his or her office. The interviews were conducted between September and December 2011.

Ethical considerations

Written informed consent was obtained from all participants, who were assured that the data would be processed anonymously. The study was approved by the ethical review board of the Dutch Society for Medical Education (Nederlandse Vereniging voor Medisch Onderwijs [NVMO]; dossier number 81).

Interviews

All interviews were conducted by the principal researcher using an interview guide based on the research questions and notions regarding the potential consequences of innovations, such as different dimensions and theories on the development of consequences (Fig. 1).2 In keeping with the research approach and the goal of the study, the interviewer asked open-ended questions on the topics in the interview guide, and also probed emerging issues that seemed of interest, for which diversion from the proposed order of topics was accepted. The interviews took 30–45 minutes, were audio-recorded and transcribed verbatim by an experienced transcriber.

image

Figure 1.  Interview guide. Mini-CEX = mini clinical evaluation exercise; OSATS = objective structured assessment of technical skills

Download figure to PowerPoint

Analysis

We analysed the data using template analysis.26 This is a supporting technique for the analysis of qualitative data, the characteristics of which make it suitable for use in a study using a constructivist approach that is guided by theory. It involves creation of a template, which is a schema of (coded) themes that are identified as important in the data and represents the relationships between these themes as recognised in analysis. It enables researchers to explicate their assumptions (i.e. from existing theory) about possible themes in the data, without having these assumptions restrict the process of analysis to these assumptions. Thus, the analysis starts from an ‘initial template’ containing a priori themes which can be based on relevant literature, themes derived from initial coding of part of the dataset and on researchers’ own assumptions. This template is then modified by iteratively adding, deleting and reorganising themes as coding continues.

Open coding of the data and construction of the templates was conducted by the main researcher (JPIF). The initial template consisted of theoretical topics as used for the interview guide combined with themes that had resulted from analysis of the first two interviews. Based on this initial template, the consecutive interviews were analysed by JPIF and the template modified in the process. To prevent the premature narrowing of ideas, identified themes and relationships were discussed with the whole research team at the points of analysis of interviews 3, 6 and 11. To this purpose, the seventh transcript was also analysed by a second researcher (MW), using open coding which was compared with the template generated by the principal researcher. Discussion of discrepancies slightly altered the relations between themes but yielded no new themes. After 15 interviews, theoretical saturation of the data was reached as no new insights were emerging. The inclusion of new participants was stopped, but two further interviews had already been conducted. A discussion by JPIF, MW, NvdL and FS of the template and the relationships among the categories led to modification of the template: an initial division of effects based on individual versus group level was abandoned. After examining the applicability of the modified template to all 15 interviews, JPIF adjusted the wording of the modified template to better fit daily practice vocabulary. After JPIF had applied the final template to the transcripts of interviews 16 and 17, the template was agreed on by the research team.

Results

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. References

The total of 17 participants included seven trainees and 10 consultants (including four lead consultants), and represented four different specialties, eight different departments and five hospitals (Table 1).

Table 1.   Characteristics of participants
 Consultants, nTrainees, nTotal, n
Internal medicine 2 24
Paediatrics 3 25
Obstetrics and gynaecology 3 14
Surgery 2 24
Total10 (7 men)7 (5 men)17
Age, years, mean (range)48 (35–61)36 (30–44) 
Time in present position, years, mean (range)12 (1–27) 3.5 (2–6) 

We consecutively present the six different, albeit interrelated, domains of effects of WBA that resulted from the analysis of participants’ reports: sentiments; dealing with the innovation; specialty training; teaching and learning; workload and tasks, and patient care. These are illustrated by examples and quotations from the interviews. No differences were found between consultants and trainees, or among specialties. In fact, different participants made complementary contributions to the range of effects in each domain.

Sentiments

Participants expressed sentiments that related to their affinity with WBA.

Affinity

Some participants expressed a clear understanding of the ideas underpinning WBA, such as direct observation and the documentation of focused feedback to promote learning, and felt the innovation made sense and was appropriate:

‘It makes you notice things at an earlier stage, which enables you to correct things and also, yes, make you provide a more nuanced training.’ (Consultant 4)

As the objectives of WBA and its place in the training programme were consistent with or complementary to participants’ natural approach to education, these participants seemed to incorporate WBA instruments into their work routines and did not feel constrained by the mandatory use of standard assessment forms. Rather, these participants indicated that they customised their use of the forms to match different situations in the workplace.

Other participants said they understood the concepts behind WBA and subscribed to its goals, but found the instruments quite unfamiliar and not compatible with their customary approaches to feedback and assessment. They mentioned that although they experienced WBA as quite demanding, they incorporated it into their teaching because they considered it to be worth the effort.

Participants who did not really understand the objectives of WBA said that they adhered to their customary approaches and used the WBA formats as obligatory add-ons. They perceived WBA as a formal exercise with little educational value and experienced it as a burden:

‘In my opinion, you learn this profession by doing. It’s a craft, we shouldn’t complicate things: you need to see a lot and do a lot. Feedback follows naturally. I can’t squeeze everything into forms.’ (Consultant 1)

Affinity with WBA was recognised in analysis as extending beyond a characteristic of individual participants. Individual affinity and sentiments in the training groups seemed to be interrelated. Training groups, representing social systems in which the lead consultant in each played a prominent role, appeared to develop a shared attitude towards WBA. In addition, individual and group affinity with WBA impacted on other effects of working with WBA.

Emotions

Participants’ emotions on the topic seemed to be mostly related to positive or negative experiences of WBA and its perceived value. Affinity with WBA was related to positive emotions, such as satisfaction with effective teaching and learning, pleasure from a conversation about the specialty and satisfaction with the good organisation of training in the department:

‘I like it [OSATS]. I liked it as a trainee, and now as a consultant I still like it. [...] it’s good to talk through the procedure together beforehand.’ (Consultant 5)

Negative emotions related mainly to the experience of an imbalance between the burden of conducting regular mandatory assessments using standardised instruments and the perceived (low) value of the assessments. Frustration and irritation were expressed mainly, but not exclusively, by participants who had little affinity with WBA. Most consultants mentioned emotions such as irritation or guilt when trainees asked for assessment at moments of high time pressure. Trainees reported feeling uneasy about asking a clearly reluctant consultant to assess them, and tense when being observed or receiving feedback. Some consultants were apprehensive when they had to give difficult feedback.

Dealing with the innovation

Participants reported customising WBA to fit their personal preferences. Their experiences with this innovation shaped their expectations and anticipation of any future innovations.

Shaping the use of WBA

Acknowledging that WBA was an innovation, participants assumed it would take time and practice to achieve optimal results:

‘I think that at first people thought: “Oh my, another load!” [...] But not anymore, I think. Because by now everyone knows that it actually doesn’t take much time, and that it does add value.’ (Trainee 2)

Individually and in group interactions, participants deliberated about the acceptance and practical implementation of WBA and the experiences of other groups. Consultants indicated that they adapted their usage of WBA to fit conditions in the workplace. One gynaecologist reported that if she did not do an OSATS immediately after a laparoscopic procedure, she would watch the video of the procedure later with the trainee in order to give concrete feedback. Participants revealed that WBA was mainly used for elements of training that were considered to be core components of the specialty in question. Most surgical participants, for example, considered the relevance of assessing trainee performance in an out-patient clinic to be low if the trainee had already mastered the technical skills required, such as suturing and physical examination:

‘And actually, yes, it is just expected of you that you’re capable of doing that [out-patient consultations].’ (Trainee 6)

Anticipating future innovations

Attitudes to future innovations appeared to be shaped by participants’ experiences with the current innovation. Participants who felt their group was successfully managing the use of WBA and understood and valued its contribution to training voiced no explicit misgivings at the prospect of further educational innovations. Participants who had experienced significant difficulties with the implementation of WBA, however, were more likely to express strong aversion to this prospect. To most participants, WBA was only one among many innovations in PGME, the rationales for which were not always clear to them. A frequently mentioned barrier to acceptance of innovations was the perceived lack of scientific evidence in support of their value:

‘With this [WBA] as well, I think you should do much more research [...] instead of changing things without reviewing them.’ (Consultant 9)

Specialty training

Consultants and trainees voiced increased interest in matters relating to specialty training, which seemed to be related to the introduction of WBA.

Specialty training as an area of interest

In analysis, the researchers noticed that the implementation of WBA drew attention to education and training. Trainees and consultants engaged in formal and informal conversations about ways to accommodate WBA in their work. The introduction of WBA made specialty training a topic of discussion in departments. Some participants noted that this type of attention promoted recognition of PGME as an area of interest in its own right:

‘I think there is increasing awareness that, if you really want to learn something during training, that you really have to be in charge. I think that’s what’s going on.’ (Trainee 2)

Shaping specialty training

The growing recognition of PGME as a field of interest stimulated training groups to discuss education, compare their activities with those of other training groups and consider matters of consent for content and activities. Individual trainers and trainees and training groups as social units appeared to be customising training activities to fit within their practice routines. Several participants, for example, pointed to an emerging shared value in their training group: ‘Good training involves WBA’, ‘Good WBA depends on good feedback’, ‘Good feedback depends on the application of the Pendleton rules, so if you want good training you should always use the Pendleton rules for feedback’.

Teaching and learning

Workplace-based assessment was regarded by the participants as both a stimulus and a hindrance to teaching and learning. It was said to stimulate the learning of trainees by promoting higher quality and more frequent feedback, and the WBA instruments, the mini-CEX form in particular, were considered to stimulate consultants to give competency-oriented and specific feedback. The provision of this type of feedback was generally considered to require practice and training:

‘A standard structure, that’s the essence. So everyone has the mini-CEX’s structure in mind, that you have to focus on specific competencies [...] Maybe another structure would be as good or even better, I don’t know, but a standard structure that you can look up, on the computer, that’s important.’ (Trainee 1)

Consultants and trainees said that writing down comments when discussing an observed activity stimulated precision and comprehension of feedback. They also mentioned that WBA instruments or structure seemed to encourage consultants to report poor performance, something they might have avoided previously. Our analysis of the data showed these effects to be strongly associated with an affinity for WBA. Those with less affinity usually expressed an inability to fit their comments into the prescribed structure of the instruments and consequently did not give frequent feedback. Some of them felt that feedback was deteriorating because of the compulsory use of WBA instruments:

‘I think they’re annoying forms to fill out. And sometimes that makes me think: let’s just skip it this time.’ (Consultant 1)

The structure imposed by WBA was considered by most participants to generate more frequent observation-based feedback and more feedback on inadequate performance:

‘Afterwards, they always tell you what went well and then also what went badly. Always. It’s never the case anymore that you only hear what you did well. That [has] absolutely changed in the past 2 years.’ (Trainee 2)

Because trainees whose performance was generally adequate or even exceptional received more feedback in the new system, some participants felt that trainees made more progress from an earlier stage of training:

‘They get feedback more quickly and can correct themselves when they do something wrong.’ (Consultant 4)

If working properly, electronic recording of WBA was considered to afford a good overview of trainees’ strengths, weaknesses and areas requiring improvement, but when programs were slow, crashes frequent or computers not readily available, electronic recording was experienced by participants as a hindrance to assessment.

According to some participants, the recording of focused feedback on all competencies helped trainees and (lead) consultants to pinpoint strengths and weaknesses, and to formulate focus points for training.

A good overview of their performance boosted the self-confidence of some trainees. Moreover, the availability of solid information from the review of assessments to guide progress interviews increased the value of these interviews in the eyes of lead consultants and trainees.

Focus of educational attention

In analysis, the mandatory nature of WBA was recognised as affecting participants’ focus on trainees’ activities. Trainees and consultants reported that they actively looked for opportunities in the workplace to ‘get one [an assessment] done’. They experienced that WBA was easier to arrange for some activities than for others, leading to practical considerations rather than educational relevance tending to direct educational attention. More specifically, tasks not related to patient encounters, such as handovers and presentations, were cited as opportunities for WBA because they involved the simultaneous presence of trainees and consultants, and were not subject to strict time constraints:

‘So routinely, after a nightshift, you get a mini-CEX for your patient handover, how it went.’ (Trainee 3)

By contrast, it took considerable effort to arrange the observation and discussion of patient contacts that were routinely performed by trainees on their own such as out-patient consultations. This logistical challenge discouraged the frequent assessment of trainees’ tasks in patient care:

‘I just don’t know how to arrange that, if I’m not there together with a surgeon already. Very often, that’s just not the case.’ (Trainee 6)

In a similar vein, participants from surgical specialties reported that mainly logistical considerations led to OSATS being conducted more easily and more frequently than mini-CEXs. This appeared to be strongly influenced by the culture in the training group: groups that considered surgical skills to represent the core components of their specialty were less willing to arrange for the assessment of other activities, as a result of which the mini-CEX was largely ignored and reduced to a mere checklist exercise.

Workload and tasks

Workplace-based assessment influenced users’ tasks and responsibilities and the workload they experienced.

Workload

All participants regarded conducting WBA as an extra task, but views of the related workload differed. Those with an affinity for WBA experienced less workload and considered tasks easier to perform or well worth the extra effort. This also applied for participants from departments in which clinical work was organised to create natural opportunities and time for assessment:

‘There has to be an opportunity during supervision. If there is enough time or at least set moments for supervision, than you can ask them: “Could you please fill out a mini-CEX?”’ (Trainee 5)

Both trainees and supervisors expressed a preference to be relieved of the workload created by their joint responsibility for WBA, with each preferring arrangements to be made by the other party.

Some trainees experienced a WBA-related reduction of workload attributable to the insight it afforded into their performance and learning goals.

Task allocation

Certain informal practices in relation to the allocation of WBA tasks in training groups were mentioned. Lead consultants were reported by everyone as conducting many assessments in order to set a good example, presumably because they feel responsible ensuring that WBA is implemented successfully. Trainees sought assessment mainly from lead consultants or from consultants they knew to be in favour of WBA who appeared to be least bothered by these requests and to give the best feedback:

‘Some consultants […] are more inclined to sit down at the computer and take time to discuss it. Well, and the lead consultant himself is also mini-CEX-minded. For the rest, it differs per consultant.’ (Trainee 3)

Balancing care and training

Participants’ struggles to balance patient care and teaching usually turned out unfavourably for training activities, which participants confessed to skipping or shortening. Participants with a strong commitment to training expressed regret over missed training opportunities, which again might increase their workload.

Patient care

Supporters of WBA felt it helped trainees to provide better care at an earlier stage in their training. They saw improvements in all competencies and skills. Some noted that WBA also met current societal demands by its focus on generic competencies, particularly patient-centred communication, the only aspect of patient care that participants considered likely to benefit from WBA in the long run. Consultants and trainees reported experiences in which observation caused trainees to perform more awkwardly or more correctly than usual. More experienced trainees said these effects diminished as they became used to observation. As a result of more frequent observation, patients were confronted more often with the presence of an extra doctor:

‘It feels unnatural, you’re not used to it; logistics-wise it’s often inconvenient. And patients, they automatically start to talk to the person with most grey hair.’ (Trainee 4)

Participants speculated that effects on patients ranged from the causing of confusion to the provision of reassurance.

Discussion

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. References

We focused on WBA to explore different types of effects of innovations in PGME as perceived by the users of this innovation. Six domains of interrelated effects were distinguished, referring to: sentiments; dealing with the innovation; specialty training; teaching and learning; workload and tasks, and patient care.

Comparison with the literature

The effects identified in this study extend beyond the range of intended effects, which, in the case of WBA, include the facilitation and documentation of learning.16 This finding is in line with Rogers’ proposal to consider unintended, undesired and unexpected effects of innovations.7 The interrelatedness of the effect domains underlines the relevance of considering both intended and unintended consequences of innovations.

Several domains of perceived effects identified in this study about a medical education innovation are comparable with effects that have been previously recognised as impacts of innovations in health care practice, such as effects on task allocation, workflow issues, and sentiments and emotions.27 This consistency in the affected domains seems to indicate that, regardless of the nature of the innovation, some aspects of the daily practice of health care are particularly susceptible to effects of an innovation.

Of particular relevance to the meaningful use of innovation was the effect of users’ affinity with WBA. An explanation for the pervasiveness of this effect may be found in theory on the diffusion of innovations, particularly in relation to the notion that every actor has a certain probability of adopting an innovation.5 Our results suggest that adoption is affected by user affinity. The broad impact of affinity is also supported by the social psychological notion that beliefs influence behaviour.28 Diffusion of innovations theory further states that the probability of adoption can be altered by ‘communication and influence’,5 which may explain the strong effect on user affinity of lead consultants’ attitudes.

High WBA-related workload was not experienced exclusively by participants with low affinity. It also occurred when departments failed to incorporate WBA into work schedules. This finding indicates that adapting departmental organisation could accommodate the implementation of an innovation. This is in line with occupational psychology’s recognition that a supportive environment can motivate and engage people.29 Workload appears to be an important area to consider in innovations in specialty training.

Some of the effects, specifically users’ sentiments about the innovation and users moulding of their use of the innovation to fit their personal beliefs, resonate with effects reported in change management, both in general and in medical education specifically.1,3,30 This aptly illustrates how the effect domains we found relate to different aspects of innovations, such as their implementation and the way users incorporate the innovation into work routines. To anticipate and deal with effects such as negative emotions, it may be advisable to determine whether these are due to the change (management) or are caused by characteristics inherent to the innovation.

The reported tendency to focus on assessability rather than on educational and professional relevance of an activity emphasises the importance of maintaining a good fit between innovation and practice, and suggests that the mandatory implementation of an innovation may not be the best way to promote its meaningful use in specialty training.

Strengths and limitations

Broadening the perspective on consequences of changes in medical education, we explored a wide range of effects of an innovation in PGME. We appropriately conducted an exploratory study, with some guidance from existing theory. Template analysis enabled the researchers to give the analysis a theoretical foundation without being restricted by it. The resulting effect domains complement existing theory in that they do not require to be labelled as ‘unintended’ or ‘unexpected’.7

Because of our focus on the users of one particular innovation, the resulting domains are not necessarily exhaustive and studies of other innovations or from different perspectives may reveal different effect profiles. However, as our findings are supported by various theoretical perspectives, we expect that our conclusions bear some relevance to other innovations in PGME and other settings.

We found no consistent differences between male and female participants. However, the over-representation of male participants in this study may have influenced the present findings as a result of between-gender differences in, for example, perception and coping.31

Future research

Further research should determine whether the domains we identified also apply to other types of innovation and whether different perspectives yield additional domains of effects. It also seems worthwhile to examine which types of effect develop under which conditions in order to facilitate the optimal tailoring of the implementation of innovations to specific circumstances.

Implications for practice

Educationalists, administrators and clinicians who design and implement innovations in PGME should be aware that innovations may trigger a variety of effects in the workplace. Given the interrelatedness of the different effects, we suggest that all potential effects deserve careful attention. By looking at the intended and unintended effects of an innovation in medical education, this study offers those involved in current and future changes a framework for recognising the potential pitfalls of an implementation by directing their attention to six domains of effects of innovations.

Contributors:  all authors made substantial contributions to the study conception and design. JPIF acquired all data. JPIF analysed and interpreted the data in collaboration with PWT, MW, NvdL and FS. JPIF wrote the first draft of the article, with contributions from PWT. All authors contributed to the subsequent critical revision of the article and approved the final manuscript for publication.

Acknowledgements:  the authors thank Mereke Gorsira, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands, for editing the manuscript for submission.

Funding:  none.

Conflicts of interest:  none.

Ethical approval:  this study was approved by the ethical review board of the Dutch Society for Medical Education (Nederlandse Vereniging voor Medisch Onderwijs [NVMO]; dossier no. 81).

References

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. References