SEARCH

SEARCH BY CITATION

Keywords:

  • effectiveness;
  • innovative culture;
  • long-term care;
  • nurses;
  • nursing;
  • patient safety;
  • quality improvement collaborative

Abstract

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. Acknowledgement
  9. Contributions
  10. Conflict of interest
  11. Ethical approval
  12. Funding
  13. References
  14. Appendix

Aims and objectives.  Explore effectiveness of 11 collaboratives focusing on 11 different topics, as perceived by local improvement teams and to explore associations with collaborative-, organisational- and team-level factors.

Background.  Evidence underlying the effectiveness of quality improvement collaboratives is inconclusive and few studies investigated determinants of implementation success. Moreover, most evaluation studies on quality improvement collaboratives are based on one specific topic or quality problem, making it hard to compare across collaboratives addressing different topics.

Design.  A multiple-case cross-sectional study.

Methods.  Quality improvement teams in 11 quality improvement collaboratives focusing on 11 different topics. Team members received a postal questionnaire at the end of each collaborative. Of the 283 improvement teams, 151 project leaders and 362 team members returned the questionnaire.

Results.  Analysis of variance revealed that teams varied widely on perceived effectiveness. Especially, members in the Prevention of Malnutrition and Prevention of Medication Errors collaboratives perceived a higher effectiveness than other groups. Multilevel regression analyses showed that educational level of professionals, innovation attributes, organisational support, innovative culture and commitment to change were all significant predictors of perceived effectiveness. In total, 27·9% of the individual-level variance, 57·6% of the team-level variance and 80% of the collaborative-level variance could be explained.

Conclusion.  The innovation’s attributes, organisational support, an innovative team culture and professionals’ commitment to change are instrumental to perceived effectiveness. The results support the notion that a layered approach is necessary to achieve improvements in quality of care and provides further insight in the determinants of success of quality improvement collaboratives.

Relevance to clinical practice.  Understanding which factors enhance the impact of quality improvement initiatives can help professionals to achieve breakthrough improvement in care delivery to patients on a wide variety of quality problems.


Introduction

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. Acknowledgement
  9. Contributions
  10. Conflict of interest
  11. Ethical approval
  12. Funding
  13. References
  14. Appendix

Quality improvement collaboratives (QICs) are increasingly being used to improve quality of care. The Breakthrough method developed by the Institute for Healthcare Improvement (2003) has been one of the major instruments put to use in such collaboratives. In Breakthrough QICs, teams from different organisations join forces to improve care on a certain topic within a set time-frame, steered and supported by a faculty team. These teams will develop and implement improvement actions geared to their own organisations and client groups. Best practices or evidence-based interventions are the usual starting points and teams will learn about these at national conferences organised to this purpose. QICs are expected to enhance quality and efficiency of care by acting as a ‘learning laboratory’ stimulating and implementing innovations.

The evidence underlying the effectiveness of QICs is inconclusive (Leatherman 2002, Ovretveit 2002, Cretin et al. 2004, Schouten et al. 2008) and few studies investigated determinants of success (Mills & Weeks 2004, Neily et al. 2005, Dückers et al. 2009). Moreover, most evaluation studies on QICs are based on one specific topic, making it hard to compare across collaboratives addressing different topics. The objective of our study is to explore effectiveness of 11 collaboratives focusing on 11 different topics, as perceived by local improvement teams and to explore associations with collaborative-, organisational- and team-level factors. The results contribute to a better insight into the mechanisms underlying QICs and factors that enhance success.

To understand the operating mechanism of QICs, the ‘chain of action’ framework developed by Cretin et al. (2004) is used, which suggests a layered approach is needed to improve quality. The proposed chain of action begins with participating teams and their environment, the latter comprising the organisational context and the broader context of the collaborative itself. To explain perceived effectiveness, we examine collaborative-level factors, the organisational context of the team and team-level factors.

At the collaborative level, several conditions should be in place for teams to be effective. First, we hypothesise that if new working methods are perceived by professionals as relatively beneficial, compatible with norms and values, easy to learn and implement, allow for experimentation and have observable results, the implementation process is expected to be more successful (Rogers 1995). Second, it is expected that stimulating participants’ improvement efforts requires (1) a challenging and achievable collaborative target, (2) appropriate measures and usable monitoring tools that help teams make stepwise changes guided by measured results and keep them focused on the collaborative target (Øvretveit 2002, Øvretveit & Gustafson 2002, Øvretveit et al. 2002, Dückers et al. 2009) and (3) program management support (Benn et al. 2009, Dückers et al. 2009, Nembhard 2009).

According to the ‘chain of action’ framework, commitment to quality improvement, organisational support and organisational culture are considered important organisational-level conditions (Cretin et al. 2004, Lin et al. 2005). The more organisations involve their teams with quality improvement activities, the more the professionals will be committed to implementing changes and the more positive their perceptions of effectiveness will be. Also, organisational support in terms of leadership and active involvement of top management motivates professionals to achieve improvement (Gustafson et al. 2003, Mills & Weeks 2004, Dückers et al. 2009, Kaplan et al. 2010). Culture conveys the norms, values, beliefs and behaviours of an organisation, reflecting ‘how we do things around here’. The competing values framework distinguishes four types of culture: group (teamwork and participation), developmental (risk-taking, innovation and change), hierarchical (rules, regulations and bureaucracy) and rational (efficiency, goal attainment and achievement) (Zammuto et al. 2000, Shortell et al. 2004). Some studies suggest that organisations are most effective when a group culture is dominant (Lin et al. 2005). Shortell et al. (2004), however, suggest that each of the four types of cultures may contribute to effective quality improvement. Our hypothesis therefore is that the relative balance among the four culture types is associated with perceived effectiveness.

On the team level, or workgroup level as it is called in the ‘chain of action’ framework, compositional characteristics such as team size, educational level and presence of management have been found to play a role (Fried et al. 2000, Shortell et al. 2004). But also commitment to change and innovative culture are expected to be key determinants (Lin et al. 2005, Lemmens et al. 2009). Professionals who (1) are committed to change, (2) value the outcomes associated with successful implementation of changes in care processes and (3) believe that effort and implementation will lead to the targeted outcomes are key to successful improvement of quality of care. Innovative culture, conceptualised as social expectations of team members, may be more or less conducive to creativity and can facilitate implementation by generating social approval when working together effectively and acting quickly (Caldwell & O’Reilly 2003). To conclude, we expect that differences in perceived effectiveness can be explained by the aforementioned collaborative-, organisational- and team-level characteristics.

Methods

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. Acknowledgement
  9. Contributions
  10. Conflict of interest
  11. Ethical approval
  12. Funding
  13. References
  14. Appendix

Setting and design

This multiple-case cross-sectional study included quality improvement teams participating between 2006–2009 in 11 QICs which were part of a national Dutch program called ‘Care for Better’. Each collaborative focused on one specific quality topic. These were: pressure ulcers, ill-nutrition, prevention of sexual abuse, medication safety, fall prevention, problem behaviour, client autonomy and control, social participation, recovery-oriented care, somatic comorbidity of psychiatric clients and outreach care (Table 1 and Strating et al. 2008, 2011). Organisations from the following sectors participated: nursing homes, residential care homes, home care, care for people with a mental handicap and care for people with a physical disability.

Table 1. Overview of Care for Better collaboratives
CollaborativeTargetTeam levelIndividual level
No. of participating teams in QICNo. of teams with surveyResponseteams to survey (%)No. of participating team members in QICNo. of team members responding to surveyResponse team members to survey (%)
Prevention of pressure ulcersLowering the prevalence of pressure ulcers by 50%251560·01003434·0
Prevention of malnutritionLowering the prevalence of malnutrition by 40%383181·61527750·7
Prevention of sexual abuseScore 8 at a scale of 1–10 at each measure191368·4763850·0
Reducing medication errorsLowering the prevalence of medication errors by 30%482858·31447149·3
Reducing problem behaviourLowering the prevalence of problem behaviour261765·41044139·4
Prevention of fall incidentsLowering the prevalence of fall incidents by 30%242083·3966264·6
Improving autonomy and controlSubstantial improvement in autonomy and quality of life of clients452453·31355641·5
Social psychiatric careDecrease in problems increase in social functioning251456·01003030·0
Improving recovery-oriented careIncrease in quality of life increase in recovery-oriented services251872·01005050·0
Enhancing social participationDecrease in loneliness262180·81045754·8
Screening somatic co-morbidityAttaining normal BMI attaining normal systolic pressure181477·8723244·4
Total 31921567·4118354846·3

Program management was in the hands of the long-term care knowledge institute Vilans and it was commissioned by ZonMw, the main funding agency of health research in the Netherlands. As a research team, we were asked to describe the processes and effects of the collaboratives for clients and participating teams and to describe which interventions were actually carried out.

Set up of the quality improvement collaboratives

Each collaborative was led by a faculty team consisting of a program leader and other experts on the selected quality improvement topic. The improvement teams from the participating organisations were invited to attend four national conferences offering workshops and sessions where questions could be posed to other teams or to experts. The improvement teams developed and executed their interventions under the guidance of process counsellors. They used the Plan-Do-Study-Act cycle: carrying out small scale actions, measuring if the actions led to the expected outcomes and, if not, adjusting the actions.

Data collection and measures

As part of a larger evaluation study, team members received a postal questionnaire within one week after the last collaborative conference. Teams typically comprised five members, one of which was team leader. In total, 548 team members (about 46%, on average 2·6 per team) returned a questionnaire (see for response per collaborative Table 1). These 548 respondents represented 215 teams of the 319 participating teams (about 67% at the team level).

The questionnaire mostly consisted of existing validated measurements instruments that have been used before in quality improvement projects (Appendix). Most instruments were validated in health care and extensively tested in previous studies. Internal consistency of each scale based on our study results is represented by Cronbach’s alpha and is included in Table 2. Scores on all items of a scale were summed and divided by the number of items and higher scores indicate a higher degree of the underlying concept.

Table 2. Overview of theoretical constructs and instruments per variable
 No. of itemsPotential rangeActual rangeCronbach’s alpha
Perceived effectiveness41–51·25–5·00·82
Innovation’s attributes101–52·6–5·00·68
Program management expertise51–72·5–7·00·86
Advisor support41–71·5–7·00·84
Achievability41–71·75–7·00·77
Challenging targets11–71·0–7·0 
Measurability41–71·0–7·00·86
Quality improvement commitment81–52·27–5·00·85
Organisational support131–71–70·90
Cultural balance200–10–0·82 
Innovative culture151–51·93–3·620·81
Commitment to change181–24544–245 
Dependent variable

Perceived team effectiveness was assessed by four questions, using a five-point response scale (Lemieux-Charles et al. 2002, Lemieux-Charles & McGuire 2006). These questions assessed the extent to which each team member: (1) believed the team’s overall performance met expectations, (2) was satisfied with his/her experience as a team member, (3) felt positive about their experience and (4) would be willing to work in a similar team in the future. A higher score indicates a higher level of perceived effectiveness.

Independent variables at the collaborative level
  •  Innovation attributes were assessed with 10 items on the innovation’s relative benefit, compatibility, complexity and observability (Vos et al. 2008). Items were rated by each team member on a scale of 1 (totally disagree)–5 (totally agree) and summed to form one score.
  •  Program management expertise on breakthrough methodology and the collaborative topic, provision of information and advice was rated by project leaders with five items on a scale of 1–7 (Dückers et al. 2008). An example statement: ‘program management had sufficient expertise on the improvement methods’.
  •  Advisor’s support was assessed by four items. Project leaders rated the extent to which their advisor gave advice that was good and specific to the team’s needs and problems (Dückers et al. 2008). Four items were rated on a scale of 1–7. An example statement: ‘Our advisor was sufficiently responsive in the design of our action plan, implementation of improvement actions and measurements’.
  •  Achievability was assessed by four statements. Example statements are: ‘collaborative targets are achievable’ and ‘program management made clear how to achieve collaborative targets’ (Dückers et al. 2008). Rating was on a seven-point scale ranging from ‘strongly disagree’ to ‘strongly agree’. Higher scores indicated that team leaders perceived a higher degree of achievability of the collaborative’s targets.
  •  Challenging targets was assessed by project leaders who rated whether ‘Program management set high expectations with regard to performance and improvement possibilities’ (Dückers et al. 2008). Team leaders rated this statement on a seven-point scale ranging from ‘strongly disagree’ to ‘strongly agree’.
  •  Measurability was assessed by four statements. Example statements are: ‘measuring indicators helps to monitor progress’ and ‘there were clear agreements on measuring central indicators’ (Dückers et al. 2008). Rating was on a seven-point scale ranging from ‘strongly disagree’ to ‘strongly agree’. Higher scores indicated perception of a higher degree of measurability.
Independent variables at the organisational level
  •  Quality improvement commitment was assessed in the project leaders’ survey with eight items formulated by the European foundation for quality management (Shortell et al. 1995). Rating was on a five-point scale ranging from ‘strongly disagree’ to ‘strongly agree’. Example statements were ‘Realising improvements is rewarded in this organisation’ and ‘Our board of directors is actively involved in quality improvement’.
  •  Organisational support was assessed by 13 items of existing questionnaires (RAND 1999) on availability of time and means and on the degree of encouragement from top management. Rating was on a seven-point scale ranging from ‘strongly disagree’ to ‘strongly agree’. An example statement: ‘Senior management encouraged staff to improve their performance’.
  •  Organisational culture was assessed in line with the competing values framework (Shortell et al. 1995, Zammuto et al. 2000). Team members distributed 100 points across four sets of organisational statements (representing the culture types) according to descriptions that best fit their organisation. The Blau Index of heterogeneity (Blau 1977) was calculated to assess the level of balance between the four culture types. A score of 1 indicates that points were apportioned in a 25/25/25/25 pattern and indicates an optimal balance.
Independent variables at the team level
  •  Team composition characteristics were based on individual socio-demographic characteristics. Education level was assessed by a 0–7 point ordinal scale, higher scores indicating a higher educational level. A variable indicating whether a manager was part of the team was computed. Project leaders were asked whether any changes (dropouts or new team members) occurred.
  •  Innovative culture of the team was assessed by 15 items of the Group Innovation Inventory (Caldwell & O’Reilly 2003, Strating & Nieboer 2010, Nieboer & Strating 2011). Respondents were asked to answer statements on a five-point scale ranging from ‘strongly disagree’ to ‘strongly agree’. Higher scores indicated a more innovative culture. An example statement: ‘The attitude around here is that when you are trying new things, mistakes are a normal part of the job’.
  •  Commitment to change was assessed by 28 items with a seven-point rating scale based on the expectancy measurement for motivation developed by Vroom (1995). Three subscales were computed: expectancy (perceived probability that effort will lead to good performance), instrumentality (perceived probability that good performance will lead to desired outcomes) and valence (value that an individual personally places on these outcomes). A composite measure was calculated as the product of valence, instrumentality and expectancy. Example items are ‘how important do you find making changes that improve processes of care?’, ‘success in implementing changes in care will help improve quality of care’ and ‘exerting effort will help implement changes in care for clients’.

Analysis

Owing to missing data on one or more of the variables, a sample of 513 was used for the analyses. We examined Spearman or Pearson correlations. Because of the hierarchical structure of the data (individuals are nested within teams and within collaboratives), a normal regression design would lead to estimation errors. We thus employed multi-level techniques (mixed models option spss 17; SPSS Inc., Chicago, IL, USA). We first estimated an empty model (0), which reflected variation in the intercept. To assess the extent to which variance should be ascribed to the team or collaborative rather than individual level, collaboratives served as level-3 and teams as level-2 units (model 1). In the models thereafter, we entered the independent variables as fixed effects in separate steps. As individual socio-demographic and team composition characteristics are expected to influence perceived effectiveness as well as other independent variables, these were included first. Following the theoretical model, collaborative-, organisation- and team-level variables were entered in the separate steps. Results were considered statistically significant when two-sided p-values were ≤0·05. Deviance tests or likelihood ratio tests compared the relative fit of the different models. The difference in deviance of two nested models has a chi-square distribution with degrees of freedom equal to the number of extra parameters in the larger model.

Results

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. Acknowledgement
  9. Contributions
  10. Conflict of interest
  11. Ethical approval
  12. Funding
  13. References
  14. Appendix

Sample characteristics

Respondents had a mean age of 44 and were mostly female (72·9%). About 53% had completed tertiary education, 15% had a university degree. About 85% had been working for more than three years in the organisation and 66·7% worked more than 29 hours per week. Teams consisted of medical assistants (6·1%), nurses (25·8%), social workers (7·8%), medical specialists (7·3%), paramedical professionals (9·2%), quality staff (10·9%) and lower and middle managers (32·9%).

Descriptive statistics

Table 3 provides descriptive statistics (mean, standard deviation or percentages) of all variables. Mean scores on the collaborative-level variables were moderate with mean score varying between 4·01 and 5·34 on a seven-point scale. With respect to organisational factors, commitment to quality improvement had a mean of 3·81 on a five-point scale, indicating a relatively high rate of professional involvement in quality improvement and team member training. The mean score on organisational support was relatively low (4·05 on a seven-point scale). With respect to cultural balance, a mean of 0·67 on a 0–1 range indicated that most respondents perceived a moderate balance between the four types of culture in their organisation. At the team level, innovative culture was moderate with a mean of 3·62 (on five-point scale) and commitment to change was relatively low with a mean of 123·52 compared with the theoretical range of 1–245.

Table 3. Descriptive statistics and correlation with perceived effectiveness (nteams = 215) (nindividual = 513)
Demographic characteristics n % or mean (SD)Correlation
  1. Two-tailed p-values.

Gender
 0 male13927·1%0·02
 1 female37472·9% 
Position
 0 manager/quality staff22243·3%−0·09*
 1 frontline professionals21942·7% 
Age51344·20 (9·69)0·02
Educational level5135·47 (1·23)−0·12*
Team composition characteristics
Stable team
 0 no15471·6%−0·14
 1 yes6128·4% 
Manager in the team
 0 no9443·7%0·17*
 1 yes12156·3% 
% females in the team21573·0 (33·7)0·09
Mean educational level of the team2155·54 (0·90)−0·13
% tertiary education of the team21570·2 (34·2)−0·13
Collaborative-level variables
 Innovation’s attributes5134·01 (0·45)0·44**
 Program management expertise2155·34 (0·96)0·15**
 Advisor support2154·95 (1·42)0·11*
 Achievability2155·01 (0·95)0·17**
 Challenging targets2154·24 (1·32)0·02
 Measurability2154·98 (1·17)0·13**
Organisation-level variables
 Quality improvement commitment2153·81 (0·60)0·08
 Organisational support5134·05 (1·10)0·42**
 Cultural balance5130·67 (0·09)−0·07
Team-level variables
 Innovative culture5133·62 (0·41)0·48**
 Commitment to change513123·52 (39·13)0·35**

Perceived effectiveness varied between the 11 collaboratives (Table 4). High average scores were found especially in the Prevention of Malnutrition and Prevention of Medication Errors collaboratives. Reducing Problem Behaviour and Social Psychiatric Care scored relatively low on perceived effectiveness.

Table 4. Descriptive statistics on perceived effectiveness for each collaborative
  n MeanSD
  1. Analysis of variance F = 4·51 and = 0·000.

Prevention of medication errors674·190·65
Prevention of malnutrition724·160·67
Improving recovery-oriented care484·010·71
Prevention of fall incidents593·980·62
Improving autonomy and control523·970·59
Prevention of pressure ulcers313·950·59
Enhancing social participation303·890·65
Screening of somatic comorbidity523·780·71
Prevention of sexual abuse363·810·79
Reducing problem behaviour363·650·64
Social psychiatric care303·430·89
Total5133·940·70

Associations between independent and dependent variables

Frontline professionals and respondents with a higher educational level scored lower on perceived effectiveness (Table 3). Teams with a manager in the team scored higher. Of the collaborative-level variables, all but challenging targets were significantly positively associated with perceived effectiveness. Of the organisational-level variables, only organisational support had a significant correlation. Both team-level variables were significantly correlated with perceived effectiveness, innovative culture having the highest correlation coefficient of 0·48.

Table 5 shows the results of the multilevel regression analysis. The first empty model served as a baseline with just intercepts. Model 1 shows that about 33% of the variance could be attributed to differences between teams and 5% to differences between collaboratives. Model 2 shows that respondent’s educational level, innovation’s attributes, organisational support, innovative team culture and commitment to change have positive effects on perceived effectiveness. In total, 27·9% individual-level variance, 57·6% team-level variance and 80% collaborative-level variance could be explained.

Table 5. Hierarchical linear multilevel analyses on perceived effectiveness (n = 513)
Model012345
BSEBSEBSEBSEBSEBSE
  1. *p < 0·05 **p< 0·01 $ 0·05 > p < 0·10, all two-sided tests.

  2. 1((variance individual-level model 1 − variance individual-level final model)/variance individual-level model 1)*100 = 0·61 − 0·44/0·61*100 = 27·9%.

Constant0·010·05−0·060·090·330·360·64$0·340·58$0·330·93**0·31
Gender    0·070·110·000·100·050·100·000·09
Age    0·000·000·000·000·000·00−0·010·00
Educational level    −0·07$0·09−0·10**0·04−0·09**0·04−0·11**0·03
Position    −0·130·09−0·090·09−0·050·08−0·090·08
Stable team    0·120·130·160·120·020·12−0·050·11
Collaborative-level variables
Innovation’s attributes           0·34**0·050·29**0·050·24**0·04
Program management expertise           0·120·110·030·11−0·040·10
Advisor support           −0·070·090·020·090·050·08
Achievability           0·140·090·120·080·14$0·07
Challenging targets           0·060·060·020·060·000·06
Measurability           −0·010·07−0·040·07−0·060·06
Organisation-level variables
Quality improvement commitment          −0·040·06
Organisational support          0·18**0·05
Cultural balance          −0·060·04
Team-level variables
Innovative culture          0·24**0·05
Commitment to change          0·16**0·04
−2 log likelihood1264·51 1214·68 1042·63 922·37 954·21 922·04 
Variance individual level0·970·060·610·050·550·050·510·050·480·050·440·05
Variance team level  0·330·070·360·080·220·060·190·050·140·05
Variance collaborative level  0·050·040·030·030·030·030·010·020·010·02
Explained individual level          27·9% 
Explained variance team level          57·6% 
Explained variance collaborative level          80·0% 

Discussion

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. Acknowledgement
  9. Contributions
  10. Conflict of interest
  11. Ethical approval
  12. Funding
  13. References
  14. Appendix

The evidence underlying the effectiveness of QICs is inconclusive (Leatherman 2002, Ovretveit 2002, Cretin et al. 2004, Schouten et al. 2008) and few studies investigated determinants of implementation success (Mills & Weeks 2004, Neily et al. 2005, Dückers et al. 2009). Moreover, most evaluation studies are based on one specific topic, making it hard to compare across collaboratives addressing different topics. The objective of our study was to explore effectiveness of 11 collaboratives focusing on 11 different topics, as perceived by local improvement teams and to explore associations with collaborative-, organisational- and team-level factors. The results partly supported the theoretically proposed associations. As suggested by several authors, the nature of topic or quality problem addressed is critically important to a collaborative’s effectiveness (Øvretveit & Gustafson 2002, Wilson et al. 2003). Respondents in the Prevention of Malnutrition and Prevention of Medication Errors collaboratives perceived effectiveness as considerably higher than respondents in other collaboratives, those in Reducing Problem Behaviour and Social Psychiatric Care perceived effectiveness as considerably lower.

In preparing and organising a QIC program, managers should carefully consider the type of quality problem or topic addressed and researchers investigating effectiveness of QICs should also take this into account. At the collaborative level, the innovation’s attributes are key in explaining implementation success. The more the new working methods were perceived by professionals as – having relative benefit, being compatible with norms and values, not difficult to learn and implement and leading to observable results – the more the implementation process was perceived as successful. In contrast with previous studies (Øvretveit 2002, Øvretveit & Gustafson 2002, Øvretveit et al. 2002, Benn et al. 2009, Dückers et al. 2009, Nembhard 2009), the other collaborative-level factors – program management expertise, advisor support, achievability, challenging targets and measurability – were not significant predictors. Although most of these variables showed significant associations with perceived effectiveness in the univariate analyses, the associations disappeared in the multivariate analyses owing to the strong effect of the innovation’s attributes.

At the organisational level, the findings suggest that for teams to perceive a higher impact of their improvement efforts, organisational support is crucial. In line with previous studies (Gustafson et al. 2003, Mills & Weeks 2004, Dückers et al. 2009), organisational support – conceptualised as making time, finances, means and instruments available and having a manager who shows interest, coaches and encourages professionals – is important to achieve improvement. In contrast with previous studies (Meterko et al. 2004, Shortell et al. 2004, Lin et al. 2005, Hann et al. 2007), the other organisational-level variables, quality improvement commitment and organisational culture, were not identified as determinants of perceived effectiveness.

Commitment to change and innovative culture are both significant predictors of perceived effectiveness on the team level. Professionals who attach importance to the outcomes of quality improvement and believe they can achieve them are associated with higher perceived effectiveness. Innovative culture, however, was the stronger predictor. Teams with high social expectations – trying new ways of doing things, taking risks, tolerating mistakes – facilitate implementation.

Limitations

The cross-sectional design hampered our ability to draw causal inferences. Our results establish a significant association, which is an important step that prompts further studies to identify directionality. Second, the overall moderate response on the evaluation survey and the rather low number of respondents per team (2·6) may have led to some selection bias. During the collaborative program, many team members held other jobs or left the organisation. Given the dynamics in the field with new (compulsory) policies, reorganisations or mergers, not many respondents were available for this study. Third, we used self-reported instruments to assess organisational- and team-level factors and perceived effectiveness. Professionals’ perceptions of effectiveness may have been influenced by expectations and positive feelings of working together. Although such a measure can introduce bias, the considerable variation between teams and collaboratives left room for explanation by organisational- and team-level factors.

Unfortunately, no single measure for objective effectiveness could be computed across all 11 collaboratives owing to the diversity in topic, content and outcome indicators. Neglecting the content of the indicator and gathering the different indicators together would lead to misleading results. We thus used the perceived effectiveness of team members as an indicator of the collaboratives’ overall impact. Other accomplishments and effects not measured by outcome indicators may well be perceived, especially in the context of service delivery. Team members, for example, may have noticed how patients benefited or how professionals learned new working practices and routines. Distinguishing a subjective part as a conceptualisation of effectiveness allows us to compare the effectiveness of collaboratives that address different types of problems. For future research, it would be interesting to investigate to what extent perceived effectiveness is related to changes in objective outcome indicators and what different determinants of success may play a role. Although these general limitations may have somewhat influenced the reported results, they allowed us to compare a wide range of QICs, a major strength of the study.

Conclusion

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. Acknowledgement
  9. Contributions
  10. Conflict of interest
  11. Ethical approval
  12. Funding
  13. References
  14. Appendix

The results support the notion that a layered approach is necessary to achieve improvements in quality of care and provide further insight in the determinants of success of QICs. By evaluating 11 different QICs, our study provides insight in how collaborative-, organisational- and team-level factors may play a role in perceived effectiveness of different collaboratives. The innovation’s attributes, organisational support, an innovative team culture and professionals’ commitment to change are instrumental to perceived effectiveness.

Relevance to clinical practice

Understanding which factors enhance the impact of quality improvement initiatives can help professionals to achieve breakthrough improvement in care delivery to patients on a wide variety of quality problems.

Contributions

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. Acknowledgement
  9. Contributions
  10. Conflict of interest
  11. Ethical approval
  12. Funding
  13. References
  14. Appendix

Study design: MMHS, APN; data collection and analysis: MMHS, APN and manuscript preparation: MMHS, APN.

Ethical approval

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. Acknowledgement
  9. Contributions
  10. Conflict of interest
  11. Ethical approval
  12. Funding
  13. References
  14. Appendix

As the quality improvement initiatives were part of daily practice and were initiated and implemented by health care organisations and not by the evaluation research team, no ethics approval was required in the current ethical regime in The Netherlands.

Funding

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. Acknowledgement
  9. Contributions
  10. Conflict of interest
  11. Ethical approval
  12. Funding
  13. References
  14. Appendix

The Care for Better program and evaluation study are funded by the Netherlands Organization for Health Research and Development (ZonMw grantnr 5942 and 60-60900-96-005). The evaluation study researchers are independent of the funding organisation.

References

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. Acknowledgement
  9. Contributions
  10. Conflict of interest
  11. Ethical approval
  12. Funding
  13. References
  14. Appendix
  • Benn J, Burnett S, Parand A, Pinto A, Iskander S & Vincent C (2009) Perceptions of the impact of a large-scale collaborative improvement programme: experience in the UK Safer Patients Initiative. Journal of Evaluation in Clinical Practice15, 524540.
  • Blau PM (1977) Inequality and Heterogeneity: A Primitive Theory of Social Structure. Free Press, New York, NY.
  • Caldwell DF & O’Reilly CA (2003) The determinants of team-based innovation in organisations. The role of social influence. Small Group Research34, 497517.
  • Cretin S, Shortell SM & Keeler EB (2004) An evaluation of collaborative interventions to improve chronic illness care. Framework and study design. Evaluation Review28, 2851.
  • Dückers MLA, Wagner C & Groenewegen PP (2008) Developing and testing an instrument to measure the presence of conditions for successful implementation of quality improvement collaboratives. BMC Health Services Research8, 172.
  • Dückers MLA, Spreeuwenberg P, Wagner C & Groenewegen P (2009) Exploring the black box of quality improvement collaboratives: modelling relations between conditions, applied changes and outcomes. Implementation Science4, 74.
  • Fried B, Rundall T & Topping S (2000) Groups and teams in health service organisations. Health Care Management: Organisation Design and Behaviour , 154190.
  • Gustafson DH, Sainfort F, Eichler M, Adams L, Bisognano M & Steudel H (2003) Developing and testing a model to predict outcomes of organizational change. Health Services Research38, 751776.
  • Hann M, Bower P, Campbell S, Marshall M & Reeves D (2007) The association between culture, climate and quality of care in primary health care teams. Family Practice24, 323329.
  • Institute for Healthcare Improvement (2003) The Breakthrough Series: IHI’s Collaborative Model for Achieving Breakthrough Improvement . IHI, Cambridge, MA.
  • Kaplan HC, Brady PW, Dritz MC, Hooper DK, Linam WM, Froehle CM & Margolis P (2010) The influence of context on quality improvement success in health care: a systematic review of the literature. The Milbank Quarterly88, 500559.
  • Leatherman S (2002) Optimizing quality collaboratives. Quality and Safety in Health Care11, 307.
  • Lemieux-Charles L & McGuire WL (2006) What do we know about health care team effectiveness? A review of the literature. Medical Care Research and Review63, 263300.
  • Lemieux-Charles L, Murray M, Baker GR, Barnsley J, Tasa K & Ibrahim SA (2002) The effects of quality improvement practices on team effectiveness: a mediational model. Journal of Organizational Behavior23, 533553.
  • Lemmens K, Strating M, Huijsman R & Nieboer A (2009) Professional commitment to changing chronic illness care: results from disease management programmes. International Journal for Quality in Health Care21, 233242.
  • Lin MK, Marsteller JA, Shortell SM, Mendel P, Pearson M, Rosen M & Wu SY (2005) Motivation to change chronic illness care: results from a national evaluation of quality improvement collaboratives. Health Care Management Review30, 139156.
  • Meterko M, Mohr DC & Young GJ (2004) Teamwork culture and patient satisfaction in hospitals. Medical Care42, 492498.
  • Mills PD & Weeks WB (2004) Characteristics of successful quality improvement teams: lessons from five collaborative projects in the VHA. Joint Commission Journal on Quality and Safety30, 152162.
  • Neily J, Howard K, Quigley P & Mills PD (2005) One-year follow-up after a collaborative breakthrough series on reducing falls and fall-related injuries. Joint Commission Journal on Quality and Patient Safety31, 275285.
  • Nembhard IM (2009) Learning and improving in quality improvement collaboratives: which collaborative features do participants value most?Health Services Research44, 359378.
  • Nieboer AP & Strating MMH (2011) Innovative culture in long-term care settings: the influence of organizational characteristics. Health Care Management Review, in press.
  • Ovretveit J (2002) Action Evaluation of Health Programmes and Changes: A Handbook for a User-Focused Approach. Radcliffe Medical Press, Oxford.
  • Øvretveit J (2002) How to run an effective improvement collaborative. International Journal for Health Care Quality Assurance15, 192196.
  • Øvretveit J & Gustafson D (2002) Evaluation of quality improvement programmes. Quality and Safety in Health Care11, 270275.
  • Øvretveit J, Bate P, Cleary P, Cretin S, Gustafson D, McInnes K, McLeod H, Molfenter T, Plsek P & Robert G (2002) Quality collaboratives: lessons from research. Quality and Safety in Health Care11, 345351.
  • RAND (1999) Improving Chronic Illness Care Evaluation. Healthcare Organization Survey for Breakthrough Series (BTS) Team Members. RAND, Santa Monica, CA.
  • Rogers EM (1995) Diffusion of Innovation. The Free Press, New York, NY.
  • Schouten LMT, Hulscher M, Everdingen JJE, Huijsman R & Grol R (2008) Evidence for the impact of quality improvement collaboratives: systematic review. British Medical Journal336, 14911494.
  • Shortell SM, O’Brien JL, Carman JM, Foster RW, Hughes EF, Boerstler H & O’Connor EJ (1995) Assessing the impact of continuous quality improvement/total quality management: concept versus implementation. Health Services Research30, 377401.
  • Shortell SM, Marsteller JA, Lin M, Pearson ML, Wu SY, Mendel P, Cretin S & Rosen M (2004) The role of perceived team effectiveness in improving chronic illness care. Medical Care42, 10401048.
  • Strating MMH & Nieboer AP (2010) Norms for creativity and implementation in healthcare teams: testing the group innovation inventory. International Journal for Quality in Health Care6, 18.
  • Strating MMH, Zuiderent-Jerak T, Nieboer AP & Bal RA (2008) Evaluating the Care for Better Collaborative. Results of the First Year of Evaluation. Institute of Health Policy and Management, Rotterdam.
  • Strating MMH, Nieboer AP, Zuiderent-Jerak T & Bal RA (2011) Creating effective quality improvement collaboratives: a multiple case study. Quality and Safety in Health Care20, 344350.
  • Vos L, Dückers M & Wagner C (2008) Evaluation Better Faster Pillar 3: Results of an Improvement Programme for Hospitals. Nivel, Utrecht [in Dutch].
  • Vroom VH (1995) Work and Motivation. Jossey-Bass, Co., San Francisco.
  • Wilson T, Berwick DM & Cleary PD (2003) What do collaborative improvement projects do? Experience from seven countries. Joint Commission Journal on Quality and Patient Safety29, 8593.
  • Zammuto RF, Gifford G & Goodman EA (2000) Managerial ideologies, organisation culture and the outcomes of innovation: a competing values perspective. In The Handbook of Organizational Culture and Climate (Ashkanasy NM, Wilderom C & Peterson MF eds). Sage Publications, Inc, Thousand Oaks, CA, pp. 261–278.

Appendix

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. Acknowledgement
  9. Contributions
  10. Conflict of interest
  11. Ethical approval
  12. Funding
  13. References
  14. Appendix
Measurement instruments

Perceived effectiveness

Lemieux-Charles L, Murray M, Baker GR, et al. (2002) The effects of quality improvement practices on team effectiveness: a mediational model. Journal of Organizational Behavior23, 533–553.

  Strongly disagreeStrongly agree
I am satisfied with my experience as a team member12345
I feel positive about my experience in the team12345
I am willing to work in a similar team in the future12345
I believe the team’s overall performance met (my) expectations12345

Innovation attributes

Vos L, Dückers M, Wagner C (2008) Evaluation Better Faster pillar 3: results of an improvement programme for hospitals [in Dutch].

The new improvement and working methods of Care for BetterStrongly disagreeStrongly agree
Are applicable to our division12345
Match with our needs12345
Are transferred in an appropriate manner12345
Are clear12345
Are relevant to our division12345
Are difficult to learn12345
Are difficult to implement12345
Have a favourable balance between costs and benefits12345
We perceive risks on the project12345
Will lead to observable results12345

Program management expertise

Dückers ML, Wagner C, Groenewegen PP (2008) Developing and testing an instrument to measure the presence of conditions for successful implementation of quality improvement collaboratives. BMC Health Serv Res8, 172.

Program managementStrongly disagreeStrongly agree
Explained the improvement methods well1234567
Provided clarity on the purpose and approach of the project1234567
Gave a sufficiently tailored instruction1234567
Had sufficient expertise on the quality improvement topic1234567
Had sufficient expertise on the improvement methods1234567

Advisor support

Dückers ML, Wagner C, Groenewegen PP (2008) Developing and testing an instrument to measure the presence of conditions for successful implementation of quality improvement collaboratives. BMC Health Serv Res8, 172.

Our advisorStrongly disagreeStrongly agree
Had regular telephone contact1234567
Was sufficiently responsive in the design of our action plan, implementation of improvement actions and measurements1234567
Was sufficiently responsive to our questions and problems we ran in to1234567
Stimulated us to report the results and progress monthly1234567

Achievability

Dückers ML, Wagner C, Groenewegen PP (2008) Developing and testing an instrument to measure the presence of conditions for successful implementation of quality improvement collaboratives. BMC Health Serv Res8, 172.

 Strongly disagreeStrongly agree
Collaborative targets are achievable1234567
Program management made clear how to achieve collaborative targets1234567
Program management offered a standardised set of indicators to monitor progress and compare results1234567
Program management offered good practices and evidence on achievable results1234567

Challenging targets

Dückers ML, Wagner C, Groenewegen PP (2008) Developing and testing an instrument to measure the presence of conditions for successful implementation of quality improvement collaboratives. BMC Health Serv Res8, 172.

 Strongly disagreeStrongly agree
Program management set high expectations with regard to performance and improvement possibilities1234567

Measurability

Dückers ML, Wagner C, Groenewegen PP (2008) Developing and testing an instrument to measure the presence of conditions for successful implementation of quality improvement collaboratives. BMC Health Serv Res8, 172.

 Strongly disagreeStrongly agree
Progress is measured continuously1234567
Timely and accurate progress information was available at all times1234567
Measuring indicators helps to monitor progress1234567
There were clear agreements on measuring central indicators1234567

Quality improvement commitment

Shortell SM, O’Brien JL, Carman JM, et al. (1995) Assessing the impact of continuous quality improvement/total quality management: concept versus implementation. Health Serv Res30, 377–401.

  Strongly disagreeStrongly agree
Agree
Staff is involved in developing plans for improving quality12345
Staff is given the opportunity to improve quality12345
Staff has the authority to correct problems in their area when quality standards are not being met12345
Staff is supported when they take necessary risks to improve quality12345
The organisation has an effective system for employees to make suggestions to management on how to improve quality12345
Staff is given education and training in how to identify and act on quality improvement opportunities12345
Staff is given the needed education and training to improve job skills and performance12345
Staff is rewarded and recognised (e.g., financially and/or otherwise) for improving quality12345

Organisational support

RAND (2010) Improving Chronic Illness Care Evaluation. Healthcare Organization Survey for Breakthrough Series (BTS) Team members.

  Strongly disagreeStrongly agree
Senior management pays attention to the activities of the improvement team1234567
Senior management acted as coach to our improvement team1234567
Senior management encouraged staff to improve their performance1234567
Senior management provides good feedback on the work of our improvement team1234567
Senior management was open for criticism1234567
Senior management gave us time to reflect up on our work1234567
Senior management gave us time to try new working methods1234567
I am satisfied about the way senior management supported our team1234567
Our team had enough time to implement the changes1234567
Our team had enough manpower to execute the project1234567
Our team had enough resources to make the project successful1234567
Our team members had the skills necessary to make the project successful1234567

Organisational culture

Zammuto RF, Gifford G, Goodman EA (2000) Managerial ideologies, organisation culture and the outcomes of innovation: a competing values perspective. In The Handbook of Organizational Culture and Climate (Ashkanasy NM, Wilderom C & Peterson MF eds). Sage Publications, Inc., Thousand Oaks, CA.

Shortell SM, O’Brien JL, Carman JM, et al. (1995) Assessing the impact of continuous quality improvement/total quality management: concept versus implementation. Health Serv Res30, 377–401.

Instructions

These questions relate to the type of organisation that your institution is most like. Each of these items contains four descriptions of healthcare organisations. Please distribute 100 points among the four descriptions depending on how similar the description is to your organisation. None of these descriptions is any better than the others; they are just different. For each question, please use all 100 points. For example: In question 1, if Organization A seems very similar to mine, B seems somewhat similar, and C and D do not seem similar at all, I might give 70 points to A and the remaining 30 points to B. Please note that these questions pertain to the overall organisation of which you are a part, not to your individual team or unit.

Organisation character (please distribute 100 points)

  • 1
     _______ Organisation A is a very personal place. It is a lot like an extended family. People seem to share a lot of themselves.
  • 2
     _______ Organisation B is a very dynamic and entrepreneurial place. People are willing to stick their necks out and take risks.
  • 3
     _______ Organisation C is a very formalized and structured place. Bureaucratic procedures generally govern what people do.
  • 4
     _______ Organisation D is very production oriented. A major concern is with getting the job done. People aren’t very personally involved.

Total = 100 points

Organisation’s managers (please distribute 100 points)

  • 5
     _______ Managers in organisation A are warm and caring. They seek to develop employees’ full potential and act as their mentors or guides.
  • 6
     _______ Managers in organisation B are risk-takers. They encourage employees to take risks and be innovative.
  • 7
     _______ Managers in organisation C are rule-enforcers. They expect employees to follow established rules, policies, and procedures.
  • 8
     _______ Managers in organisation D are coordinators and coaches. They help employees meet the organisation’s goals and objectives.

Total = 100 points

Organisation cohesion (please distribute 100 points)

  • 9
     _______ The glue that holds organisation A together is loyalty and tradition. Commitment to this organisation runs high.
  • 10
     _______ The glue that holds organisation B together is commitment to innovation and development. There is an emphasis on being first.
  • 11
     _______ The glue that holds organisation C together is formal rules and policies. Maintaining a smooth running operation is important here.
  • 12
     _______ The glue that holds organisation D together is the emphasis on tasks and goal accomplishment. A production orientation is commonly shared.

Total = 100 points

Organisation emphases (please distribute 100 points)

  • 13
     _______ organisation A emphasizes human resources. High cohesion and morale in the organisation are important.
  • 14
     _______ organisation B emphasizes growth and acquiring new resources. Readiness to meet new challenges is important.
  • 15
     _______ organisation C emphasizes permanence and stability. Efficient, smooth operations are important.
  • 16
     _______ organisation D emphasizes competitive actions and achievement. Measurable goals are important.

Total = 100 points

Organisation rewards (please distribute 100 points)

  • 17
     _______ organisation A distributes its rewards fairly equally among its members. It’s important that everyone from top to bottom be treated as equally as possible.
  • 18
     _______ organisation B distributes its rewards based on individual initiative. Those with innovative ideas and actions are most rewarded.
  • 19
     _______ organisation C distributes its rewards based on rank. The higher you are, the more you get.
  • 20
     _______ organisation D distributes its rewards based on the achievement of objectives. Individuals who provide leadership and contribute to attaining the organisation’s goals are rewarded.

Total = 100 points

Innovative culture

Caldwell DF, O’Reilly CA (2003) The determinants of team-based innovation in organisations. The role of social influence. Small Group Research34, 497–517.

Strating MMH, Nieboer AP (2010) Norms for creativity and implementation in healthcare teams: testing the group innovation inventory. International Journal for Quality in Health Care6, 1–8.

  Strongly disagreeStrongly agree
Risk taking is encouraged around here12345
Management provides rewards and recognition for innovation and trying new things12345
Mistakes are a normal part of trying something new12345
People have great freedom to act to make necessary changes around here12345
The attitude around here is that when you are trying new things, mistakes are a normal part of the job12345
In our group, there is a great deal of openness in sharing information12345
People in our group encourage each other to try new things12345
Decisions in our group are made quickly12345
Management encourages people to try new things.12345
Members of our group listen carefully to the views of others12345
In our group we expect others to take initiative and get things done even if a person is not formally responsible12345
Our group is flexible and adapts quickly to new opportunities.12345
In our group we try to reach a consensus about important decisions12345
Once a decision is made, we implement it quickly12345
Our group has sufficient autonomy to implement new ideas without clearance from above.12345
People in this organisation are willing to try new things12345
It may go wrong when trying to grant wishes of individual clients12345
In this organisation we are always looking for other ways to organise our work in order to provide better care12345

Commitment to change

Vroom VH (1995)Work and Motivation. Jossey-Bass, Co., San Francisco, CA.

Exerting effort (e.g., time and resources) willStrongly disagree     Strongly agree
Help you implement changes in care1234567
Success in implementing changes in careStrongly disagree     Strongly agree
Help you improve quality of care for patients1234567
Help you improve patient satisfaction with their care1234567
Help you improve productivity/efficiency1234567
Help improve patient clinical outcomes1234567
Help you involve patients with their own care1234567
Help improve continuity of care1234567
Allow you opportunities to use your skills and abilities better1234567
Help you get recognition (i.e., praise, promotion, etc.) from your superiors1234567
Help you feel that you have accomplished something worthwhile1234567
Exerting effort (e.g., time and resources) willStrongly disagree     Strongly agree
Help you to adopt the PDSA improvement process1234567
Success in adopting the PDSA improvement process willStrongly disagree     Strongly agree
Enable your team to make changes that improve the processes of care1234567
Enable process changes to be spread to other parts of the organisation1234567
Enable the team to gain support for process changes1234567
Enable your team to adapt the collaborative improvement methods to their needs1234567
How important are the following to you?Strongly disagree   Strongly agree  
Improving quality of care for patients12345  
Improving patient satisfaction with their care12345  
Improving productivity/efficiency12345  
Improving patient clinical outcomes12345  
Involving patients with their own care12345  
Improving continuity of care12345  
Having opportunities to use your skills and abilities better12345  
Getting recognition (i.e., praise, promotion, etc.) from your superiors12345  
Feeling that you have accomplished something worthwhile12345  
Making changes that improve the processes of care12345  
Spreading process changes to other parts of the organisation12345  
Gaining support for process changes12345  
Adapting collaborative improvement methods to your team’s needs12345  

The Journal of Clinical Nursing (JCN) is an international, peer reviewed journal that aims to promote a high standard of clinically related scholarship which supports the practice and discipline of nursing.

For further information and full author guidelines, please visit JCN on the Wiley Online Library website: http://wileyonlinelibrary.com/journal/jocn

Reasons to submit your paper to JCN:

High-impact forum: one of the world's most cited nursing journals, with an impact factor of 1·118 – ranked 30/95 (Nursing (Social Science)) and 34/97 Nursing (Science) in the 2011 Journal Citation Reports® (Thomson Reuters, 2011)

One of the most read nursing journals in the world: over 1·9 million full text accesses in 2011 and accessible in over 8000 libraries worldwide (including over 3500 in developing countries with free or low cost access).

Early View: fully citable online publication ahead of inclusion in an issue.

Fast and easy online submission: online submission at http://mc.manuscriptcentral.com/jcnur.

Positive publishing experience: rapid double-blind peer review with constructive feedback.

Online Open: the option to make your article freely and openly accessible to non-subscribers upon publication in Wiley Online Library, as well as the option to deposit the article in your preferred archive.