Building District-Level Capacity for Continuous Improvement in Maternal and Newborn Health

Authors

  • Kim Ethier Stover MA,

    Corresponding author
    • Address correspondence to Kim Ethier Stover, MA, Maternal and Neonatal Health in Ethiopia Partnership, University Research Co., LLC, 7200 Wisconsin Ave, Suite 600, Bethesda, MD 20814. E-mail: kethier@urc-chs.com

    Search for more papers by this author
  • Solomon Tesfaye MD, MPH,

  • Aynalem Hailemichael Frew RN, MPH,

  • Hajira Mohammed MSc,

  • Danika Barry MPH,

  • Lamesgin Alamineh MPH,

  • Abebe Teshome BSc,

  • Kenneth Hepburn PhD,

  • Lynn M. Sibley CNM, RN, PhD


Abstract

Introduction

The Maternal and Newborn Health in Ethiopia Partnership (MaNHEP) adapted a collaborative improvement strategy to develop woreda (district) leadership capacity to support and facilitate continuous improvement of community maternal and neonatal health (CMNH) and to provide a model for other woredas, dubbed “lead” woredas. Community-level quality improvement (QI) teams tested solutions to improve CMNH care supported by monthly coaching and regular meetings to share experiences. This study examines the extent of the capacity built to support continuous improvement in CMNH care.

Methods

Surveys and in-depth interviews assessed the extent to which MaNHEP developed improvement capacity. A survey questionnaire evaluated woreda culture, leadership support, motivation, and capacity for improvement activities. Interviews focused on respondents’ understanding and perceived value of the MaNHEP improvement approach. Bivariate analyses and multivariate linear regression models were used to analyze the survey data. Interview transcripts were organized by region, cadre, and key themes.

Results

Respondents reported significant positive changes in many areas of woreda culture and leadership, including involving a cross-section of community stakeholders (increased from 3.0 to 4.6 on 5-point Likert scale), using improvement data for decision making (2.8-4.4), using locally developed and tested solutions to improve CMNH care (2.5-4.3), demonstrating a commitment to improve the health of women and newborns (2.6-4.2), and creating a supportive environment for coaches and QI teams to improve CMNH (2.6-4.0). The mean scores for capacity were 3.7 and higher, reflecting respondents’ agreement that they had gained capacity in improvement skills. Interview respondents universally recognized the capacity built in the woredas. The themes of community empowerment and focused improvement emerged strongly from the interviews.

Discussion

MaNHEP was able to build capacity for continuous improvement and develop lead woredas. The multifaceted approach to building capacity was critical for the success in creating lead woredas able to serve as models for other districts.

INTRODUCTION

The Maternal and Newborn Health in Ethiopia Partnership (MaNHEP) was a 3.5-year (November 2009-May 2013) learning project funded by the Bill and Melinda Gates Foundation. The project's overall aim was to learn how best to ensure that a package of evidence-based community maternal and newborn health (CMNH) practices could be provided to “every woman, in time, every time” during birth and the early postnatal period when women and newborns are at greatest risk of death.[1] Timely and consistent delivery of this package during birth and the early postnatal period was expected to improve both maternal and newborn outcomes.[2] As described elsewhere, increased provision of the CMNH package was to be accomplished by building competence, confidence, and teamwork among health extension workers, community health development agents, traditional birth attendants (TBAs),[3, 4] and the public health provider network to deliver CMNH service—and by increasing the demand for CMNH care by women and families and improving their self-care practices.[5, 6]

Under the leadership of the Federal Ministry of Health, MaNHEP worked in 6 woredas (rural districts) of 2 of the country's pastoral regions—Amhara and Oromiya—to strengthen implementation of its CMNH program and to work toward achievement of Millennium Development Goals 4 and 5 to reduce child and maternal mortality.[1] MaNHEP was led by Emory University in collaboration with University Research Co., LLC; JSI Research & Training Institute, Inc.; and Addis Ababa University. MaNHEP's efforts to ensure that every woman in the project's pilot woredas received appropriate antenatal, birth, and immediate postnatal care are described elsewhere.[1]

In addition to its training, community education, and behavior change communication strategies, MaNHEP used an improvement approach (also referred to as quality improvement [QI]) as a key strategy to meld both of the project's foci: supply and demand. The improvement approach sought to help communities develop a system for providers and women, families, and community members to work together to codevelop contextually appropriate approaches to ensure that CMNH care reached women and their newborns “in time, every time.” This article describes the methods by which and the extent to which MaNHEP was able to develop the capacity of coaches and teams to support continuous improvement in CMNH care.

Quick Points

  • The Maternal and Newborn Health in Ethiopia Partnership (MaNHEP) built support and capacity for continuous improvement in 6 woredas (districts) in Ethiopia by developing leaders at woreda and health center levels and community-level quality improvement teams through classroom training, on the job training, and continuous support and feedback over time.
  • Respondents reported significant, positive changes in many of characteristics of woreda culture and leadership that support improvement work such as facilitative supervision, representative community teams, data for decision making, locally developed solutions, a commitment to improving health of women and newborns, and a supportive environment.
  • Interviews confirmed that the MaNHEP approach contributed to improved coordination and supervision, focused improvement efforts, community empowerment, and an ability to continue with improvements in care for mothers and newborns.
  • On a community level, MaNHEP's application of rapid-cycle process improvement approach, used throughout the world for clinical and organizational improvement, provides an example of the powerful results that can be achieved when providers at all levels of care and community members come together to improve both clinical care and public health concerns.
  • This approach demonstrates the importance of intentionally building organizational culture and leadership to create an environment that enables improvement through the engagement of all stakeholders.

MaNHEP IMPROVEMENT STRATEGY

MaNHEP's improvement strategy adapted a collaborative improvement approach used in other low- and middle-income countries[7-9] to develop and test solutions to improve 4 key areas deemed necessary to ensure that women and newborns receive care during birth and the early postnatal period: 1) pregnancy identification; 2) antenatal care registration; 3) participation in birth preparation training through CMNH family meetings (meetings of a woman, her family, a TBA, and a community health development agent to review best practices around birth); and 4) labor and birth notification to a health extension worker and postnatal care within 48 hours of birth by a health extension worker. Collaborative improvement approach involves multiple teams, called QI teams, working on common aims meeting regularly to learn from one another. Although there have been many adaptations of the approach for low- and middle-income settings, MaNHEP's adaptation was one of the first applications to the community level, with QI teams based at and run by community members rather than health facility staff.[10]

The MaNHEP improvement strategy sought to facilitate the development of community and health system processes to support delivery of CMNH care, effectively creating lead woredas or model districts. A lead woreda is one that is committed and able to continuously improve CMNH care and service delivery to meet the needs of childbearing families. MaNHEP sought to work intensively with a limited number of woredas, developing their skills to improve CMNH. The woredas served as regional learning laboratories that would allow MaNHEP and the Ethiopian Federal Ministry of Health to understand how this approach could be best integrated into the existing ministry system so that, if the approach was successful, it could be spread to other woredas.

Together with the Ethiopian Federal Ministry of Health and the Amhara and Oromiya Regional Health Bureaus, the project selected 6 woredas (3 in each region) to participate in the learning project. The project focused on a vertical slice of the health system in each woreda to create a support network linked to existing organizational structures.[11] The slice involved key personnel from each level of the health system: regional health bureau; zonal health departments; woreda health offices; 2 health centers within each woreda; and 7 to 10 kebele (subdistrict) health posts linked to the 2 health centers (a total of 51 kebeles), who were responsible for ensuring childbearing families received CMNH care. Figure 1 shows the Ethiopian Ministry of Health health care delivery system. MaNHEP focused on the primary health care and kebele levels. Health extension workers focus on prevention, including bed nets; sanitation; safe and clean birth; basic antenatal care and postnatal care; breastfeeding; immunization of children and mothers; family planning information and services; and malaria, diarrhea, sepsis, and pneumonia case management.

Figure 1.

Overview of the Ethiopian Ministry of Health Structure

Reprinted with permission of the Bill & Melinda Gates Foundation.

The improvement strategy required training and deployment of woreda-level coaches and kebele-level QI teams. Coaches were drawn from woreda health office and health center personnel who were already supervising the health extension workers and community health development agents. The QI team members were selected by the kebeles and comprised of 12 to 20 members representing a cross-section of stakeholders, including health extension workers, community health development agents, TBAs, pregnant women, families, community elders, representatives of community-based organizations such as agricultural and women's associations, and local administrators. Coaches provided monthly support to QI teams, assisting them to implement improvement activities. QI teams each selected a leader who organized the QI teams and facilitated the relationship between QI teams and coaches. Neither coaches nor QI team members received any incentives for this work. They did receive reimbursement for travel expenses for attendance at learning sessions.

Whereas the MaNHEP program of CMNH family meetings was being implemented with pregnant women and their family caregivers on an ongoing basis,[5] QI teams continuously worked on developing and testing potential solutions to improve CMNH care so that women would reliably receive the care they required. QI teams used an improvement model to identify interventions designed to improve the 4 key areas.[12] This model guides the team to ask 3 questions focused on the Ethiopian Federal Ministry of Health priorities: 1) What are we trying to accomplish (aim)? 2) How will we know that a change is an improvement (measure)? 3) What changes can we make that will result in improvement (potential solutions)? Within each key area, the QI teams identified barriers to providing care in their own communities. They brainstormed or used ideas from other QI teams and prioritized potential solutions (change ideas). They then implemented plan-do-study-act (PDSA) cycles to test these change ideas and determine if they led to improvement based on data they had collected and reviewed monthly.

BUILDING AND SUPPORTING IMPROVEMENT CAPACITY

MaNHEP provided initial training and regular reinforcement and augmentation of improvement principles and methods in regular workshops, known as learning sessions, from December 2010 to February 2013. These regional learning sessions were attended by health authorities, coaches, and 2 to 3 QI team representatives (average attendance per region, 102). We introduced participants to basic improvement skills at the first regional learning session and provided an additional day of training for coaches to discuss their roles and develop facilitation skills. Subsequently, participants met every 4 to 6 months to learn more about improvement methods and to share progress, discuss successful and unsuccessful solutions tested, and develop initial plans to test or adapt change ideas. At each learning session, coaches received feedback and more advanced training on monitoring and data analysis.

MaNHEP also facilitated day-long woreda-level learning sessions for coaches and QI teams (average attendance, 62). These workshops, which included almost all QI team members, allowed members to share and learn from others. They also provided an opportunity for woreda administration and woreda health office representatives to address and improve common issues within and across the woredas (eg, to provide substitute health extension workers for kebeles without a health extension worker due to illness or maternity leave).

During the first project year, MaNHEP staff and coaches jointly conducted monthly visits to support the QI teams. MaNHEP staff modeled the coaching process and coached the coaches, discussing the goals and approaches for QI team meetings, supporting their conduct of the meetings, and debriefing them on the focus areas for follow-up visits. As coaches became more confident, they functioned more independently, conferred with MaNHEP staff to discuss progress and challenges, and sought feedback. MaNHEP staff continued to attend coaching visits on a limited or targeted basis. In most woredas, steering committees made up of coaches, woreda administration, and woreda health office and MaNHEP staff met monthly to discuss overall progress, challenges, and next steps.

METHODS

Surveys and individual interviews assessed the extent to which MaNHEP developed lead woreda improvement capacity. All procedures and instruments were vetted by institutional review boards at Emory and Addis Ababa Universities, as well as by the Ethiopian Federal Ministry of Health and the Amhara and Oromiya Regional Health Bureaus.

Survey

A questionnaire was developed in English and translated into Amharic and Afan Oromo in November 2012 to retrospectively assess key respondents’ perceptions of MaNHEP's 2-year progress in enhancing local and regional improvement capacity. Questionnaire design drew on established improvement science conceptual frameworks.[13-15] The questionnaire was administered to all QI team leaders (n = 50), health center coaches (n = 15), and woreda health office coaches and leaders (n = 19). The woreda health office leaders provided political support and guidance to the project and participated in learning sessions, but they were not otherwise targeted for capacity-building or responsible for improvement activities.

The questionnaire principally sought information in 4 areas: 1) perceptions of woreda culture and leadership for improvement activities before MaNHEP; 2) perceptions of woreda culture and leadership for improvement activities after MaNHEP; 3) motivation for participation in improvement work; and 4) self-assessed capacity for improvement work. Parts 1 and 2 asked participants to respond to Likert scale items about woreda culture and leadership (1, strongly disagree; 5, strongly agree) (Table 1). Woreda culture was defined as the environment and support for implementing improvement activities. Leadership was defined as actual administrative and leadership actions taken to support improvement activities. Part 3 asked participants to freely list all of the factors that motivated them to work on improving CMNH. Part 4 asked respondents to rate their own current improvement capacities across a taxonomy of 4 domains: I know how to do; I have done; I have led others to do; and I can apply this to a new area.[16] Participants were asked 5-item Likert scale questions about organizing and running QI teams; overcoming team communication problems; identifying priority improvement problems and settings aims; collecting and plotting improvement data; using a flow chart to analyze care process and identify barriers; developing, prioritizing, and planning tests of solutions/changes; reviewing results of a test; determining follow-up actions; and restarting improvement following a plateau in performance.

Table 1. Topics for Woredaa Culture and Woreda Leadership Survey of Woreda Coaches, Health Center Coaches, and Quality Improvement Team Leaders
Woreda CultureWoreda Leadership
  1. Abbreviations: CMNH, community maternal and neonatal health; QI, quality improvement.

  2. a

    District.

  3. b

    Subdistrict.

Uses facilitative supervision styleCommitment to improve CMNH
Involves cross-section of community stakeholders in QI teamsArticulated approach to improve CMNH
Uses QI data for decision makingAdministrative support to improve CMNH
Uses QI teams’ locally developed and tested solutions to improve CMNH careResults-oriented accountability for CMNH
Emphasizes freedom of QI teams to make and learn from mistakesEmphasis on local data to improve CMNH
Promotes sharing between kebeleb QI teams for cross-learningCreates supportive environment to improve CMNH
 Participate in improvement activities

Two trained interviewers administered the survey in November 2012 at locations convenient to respondents. After obtaining written informed consent, the interviewer read each question aloud, allowing participants time to privately mark responses and clarifying questions as needed. Surveys were read aloud to simplify the administration of the survey due to low levels of literacy. Data were entered and cleaned in Microsoft Excel 2010.[17] Descriptive statistics and bivariate analyses by respondent type (QI team member, health center coach, woreda health office coach), and region were performed for all variables in SPSS version 20 (Chicago, IL), using 2-sided chi-square or student t tests (α = .05).[18] Additionally, differences in average before versus after Likert scale responses on perceived woreda culture and leadership questions were assessed.

Mean Likert scale scores were calculated for each of the 4 improvement capacity domains (know how, have done, lead others, and can apply to new content area). Then a separate multivariate linear regression model was used to predict which respondent factors were associated with a higher mean Likert scale score for each domain. Key exposure variables of interest were the number of months served as a coach or QI team member (categorized in 6-month intervals) and the number of woreda and regional learning sessions attended. Respondent type, region of residence, gender, age, and years of school attended were included as control variables. There was a high level of missing data for regional and woreda learning session attendance, particularly among QI team members in Oromiya. Thus, the models were rerun with an imputed value of “none,” as well as with these variables omitted. All modeling was performed with SAS version 9.3 (Cary, NC).[19] Parameter estimates and P values were used to interpret the model variables, and the adjusted R[2] was used to assess the model's explanatory power.

The free-listed motivation factors were translated into English. One author (ST) developed a code book of themes and then coded the individual responses based on these. A second author (AHF) independently reviewed the code book and coded the responses. There was 91% agreement between their coding. Disagreements were discussed and resolved. The data were entered into a Microsoft Excel file, and the frequency of each response was computed.

In-Depth Interviews

A trained qualitative researcher conducted audiotaped semistructured interviews with 22 purposively sampled individuals representing the Amhara and Oromiya regional health bureaus (n = 4), zonal health departments (n = 2), woreda administration (n = 5), woreda health offices (n = 3), woreda coaches (n = 4), and kebele QI team leaders (n = 4). An interview guide tailored to each type of respondent sought information on respondents’ understanding and perceived value of the MaNHEP improvement approach; challenges perceived in implementing the approach; perception of woreda staff (if a woreda, zonal, or regional administrator) or personal capacity (if a QI team leader or woreda QI coach) for implementing the approach; perceived changes in relationships between kebeles, health centers, and woredas as a result of implementing the approach; and the ability to implement the approach for another priority area. The interview guides were translated by MaNHEP staff into Amharic and Afan Oromo and pretested. The researcher transcribed and translated each audiotape into English, checking the final transcript against the audiotape for fidelity.

One of the authors (KES) used a constant comparative iterative process to repeatedly review transcripts and code them for key themes, and a second author (ST) independently reviewed them. We organized themes by region and respondent group, as well as by topic of inquiry, and compared observations within and between regions (by respondent group and topic).

RESULTS

Survey

Most QI team members were male, older, and less educated compared with the woreda leaders and health center staff. Amhara health center coaches had a significantly higher level of education than those in Oromiya (100% with diploma/bachelor's degree vs 67%; P = .01) (Table 2). Most health center coaches and QI team members had participated in improvement activities for at least 2 years. Whereas most of the coaches participated in the initial training, their length of involvement varied by region: Amhara woreda coaches served for significantly less time than Oromiya coaches (38% vs 100% served ≥ 24 months; P = .01). The proportion of health center and woreda coaches attending the original coaches’ training and ongoing coaches meetings was higher in Oromiya than in Amhara. All of the QI team members, health center, and woreda coaches from Oromiya attended at least one regional and woreda-level learning session, whereas 29% of QI team members from Amhara never attended any regional learning sessions as a representative.

Table 2. Selected Characteristics of Sampled Respondents by Region
 n (%)
 AmharaOromiya
 QI TeamHealth CenterWoredaaQI TeamHealth CenterWoreda
Characteristic(n = 24)(n = 9)(n = 8)(n = 26)(n = 6)(n = 11)
  1. Abbreviations: QI, quality improvement.

  2. a

    District.

Female2 (8.3)3 (33.3)2 (25.0)3 (11.5)2 (33.3)0 (0.0)
Age in years      
20-344 (16.7)9 (100)3 (50.0)12 (46.2)4 (80)7 (63.6)
>3520 (83.3)0 (0.0)3 (50.0)14 (53.8)1 (20)4 (36.4)
Education level      
None or any primary13 (54.2)0 (0.0)0 (0.0)8 (30.8)0 (0.0)0 (0.0)
Any secondary8 (33.4)0 (0.0)0 (0.0)13 (50.0)0 (0.0)0 (0.0)
Certificate3 (12.5)0 (0.0)0 (0.0)5 (19.2)2 (33.3)0 (0.0)
Diploma/Bachelor's degree0 (0.0)9 (100)8 (100)0 (0.0)4 (66.7)11 (100)
Months served as QI team leader/coach      
6-124 (16.7)2 (22.2)1 (12.5)0 (0.0)0 (0.0)0 (0.0)
13-233 (12.5)1 (11.1)4 (50)2 (8.0)4 (66.7)0 (0.0)
>2417 (70.8)6 (66.6)3 (37.5)23 (92)2 (33.3)11 (100)
Participated in original coaches trainingNA8 (88.9)7 (87.5)NA6 (100)11 (100)
Number of coaches' meetings attended      
2NA2 (22.2)2 (28.6)NA0 (0.0)0 (0.0)
4 5 (55.5)2 (28.6) 3 (50.0)2 (18.2)
>5 2 (22.2)3 (42.9) 3 (50.0)9 (81.8)
Number of regional learning sessions attended      
07 (29.2)2 (22.2)0 (0.0)0 (0.0)0 (0.0)0 (0.0))
15 (20.8)2 (22.2)4 (50.0)2 (8.0)4 (66.7)1 (10.0)
25 (20.8)2 (22.2)0 (0.0)4 (16.0)1 (16.7)6 (60.0)
>37 (29.2)3 (33.3)4 (50.0)19 (76.0)1 (16.7)3 (30.0)
Number of woreda learning sessions attended      
02 (8.3)0 (0.0)0 (0.0)0 (0.0)0 (0.0)0 (0.0)
110 (41.7)0 (0.0)2 (25.0)1 (4.0)0 (0.0)1 (9.1)
28 (33.0)4 (44.4)4 (50.0)9 (36.0)2 (33.3)2 (18.8)
>34 (16.7)5 (55.6)2 (25.0)15 (60.0)4 (66.7)8 (72.7)

Woreda Culture for Improvement

Overall, respondents reported significant positive changes in many characteristics of woreda culture that support improvement work. The most striking increases across all groups and regions were reported for “using a facilitative supervision style” (increased from mean 3.0 to 4.5 on a 5 point Likert scale); “involving a cross-section of community stakeholders” (3.0-4.6); “using improvement data for decision making” (2.8-4.4); and “using locally developed and tested solutions to improve CMNH care” (2.5-4.3). Responses varied somewhat by respondent subgroup and region. Amhara QI team responses reflect smaller perceived changes across characteristics compared with their Oromiya counterparts. However, all before and after changes were significant at the P value less than .001 level for both regions, with the exception of nonsignificant changes in responses to the item “emphasizes freedom of QI teams to make and learn from mistakes” in Amhara.

Amhara health center coaches did reflect significant perceived change in 4 out of 6 categories. Their most significant perceived change was for “using locally developed and tested solutions to improve CMNH care” (1.9-3.9). Oromiya health center coaches’ change scores were not significant for any category; this may be because they assigned relatively high before as well as after scores and consistently ranked themselves more highly in improvement capacity than any other group. In contrast, at the woreda level, Oromiya woreda coaches’ perception of culture change is significant for all categories, whereas in Amhara there is no significant change although coaches’ after scores are higher than their before scores. Of note, the Oromiya woreda coaches’ scores showed the biggest improvement for “using improvement data for decision making” and “using locally developed and tested solutions to improve CMNH care” (both increased from 3.0 to 4.5). Overall, these scores reflect a perceived uptake by the woredas of the improvement principles and approaches that were introduced by MaNHEP.

Woreda Leadership for Improvement

Overall, respondents reported significant positive changes in characteristics of woreda leadership, most notably in “commitment to improve the health of women and newborns” (2.6-4.2) and “creating a supportive environment for coaches and QI teams to improve CMNH” (2.6-4.0). Responses again varied by region and subgroup. QI teams in both regions perceived significant positive changes in woreda leadership across all characteristics; however, Oromiya QI teams reported consistently higher endpoint scores. Oromiya health center coaches reported no significant changes; nevertheless, their responses reflect higher values—both before and at MaNHEP's endpoint (4.7-4.9) —compared with their Amhara counterparts. Amhara health center coaches’ responses all reflected significant change, but after scores were 2.8 or lower, with the exception of “participate in improvement activities” (1.9-3.9). Oromiya woreda coaches’ perception of changes in woreda leadership was positive and significant across all characteristics. Of particular note were changes in “administrative support to improve CMNH” (2.8-4.2), “results-oriented accountability for CMNH” (2.7-4.2), and “participation in improvement activities” (2.6-4.4). Amhara woreda coaches’ responses were not significant or particularly high (2.9-3.5).

Motivation to Engage in Improvement for CMNH

Across all respondents, the 4 most frequently mentioned motivating factors were “supporting the campaign to avert deaths and reach Millennium Development Goals”(42%),“change observed after project intervention” (33%), “high number of mothers and newborn deaths in the community” (31%), and “personal experience with maternal or newborn deaths” (27%). Other factors combined were mentioned by less than 24% of respondents. The order of factors varied by region and respondent subgroup, but the 4 main factors consistently ranked among the top 4 in both regions and by all groups of respondents.

Self-Assessed Capacity to Implement Improvement

Overall, the average scores were 3.7 and above, reflecting agreement by respondents in each of the 3 cadres (QI team member, health center coach, and woreda health administrator) that their level of capacity in the improvement skills shown in Table 3 had improved significantly. There is a statistically significant difference in the level of agreement for similar cadres between the 2 regions. Agreement was less for Amhara QI team leaders and health center coaches compared with their Oromiya counterparts. For QI team leaders, the regional differences were significant for all skill levels. For health center coaches, the differences were significant for 3 of the 4 skill levels. There were no significant regional differences in the woreda coaches’ scores.

Table 3. Average Level of Agreement on Capacity to Implement Core Improvement Skills by Type of Respondent and Region
 Amhara, meanOromiya, meanRegional Comparisons
 QI TeamHCWoredaaQI TeamHCWoredaP value
Skills(n = 21)(n = 8)(n = 5)(n = 23)(n = 6)(n = 10)QIHCWoreda
  1. Abbreviations: HC, health center; QI, quality improvement; SD, standard deviation. Average level of agreement on a 5 point Likert scale (1, strongly disagree; 5, strongly agree).

  2. a

    District.

Know how4.13.84.24.74.94.60.010.030.31
Have done3.73.73.84.75.04.40.0010.010.20
Lead others3.73.54.24.75.04.4< 0.0010.010.73
Can apply to new content area4.04.03.74.74.94.60.0030.080.15
Average4.03.74.34.75.04.50.0010.020.67

In the linear regression analysis, the models with imputed values of “none” for missing learning session attendance variables explained 7%, 15%, 21%, and 14% of the variation in respondents’ mean Likert scale score for the “know how,” “have done,” “led others,” and “can apply to new content area” items, respectively. None of the variables in the model was significant for the “know how” items. Only region was significantly associated with the mean score for the “have done” items. Amhara respondents on average had a score that was 0.5 points lower than Oromiya respondents (P = .04). Region of residence and months served as coach were significantly associated with the mean score for the “led others” items. Amhara respondents on average had a score that was 0.6 points lower than Oromiya respondents (P = .02). A greater number of months served as a coach/team member was also associated with a higher score, with each 6-month increase in service associated with a 0.4-point increase in mean score (P = .03). The longer period of serving as a coach provides additional exposure to training, mentoring, and peer learning opportunities designed to increase competency in improvement. Coaches coming on midway through the activity needed to catch up on skills. This also has an effect on a QI team that may be supported by a new coach who is less able to guide them. Only region of residence is significantly associated with the mean score for the “can apply to new content area” items. Amhara respondents on average had a score that was 0.5 points lower than Oromiya respondents (P = .04). Results for the models that omitted the learning session attendance variables due to data collection problems were similar to the above findings.

In-Depth Interviews

All 22 respondents had been in their current position for at least one year, and most had 6 to 8 years of experience. All respondents highlighted positive results of MaNHEP's improvement strategy including increased facility births, reduced maternal and newborn deaths, and new processes of care delivery—especially identification of pregnant women. They universally recognized the capacity built in the woredas, particularly in new CMNH-care knowledge and skills. Most respondents referenced some or all of the key improvement skills. The themes of community empowerment and focused improvement emerged strongly from the interviews across a variety of questions. These and other themes are discussed and illustrated below.

Focused Improvement

Multiple respondents from different levels mentioned that, before the MaNHEP improvement approach was introduced, they had been working on government programs trying to improve all areas at once. They felt that the approach of finding specific weak areas and working in a focused way to improve those was beneficial. The QI team leaders all said that before MaNHEP they had general health promotion activities but no special focus on CMNH care and saving lives. As one regional level health official noted, “We were working, but it was not focused like MaNHEP. They came with system which enables you [to know] where you should focus and at what time you should focus and what are the needs on this regards.”

Improvement in Coordination and Supervision

Almost all agreed that the interaction and coordination between the woreda health office, health center, health post, and community had improved. The QI team leaders’ perceptions were mixed about changes in supervision. Although all felt that their woreda health office supervisors were supportive and provided feedback, one team leader raised a problem of treating volunteer team members such as employees, and one raised the low frequency of woreda staff visits. Amhara woreda coaches reported improved supervisory visits that now included discussions instead of mere collection of reports. As one QI team leader mentioned, “People from woreda will share their observation [of] other kebeles and guide us to improve our weak part. They are helping us to advance our knowledge and skills. But, previously there was no close follow-up.”

Community Empowerment

Empowerment of communities to solve problems for themselves was a frequently mentioned component of the MaNHEP approach. One regional-level health official noted, “MaNHEP was able to create knowledge and a system that enables the community to identify and solve their problems by themselves…. People are able to have firm standing on one thing, ‘We can solve our problems by ourselves.’

Understanding of Improvement

Respondents mentioned most of the key aspects of the improvement approach, but emphasis varied by cadre and region. Woreda coaches and QI teams homed in on the details of MaNHEP's efforts to strengthen their capacities. Woreda health office personnel articulated a somewhat better grasp of the improvement strategy than did other woreda administrators, more so in Oromiya than in Amhara, where outcomes of the process (eg, new ways to identify pregnant women) were of greater interest. Oromiya QI team leaders identified the “MaNHEP approach” with teaching CMNH family meetings, whereas Amhara QI team leaders focused on the continuous “evaluation” of activities. One woreda coach perfectly captured the process:

[The] model helps them to plan, set goals, implement their plan, and evaluate the progress. This is a cyclical process…. If we have a meeting [scheduled for] next month they will prepare a new plan, and, based on previous performance, they will prepare a new plan. At the end they will bring the report of their performance and again they will sit and discuss the progress—what was done and what are the problems. They are using this cycle every month.

A QI team leader reported on the improvement approach at work:

When we started the work we tried to identify pregnant women using Ider [local social insurance] and mourning, but it was not effective thus we shifted to other options like observing women at water fetching points, asking the family of her husband and close friends. In these ways we are able to identify those shy pregnant women. These kinds of ideas generated from our discussion.

Capacity for CMNH Care

All acknowledged that coaches had developed much better CMNH care capacity and knowledge and that community CMNH capacity had been strengthened. The QI team leaders did not distinguish between the MaNHEP improvement strategy and the MaNHEP CMNH training they had received.

Ability to Continue

Respondents agreed that they would be able to continue the CMNH care work started under MaNHEP. All believed that these activities should spread to other kebeles, woredas, and zones. Coaches urged that the activities be built into plans and budgets to cover costs (eg, for travel and materials). Despite one coach's concern that workload might exceed capacity, one woreda official expressed confidence in maintaining the momentum that MaNHEP had built:

It just needs commitment because basic trainings [have been] given…. Moreover, our staffs also have different trainings and experience sharing from other woredas and region. There has been different capacity building. Therefore, I don't see anything [to prevent] us from continuing this thing…. What MaNHEP showed us is something wonderful.

Spread to New Areas

Respondents linked spread to the recently implemented health development army, something MaNHEP had discussed with them. One Oromiya respondent mentioned that there was at least one new area in which they were already using ideas generated by the community to make improvements. An Amhara respondent suggested the use of QI team-developed identification approaches in new areas to identify other people in need of services (eg, tuberculosis, immunizations).

DISCUSSION

Survey and interview data presented in this article and results from improvement efforts outlined in other articles confirm that, overall, MaNHEP was able to build health system capacity for continuous improvement in 6 districts of Ethiopia.[1, 4-6] Clear advances were reported in the enhanced receptivity for improving CMNH care in the project woredas and in the actions of leaders to implement and extend those advances. Those involved at all levels indicated they not only knew how to lead the improvement process, and had actually done so, but that they were able—or felt themselves competent—to lead the spread of the improvement process to other areas. Of equal importance, all respondents expressed congruent motives for involvement in the improvement process. Improved results from monthly monitoring by QI teams confirm that improvement capacity was built. For example, the proportion of newly identified pregnant women who attended their first antenatal care visit each month increased from 38% from November 2010 to 90% in October 2012.[20] Other MaNHEP reports indicate overall project success, including enhanced care provider capacity and service demand and use.[1, 4-6] Change ideas were synthesized, and their effectiveness ranked and then packaged to share among the 51 kebeles and to support spread to new areas.[21]

MaNHEP achieved support for improvement by developing leaders at the woreda level and working within existing government structures rather than by creating a parallel support system. This strategy is in line with suggestions emphasizing the need for organizational and management support for sustaining new ideas developed by QI teams and for longer-term changes to foster an innovative culture within an organization.[22] Kebele-level QI teams clearly reflected differences in the ways that woreda leaders foster a supportive environment for improvement. Perceived effectiveness of an intervention is one important part of creating a culture of continuous improvement and innovation,[22] and one of the key motivating factors for team members was seeing changes in the way that care was delivered after the project.

Differences in health system performance existed between the 2 regions from the onset of the project.[23] Regional comparisons showed that, although both groups reflected significant changes in culture, leadership, and competency for improvement, Oromiya continued to be the stronger region. The regional differences may reflect differential levels of attendance in improvement activities (less by Amhara than Oromiya) and can help guide and highlight how to focus capacity building and improvement efforts. Additional research could be done into the factors that contribute to these differences related to economics, geography, politics, leadership, or other factors.

The multifaceted approach to building capacity through classroom training, on-the-job training, and continuous support and feedback served to strengthen and build capacity over time rather than relying on a one-off training input. Loevinsohn et al discuss the value of systematic supervision with objective measures and feedback to improve quality of care.[24] After regular coaching and support, QI teams gradually achieved a level of independence to test changes, monitor their own data, and improve care in multiple areas simultaneously.

The study had several limitations. Because the survey involved retrospective judgments, responses to items about woreda culture and leadership prior to MaNHEP may be subject to recall bias. The self-assessment of improvement skills has not been externally verified. Because health center coaches were not included in the interviews, it is not possible to compare their self-assessment with their qualitative responses. Furthermore, there was a high level of missing data for the learning session attendance of Oromiya QI team members. Learning sessions provided opportunities for refresher training and peer-to-peer learning and support, so the hypothesis is that more learning sessions attended would have been associated with higher self-reported competency scores. Because only one QI team member was surveyed per kebele team, learning session attendance was expected to be low; only one to 2 team representatives can attend a given learning session. A larger sample of QI team members may be needed to better assess the effect of learning session attendance for this group. Small sample sizes overall may have also limited our power to detect significant effects in the regression analyses.

CONCLUSION

The changes seen in leadership, culture, motivations of different cadres, self-assessment of capacity, and reflections of all levels and cadres in interviews indicate that the project was successful in creating lead, woreda learning laboratories that provided enabling environments to improve care. The rapid-cycle process improvement approach employed in MaNHEP is used throughout the world to facilitate clinical and organizational improvement. The application to the community level provides an example of the powerful results that can be achieved when providers at all levels of care and community members come together to improve both clinical care and public health concerns. In addition, the approach used demonstrates the importance of intentionally building organizational culture and leadership to create an environment enabling improvement through the engagement of all stakeholders.

There was consensus that the MaNHEP model—the combined CMNH family meeting and improvement approach—should be spread to new geographic areas. MaNHEP worked closely with regional, zonal, and woreda leaders to determine how its activities could be integrated into emerging governmental structures. Whether local authorities will be able to leverage uptake of the lead woreda approach into other zones and regions is beyond the scope of this evaluation and the MaNHEP project. Further research is merited on this important question.

CONFLICT OF INTEREST

The authors have no conflicts of interest to disclose.

ACKNOWLEDGMENTS

We would like to thank the Bill and Melinda Gates Foundation for their support of the Maternal and Newborn Health in Ethiopia Partnership (MaNHEP) and MaNHEP partners including national, regional, zonal, and woreda Ministry of Health; Emory University; JSI Research & Training Institute, Inc.; University Research Co., LLC; and Addis Ababa University. We are very grateful to Mr. Robel Yirgu and Mr. Elias Merdessa for their work as interviewers for the survey, and to Mr. Tewdros Getachew for his work on the qualitative component of the study in conducting interviews, transcription, and translation. We would like to acknowledge Dr. Nigel Livesley for his review of tools and articles. We thank Abebe Gebremariam Gobezyehu, Lelise Tadesse, Hannah Tesema, Yeshiwork Aklilu, Abera Biadgo, and Binyam Fekada Desta of the MaNHEP team for their roles in building capacity for improvement. Finally, we want to recognize all the hard work of the quality improvement teams, guide teams, coaches, and leadership in our project kebeles, woredas, and regions.

Biographies

  • Kim Ethier Stover, MA, is Senior Quality Improvement Advisor for the Maternal and Newborn Health in Ethiopia Partnership. An employee of University Research Co., LLC, she is based in Bethesda, Maryland.

  • Solomon Tesfaye, MD, MPH, is Quality Improvement Advisor for the Maternal and Newborn Health in Ethiopia Partnership. An employee of University Research Co., LLC, he is based in Addis Ababa, Ethiopia.

  • Aynalem Hailemichael Frew, RN, MPH, is the Oromiya Regional Program Manager for the Maternal and Newborn Health in Ethiopia Partnership. An employee of JSI Research & Training Institute, Inc., she is based in Addis Ababa, Ethiopia.

  • Hajira Mohammed, MSc, is the Maternal and Newborn Health in Ethiopia Partnership Monitoring and Evaluation Officer. An employee of JSI Research & Training Institute, Inc., she is based in Addis Ababa, Ethiopia.

  • Danika Barry, MPH, is the Maternal and Newborn Health in Ethiopia Partnership Research Coordinator at the Nell Hodgson Woodruff School of Nursing at Emory University in Atlanta, Georgia.

  • Lamesgin Alamineh, MPH, was a Maternal and Newborn Health Specialist for the Maternal and Newborn Health in Ethiopian Partnership. An employee of JSI Research & Training Institute, Inc., he is based in Bahir Dar Amhara Region, Ethiopia.

  • Abebe Teshome, BSc, is a Maternal and Newborn Health Specialist for the Maternal and Newborn Health in Ethiopia Partnership. An employee of JSI Research & Training Institute, Inc., she is based in Addis Ababa, Ethiopia.

  • Kenneth Hepburn, PhD, is a Professor at the Nell Hodgson Woodruff School of Nursing at Emory University in Atlanta, Georgia. He is a member of the Maternal and Newborn Health in the Ethiopia Partnership Project Advisory Committee.

  • Lynn M. Sibley, CNM, RN, PhD, FACNM, FAAN, is Professor at the Nell Hodgson Woodruff School of Nursing and Rollins School of Public Health at Emory University in Atlanta, Georgia. She is Principal Investigator for the Maternal and Newborn Health in Ethiopia Partnership.

Ancillary