Corresponding Author Dr. Sennen Hounton, Centre Muraz, PO Box 298 Bobo-Dioulasso, Burkina Faso. Tel.: +226 20 97 26 30; Fax: +226 20 97 01 77; E-mail: firstname.lastname@example.org
Introduction This paper aims to describe the design, methods and approaches used to assess the effectiveness and cost-effectiveness of the Skilled Care Initiative in reducing pregnancy-related and perinatal mortality in Ouargaye district, Burkina Faso.
Methods The evaluation used a quasi-experimental design, mixed methods and a composite of tools to compare mortality and severe morbidity (near-miss) of women in reproductive age, perinatal mortality, facility functionality, perceived quality of care, utilisation of maternal health services, and costs borne by families and the health care system for maternal health care in Ouargaye and Diapaga districts. Structured questionnaires and interview guides were developed, pre-tested and piloted prior to the main survey. The evaluation was carried out from January to July 2006. A household census was used to retrospectively assess pregnancy-related and perinatal mortality over the previous 5 years, and causes of pregnancy-related death were identified using a newly developed and tested probabilistic model for interpreting verbal autopsy data. Data were directly entered into Personal Digital Assistant devices at the point of interview. Analyses included univariate and multivariate regressions and incremental cost-effectiveness ratios.
Results A population census covering over half a million people, three qualitative surveys and facility surveys in 47 health centres have been carried out.
Conclusion A partnership with key stakeholders and the use of mixed methods proved feasible for evaluating complex safe motherhood strategies, and the use of hand-held computers proved possible for direct data capture, even in this remote rural environment.
Pregnancy-related death remains the health indicator with the greatest disparity between the developed and the developing countries (Ronsmans & Graham 2006). Achieving the fifth Millennium Development Goal (MDG) is challenging and requires reliable tools and methods to track and document progress and to identify most cost-effective safe motherhood strategies. This assessment was conducted by Immpact, a global safe motherhood research initiative developed in response to this need, and aiming to improve methods and tools for measuring pregnancy-related mortality, identify cost-effective strategies for reducing pregnancy-related and perinatal mortality and build capacity for robust evaluation and evidence-based decision-making (Hounton et al. 2005). To generate evidence on the effectiveness and cost effectiveness of existing strategies, Immpact (http://www.immpact-international.org) has worked in countries specifically selected to address a broad mix of contexts and safe motherhood issues. In Burkina Faso, Family Care International’s (FCI) Skilled Care Initiative (SCI), developed and implemented in various ways across the Ouargaye district in Burkina Faso, was selected for evaluation in a participative process (Madi et al. 2007). The SCI was implemented using a quasi-experimental design with Diapaga as the comparison health district. This paper is part of a series of papers on the effects of the SCI in reducing pregnancy-related and perinatal mortality. It aims to describe and discuss the implementation process and methodological approaches used during this evaluation.
The evaluation setting
This evaluation was conducted in Burkina Faso, one of Immpact’s focus countries and one of the poorest countries in the world. Burkina Faso is situated in the heart of West Africa and is ranked 174/177 on the Human Development Index in the 2006 United Nations Development Program Human Development Report. The country is characterised by very low educational levels, poor health status and a high maternal mortality ratio of 484 per 100 000 live births (Macro International 2000). Under-five child mortality, infant mortality and neonatal mortality rates were 184 deaths per 1000 live births, 83 per 1000 live births and 31 deaths per 1000 live births respectively (Macro International 2004).
The SCI was implemented in various ways across Ouargaye Health District, while the nearby Diapaga Health District served as a comparator. The location of these Districts in south-eastern Burkina Faso is shown in Figure 1. Ouargaye town is some 230 km from the national capital, Ouagadougou, and the District borders Togo to the south. Diapaga town is some 450 km from Ouagadougou, and the District borders Benin to the south and Niger to the east. Both Ouargaye and Diapaga towns have district hospitals with surgical and in-patient facilities. The area directly between the two districts is Kompienga District and includes a sparsely populated National Park.
Table 1 describes some key characteristics of Ouargaye and Diapaga Districts at the start of the SCI. Some considerations used by FCI to select the comparison district were: (i) being in the same region but sharing no borders with the intervention district, (ii) having a similar proportion of skilled attendance at delivery and (iii) having similar levels of poverty. Ouargaye is a remote, rural district lacking electricity, running water, good roads and an effective transport system. SCI sought to upgrade the district hospital and 13 health centres, and conduct community activities across the District; there was also a UNFPA-sponsored reproductive health project that upgraded four of the remaining health centres in Ouargaye district. The UNFPA-sponsored project was similar to SCI in terms of supply-side activities (equipment, drugs, training for providers). A further non-Governmental Organisation, World Neighbours, also developed a reproductive health project with focus on family planning (condoms and contraceptive implants), prevention of malaria (anti-malarial drugs, iron), and traditional birth attendants in some villages associated with five health centres (some of which were also supported by SCI). All of these activities occurred at different stages in the 5-year period, during which time two additional health centres were also built and commissioned.
Table 1. Baseline characteristics (2001) of the Ouargaye and Diapaga Districts, Burkina Faso
Average distance to Ouagadougou (km)
Male:female sex ratio
Average distance to facility, within catchment area (km)
In the comparison district, a UNICEF-sponsored cost-sharing mechanism aiming at decreasing financial and geographical barriers to emergency obstetric care was implemented from 2001. The UNFPA-sponsored project (also found in parts of Ouargaye District) was implemented in four health centres in 2002 and two more in 2004. In addition, there was a weekly community radio broadcast featuring safe motherhood health education programmes and a large (∼4000 deliveries per year) missionary health centre.
Implementing the evaluation
Eliciting the research question.
It is well recognised that ownership of contextually relevant research questions is essential to translating research results into policy and practice (Sassi 2002; Gibson et al. 2004). Prior to launching Immpact research activities, a national steering committee was established to elicit the priority research questions and to review the research proposal. This participative process was deemed necessary to ensure relevance and congruence of the evaluative research to Burkina Faso and to foster ownership of results. Detailed descriptions of the process and results are reported elsewhere (Madi et al. 2007; Ministry of Health 2002; Ministry of Health 2002). This participative process led to the selection of research questions and identification of the existing opportunity for evaluation. The main objectives were to investigate the extent to which the SCI was effective in reducing pregnancy-related and perinatal mortality, and to assess how costs borne by the health system and households for maternal care were affected by the SCI.
Understanding the intervention and the context.
The SCI was a 5-year (2001–2005) integrated safe motherhood programme funded by the Bill & Melinda Gates Foundation and implemented by FCI in selected districts within Burkina Faso, Kenya and Tanzania. The overall goal was to contribute towards pregnancy-related mortality reduction by ensuring skilled care for every woman before, during and after childbirth. The project aimed to increase rates of skilled attendance by at least 10% in the project districts. Project activities focused on two main areas: improving the availability and quality of maternity care, and promoting increased utilisation of maternity services. It also aimed to strengthen obstetric care at the first referral level – the district hospital – to where women with more serious complications should be referred.
District-level interventions were complemented by national-level activities aimed at strengthening government policies and strategies related to maternal health, including the development of clear standards and protocols for obstetric care.
In Burkina Faso, the SCI was implemented in Ouargaye Health District. To improve the quality and availability of maternity care, FCI worked in the district hospital and, on a phased basis, in 13 of the then 19 health centres, while the remaining six health centres received support from other donors. Activities included infrastructure improvements, such as the provision of solar panels for electricity in the maternity departments and the health care providers’ houses, training of all maternity care providers in obstetric best practices and interpersonal communication, provision of essential maternity equipment and supplies; and strengthening of the referral system by donating an ambulance and radio-call system and by subsidizing cost-sharing for emergency transportation. FCI also worked to strengthen health service management, supervision and health management information systems, and introduced a problem-solving approach among facility managers and staff – Client-Oriented Performance-Efficiency for Maternal Health Services (EngenderHealth 2007) – to encourage a continuous quality improvement process among staff.
To complement supply side interventions, the project launched an intensive behaviour change and community mobilisation effort throughout the District to encourage health-seeking behaviour and build critical community support for skilled care. Activities included participatory theatre and songs around five key themes: use of antenatal care, birth preparedness, skilled care at childbirth, recognising (and responding) to emergencies and early postpartum care. All was planned in close collaboration with traditional chiefs to develop and implement community action plans. A network of 186 community outreach agents was established to educate communities on skilled care and a range of promotional and educational materials produced to support the ongoing behaviour-change activities in the communities.
After administrative clearance and preparatory work, the project started its supply side activities in late 2001 in what was referred to as ‘first generation health facilities’– five health centres and the district hospital. The next phase was a scale up in early 2003 to the ‘second generation health facilities’, an additional eight health centres in the District, thus totalling coverage to 14 facilities out of then 20 in the district. The selection and training of community outreach agents started in mid 2004. The community mobilisation started in early 2005 with the participatory theatre followed by the work with the traditional leaders.
Dealing with attribution and monitoring challenges.
The complexity of this evaluation thus arises from the complexity of the intervention (different components covering parts of the intervention district at different times) and the multiplicity of interventions in the same setting, posing a real problem for attribution of effects and internal validity. Aiming to conduct a real-life assessment of an existing strategy, we used a quasi-experimental design (Ovretveit 1998; Habicht et al. 1999), and a mixed-methods study covering the project implementation period (Figure 2) to assess changes in selected outcomes and process measures within the intervention and comparison districts. Although a prospective, randomised trial approach might have improved control of confounding and reduced sources of bias, it would not have been appropriate or possible in this real-world setting and would not necessarily have increased external validity (Habicht et al. 1999; Victora et al. 2004; Blackwood 2006). Here a robust trial-based design would have been impossible because of variations between multiple stake-holders, inherent dissimilarities between the Districts and the complexity of the SCI implementation.
In spite of challenges in assessing and tracking pregnancy-related and perinatal mortality (Stanton et al. 2001; Hill et al. 2006), ensuring progress towards MDGs 4 and 5 will require that these impact indicators remain targets of all safe motherhood strategies and evaluative research. However, the effectiveness of complex public health strategies cannot solely be judged on health outcome indicators (Victora et al. 2005) because the pathway to health outcomes (in this case to pregnancy-related deaths) is a continuum (Bryce et al. 2005). Table 2 summarises the mix of outcomes and process indicators used for this complex evaluation: pregnancy-related deaths, near-miss, perinatal mortality, skilled delivery, Caesarean sections, unmet obstetric needs, outcomes after pregnancy, costs borne by families and the health system for uptake of maternal care. Given the relatively small sample size of the population of study (500 000 for both intervention and comparison districts), the low rates of health care utilization, the desirability to assess change in the period of implementation, the need for a number of deaths to find statistically significant difference, we were only left with the option of a census, albeit limited in monitoring progress in the short run (Stanton et al. 2001). Each of the methods and tools used (Table 3) is briefly described below.
Table 2. Outcomes, indicators and assessment methods for the SCI evaluation. Unmet Obstetric Needs Network (2007)
Examples of indicator
CEmOC, comprehensive emergency obstetric care.
Maternal mortality ratio Maternal mortality rate
Census, verbal autopsy
Perinatal mortality rate
Census, verbal autopsy
Severe morbidity access to CEmOC
Near miss rate, quality of care long term disability rate proportion of unmet obstetric needs
Interviews with women experiencing pregnancy between 2001 and 2005
Perinatal deaths population based Caesarean section rates Institutional delivery rates
Adult deaths survey
Interviews with heads of households
Short-term outcomes after pregnancy
Interviews with women experiencing pregnancy during past year
Physical, psychological, social consequences of pregnancy & anaemia
Household costs for maternal care
Interviews with women who had delivered within last 7 weeks
Estimates of costs for maternal care from individuals’ perspectives
Interviews with household informants re adult female deaths
Unmet obstetric need
Case extraction for women who underwent a maternal obstetric intervention for an acute maternal indication
Unmet obstetric needs
Incidence and profile of life threatening complications
Maternal death records
Reports, registers, case extraction
Estimates of institutional maternal deaths
Estimates of costs for maternal care from health systems perspective
Self-administered by providers
Proportion of maternal care within general care
Interviews with health care providers & data extraction
Health centre functionality index
Interviews with health care providers
Profiles of incentives for quality delivery care
Knowledge, attitudes, practice
Interviews with health care providers
Assessment of providers’ skills and knowledge
Interviews with verbal autopsy informants
Barriers to notification of community maternal deaths
Interviews with near-misses and families
Near-miss coping strategies
Perceived quality of care
Interviews observation (provider & user interaction)
Quality of care as perceived by users and providers
A census of all households was conducted using a general pregnancy-related mortality module recommended for data collection on household deaths (Hill et al. 2001). Collection of such data are often restricted to the 12 or 24 months prior to interview, due to concerns regarding recall bias and the dissolution of households following an adult death. For the purposes of pregnancy-related mortality measurement in the SCI evaluation, it was necessary to extend this to a 5-year period, necessitating careful assessment of potential recall bias. The census questionnaire, administered to the head of household, was designed to collect household assets, socio-demographic characteristics of all household members (age, sex, occupation, education, marital status, etc.) in every household in both districts. The household assets were used to derive wealth quintiles using the method developed by Filmer and Pritchett (2001). The exact location of each household was recorded using the Global Positioning System to permit mapping of health outcomes. Appended to the main census questionnaire were an ‘adult death’ module designed to record details of deaths during the preceding 5 years between the ages of 12 and 49 years, including information on age at death, sex, pregnancy status at death, outcomes of pregnancy and foetal/newborn survival and a ‘perinatal death’ module, designed to record, for all women aged 12–49 years, the number of pregnancies, pregnancies lasting more than 6 months, place of delivery, delivery attendant and foetal/newborn survival during the preceding 5 years. Perinatal mortality rates and population-based Caesarean section rates were calculated using this module. It should be noted that the approach used to capture perinatal deaths was designed as an add-on to the existing census questionnaire, which was needed for other data collection purposes. Although there is extensive experience of collecting data on household deaths within a 1–2 year period via censuses, no references validating this method for capturing late pregnancy losses or deaths within the first week of life were found (as age at death is generally recorded in completed months or years in a census). Data on perinatal death in surveys of women of reproductive age are believed to suffer from misclassification of both vital status at birth and gestational age, and underreporting (Stanton et al. 2006). It is reasonable to assume that reporting via a brief interview with someone other than the person who experienced that pregnancy could lead to greater data deficiencies. Consequently, the questions in the perinatal module were administered to the women who carried the pregnancies and not the head of household, the usual respondent for the census. Nevertheless, perinatal mortality measured via this approach resulted in very low perinatal mortality rates even when restricted to the two years prior to interview (∼32 per 1000 live births).
Table 3 also describes the eligibility for the three remaining household survey modules (verbal autopsy, outcomes after pregnancy and household costs). Causes and profiles of deaths among women of reproductive age were characterised using InterVA-M, a new probabilistic model for interpreting community-based verbal autopsy data, as previously reported elsewhere (Byass et al. 2006; Fottrell et al. 2007). Even when well measured, pregnancy-related mortality represents only the ‘tip of the iceberg’ regarding women’s suffering as a result of pregnancy complications (Ronsmans & Graham 2006). To capture changes in physical, psychological, social and familial diseases, disabilities or discomfort that women may suffer as results of pregnancy, a specific ‘outcomes after pregnancy’ questionnaire was developed, pre-tested and piloted. The tool was administered to women aged 12–49 years who had an experience of pregnancy during the 12 months prior to interview. The questionnaire asked about indicators of women’s own socio-economic status, capacity to undertake activities of daily life, psychological status, social and familial relation status, and discomfort as a result of pregnancy and childbirth. Finally, a finger-prick blood sample was taken for on-site estimation of anaemia at the community level among a random sample (five per enumeration zone) of eligible women. Any cases of dangerous levels of anaemia were immediately followed-up.
The approach taken to conduct a cost-effectiveness analysis in the framework of this evaluation was to estimate costs borne by women and their families as well as costs incurred by the health system for maternal care, to derive costs per specific outcomes from health systems and societal perspectives. At the household level, data were collected on delivery costs, delivery time costs, and travel time costs for delivery, financing delivery costs for women and their families, and household expenditure over the last month as a proxy for income. Ability to pay, households’ wealth and expenditure were used to check for distributional issues and whether or not households frequently incurred catastrophic health expenditure for maternal care.
A functional health centre combined with well trained and motivated staff, appropriate equipment and standard guidelines and monitoring procedures is pre-requisite for quality care and could greatly affect the incidence of pregnancy-related mortality and severe morbidity (Ronsmans & Graham 2006). The main purpose of the health facility survey within the evaluation was to investigate the quality of maternity care and the trends of selected process indicators (institutional delivery, population based Caesarean section rates and unmet obstetric needs) and the extent to which the latter are correlated with outcome measures, such as pregnancy-related deaths or perinatal deaths.
Additionally a 6-month prospective study of narrowly averted pregnancy-related deaths (near-misses), a survey of institutional reporting of pregnancy-related deaths, a health centre performance survey, an incentive survey, and a facility costing survey were conducted in both of the district hospitals and all 43 health centres. The near-miss study was carried out at the two district hospitals to describe incidence and causes of near-misses and other severe obstetric complications and to identify contextual determinants which could contribute to mortality reduction strategies. The survey of institutional reporting of pregnancy-related deaths was developed to compare facility and community levels of pregnancy-related mortality. The health centre performance survey was conducted to document levels of functionality regarding referrals, transport, communications, infrastructure, equipment, supervision, staffing and drug supply in relation to maternity care. A health service functionality index was subsequently constructed based on key variables such as staff complement, personnel knowledge, drug supply, laboratory facilities, availability of water, infrastructure, vehicles, etc. using principal component analysis. The incentive survey was conducted to document motivating or de-motivating financial as well as non-financial factors that influenced staff performance.
A better understanding of health-seeking behaviour and circumstances typically surrounding pregnancy-related deaths and severe complications is necessary for making quality delivery care available as widely as possible. The qualitative component of the SCI evaluation was designed to assess barriers and facilitators to the notification of pregnancy-related deaths in communities, health seeking behaviour around the deaths of female adults, near-miss survival coping strategies and perceived quality of maternity care from users and providers. The approaches used were individual interviews with users (43) and providers (9), focus group discussions with communities (8), and non-participative observations (6). Interview guides were developed, pre-tested and piloted prior to the main survey. NVIVO software was used for data management and analysis (QSR International, 2007).
Not having a clear intervention to non-intervention distinction between the two Districts made the conduct of cost-effectiveness analysis more difficult. Our approach at facility level was to cost maternity health services in all health facilities, to identify principal cost categories, to conduct sensitivity analyses on major cost categories, and to derive costs estimates per outcome (deaths, complications) or per institutional delivery. The principal cost categories were derived from the WHO guidelines for costing of health services (WHO 1999). The volume of maternal care work by health workers was estimated using a time allocation sheet and costs of capital items (buildings, infrastructure and vehicles) were annualised. The allocation of total costs to maternal care was subsequently derived from all cost categories other than health workers.
Figure 3 summarises the organisation put into place to carry out the SCI evaluation. Establishing good working relationships with stakeholders is a pre-requisite for large population surveys; this was facilitated by the participatory process used for setting the research agenda (Madi et al. 2007). We designed and implemented an intensive awareness strategy within communities, the local administration, the police, village chiefs and the health system prior to launching the evaluation. This was necessary so as to avoid confusion or conflict with health officials, to ensure consistent messages on the study objectives, to ensure security of people and materials and to ensure acceptance and support from the communities.
The data collection personnel were organised in three teams for the household surveys, the facility surveys and the qualitative studies. For the household surveys, each district’s team was coordinated by a demographer, assisted by a senior field work supervisor, who oversaw all administrative and technical issues related to the survey. There were a total of 209 enumeration zones (geographically delimited areas of about 1000 people), in Ouargaye and 304 in Diapaga. An average of 20 to 25 enumeration zones was assigned to a team of seven to eight interviewers led by one supervisor. Each district established a data management centre run by a data manager, assisted by four data technicians, for downloading, merging, transferring and storing data directly captured in the field using hand-held computers (PDAs), onto laptops (Byass et al. 2008). To facilitate transportation of the research team and materials, to ensure an energy source for recharging PDAs where mains electricity was unavailable and for security reasons, cars were hired for the district coordinators, data managers, data technicians and field work supervisors. For the facility surveys, one team (comprising a public health specialist, two medical officers, a midwife and a data technician) consecutively covered the district hospitals and all health centres in both districts. The qualitative studies in both districts were also carried out by a single team of a senior social scientist, two junior social scientists and six research assistants.
The overall Immpact research proposal was approved by the Ministry of Health National Health Research Ethics Committee (Ouagadougou, Burkina Faso). The specific Evaluation and Evidence Research Group protocol was approved by Centre MURAZ (Bobo-Dioulasso, Burkina Faso) Institutional Review Board. Administrative authorizations were obtained at all level of the administrative chain (Ministry of Health, Region Governorates, Regional Directorates of Health, National and Regional Hospital Directorates, Province High Commissioners, District Health Management Teams, Heads of Clinical Services in Hospitals and village community leaders).
In spite of the technical, methodological and logistical challenges in conducting this evaluation, the overall conclusion is that a mixed methods approach is necessary for a holistic evaluation of a safe motherhood strategy. Other papers in this supplement report results of applying the mixed methods described here, leading to conclusions which could only have been reached by assessing both process and outcome indicators in parallel. For example, increased rates of skilled attendance do not necessarily lead to greater uptake of emergency obstetric care, nor lives saved. By looking at near-miss events, we were able to cross-check issues of quality of care against utilisation of services. The economic analysis was also necessary to address issues of efficiency and scaling up.
The challenge of attribution will remain a constraint on evaluating field interventions. Health districts are not ‘empty vessels’ and there were real challenges to find methods for disentangling SCI effects from other competing safe motherhood interventions and to attribute results. The choice of a suitable comparator is a requirement for both quasi-experimental design and for cost-effectiveness analysis and our approach was to make use of the same comparison district previously identified by FCI (Diapaga). Diapaga at the start was similar to Ouargaye in many respects regarding socio-demographic and economic profiles (Table 1). Concerns about significant baseline differences between the two districts (for example regarding the uptake of emergency obstetric care) also inevitably arose. The difficulty in finding an appropriate control area for a quasi-experimental design is an inevitable real-life problem, not limited to conducting evaluations of safe motherhood interventions. Many systematic reviews exclude results from quasi-experimental studies because of doubts as to comparability. As far as possible, we used robust analyses to compare outcome and process measures at inception, to compare trends between 2001 and 2005, and to compare before and after intervention measures, all applied to appropriate sub-sections of the districts surveyed. To make comparisons of pregnancy-related and perinatal mortality on the basis of the 5-year recall period necessarily adopted in the survey, adjustments were made in analyses to minimise recall bias effects. These included the exclusion of 2001 data (because of recall bias in live birth data), use of multiple imputation methods for assigning unknown years of pregnancy-related death, and the use of a person-years-at-risk approach for analysing pregnancy-related mortality over time.
Although our perinatal mortality assessment resulted in improbably low perinatal mortality rates, our results were similar to those obtained from the 2003 Burkina Faso Demographic and Health Survey (total fertility rate of 6.2 vs. 6.0 and perinatal mortality rate of 32 per 1000 vs. 33 per 1000). This raises questions as to whether cross-sectional methods can reasonably elicit perinatal events, or whether more intensive longitudinal studies are indicated.
Although this intensive mixed-methods approach and suite of studies will not always be feasible (because of time and resource constraints), progress towards improved pregnancy-related and perinatal mortality depends on reliable measurement of levels and determinants of pregnancy-related and perinatal deaths, and the effects of interventions. Not all evaluations can or should include the complete range of methods described here, but the dangers of not combining a sufficient minimum of complementary approaches need to be considered.
Overall, the mixed-methods approach used in this SCI evaluation was reasonably successful, and in this particular implementation a population of over 500 000 people were surveyed, resulting in a large database that was analysed, interpreted and written up, all within an overall 18-month period. Effectiveness and cost-effectiveness evaluations of safe motherhood strategies can be strategically important programme components, but reductions in pregnancy-related and perinatal mortality must remain as the ultimate target. Our approach and lessons learnt help may facilitate future evaluation work.
We are grateful to all the interviewers and logistic support staff who made the implementation of these methodologies possible in the field. This work was undertaken as part of an international research programme – Immpact, funded by the Bill & Melinda Gates Foundation, the UK Department for International Development, the European Commission and USAID. The funders have no responsibility for the information provided or views expressed in this paper. The views expressed herein are solely those of the authors.
Conflicts of interest
The authors have not declared any conflicts of interest.