Understanding the intervention and the context.
The SCI was a 5-year (2001–2005) integrated safe motherhood programme funded by the Bill & Melinda Gates Foundation and implemented by FCI in selected districts within Burkina Faso, Kenya and Tanzania. The overall goal was to contribute towards pregnancy-related mortality reduction by ensuring skilled care for every woman before, during and after childbirth. The project aimed to increase rates of skilled attendance by at least 10% in the project districts. Project activities focused on two main areas: improving the availability and quality of maternity care, and promoting increased utilisation of maternity services. It also aimed to strengthen obstetric care at the first referral level – the district hospital – to where women with more serious complications should be referred.
District-level interventions were complemented by national-level activities aimed at strengthening government policies and strategies related to maternal health, including the development of clear standards and protocols for obstetric care.
In Burkina Faso, the SCI was implemented in Ouargaye Health District. To improve the quality and availability of maternity care, FCI worked in the district hospital and, on a phased basis, in 13 of the then 19 health centres, while the remaining six health centres received support from other donors. Activities included infrastructure improvements, such as the provision of solar panels for electricity in the maternity departments and the health care providers’ houses, training of all maternity care providers in obstetric best practices and interpersonal communication, provision of essential maternity equipment and supplies; and strengthening of the referral system by donating an ambulance and radio-call system and by subsidizing cost-sharing for emergency transportation. FCI also worked to strengthen health service management, supervision and health management information systems, and introduced a problem-solving approach among facility managers and staff – Client-Oriented Performance-Efficiency for Maternal Health Services (EngenderHealth 2007) – to encourage a continuous quality improvement process among staff.
To complement supply side interventions, the project launched an intensive behaviour change and community mobilisation effort throughout the District to encourage health-seeking behaviour and build critical community support for skilled care. Activities included participatory theatre and songs around five key themes: use of antenatal care, birth preparedness, skilled care at childbirth, recognising (and responding) to emergencies and early postpartum care. All was planned in close collaboration with traditional chiefs to develop and implement community action plans. A network of 186 community outreach agents was established to educate communities on skilled care and a range of promotional and educational materials produced to support the ongoing behaviour-change activities in the communities.
After administrative clearance and preparatory work, the project started its supply side activities in late 2001 in what was referred to as ‘first generation health facilities’– five health centres and the district hospital. The next phase was a scale up in early 2003 to the ‘second generation health facilities’, an additional eight health centres in the District, thus totalling coverage to 14 facilities out of then 20 in the district. The selection and training of community outreach agents started in mid 2004. The community mobilisation started in early 2005 with the participatory theatre followed by the work with the traditional leaders.
Dealing with attribution and monitoring challenges.
The complexity of this evaluation thus arises from the complexity of the intervention (different components covering parts of the intervention district at different times) and the multiplicity of interventions in the same setting, posing a real problem for attribution of effects and internal validity. Aiming to conduct a real-life assessment of an existing strategy, we used a quasi-experimental design (Ovretveit 1998; Habicht et al. 1999), and a mixed-methods study covering the project implementation period (Figure 2) to assess changes in selected outcomes and process measures within the intervention and comparison districts. Although a prospective, randomised trial approach might have improved control of confounding and reduced sources of bias, it would not have been appropriate or possible in this real-world setting and would not necessarily have increased external validity (Habicht et al. 1999; Victora et al. 2004; Blackwood 2006). Here a robust trial-based design would have been impossible because of variations between multiple stake-holders, inherent dissimilarities between the Districts and the complexity of the SCI implementation.
In spite of challenges in assessing and tracking pregnancy-related and perinatal mortality (Stanton et al. 2001; Hill et al. 2006), ensuring progress towards MDGs 4 and 5 will require that these impact indicators remain targets of all safe motherhood strategies and evaluative research. However, the effectiveness of complex public health strategies cannot solely be judged on health outcome indicators (Victora et al. 2005) because the pathway to health outcomes (in this case to pregnancy-related deaths) is a continuum (Bryce et al. 2005). Table 2 summarises the mix of outcomes and process indicators used for this complex evaluation: pregnancy-related deaths, near-miss, perinatal mortality, skilled delivery, Caesarean sections, unmet obstetric needs, outcomes after pregnancy, costs borne by families and the health system for uptake of maternal care. Given the relatively small sample size of the population of study (500 000 for both intervention and comparison districts), the low rates of health care utilization, the desirability to assess change in the period of implementation, the need for a number of deaths to find statistically significant difference, we were only left with the option of a census, albeit limited in monitoring progress in the short run (Stanton et al. 2001). Each of the methods and tools used (Table 3) is briefly described below.
Table 2. Outcomes, indicators and assessment methods for the SCI evaluation. Unmet Obstetric Needs Network (2007)
|Type||Indicator||Examples of indicator||Assessment methods|
|Impact||Pregnancy-related mortality||Maternal mortality ratio Maternal mortality rate||Census, verbal autopsy|
|Perinatal mortality||Perinatal mortality rate||Census, verbal autopsy|
|Process measures||Severe morbidity access to CEmOC||Near miss rate, quality of care long term disability rate proportion of unmet obstetric needs||Prospective survey household survey retrospective survey|
|Performance||Performance of maternity unit||Cross sectional survey (functionality index)|
|Utilisation of maternal health services||Skilled attendance rate||Household survey|
|Costs||Cost||Direct costs indirect costs||Valuation of health and non-health inputs|
|Cost effectiveness analysis||Health system costs household costs Household revenue and expenditure||Cost effectiveness analysis, health system and societal perspectives|
|Qualitative||Barriers and facilitators to utilising skilled care||Financial and non-financial incentives Coping strategies for near-miss perceived quality of care||In-depth interviews focus groups Non-participative observations|
|Linkage between outcomes and process measures||Correlation||Associations of mortality and near-miss with CEmOC, skilled attendance, costs, etc.||Correlation analyses|
Table 3. Summary of tools and methods used for the SCI evaluation in Burkina Faso
|Survey type||Instrument/tool||Method||Targeted outcome|
|Community surveys||Census & socio-economic survey||Geo-referenced census (interviews with heads of households)||Mortality estimates demographic indicators wealth quintiles (poverty)|
|Perinatal survey||Interviews with women experiencing pregnancy between 2001 and 2005||Perinatal deaths population based Caesarean section rates Institutional delivery rates|
|Adult deaths survey||Interviews with heads of households||Maternal deaths|
|Short-term outcomes after pregnancy||Interviews with women experiencing pregnancy during past year||Physical, psychological, social consequences of pregnancy & anaemia|
|Household costs for maternal care||Interviews with women who had delivered within last 7 weeks||Estimates of costs for maternal care from individuals’ perspectives|
|Verbal autopsy||Interviews with household informants re adult female deaths||Maternal deaths|
|Facility surveys||Unmet obstetric need||Case extraction for women who underwent a maternal obstetric intervention for an acute maternal indication||Unmet obstetric needs|
|Near-miss*||Case extraction||Incidence and profile of life threatening complications|
|Maternal death records||Reports, registers, case extraction||Estimates of institutional maternal deaths|
|Facility costs||Costing||Estimates of costs for maternal care from health systems perspective|
|Time allocation||Self-administered by providers||Proportion of maternal care within general care|
|Performance||Interviews with health care providers & data extraction||Health centre functionality index|
|Incentive survey||Interviews with health care providers||Profiles of incentives for quality delivery care|
|Knowledge, attitudes, practice||Interviews with health care providers||Assessment of providers’ skills and knowledge|
|Qualitative surveys||Verbal autopsy||Interviews with verbal autopsy informants||Barriers to notification of community maternal deaths|
|Near-miss||Interviews with near-misses and families||Near-miss coping strategies|
|Perceived quality of care||Interviews observation (provider & user interaction)||Quality of care as perceived by users and providers|
A census of all households was conducted using a general pregnancy-related mortality module recommended for data collection on household deaths (Hill et al. 2001). Collection of such data are often restricted to the 12 or 24 months prior to interview, due to concerns regarding recall bias and the dissolution of households following an adult death. For the purposes of pregnancy-related mortality measurement in the SCI evaluation, it was necessary to extend this to a 5-year period, necessitating careful assessment of potential recall bias. The census questionnaire, administered to the head of household, was designed to collect household assets, socio-demographic characteristics of all household members (age, sex, occupation, education, marital status, etc.) in every household in both districts. The household assets were used to derive wealth quintiles using the method developed by Filmer and Pritchett (2001). The exact location of each household was recorded using the Global Positioning System to permit mapping of health outcomes. Appended to the main census questionnaire were an ‘adult death’ module designed to record details of deaths during the preceding 5 years between the ages of 12 and 49 years, including information on age at death, sex, pregnancy status at death, outcomes of pregnancy and foetal/newborn survival and a ‘perinatal death’ module, designed to record, for all women aged 12–49 years, the number of pregnancies, pregnancies lasting more than 6 months, place of delivery, delivery attendant and foetal/newborn survival during the preceding 5 years. Perinatal mortality rates and population-based Caesarean section rates were calculated using this module. It should be noted that the approach used to capture perinatal deaths was designed as an add-on to the existing census questionnaire, which was needed for other data collection purposes. Although there is extensive experience of collecting data on household deaths within a 1–2 year period via censuses, no references validating this method for capturing late pregnancy losses or deaths within the first week of life were found (as age at death is generally recorded in completed months or years in a census). Data on perinatal death in surveys of women of reproductive age are believed to suffer from misclassification of both vital status at birth and gestational age, and underreporting (Stanton et al. 2006). It is reasonable to assume that reporting via a brief interview with someone other than the person who experienced that pregnancy could lead to greater data deficiencies. Consequently, the questions in the perinatal module were administered to the women who carried the pregnancies and not the head of household, the usual respondent for the census. Nevertheless, perinatal mortality measured via this approach resulted in very low perinatal mortality rates even when restricted to the two years prior to interview (∼32 per 1000 live births).
Table 3 also describes the eligibility for the three remaining household survey modules (verbal autopsy, outcomes after pregnancy and household costs). Causes and profiles of deaths among women of reproductive age were characterised using InterVA-M, a new probabilistic model for interpreting community-based verbal autopsy data, as previously reported elsewhere (Byass et al. 2006; Fottrell et al. 2007). Even when well measured, pregnancy-related mortality represents only the ‘tip of the iceberg’ regarding women’s suffering as a result of pregnancy complications (Ronsmans & Graham 2006). To capture changes in physical, psychological, social and familial diseases, disabilities or discomfort that women may suffer as results of pregnancy, a specific ‘outcomes after pregnancy’ questionnaire was developed, pre-tested and piloted. The tool was administered to women aged 12–49 years who had an experience of pregnancy during the 12 months prior to interview. The questionnaire asked about indicators of women’s own socio-economic status, capacity to undertake activities of daily life, psychological status, social and familial relation status, and discomfort as a result of pregnancy and childbirth. Finally, a finger-prick blood sample was taken for on-site estimation of anaemia at the community level among a random sample (five per enumeration zone) of eligible women. Any cases of dangerous levels of anaemia were immediately followed-up.
The approach taken to conduct a cost-effectiveness analysis in the framework of this evaluation was to estimate costs borne by women and their families as well as costs incurred by the health system for maternal care, to derive costs per specific outcomes from health systems and societal perspectives. At the household level, data were collected on delivery costs, delivery time costs, and travel time costs for delivery, financing delivery costs for women and their families, and household expenditure over the last month as a proxy for income. Ability to pay, households’ wealth and expenditure were used to check for distributional issues and whether or not households frequently incurred catastrophic health expenditure for maternal care.
A functional health centre combined with well trained and motivated staff, appropriate equipment and standard guidelines and monitoring procedures is pre-requisite for quality care and could greatly affect the incidence of pregnancy-related mortality and severe morbidity (Ronsmans & Graham 2006). The main purpose of the health facility survey within the evaluation was to investigate the quality of maternity care and the trends of selected process indicators (institutional delivery, population based Caesarean section rates and unmet obstetric needs) and the extent to which the latter are correlated with outcome measures, such as pregnancy-related deaths or perinatal deaths.
Additionally a 6-month prospective study of narrowly averted pregnancy-related deaths (near-misses), a survey of institutional reporting of pregnancy-related deaths, a health centre performance survey, an incentive survey, and a facility costing survey were conducted in both of the district hospitals and all 43 health centres. The near-miss study was carried out at the two district hospitals to describe incidence and causes of near-misses and other severe obstetric complications and to identify contextual determinants which could contribute to mortality reduction strategies. The survey of institutional reporting of pregnancy-related deaths was developed to compare facility and community levels of pregnancy-related mortality. The health centre performance survey was conducted to document levels of functionality regarding referrals, transport, communications, infrastructure, equipment, supervision, staffing and drug supply in relation to maternity care. A health service functionality index was subsequently constructed based on key variables such as staff complement, personnel knowledge, drug supply, laboratory facilities, availability of water, infrastructure, vehicles, etc. using principal component analysis. The incentive survey was conducted to document motivating or de-motivating financial as well as non-financial factors that influenced staff performance.
A better understanding of health-seeking behaviour and circumstances typically surrounding pregnancy-related deaths and severe complications is necessary for making quality delivery care available as widely as possible. The qualitative component of the SCI evaluation was designed to assess barriers and facilitators to the notification of pregnancy-related deaths in communities, health seeking behaviour around the deaths of female adults, near-miss survival coping strategies and perceived quality of maternity care from users and providers. The approaches used were individual interviews with users (43) and providers (9), focus group discussions with communities (8), and non-participative observations (6). Interview guides were developed, pre-tested and piloted prior to the main survey. NVIVO software was used for data management and analysis (QSR International, 2007).
Not having a clear intervention to non-intervention distinction between the two Districts made the conduct of cost-effectiveness analysis more difficult. Our approach at facility level was to cost maternity health services in all health facilities, to identify principal cost categories, to conduct sensitivity analyses on major cost categories, and to derive costs estimates per outcome (deaths, complications) or per institutional delivery. The principal cost categories were derived from the WHO guidelines for costing of health services (WHO 1999). The volume of maternal care work by health workers was estimated using a time allocation sheet and costs of capital items (buildings, infrastructure and vehicles) were annualised. The allocation of total costs to maternal care was subsequently derived from all cost categories other than health workers.
Figure 3 summarises the organisation put into place to carry out the SCI evaluation. Establishing good working relationships with stakeholders is a pre-requisite for large population surveys; this was facilitated by the participatory process used for setting the research agenda (Madi et al. 2007). We designed and implemented an intensive awareness strategy within communities, the local administration, the police, village chiefs and the health system prior to launching the evaluation. This was necessary so as to avoid confusion or conflict with health officials, to ensure consistent messages on the study objectives, to ensure security of people and materials and to ensure acceptance and support from the communities.
The data collection personnel were organised in three teams for the household surveys, the facility surveys and the qualitative studies. For the household surveys, each district’s team was coordinated by a demographer, assisted by a senior field work supervisor, who oversaw all administrative and technical issues related to the survey. There were a total of 209 enumeration zones (geographically delimited areas of about 1000 people), in Ouargaye and 304 in Diapaga. An average of 20 to 25 enumeration zones was assigned to a team of seven to eight interviewers led by one supervisor. Each district established a data management centre run by a data manager, assisted by four data technicians, for downloading, merging, transferring and storing data directly captured in the field using hand-held computers (PDAs), onto laptops (Byass et al. 2008). To facilitate transportation of the research team and materials, to ensure an energy source for recharging PDAs where mains electricity was unavailable and for security reasons, cars were hired for the district coordinators, data managers, data technicians and field work supervisors. For the facility surveys, one team (comprising a public health specialist, two medical officers, a midwife and a data technician) consecutively covered the district hospitals and all health centres in both districts. The qualitative studies in both districts were also carried out by a single team of a senior social scientist, two junior social scientists and six research assistants.
The overall Immpact research proposal was approved by the Ministry of Health National Health Research Ethics Committee (Ouagadougou, Burkina Faso). The specific Evaluation and Evidence Research Group protocol was approved by Centre MURAZ (Bobo-Dioulasso, Burkina Faso) Institutional Review Board. Administrative authorizations were obtained at all level of the administrative chain (Ministry of Health, Region Governorates, Regional Directorates of Health, National and Regional Hospital Directorates, Province High Commissioners, District Health Management Teams, Heads of Clinical Services in Hospitals and village community leaders).