Undertaking a complex evaluation of safe motherhood in rural Burkina Faso
Corresponding Author Prof. Wendy J. Graham, Immpact, Health Sciences Building, University of Aberdeen, Foresterhill, Aberdeen AB25 2ZD, Scotland, UK. Tel.: +44 1224 555850; Fax: +44 1224 555704; E-mail: firstname.lastname@example.org
Evaluations of composite health interventions, such as those attempting to make motherhood safer, are by definition complex, but nevertheless regarded as essential to informing progress in global health. This paper introduces a series of reports which set out the basis of Family Care International’s Skilled Care Initiative in rural Burkina Faso, go on to describe strategies and methods for evaluating it, and present evaluation findings in terms of pregnancy outcomes, utilisation and effects of the intervention and economics. Although there were encouraging findings, no ‘magic bullets’ emerged from these studies, illustrating the reality that sustained and increasing resources are needed to achieve safe motherhood for all. There is no cheap or short-cut solution.
There are many discrete interventions which can help to prevent the tragedy of maternal death. These differ enormously and in many respects: their modes of working, resource requirements, timing, consequences and effectiveness. These interventions, however, also share a common inadequacy: none alone can achieve a significant impact on maternal mortality at a population level. By 2015, the target of Millennium Development Goal 5 is to achieve a 75% reduction in maternal mortality. History provides examples of such dramatic progress (De Brouwere et al. 1998) and also clear indications that composites of interventions are needed, delivered through functioning health systems. These major composite intervention strategies are necessarily challenging to design and implement, and to robustly evaluate for effectiveness and cost-effectiveness (Milne et al. 2004).
The importance of evaluating all major health initiatives is now undisputed. Indeed, it is advocated that around 10% of the budget for such initiatives is devoted to evaluation (Savedoff et al. 2006). Most interventions for composite health problems require complex evaluations to capture the multiple inputs, processes, outputs and outcomes as well as the context of implementation. Such complexity to an assessment is a natural reflection of the real world and not necessarily a bad thing. There are, however, risks arising from the sheer volume and layering of information from complex evaluations. Inconclusive findings may, for example, be misinterpreted as evidence that the health problem is intractable or the interventions ineffective or undeliverable. Decision-makers who could have used the evidence may be unwilling or unable to navigate the labyrinths of conclusions, and the value-for-money of the evaluation may rightly be called into question. These negative risks can be reduced by taking the extra step of distilling the mass of findings into policy-relevant messages (Campbell Collaboration, 2005). This distillation process is necessarily simplifying, and should reflect a dialogue between the generators of the evidence and those who may act upon it (Bryce et al. 2005). In this Supplement, the experience and findings of a complex evaluation in Burkina Faso are reported. Safe motherhood was the complex problem being targeted, and the intervention strategy focused on skilled attendance at delivery (Graham et al. 2008).
Complex evaluations are, by definition, major undertakings, from conception through to translation and communication. The experience reported here reflects a study period of over 4 years. More than half this time was devoted to planning and set-up: clarifying the key questions, ensuring ownership of the evaluation by Government decision-makers, and forging relations with the implementing partners – Family Care International and the study district health authorities. This distribution of time is both typical and essential (Pawson & Tilley 1997), but also invariably results in the data capture phase being highly intensive and tightly time-bound, with limited opportunity for interim analysis. In Burkina Faso, this was the experience. The vast majority of the data reported in this Supplement was gathered from February to May 2006, according to the study design and methods described by Hounton et al. (2008a). Such a short period for capture of population-based data was made possible by the intensive efforts of the fieldwork teams and by the use of Personal Digital Assistants (PDA), which streamlined considerably the data management processes. Byass et al. (2008) report the experience of using this technology in Burkina Faso to gather interview data from over 500 000 people in about 16 weeks. The primary health outcome captured during these visits was maternal deaths in the last 5 years, and the profile of mortality for the study districts is reported by Bell et al. (2008). To complement the findings on death, data were also gathered on the sequelae for women experiencing serious complications or stillbirths, including impact on quality of life and health status. The skilled attendance intervention sought specifically to increase uptake of care by women at the time of delivery, in others words to increase demand, and also addressed supply-side issues by upgrading the functionality of health centres and the district hospital. Process measurement was thus an essential component of the evaluation, and these findings are reported by Hounton et al. (2008b). The three remaining papers in the Supplement weave together the conclusions in terms of the effectiveness (Hounton et al. 2008c) and the costs (Newlands et al. 2008) of the intervention strategy, and the programmatic implications (Meda et al. 2008). So what emerges from the translation of the rich detail in the individual papers?
‘Seeing the wood for the trees’
To help grapple with the wealth of findings from complex evaluations, it can be useful to separate results into three main categories: confirmatory, contradictory and new. Each of these has different values and implications. Given the resources involved in complex evaluations, it is often presumed that completely new knowledge will emerge – answers to questions, fresh insights and recommendations for change. This desire for ‘new’ findings can lead to unrealistic expectations of public health interventions (Pawson & Tilley 1997), as well as the neglect of those results which support the existing knowledge base or indeed challenge it. Such confirmatory or contradictory findings can be hard to publish, and this publication bias adds further to the scramble to find something new. In the complex evaluation reported in this Supplement, most of the results are confirmatory, some are contradictory, and – as is typical – fewer still are new. Together, they help to build an understanding of the skilled attendance complex intervention, and importantly the context in which it was implemented and evaluated. Here, we give a few illustrative examples of confirmatory, contradictory and new findings.
The process indicators reported (Hounton et al. 2008b,c; Newlands et al. 2008) provide confirmation of the barriers and facilitators of uptake of delivery care by women in this population. Some of the relationships were straightforward, such as the negative association between uptake and distance to health centres, and others reflect a more complex picture, such as the differential influence of fees. What is clear is that marked increases in the proportion of deliveries in health facilities with locally-defined skilled attendants can be shown over comparatively short periods, so confirming this proportion as a sensitive monitoring indicator. The indicator also performs as expected with regard to poverty quintiles, with the highest uptake in the least poor. A contradictory finding, however, is the relationship seen between poverty quintiles and maternal mortality, particularly, in the comparison district (Bell et al. 2008). Other primary studies (Ronsmans et al. 2007) and secondary analysis of international datasets (Graham et al. 2004) suggest a clear poverty gradient in the risk of maternal deaths. Whereas confirmatory findings offer reassurance of the validity of the data, contradictory results prompt the question of whether they are artefacts of measurement or genuine challenges to expected relationships, prompting further analysis and interpretation as well as additional studies. In terms of examples of new findings, Bell et al. (2008) reveal how levels of maternal mortality vary considerably sub-nationally and between comparatively close geographic areas – variability previously undocumented for Burkina Faso. Similarly, poor pregnancy outcomes are shown here to impinge significantly on the quality of women’s lives. Finally, a new result of a somewhat different kind is the feasibility of using PDA technology in a remote and challenging field setting (Byass et al. 2008).
What does it all mean – for Burkina Faso?
Complex evaluations tell a story. The story here includes good news: communities are responsive to concerted efforts to raise awareness of the importance of delivering in an institution from which referral to hospital should be easier in the event of an emergency. Women respond in numbers which produce significant upward trends in the target indicator. For Burkina Faso, the main strategic objective of the 2004–2007 national safe motherhood programme is to reach 50% of deliveries in institutions (Direction de la Santé de la Famille 2003). The evaluation reported here shows this was achieved in 2005 for the intervention district of Ouargaye. It would thus seem reasonable to expect that in parts of the country with similar geographic and health service parameters, including community mobilization, the Government target of 50% may be achievable. The generalization to a national level requires greater caution. The lowest uptake seen in Ouargaye was 22% and among women living more than 15 km from the nearest health centre, which suggests the target is less likely to be met in areas even more remote from facilities.
The good news, however, is not the whole story. The increased demand for delivery care was not uniform and marked differentials remain. Uptake declined with increases in distance from health centres and in poverty levels, as measured by asset ownership. Of course, these two parameters are related: the poorest communities are often the most remote and travel costs – in time and money – an even bigger deterrent. Distance and poverty are thus limiting factors for achieving equitable skilled attendance at delivery and ultimately universal access. This focus of low uptake does, however, point to a potential way forward through geographical targeting of incentives. In Nepal, for example, a country with total expenditure on health as a percentage of GDP similar to Burkina Faso (World Health Organization 2005), the Government has introduced a voucher scheme to pay women to come for delivery care, with those coming furthest tending to be the poorest and getting paid more (Borghi et al. 2006).
An increased uptake of skilled attendance at delivery will not impact on health outcomes in the mother or baby unless the quality of care received and the availability of referral is adequate. This is intuitively obvious. Thus, the supply side of the health system must be boosted in unison and in proportion to the demand creation (MacDonagh 2005). This was the aim of the Skilled Care Initiative (SCI) intervention as described later (Graham et al. 2008; Hounton et al. 2008a). Direct and indirect markers of met need for life-saving interventions, however, suggest that the intensity and duration of inputs were limited. Population levels of Caesarean sections remained incredibly low at 0.4% and maternal mortality showed no evidence of association with the SCI intervention. The puzzling finding here is the lack of a significant poverty gradient in either Caesarean section or in maternal mortality. Although asset-ownership could be challenged as a basis for capturing poverty (Morris et al. 2000), the presence of the predictable relationship with skilled attendance uptake suggests this is not the obvious problem here. An alternative possible explanation relates to quality of care and referral for emergency obstetric intervention. Delays in detection of complications, in appropriate first-line management, in transfer and in receiving life-saving interventions will affect survival prospects of all women, regardless of their poverty status. Financial barriers to paying for life-saving intervention will be an added obstacle for the poorest women, hence the introduction of cost-sharing mechanisms in some parts of the country, including Diapaga District in 2001 and Ouargaye in 2004. Improved access to life-saving interventions at first level referral facilities in Burkina Faso clearly remains a priority for all women. However, an additional recommendation would be the phased introduction of fee exemptions and transport for women from the poorest areas, initially for emergency care, and subsequently backed-up by incentives to attend for normal deliveries at health centres. A detectable impact on maternal and perinatal mortality could reasonably be expected from such an enhanced demand-side strategy for skilled attendance, when combined with community mobilization (Moore et al. 2002) and with a strengthened supply-side. The complex evaluation undertaken in the two districts of Burkina Faso provides information not only to aid the design of such an enhancement, but also as a strong baseline for its subsequent monitoring and impact evaluation. Interestingly, the Government of Burkina Faso in 2006 introduced a policy for reducing the costs of Caesarean sections and normal deliveries by 80%; it is crucial that the effects of this approach to reducing financial barriers are robustly assessed.
What does it all mean – for safe motherhood?
After 20 years of the international safe motherhood initiative, the findings from Burkina Faso have heightened and wider significance. Skilled attendance at delivery is an intuitively obvious priority since the intrapartum period is the time of greatest risk to mother and baby (Ronsmans & Graham 2006). The expectation from interventions designed to increase coverage of deliveries with skilled attendance is that of ‘progress’, with reduced maternal mortality being one marker. The complex evaluation in Burkina Faso shows this to be a complex relationship. Incremental gains in coverage do not necessarily translate into reduced maternal mortality. This is not bad news about skilled attendance. Rather it seeks to remind us that the quality of ‘skills’ is crucial, that providers alone are necessary but not sufficient, and that whilst emergency obstetric care is assumed to be integral to skilled attendance, adequate resources – of trained staff, equipment and operating facilities – are often not provided to make this a reality (Bullough et al. 2005). More cases of maternal death than of Caesarean section provides clear evidence of unmet need (De Brouwere et al. 2002) for emergency care in this south-eastern part of Burkina Faso. Whilst the configuration of some aspects of delivery care will clearly need to vary setting to setting, since no one-size-fits-all, the requirement for prompt access to emergency intervention is universal (Koblinsky et al. 2006). In Burkina Faso, auxiliary midwives will remain the principal delivery attendants for the foreseeable future, in view of the acute shortage of professional midwives, and with health centres, the facilities for deliveries closest to women. This configuration has parallels with the health centre intrapartum care strategy recently suggested as optimal from research evidence (Campbell & Graham 2006). Such a strategy encompasses emergency care and demand-side considerations, including community mobilization. In Burkina Faso, financial barriers are a major deterrent to uptake of delivery care and coincide with distance obstacles, emphasizing the need to consider geographical targeting of, for example, transport interventions or incentives to health workers for remote postings. Such pointers to enhancements are legitimate and desired outcomes of complex evaluations which are conducted independently of and alongside intervention strategies. These are incremental gains in knowledge rather than breakthroughs or magic bullets, and emphasize the reality of the sustained resource inputs needed to achieve safe motherhood for all. There is no cheap or short-cut solution.
This work was undertaken as part of an international research programme – Immpact, funded by the Bill & Melinda Gates Foundation, the UK Department for International Development, the European Commission and USAID. The funders have no responsibility for the information provided or views expressed in this paper. The views expressed herein are solely those of the authors.
Conflicts of interest
The authors have declared no conflicts of interst.