Linking reductionist science and holistic policy using systematic reviews: unpacking environmental policy questions to construct an evidence-based framework

Authors


*Correspondence author: E-mail: a.s.pullin@bangor.ac.uk

Summary

1. There is a mismatch between broad holistic questions typically posed in policy formation and narrow reductionist questions that are susceptible to scientific method. This inhibits the two-way flow of information at the science-policy interface and weakens the impact of applied ecology on environmental policy.

2. We investigate the approaches to building policy in the health services as a model to help establish a framework in applied ecology and environmental management by which reductionist science can underpin decision making at the policy level.

3. A comparison of policy documents in the health and environmental sectors reveals many similarities in identifying approaches and specific interventions that might achieve policy objectives. The difference is that in the health services, information on the effectiveness of potential interventions is far more readily available through the collaborative process of systematic review.

4.Synthesis and applications. Decision makers are increasingly looking to produce policies that are shaped by evidence through evidence-based policy making. The approach that we outline here provides a framework for structuring systematic reviews to deliver the evidence on key policy issues in a way that will see a faster return and provide better use of the systematic review methodology in environmental management.

Introduction

The drive toward evidence-based policy making (e.g. Bullock, Stanley & Mountford 2001) requires that evidence is gathered on the effectiveness of alternative solutions to problems. The formation of policy may begin with the basic question ‘how do we achieve the desired policy outcome?’ Policy questions like this are often ‘big’ and ‘holistic’ in the sense of being inclusive of all the multiple steps, interventions and their interactions as well as demanding solutions adaptable to changing conditions. Consequently scientists do not see them as directly answerable or susceptible to experimentation and it is difficult to identify evidence that points to specific solutions. At the same time, policy formers may find potentially informative scientific papers or reports too reductionist and therefore lacking direct relevance to their questions as well as being (sometimes) inaccessible, time consuming and unrewarding to read. Consequently there often appears to be a mismatch between specific questions addressed by science (in the peer-reviewed literature) and broad issues addressed by policy (in strategies and implementation plans).

There is currently an effort to identify important policy-relevant questions that might be informed by scientific evidence in areas such as biodiversity conservation (e.g. Sutherland et al. 2006). Whilst the hope is that scientists will be interested in addressing some of these questions through original research or undertaking systematic reviews of the evidence, they may initially be discouraged by the generality of many. For example, of the 100 ecological questions of high policy relevance in the UK identified by Sutherland et al. (2006), question 41 ‘How can we manage microbial ecology to control invasive plant species?’ clearly falls into this category. Whilst the issue of invasive species and their control is clearly an important one for policy makers, the first part of the question is essentially just a general appeal for microbial ecologists to come up with some sort of solution to the problem. The role of microbial ecology in enabling control can, however, be expected to vary between species and habitats.

Clearly policy-relevant question setting is a generic problem at the science-policy interface and is not confined to applied ecology. In some fields, such as the health services, evidence-based frameworks have been established to provide evidence on the effectiveness of specific interventions for use in practice (e.g. Cochrane Collaboration; http://www.cochrane.org). More recently, following calls for a similar evidence-based approach to examine the effectiveness and impact of environmental management and conservation interventions (Pullin & Knight 2001; Sutherland et al. 2004), a Cochrane-style not-for-profit organization is in operation with over 80 systematic reviews either completed or in progress (see http://www.environmentalevidence.org). Despite this evidence of ‘proof of concept’ in terms of evaluating effectiveness of individual interventions, the question remains to what extent can these frameworks help to address big policy questions? In theory, policy questions could be ‘unpacked’ to reveal the scientific questions that underpin them, but the complexity underlying the development of environmental policy might be seen as inhibitory to this process, let alone gathering the evidence on individual components and ‘re-packing’ to construct an evidence-base. Nevertheless, Sutherland et al. (2006) recognized that many of the questions they identified would need to be disaggregated and then repacked before scientists could begin to address them.

Major differences in the science and practice of medicine and environmental management have been addressed by Pullin & Knight (2001). In this paper, we explore an approach to evidence-based policy-making used in the health services and examine its application to the environmental sector using specific case studies. We find that there is a clear link between specific ‘building blocks’ of scientific evidence and ‘big’ policy and that these links become clearer as consensus develops on the key evidence through systematic review and evidence-based practice. We also find potential for this to be a two-way process as the knowledge gaps, identified when constructing policy, feed back into research programmes.

Constructing policy frameworks

Beside primary research, the key methodology in evidence-based frameworks is systematic review (Pullin & Knight 2001; Pullin & Stewart 2006). The methodology seeks to synthesize all available good quality evidence on the effectiveness of an intervention. Whilst this has been shown to revolutionize effectiveness in medical practice (Stevens & Milne 1997), the problem in policy formation is to make the link from evidence supporting specific interventions to constructing broader policy that involves a whole suite of different interventions.

In the UK, the Department of Health has developed policy, in the form of National Service Frameworks, for tackling a number of key health problems. Each National Service Framework started with a big policy question, such as ‘How best to reduce premature (<75 years) mortality because of Coronary Heart Disease (CHD) in the UK population?’ CHD and stroke (collectively termed cardiovascular disease) are major causes of premature death in the UK. The Department of Health sought explicit application of the evidence base to ensure consistent and effective interventions to reduce CHD. As part of its guiding values and principles it states ‘CHD policies are to be based on the best available evidence; wherever possible, these will be well-conducted, updated systematic reviews of valid, relevant evidence’ (Department of Health 2000). The policy question was divided into a number of key approaches to provide potential solutions, principally primary prevention (preventing the development of ‘risk factors’ for CHD), secondary prevention (preventing the transition from risk to actual disease), treatment (for example, of heart attack) and tertiary prevention (preventing further events and minimizing the impact of disease on quality of life). Each of these was further broken down into specific possible interventions. It is at this level that evidence can be gathered in the form of systematic reviews of effectiveness so that each alternative intervention can be considered objectively. By using this approach the overall strategy for reducing mortality from CHD was built up into a hierarchy of interventions designed to operate at either population or patient level; from primary prevention through to treatment.

For example, smoking is well established as a major risk factor for cardiovascular disease, so reducing the prevalence of smoking in the population was key to the primary prevention element of the strategy. The effectiveness of different smoking cessation interventions has been measured in a number of systematic reviews (see the Cochrane Library at http://www.cochrane.org and http://www.medicine.ox.ac.uk/bandolier/booth/booths/cardiac.html for past and recent reviews) and on the basis of their findings (Raw, McNeill & West 1998), the CHD National Service Framework included local provision of smoking cessation services using nicotine replacement therapy.

At the other end of the scale, if the patient has already developed CHD, the National Service Framework requires provision of cardiac rehabilitation as a tertiary prevention measure, on the basis of systematic reviews which have demonstrated that this intervention can reduce the likelihood of dying from heart disease (Jolliffe et al. 2000). Other interventions that were included in the CHD National Service Framework on the basis of evidence from systematic reviews and meta-analyses were: increased use of cholesterol-lowering drugs (statins) for secondary prevention; making thrombolytic therapy (clot-busting drugs) routine emergency treatment for heart attack; and, increased use of aspirin for secondary prevention for cardiovascular disease patients.

The Department of Health has therefore invested in these interventions as a package (they have ‘repacked’ the question to provide a policy solution), in part because of weight of evidence as demonstrated through systematic review. The most recent progress report (Department of Health 2008) summarizes the impact of the CHD National Service Framework as follows: the headline target of reducing premature death from cardiovascular disease by 40% from baseline (1999) was met 5 years early, this equates to thousands fewer premature deaths per year; the adult smoking prevalence dropped from 28% in 2000 to 22% in 2006; the estimated number of lives saved through use of statins rose from 2900 in 2000 to up to 10 000; the percentage of heart attack victims given thrombolytic therapy within 60 min of a call for help increased from 24% in 2001 to 68% in 2007.

Behind these impressive outcomes lies a very significant improvement in the local and regional organization of cardiovascular disease services. Locally partnerships were developed between the primary care trusts (commissioners of the services), primary care services, hospital trusts (both providers of services) and local patient groups to drive forward and oversee the roll-out of the CHD national service framework (NSF). These ‘local implementation teams’ were not only required to use the evidence-based explicitly to develop their local CHD NSF action plans, but also, to ensure comprehensive monitoring and evaluation of implementation. Regional cardiac networks facilitated the translation of the evidence base into practice and performance managed the process. There have therefore been wider impacts relating to local organization, delivery and capability of cardiac services (Department of Health 2008). At the national level, the role of Cardiac Czar was created to champion this evidence-based approach and to drive forward progress.

In an even more demanding context, the ‘Roll Back Malaria’ campaign was launched in 1998 by the World Health Organization, the United Nations Children’s Fund, the United Nations Development Programme and the World Bank (http://www.rollbackmalaria.org). The policy goal was to halve the mortality caused by malaria across the globe by 2010. As with the CHD National Service Framework, the strategy includes a mix of prevention and treatment. An example of a specific intervention is the use of insecticide treated bed nets for prevention of mosquito bites. Trials of insecticide-treated nets in the 1980s and 1990s showed that they reduced deaths in young children by an average of 20%. Through systematic review, Lengeler (2004) subsequently demonstrated a consensus that such nets are highly effective. Subsequent evaluation of this element of the strategy has shown it to be failing, not because of the intervention, but because it is not being adequately implemented. A large percentage of children are not actually sleeping under bed nets, primarily because of their cost.

Hence policy, which started with the big question, has been disassembled and reconstructed from the bottom-up, using an evidence-based framework, to produce national or international guidelines that advocate specific interventions. Because of the strength of the health service’s evidence base the suite of available interventions can be utilized in an adaptive way, dependent on the state of the system (individual and population), the policy context and some form of cost-benefit analysis. For example, if a health authority covers a local population that is young, but has a high incidence of smoking, it might choose to reduce CHD incidence by allocating relatively more of its limited resources to primary prevention and smoking cessation activity. If the population is ageing, tertiary care and rehabilitation may get greater spending priority.

From an applied ecologist’s perspective, health care problems may seem straightforward. They are usually evident, and are dealt with, at the level of the individual human (or sometimes the population) whereas environmental problems can be evident at or above the species level in terms of changes in relative abundance of species or changes in species interactions. Despite this fundamental difference there are some equivalent questions in environmental policy such as ‘how best to reduce the impact of invasive species?’ or ‘how to reduce carbon emissions from agriculture and forestry?’ In the former case a UK Framework Strategy for action exists for all species not native to Great Britain (Defra 2008) but each of the invasive species is the equivalent of a human disease (a disease of the environment) and could have its own framework or could be grouped by similar species. In the Framework Strategy one could unpack the policy question to address the specific questions of effectiveness of potential approaches including primary prevention, control and eradication. The integration of evidence on effectiveness of interventions at different stages enables the most cost-effective policies to be developed. For example, for some species yet to invade it may be that strong legislation preventing arrival may be the best intervention. For species already established it may be that geographically focussed intensive control efforts may be most effective to minimize impact on agriculture or native species. In the current version of the invasive species Framework Strategy, each of these approaches is recognized but the next step of identifying the most effective interventions is absent.

As a next step, each approach, or potential solution, should be considered. The potential interventions associated with each approach could then be identified (including legislation). For example, ‘What is the relative effectiveness of voluntary agreements and legislation for preventing imports of invasive plants through the horticultural trade?’ might be a question relevant to a prevention approach and is susceptible to primary research and systematic review of effectiveness. Many of the first systematic reviews in environmental management have been relevant to the control approach (Kabat, Stewart & Pullin 2006;Tyler, Pullin & Stewart 2006; McLeod, Saunders & Kabat 2008; Roberts & Pullin 2008) with knowledge gaps being a common finding. More evidence-based Framework Strategies could be developed on a needs basis by governments, international organizations or private companies. But the evidence-base must first be characterized through systematic review.

In case of considering how to reduce carbon emissions from agriculture and forestry there is again a need to unpack the question to consider a range of possible solutions in different contexts (e.g. rice production and cattle ranching) and environments. As the recent biofuel debate has highlighted (Royal Society 2008) there is also a need to take a much more holistic approach in answering the question. While substituting biofuels produced from crops for petrol might be expected to reduce greenhouse gases, recent analyses (e.g. Searchinger et al. 2008) have shown that the interplay between prices and land use may have far reaching consequences beyond those originally envisaged. The consequence is that policies to encourage biofuels and reduce emissions may have the opposite impact to that which was intended.

In the USA, policy strategies have been considered for using agriculture and forest lands to reduce greenhouse gas emissions (e.g. Richards, Sampson & Brown 2006). Here again, options are set out to achieve the desired objective: conversion (e.g. non-forestlands to forests); preservation (e.g. increasing carbon in existing forests and agricultural soils); and innovation (e.g. growing biomass to be used for energy). In addition, policy instruments are set out for encouraging these activities, including government regulation, practice-based incentives and results-based incentives. In common with the invasive species Framework Strategy, Richards et al. (2006) have stopped short of evaluating and advocating specific interventions because the key assessments of effectiveness are absent. The thousands of systematic reviews that are available to underpin policy development in the health services (see http://www.cochrane.org) have, until recently, had no equivalent in environmental management.

How big policy questions are generated is neither the subject of this paper, nor do we suggest that an evidence-based approach guarantees that science will necessarily be taken into account in forming policy. However, science underpinning policy questions can be packaged through an evidence-based framework that ensures that policy is better informed by the science. Figure 1 shows a simple process that enables big policy questions to be addressed, first in terms of potential strategies that suggest generic solutions, and secondly, within each strategy, to identify specific interventions that either have been or could be evaluated in terms of effectiveness.

Figure 1.

 Diagram illustrating the relationship between high-level policy question, potential generic solutions and individual interventions for (a) a health services example where this process has actually been completed and implemented and (b) an environmental management example for which the process is only partially complete.

Guiding and evaluating large research programmes

Effective environmental management is a multidisciplinary activity involving actions from local practical management to global policy intervention. Consequently research into environmental management, and environmental change in particular, is increasingly funded through large interdisciplinary, policy-relevant programmes. Despite their ambitions these programmes have limited resources, and funding agencies must decide which research projects will most effectively contribute to the holistic programme objectives. Presumably funding allocation would be an easier decision if funders knew (or there was consensus on) which ‘building blocks’ of evidence are already in place and conversely where the important gaps in evidence lie. The factors inhibiting the flow of information between science and policy may therefore also confound the process of prioritizing research funding. Large environmental research programmes are increasingly oriented toward policy goals of wider society (e.g. UK environmental research funder’s ‘Living with Environmental Change’ programme, http://www.lwec.org.uk; Tyndall Centre for Climate Change Research, http://www.tyndall.ac.uk; Rural Economy and Land Use http://www.relu.ac.uk). Consequently the knowledge gaps identified in policy formation should also be the targets for research. The strategic employment of systematic reviews, perhaps through open calls as a pilot phase of a programme, might provide such a basis for funding decisions. This would engage the wider research community in the process and facilitate focus on the ultimate policy goals of the programme. Clearly, once knowledge gaps are identified there will be another process of prioritization on the grounds of urgency and likely costs and benefits of future research that may be more or less evidence-based; this is not addressed here. It should also be noted that the type of evidence available to inform policy (correlation vs. experimental and qualitative vs. quantitative) may change over time and that consequently policy will be informed by different forms of evidence as the science-policy interface evolves.

The construction of Review Groups to undertake multiple reviews in key areas might be encouraged through consortia funding. This is exactly what has been performed at the inception phase of the Ecosystem Services and Poverty Alleviation programme (http://www.nerc.ac.uk/research/programmes/espa). It is also essentially what the UK government Foresight programmes aim to achieve (http://www.foresight.gov.uk) in reviewing the evidence and horizon scanning on a range of issues such as land use futures and flood and coastal defence (Evans et al. 2004a,b; Thorne, Evans & Penning-Rowsell 2007). It seems logical that an objective way of making the case for research would be by reviewing the existing evidence, but within the context of the policy framework and by engaging with policy makers. This then also opens up the way to a dialogue between researchers and policy makers enabling the more effective delivery of policy through both an analytical and deliberative approach. However, undertaking a systematic review is often thought unworthy where there is a perceived lack of evidence because confirmation of this evidence gap is considered to be an unsatisfactory outcome (the review question does not get answered), but this overlooks a key role of systematic review in characterizing the evidence base and achieving consensus on what we know and do not know. Undertaking systematic reviews at the beginning of a research programme establishes a baseline of evidence to guide future research investment. Updating reviews at appropriate intervals (e.g. 3–5 years) enables measurement of progress in developing the evidence base and potentially provides an independent measure of scientific impact of the programme.

There are undoubtedly real barriers to engaging in evidence-based practice. From a researcher’s perspective, undertaking a systematic review can appear to be a substantial commitment, searching for, extracting and synthesizing all relevant information on a particular question, with little in the way of incentive or reward in comparison with primary research. Until recently, there has been no obvious source of funding or publication route within the environmental sector. The role of systematic reviews has not been recognized by environmental research funders and the track record of publications of systematic reviews or derived papers is short, providing little guarantee of recognized research output. From a policy perspective there may be a significant cultural barrier in the sense of policy formers being unfamiliar with formulating questions that can be addressed through an evidence-based framework or with being so actively involved in knowledge exchange throughout the review process. These barriers need to be broken down by demonstration of the process in recognized policy areas (i.e. environmental rather than medical) and by the concerted effort of environmental research funders to raise the profile of evidence-based practice as a collaborative process engaging both the scientific and policy communities.

Synthesis and conclusions

Good science, even when it is well packaged and accessible, does not guarantee good policy. The fact that policy questions are often ‘big’ and holistic in nature means that they are rarely tractable through individual research projects, but examples from other sectors clearly show that combining evidence from many systematic reviews can build an evidence base for policy making. With the benefit of an evidence-based framework, the critical step of deciding which knowledge gaps most urgently need filling, and what are their relative costs and benefits, becomes better informed and more objective. It may be the apparent limited value of a single systematic review (particularly one that shows a knowledge gap or poor evidence base) that discourages funding agencies from significant investment in this methodology. Focussed investment on building an evidence-based framework for key policy issues may see a faster return on investment and provide a better indicator of the longer term value of systematic review methodology in environmental management.

Acknowledgements

This work was funded through NERC grant NE/F009356/1 and Natural England contract FST20-84-037 to ASP.

Ancillary