• Open Access

The effectiveness and evaluation of conservation planning

Authors

  • Madeleine C. Bottrill,

    1.  School of Biological Sciences, The University of Queensland, St Lucia, QLD, Australia
    2.  Conservation International, 2011 Crystal Drive, Suite 500, Arlington, VA, USA
    Search for more papers by this author
  • Robert L. Pressey

    1.  Australian Research Council Centre of Excellence for Coral Reef Studies, James Cook University, Townsville, QLD, Australia
    Search for more papers by this author

  • Editor 
    Kendra McSweeney

Madeleine C. Bottrill, Conservation International, 2011 Crystal Drive, Suite 500, Arlington, VA 22202, USA. Tel: +1-310-500-0081. E-mail: m.bottrill@conservation.org

Abstract

Investment of time and resources in conservation planning has grown exponentially in recent years; yet there has been limited evaluation of the benefits and costs of investing in planning exercises. Without evaluation, investments in planning are not accountable, decisions are not defensible, and learning from past experiences is limited. Bringing together information from published literature, planning documents, and new qualitative data from interviews with planners, we describe an evaluation framework for conservation planning to more fully address the potential range of outcomes, categorized across five types of capital: natural, financial, social, human, and institutional. We review the extent to which evidence supports these types of outcomes, and finish by considering the conceptual, operational, organizational, and policy implications of improved evaluation in planning. If conservation planning seeks to be effective, adaptive, and informative, then systems of evaluation must be considered from the outset of planning processes to support learning and improvement of outcomes.

Introduction

There is a lack of evidence-based knowledge on the effectiveness of different conservation interventions, and limited monitoring and evaluation to generate better knowledge (Pullin & Knight 2001; Sutherland et al. 2004; Ferraro & Pattanayak 2006). Without evidence, justification of investment in different interventions is thought to be guided by experience at best (Cook et al. 2010) and anecdote and received wisdom at worst (Sutherland et al. 2004). To date, there has been no comprehensive reporting or evidence to demonstrate the costs and benefits of investing in conservation planning. Program evaluation, a systematic process for judging effectiveness of programs (Patton 2008), could provide a conceptual framework and a set of tools to enhance our knowledge of the value of conservation planning. In this review, we make the case for explicit program evaluation of regional-scale planning exercises—those applied across domains defined by ecoregions, catchments, or jurisdictions with extents in the order of 102 to 105 km2. Our review has three specific aims: (1) to outline the benefits and challenges of more explicit and comprehensive evaluation of conservation planning; (2) to identify the range of indicators available to evaluate the outcomes of conservation planning; and (3) to explore conceptual, operational, organizational, and policy implications of evaluating planning exercises in the future. We hope this review highlights the importance of evaluation in promoting defensible conservation planning processes, adds realism to expectations about planning and, ultimately, helps to improve conservation outcomes.

Over the last two decades, the application of conservation planning (Margules & Pressey 2000) has expanded rapidly. It has influenced conservation priorities by international organizations (Groves et al. 2002; Sanderson et al. 2002), guided policy decisions by government agencies (Airame et al. 2003; Pressey et al. 2009), and resulted in hundreds of publications in the academic literature. In particular, considerable advances have been made in producing conservation plans—maps of priority conservation areas—with hundreds of plans developed around the world at a variety of scales. However, despite several prominent success stories of planning leading to real conservation gains (Finkel 1998; Airame et al. 2003; Fernandes et al. 2005), there has been limited empirical evidence to demonstrate actual outcomes (Knight et al. 2008). Without more comprehensive evaluation of the tangible outcomes of conservation planning, investments in planning are not accountable, decisions are not defensible, and learning from past experiences is limited. This is all the more important because each of the hundreds of planning exercises undertaken entails substantial costs (Table 1).

Table 1.  Costs of conservation planning (all estimates exclude costs of applying actions)
Planning processLead organizationCost estimate (US$)DurationArea (km2)SourceDescription
Cape Action Plan for the Environment Project, South AfricaSouth African National Botanical Institute (SANBI)ca. 1 million1998–200090,000 Cowling & Pressey 2003 Project comprised four major steps: (1) situation assessment (including conservation planning component); (2) development of strategy to involve stakeholders and institutions; (3) action plan formulation, and (4) fundraising.
Great Barrier Reef Marine Park rezoning planGreat Barrier Reef (GBR) Marine Park Authorityca. 6–8 million1999–2004344,400 Osmond et al. 2010 Costs covered staff salaries, staff time, and staff operating costs for the new zoning plan for the entire GBR spread over a 5-year period. The Australian government also funded a structural adjustment package (∼$100 million) designed to assist fishers, fishery related businesses, employees, and communities affected by the rezoning.
Adirondack State Park Landscape, USAWildlife Conservation Society† Monetary value unavailable. Costs reported in relation to staff time (37 person-weeks)2000–200319,700 Didier et al. 2009 Stages of planning process were measured in relation to staff time (person-weeks). Most time devoted to selection of focal species (14 weeks); mapping of biological landscape (8 weeks); and mapping of human landscape (8 weeks)
Maputaland Transnational Conservation Plan, Mozambique/South Africa/SwazilandUniversity of Kent (DICE)ca. 215,1432003–200517,000 Smith & Leader-Williams 2006 Costs covered convening of training workshops, staff time, development of GIS databases, production of decision support system (CLUZ), and design of tri-national conservation plan through participatory workshops with key regional stakeholders followed by publicity and presentation of results.
Kimbe Bay Conservation Action Planning, Papua New GuineaThe Nature Conservancyca. 400,0002004–200713,000 Green et al. 2009 Total cost for the scientific design process in Kimbe Bay (excluding community engagement and implementation) was divided principally between scientific research (54%), staff time (35%) and workshops (10%).
California Central Coast Marine Protected Areas Network, USACalifornia Department of Fish and Gameca. 2.5 million2005–20073,000 Raab 2006 Expenditures included: (1) executive/general administration and project management (25%); (2) facilitation and outreach (18%); (3) Department of Fish and Game lead staff and costs (18%); and (4) data preparation and analysis (12%).

For the purpose of this review, we define conservation planning broadly as the design of conservation areas. Conservation areas can include IUCN protected area categories I–VI, as well as places where diverse management arrangements, such as zoning, offsets, easements, invasive species control, and restoration are applied. Most conservation planning today can be termed “systematic” (Margules & Pressey 2000), involving explicit, quantitative objectives and leading to systems of potential conservation areas with emergent properties related to complementarity and connectivity (Margules & Pressey 2000). The lack of evidence for the effectiveness of planning applies to systematic methods and to other approaches such as ranking procedures (Smith & Theberge 1987) and expert-based prioritizations (Cowling et al. 2003). We focus here on whether designing conservation areas leads to actual conservation outcomes. Specifically, therefore, we refer here to evaluation of conservation assessment (Knight et al. 2006a), excluding the evaluation of stages involving implementation of actions or subsequent management of conservation areas (stages 10 and 11 of Pressey & Bottrill 2009) covered elsewhere (Margoluis & Salafsky 1998; Ferraro & Pattanayak 2006; Hockings et al. 2006).

In the sections that follow, we discuss the rationale for, and possible approaches to, measuring the effectiveness of conservation planning. We draw extensively on literature, but also present new qualitative data on planning outcomes gathered from semistructured interviews with experts involved in developing and/or implementing planning exercises.

Qualitative data on effectiveness of planning

The qualitative data collected in semistructured interviews were intended to fill large gaps in the literature concerning perceptions of the effectiveness of planning. Interviewees were selected using snowball sampling, by which selected interviewees identify other appropriate participants (Lewis-Beck et al. 2004). Among our participants, we interviewed people involved in the development of plans (“planners,”n= 15) and people responsible for updating plans, representing stakeholders, or providing technical support in applying actions arising from conservation plans (“implementers,”n= 13). We aimed for a representative sample of people working across socioeconomic contexts and environmental realms. More details on the affiliations of participants and the interview instrument are described in the Supporting Information. The interview format followed an open framework of themed questions that were conversational and flexible (Mason 2002). Responses were analyzed using a content analysis of interview transcripts to draw out trends and frequency of responses. The interviews revealed perceptions of both positive and negative outcomes emerging from planning exercises by those intimately involved in their development and application.

Evaluation and conservation planning

Evaluation is a systematic method for collecting, analyzing, and assessing information on the effectiveness of projects and policies in relation to stated goals (Patton 2008) and takes multiple forms. The most important basic distinction in evaluation is between summative and formative approaches (Rossi et al. 2004). Formative approaches, including process evaluation and needs assessment, strengthen or improve the program being evaluated. Summative approaches, including outcome and impact evaluations, examine effects or outcomes of programs. We are primarily concerned here with the application of summative evaluation approaches. Table 2 describes definitions of key terms used in evaluation referred to in this review.

Table 2.  Key evaluation terms in the context of conservation planning
TermDefinitionExample
PurposeThe intended aim(s) of conservation planningIdentification of priority areas for management
GoalStatements about visions of what planning processes should achieve in a specific time frame, influenced by the purpose of conservation planningTo conserve a representative sample of a region's biodiversity
InputResources, including time, data or money, invested in developing the outputs and outcomes of the planning processSpatial data on land classes across a planning region
OutputProducts produced during the planning process that contribute to outcomes of conservation planningMap of potential conservation areas that meet quantitative objectives for species representation
OutcomePositive or negative changes to forms of natural, financial, social, human, or institutional capital in the region where the planning process occurs, and perhaps also in other regionsChanges in existing legislation to protect new areas identified by a conservation plan (institutional outcome), or improved coordination of conservation activities between NGOs working across the planning region (social outcome)
ImpactThe overall contribution of outcomes to the achievement of goals. Impact might be determined using counterfactual analysis, or the assessment of outcomes in the absence of conservation intervention.Avoided loss of native forest, measured in hectares, attributable to the outcomes of planning (hectares of forest that would have been lost in the absence of planning minus actual hectares lost)
IndicatorVariables used to assess, qualitatively or quantitatively, changes in or occurrences of outputs, outcomes or impactsNew skills acquired by staff following a training workshop, demonstrating increased human capital in the planning team

Over the last decade, evaluation in the conservation sector has developed to track performance of conservation interventions and policies, with approaches including results-based management and outcomes monitoring (Salafsky et al. 2002; Kapos et al. 2008). There are still, however, substantial gaps in knowledge of the effectiveness of many conservation interventions and of the advantages and limitations of different interventions. In this section, we discuss the potential merits of evaluation for conservation planning and review the extent to which evaluation is already applied.

Evaluation has multiple benefits for developers, managers and funders of planning processes (Rossi et al. 2004). First, and most importantly, evaluation can assess whether planning has met its goals and produced intended outcomes, or identify unintended consequences, positive or negative. Second, evaluation provides accountability by measuring cost-effectiveness. Third, evaluation can provide insights into which parts of the planning process contribute most to effective outcomes. Finally, evaluation can cut through and resolve data-free debates about the relative merits of different approaches to planning. Overall, evaluation can enhance institutional learning (Knight et al. 2006a), inform choices between different approaches, and build greater confidence among funders.

Existing approaches to evaluation of conservation planning

Formal evaluation of conservation planning has not been well documented and there has been limited reflection on specific exercises. Possible reasons for this lack of attention include: a tendency throughout the conservation sector to promote successes and bury failures (Redford & Taber 2000); limited staff or financial resources allocated to evaluation; a failure to track outcomes from the outset of planning processes; the protracted nature of implementation of plans; and poor connections between those responsible for developing and implementing plans (Knight et al. 2011).

Some lessons for the evaluation of conservation planning come from evaluation in the broad areas of natural resource management and land use policy and planning. Program evaluation of the Albemarle-Pamlico Estuarine Study (Korfmacher 1998) observed positive short-term outcomes such as knowledge transfer between managers and greater integration of scientific information. However, this study also cautioned that expectations about planning outcomes should be managed because demonstrating achievement of long-term objectives can be difficult, (Korfmacher 1998). Through interviews with municipal planners, Stokes (2009) found that greater public awareness and education were required to integrate biodiversity needs into local development strategies. Responses by different stakeholders engaged in coastal and marine coastal resource management emphasized the importance of fairness and legitimacy in public consultations (Dalton 2006).

For conservation planning specifically, some recent studies have documented experiences, compared approaches, assessed specialist tools, and investigated specific outcomes. Knight et al. (2008) assessed the extent to which systematic assessments have led to implemented actions. Other planners have reflected on personal or organizational experiences in particular planning regions (Medley 2004; Smith et al. 2006; Green et al. 2009), provided perspectives on planning within environmental realms (Leslie 2005; Osmond et al. 2010), or reviewed specific approaches (Didier et al. 2009; Henson et al. 2009; Morrison et al. 2009). While these studies provide useful information, they are largely narrative, context-specific, and narrow in scope and purpose. Collectively, these studies do not provide comprehensive, quantitative methods for measuring the effectiveness of planning.

The few assessments of planning exercises have tended to focus on quantifiable outputs (e.g., number of hectares reserved, maps produced, and development of software, Knight et al. 2006b) rather than indicators that directly demonstrate changes in ecological or social systems as a result of planning (Figure 1). Indicators of outcomes have focused primarily on ecological criteria (e.g., percentage of species populations in reserve systems, Klein et al. 2008). Less explored are the financial, social, and institutional effects of planning such as conflict resolution, mainstreaming of conservation thinking into public and private sectors, and leverage of funding. Furthermore, we are not aware that any study has reported impact, or what changes to biodiversity might be directly attributed to planning. Given protected areas are concentrated in areas of least value for extractive uses (Scott et al. 2001; Joppa & Pfaff 2009), even outcomes such as formal protection do not necessarily lead to net benefits for biodiversity and other natural values.

Figure 1.

Results chain illustrating linkages between processes, outputs, outcomes, and impacts of social and natural aspects of conservation planning. Processes are specific events or series of decisions and actions that deliver outputs and outcomes. Outputs are products and knowledge created during planning exercises. Outcomes are the short- and medium-term effects of the planning exercise on biodiversity and people. Impacts are the ultimate changes that people undertaking conservation planning hope to achieve. The pathway in gray illustrates a potential negative social outcome that could emerge from a planning process.

Challenges to evaluation of conservation planning

Evaluation of conservation planning is confronted by several conceptual and methodological challenges. Effective conservation planning can be defined in many ways. Outcomes can be described in visionary, qualitative terms (e.g., the planning exercise improved protection of land cover types in reserves), or more quantitatively (e.g., partnerships and collaborations between public and private stakeholders increased by 20%). Developers of plans, implementing agencies, stakeholders, and technical experts might have very different perspectives on goals and successful outcomes. In the face of this diversity, there are no established methods for evaluation of conservation planning.

Measuring outcomes also faces the difficulty of attributing benefits to aspects of planning processes, distinct from confounding factors (Ferraro & Pattanayak 2006). How, for example, can we measure the increased funding resulting only from planning, and not from unrelated advocacy or policy? The use of counterfactuals, or control areas without planning, is generally not feasible due to the complexity of physical, biological, social, and political contexts for planning. The sheer number of variables that would have to be matched, combined with limited resources devoted to evaluation, make controls impractical (Margoluis et al. 2009).

The difficulty of program evaluation is also exacerbated by the protracted implementation of most plans. Few planning exercises involve rapid, complete application of actions across regional designs that facilitate pre- and postimplementation comparisons. Examples of this minority are the Regional Forest Agreement process in New South Wales (Pressey et al. 2009) and the re-zoning of the Great Barrier Reef Marine Park (Fernandes et al. 2005). More typically, especially on private land and in community-managed inshore waters in developing countries, implementation proceeds incrementally over years or decades, during which time initial plans are modified and their influence on outcomes declines.

A framework for evaluation of conservation planning

To assess the effectiveness of conservation planning, an outcomes-based approach is needed. We propose a framework for identifying and categorizing outcomes (Table 3) against which results of planning exercises can be compared. Within this framework, potential indicators are described to track progress towards short- and medium-term outcomes. Indicators could detect changes during planning processes, perhaps extending over several years, or during the implementation of conservation actions, which could take many years.

Table 3.  Potential outcomes from conservation planning with corresponding indicators and approaches for tracking changes to indicators categorized by five forms of capital. Information on types of outcomes was derived from responses by conservation planners and implementers of plans to questions about outcomes emerging from planning processes. Also listed are frequencies of occurrence of outcomes in interview responses when mentioned by respondents (n= 28). Approaches focus on outcomes monitoring which involves tracking changes to indicators (e.g., species, budgets, attitudes) before and after planning (and perhaps during protracted planning processes)
CapitalOutcomeFrequency in responses (%)Indicator with example(s)Approach (pre- and postplanning)
NaturalRepresentation of biodiversity18Percentages of biological features occurring in new conservation areas, e.g.,% change of critical habitat occurrenceCompare number or extent of different features occurring in new and existing areas using spatial data on ecological patterns and processes
 Reduction in loss or degradation of natural values26Extent and intensity of threatening processes, e.g., deforestation; exploitationCompare extent and intensity of threatening processes occurring in conservation areas and matched areas outside conservation areas using data on distribution and intensity of resource use
 Persistence of biodiversity18Population sizes of selected species relative to minimum viable populations or metapopulationsCompare estimated population sizes and connectivity between populations across all conservation areas
 Maintenance of ecosystem services 7Rate of service provision e.g., freshwater volume (m3)Compare rates or extents of services provided
FinancialTransparency in conservation investments14Investment patterns linked to stated priorities, e.g., alignment of investments with relative priority of areas identifiedCompare locations of investments by agencies and NGOs in relation to systematically identified priorities
 Efficiency of operations14Expenditures by implementing agencies in relation to conservation benefits, e.g., amounts allocated in specific areas or actions relative to prioritiesBased on financial statements and budgetary data, compare spending relative to conservation benefits achieved
 Maximized benefit given limited budget18Benefit to biodiversity per $ spent, e.g., extent of critical habitat secured/ha given acquisition costsCompare cost-efficiency of final portfolio with minimum-cost solution or amount required to secure a random selection of conservation areas with the same total extent
 Leverage of additional funds or in-kind support21Proportion of additional funds received, e.g., % change in annual budget of implementing agency attributable to new donorsCompare sources and extent of funding received using financial statements of planning and implementing agencies
SocialCollaboration among agencies18Frequency and type of partnerships or relationships, e.g., number of project partners working togetherCompare connectedness and exchanges between actors/agencies using social network analysis
 Coordination between different actors11Alignment between projects and strategies by different actors, e.g., complementarity between priorities and activities of different conservation agencies and NGOs in planning regionCompare priorities and strategies between different agencies and NGOs
 Trust in planning process18Perceptions of planning process and outputs by stakeholders, e.g., % of stakeholders with positive view of planCompare perceived values of planning processes and products using data from questionnaires or interviews with stakeholders
 Sharing datasets between agencies25Datasets made accessible and available to different agencies in planning process, e.g., occurrence of data-sharing agreementsCompare datasets and other resources available for use by different agencies
 Shared vision29Consensus among stakeholders and planners about future outlook forCompare perceptions or positions of key stakeholders using content
   biodiversity and conservation in a region, e.g., document stating agreement by key actors and teamanalysis or interviews or questionnaires
 Attitudes of stakeholders39Disposition or feelings of stakeholders towards biodiversity and conservation; e.g., % of stakeholders that have positive attitude to local biodiversityCompare perceived values of conservation and/or biodiversity using data from questionnaires or interviews with stakeholders
HumanRaised awareness of biodiversity or conservation39Level of knowledge of biodiversity, conservation or other related issues among different individuals, e.g., % of stakeholders aware of priority biodiversity features in regionCompare extent or level of knowledge among individuals using data from interviews or questionnaires
 New knowledge of ecological or social values50Information on ecological or social values created or synthesized during planning processes, e.g., new data on opportunity costs of small landholdersCompare information available using content analysis of documents or surveys
 Learning applied in future plans25Use of new knowledge or skills applied in subsequent plans, e.g., application of new decision tool by members of planning team in subsequent planning processesCompare knowledge and skills of planning team using data generated by self-assessment questionnaires
 Behavior of decision makers14Frequency of positive or negative behavior of decision makers; e.g., voting records of local policymakersCompare behavior of decision makers, such as policymakers, using data from content analysis of records of local government authorities
InstitutionalInfluence on future decision making by organization or partners46Occurrence of decisions based on recommendations of plan, e.g., allocation of funding or other resources to specific areasCompare investments in specific area or issues using content analysis of documents/reports
 Self-sustaining strategies7Continuing implementation and planning revisions, e.g., adaptation of a regional plan to new informationInvestment in staff time and funds in updating data sets and prioritization analyses as new information becomes available
 Role of implementing agency14Ongoing implementation and updates by implementing organization, e.g., proportion of decisions and actions by one organization focused on the planCompare affiliations of people responsible for decisions and actions using content analysis of work plans, minutes of meetings, and other documents
 Consideration of conservation issues in decision making by other sectors11Frequency of considerations by local government authorities, e.g., downloads of conservation plan from website, alignment of land use decisions with priority areasCompare local government documentation of approved development proposals and protective zonings relative to identified priority areas
 Integration of priorities into policies, conventions, or legislation11Explicit consideration of priorities in policy, e.g., occurrence of areas, conservation features, and/or actions in policy statements or regulationsCompare consideration of priorities in policy using content analysis of documents or reports
 Influence on resource-use planning25Avoidance by developers of priority areas, e.g., occurrence of development applications in priority areasCompare frequency of planning applications in priority areas using content analysis of development applications
 Protected areas expanded25Establishment of new areas within protected area network, e.g., gazettal of areas under national or regional legislationCompare network of existing protected areas using agency records or content analysis of documents

Our framework is adapted from the DfiD (UK Department for International Development) Livelihoods Framework (DfID 1999), which categorizes people's assets as different types of capital: natural, financial, social, human, and physical. We have substituted physical capital (e.g., assets produced or manufactured) for institutional capital (e.g., change to the structure or function of institutions). Capital is the product of investment, yielding a flow of benefits over time. This framework is appropriate to conservation planning for several reasons. First, it has been applied and tested in evaluation of conservation strategies such as Integrated Conservation and Development Projects (ICDPs) (Garnett et al. 2007). Second, it expands evaluation beyond ecological assets to include the other potential outcomes of planning. Finally, it recognizes that different stakeholders will value different outcomes and so accommodates diverse views of effectiveness. For each type of capital, below, we describe potential outcomes and their relationship to goals and expectations, drawing on literature and our interviews.

Natural capital

Natural capital is the stock and flow of goods and services provided by ecosystems, including the diversity of species, regulating processes, and supporting services (Costanza et al. 1997). Conservation and restoration of natural capital are key motivations for conservation planning, reflected by frequently stated goals to safeguard biodiversity from threatening processes (Margules & Pressey 2000) and maintain ecosystem services (Egoh et al. 2007; Cowling et al. 2008).

Only a quarter or less of responses explicitly noted outcomes for biodiversity persistence, reduction in threats, or maintenance of ecosystem services (Table 3). Evaluation of changes in natural capital from planning is constrained by the time lags needed to observe changes that occur beyond the lifespan of most planning exercises. Reporting on conservation planning has therefore focused primarily on outputs such as achievement of objectives for conservation features in potential conservation areas (e.g., Rebelo & Siegfried 1992; Pressey & Tully 1994; Hansen et al. 2011). Reporting on conservation actions resulting from planning has been rare, and either short-term (e.g., Knight et al. 2011) or covering a few regions where complete and rapid application of conservation areas increased coverage of species or ecosystems (e.g., California Marine Life Protection Act planning initiative, Gleason et al. 2010).

Even these few studies of positive outcomes are limited, however. Implementation of actions might not necessarily lead to positive trends in the abundance and composition of species of concern (Kapos et al. 2009). Protection of areas of little interest for extractive uses (Scott et al. 2001) could result in a failure to protect imminently threatened features elsewhere. Moreover, there is limited evidence of ecological outcomes in the extensive parts of the world where application of actions occurs gradually over many years. In a few cases, negative effects of planning on natural capital have been documented, such as preemptive deforestation of areas identified in planning documents (Murray & Wear 1998; Venter et al. 2010). Overall, reporting on the influence of conservation planning on natural capital has been weak, concentrating on outputs and with little demonstration of impact.

Financial capital

Financial capital is the gain or savings of cash, property or goods that represent the wealth or economic value of an individual or organization. Improved financial capital resulting from planning might include mobilized funding or leverage of additional funds for conservation, measurable by changes in budget allocations, operating costs, and funding levels. Financial outcomes might also reflect cost savings or efficiency of operations due to priority-setting and strategic allocation of resources, given that cost-efficiency is a key principle of systematic conservation planning (Carwardine et al. 2008).

Nongovernment organizations have acknowledged financial benefits from planning in the form of raised awareness among potential donors (Morrison et al. 2009). These benefits have not, however, been rigorously documented beyond highlighting by our interviewees (Table 3). New funding or leveraged support might emerge only after the planning process is complete, although some funds might flow in the short term based on successes during early stages of planning. It might also be difficult to attribute financial gains to specific planning exercises relative to other potential influences, including those beyond the control of the organizations that benefit. Reporting of cost savings as a result of planning has largely been documented as the potential efficiency of conservation designs that avoid, where possible, areas with high acquisition or opportunity costs (Ando et al. 1998; Stewart et al. 2003). This benefit is an output, analogous to the potential, not actual, gain in natural capital arising from well designed but notional conservation areas.

Conservation planning also has potential negative effects on cost-efficiency, such as increased land values resulting from areas earmarked for action (Armsworth et al. 2006; Polasky 2006). Planning itself also has financial and opportunity costs: the staff and funds invested in planning could have been directed to other activities. Retrospective analyses measuring actual cost savings from planning versus business-as-usual spending have not been applied to implemented conservation actions. If financial efficiency is an intended consequence of planning, then greater efforts are needed to demonstrate actual financial outcomes.

Social capital

Social capital represents the relationships and interactions between individuals and groups with productive benefits (Pretty & Ward 2001). Planning processes and decision-making arrangements invariably shape social capital through activities such as stakeholder consultation and negotiations, expert workshops, and interactions within the planning team. A widely cited purpose of the planning process itself is to bring stakeholders together to share a common vision (Loucks et al. 2004; Knight et al. 2006b). Conservation planning is a social process in that conservation aims ultimately to alter the attitudes and behavior of people through incentives or regulation (Cowling et al. 2004; Cowling and Wilhelm-Rechmann 2007). Planning can also influence the priorities of other actors working on conservation within planning regions, recognizing the importance of cooperation and collaboration (Groves et al. 2002).

Changes in social capital can be defined broadly in four categories: reciprocity and exchanges, norms, connectedness, and trust (Pretty & Ward 2001), all of which were highlighted by responses in our interviews (Table 3). Reciprocity and exchanges involve sharing ideas and information between individual actors and organizations (e.g., Klein et al 2008) or collaborations between organizations (e.g., Higgins et al. 2007). Planning could, for example, positively alter the attitudes of policy makers towards biodiversity and conservation, an outcome frequently cited by our interview respondents (Table 3). Norms refer to mutually agreed values that place group interests above those of individuals. An example is reaching agreement among stakeholders on common goals for conserving biodiversity (e.g., Gonzales et al. 2003). Connectedness refers to relationships between individuals and social groups that are influenced by planning, perhaps through conflict resolution during stakeholder workshops. Finally, trust is exemplified by acceptance of the results of planning processes by individuals or institutions.

Positive changes in social capital are largely documented within case studies of specific planning exercises (The Nature Conservancy 2003; Henson et al. 2009; Knight et al. 2011). These examples demonstrate interactions and relationships during the planning process, but there are no quantitative assessments of different aspects of social capital across multiple exercises or approaches, or of the extent to which these effects were achieved or sustained. While improved social capital is generally not a primary motivation for conservation planning, the need for more explicit monitoring and evaluation of social variables is underlined by increasing recognition of social influences on the achievement of planning goals (Cowling & Wilhelm-Rechmann 2007; Ban et al. in press). Negative social outcomes of planning are also possible. These could include opposition to recommendations, fatigue from lengthy negotiations, perceptions that sharing of personal information was not repaid with respect, alienation of local stakeholders by external agencies (Smith et al. 2009), or conflict between resource users and conservationists (Gleason et al. 2010).

Human capital

Human capital comprises knowledge or skills that enable people to develop strategies to achieve their objectives (DfID 1999), providing the foundation for the other four types of capital. In conservation planning, skills and knowledge might be improved through discussion about objectives or training or application in analytical methods or decision-support tools. New knowledge of ecological and social values, for example, was the most frequently cited outcome of conservation planning highlighted by our interviewees (Table 3). Sharing datasets from different agencies and combining data layers might also enhance knowledge of the social and ecological conditions of a planning region. Changes in human capital tend to be reported as outputs such as numbers of people trained or items of educational materials distributed. Less emphasis has been given to demonstrable outcomes such as changes in skills, increase in knowledge, and how these improvements might affect other planning outcomes, such as increased support for conservation actions, collaboration among partners, or better executed planning processes.

Institutional capital

Institutions are formal and informal rules that guide the behavior and decisions of individuals and organizations (Ostrom 1990). Institutional capital is the capacity, structure, or functioning of institutions through formal means (e.g., laws and regulations) or informal arrangements (e.g., cultural norms applied in governing natural resource uses) (Platje 2008). Almost half of our interview respondents noted influences of planning on the structure, policy, or practice of planning agencies or other partners (Table 3).

Conservation planning has the potential to influence institutional outcomes such as policies and practices, expectations of groups of people, and the priorities of organizations through lessons learned about effective or ineffective approaches. Longer-term outcomes in natural capital might be facilitated by changes to institutional thinking or structures, perhaps reflected in strengthening of legislation to protect areas or avoidance of priority conservation areas by developers. Some evidence demonstrates that planning exercises have influenced legislation and regulations, in particular for the establishment of protected areas (Pressey 1998; Fernandes et al. 2005; Gleason et al. 2010). Some organizations have embraced systematic approaches to planning as part of their missions in response to arguments for the potential of these methods combined with successful applied examples (e.g., the Nature Conservancy 2011). Notwithstanding the few success stories related to institutional capital, changes in formal or informal institutions arising from planning have not been demonstrated convincingly or comprehensively.

Implications for improved evaluation of conservation planning

There are substantial potential benefits from evaluation in terms of accountability, demonstrating impact, organizational learning, and maximizing benefits. At the same time, evaluation requires new thinking about the nature of the planning process itself and how it is approached. We explore below the conceptual, operational, organizational, and policy implications of improved evaluation of conservation planning.

Conceptual implications

Lack of agreement persists on how to define the effectiveness of conservation planning. The planning process includes not only the design of notional conservation areas (the assessment phase) but also subsequent implementation of actions and ongoing management of conservation areas (Pressey & Bottrill 2009). Limiting evaluation to outputs (Figure 1), where most demonstrations of potential benefits have focused, restricts evaluation to the assessment stages of planning, reinforcing the insulation between spatial analysis and the outcomes and impact that developers and implementers of plans say they want to achieve (Bottrill et al. 2012). To frame appropriate and informative evaluation questions, indicators relevant to the ultimate goals of planning must be developed and applied. In addition, the focus of evaluation has been concentrated unduly on aspects of natural capital, including implementation of conservation areas. Not only is direct implementation of actions an incomplete indicator of benefits to natural capital (Pressey & Bottrill 2009) but the four other types of capital (Table 3) warrant their own attention. Extending evaluation to financial, social, human and institutional outcomes will avoid underestimating the benefits of planning and motivate improved approaches that maximize benefits defined more comprehensively.

Operational implications

Our framework suggests indicators to reflect outcomes of planning (Table 3), but indicators alone do not translate into a rigorous and informative evaluation strategy. Like the process of conservation planning itself, an evaluation system is an integrated series of analyses and decisions. Frameworks and operational models for conservation planning, (e.g., Knight et al. 2006a; Pressey & Bottrill 2009), do not explicitly consider evaluation. An evaluation strategy would formulate key evaluation questions, set objectives for evaluation, identify appropriate timing for collecting and analyzing data, and articulate ways of resourcing and providing incentives for evaluation. The advantages of considering evaluation early and explicitly in the planning process include timely collection of required data and ownership of evaluation results by stakeholders. Given the potential applications of evaluation throughout the planning process and its importance in shaping decisions, an evaluation strategy should be part of the initial scoping stage of planning, with adjustment as planning proceeds (Pressey & Bottrill 2009).

Organizational implications

Measuring potential outcomes of planning appears to be a large task (Table 3), even if we have only begun to understand its scope. As with planning itself, evaluation is often viewed as an activity that diverts valuable resources from conservation actions. Organizations must therefore consider the value of knowledge gained from evaluation versus the cost of gaining that knowledge. Rigorous and effective evaluation of planning will require organizations to commit not only considerable time and resources but to embrace learning in an explicit framework. Perhaps most challenging is the prospect of exposure to internal and, in many cases, external scrutiny. Overwhelmingly, reporting of conservation outcomes has promoted successes and buried failures (Redford & Taber 2000). Reasons for this lack of transparency stem from a fear of losing donor funding and weak incentives for critical assessment or experimentation (Kapos et al. 2008). Stronger incentives could come from government and donors themselves, if not from within organizations, in the interests of maintaining organizational knowledge and learning from mistakes to make conservation planning more effective.

Policy implications

Transparency of results associated with evaluation also has consequences for policymakers. Inadequate evidence of planning effectiveness has serious implications for the accountability of large sums spent on conservation planning by government agencies and NGOs (Table 1), and could undermine confidence in identified priorities. Public accountability becomes an issue when governments and agencies make ad hoc decisions about spending on the environment. Audits of such decisions, like recent ones in Australia (Auditor-General of Queensland 2010; Victorian Auditor-General 2011) need to be broadened to consider not only management of established protected areas but the effectiveness of the whole process of conservation planning. Very large government spending programs, such as Australia's Natural Heritage Trust, should also be scrutinized with the framework we propose here. Accountability and evaluation have been embraced by other public sectors, such as public health and education, yet conservation has been slow to adopt evidence-based approaches (Pullin & Knight 2001). Obstacles to progress in evaluation of conservation planning include lack of data on social and ecological outcomes, limited sharing of data or lessons learned for proprietary reasons (Keene & Pullin 2011), and the complexity of the social, economic, and ecological contexts for conservation (Margoluis et al. 2009). However, Keene & Pullin (2011) suggest that, with improved data flows and self-organization, the conservation sector might overcome these obstacles to achieve an “effectiveness revolution.”

Acknowledgments

The authors are grateful to Angela Guerrero, Andrew Knight, Morena Mills, and three anonymous reviewers for comments on an earlier manuscript. MCB thanks Dan Segan, Andrew Knight, Michael Mascia, Hugh Possingham, Ken Vance-Borland, and our interview participants for inspiring discussions that influenced the concepts presented here. MCB was supported by a Northcote Graduate Scholarship and funding from the Australian Commonwealth Environment Research Facility. RLP acknowledges support from the Australian Research Council.

Ancillary