Evaluating participation in water resource management: A review


Corresponding author: G. Carr, Centre for Water Resource Systems, Vienna University of Technology, Karlsplatz 13, AT-1040 Vienna, Austria. (carr@waterresources.at)


[1] Key documents such as the European Water Framework Directive and the U.S. Clean Water Act state that public and stakeholder participation in water resource management is required. Participation aims to enhance resource management and involve individuals and groups in a democratic way. Evaluation of participatory programs and projects is necessary to assess whether these objectives are being achieved and to identify how participatory programs and projects can be improved. The different methods of evaluation can be classified into three groups: (i) process evaluation assesses the quality of participation process, for example, whether it is legitimate and promotes equal power between participants, (ii) intermediary outcome evaluation assesses the achievement of mainly nontangible outcomes, such as trust and communication, as well as short- to medium-term tangible outcomes, such as agreements and institutional change, and (iii) resource management outcome evaluation assesses the achievement of changes in resource management, such as water quality improvements. Process evaluation forms a major component of the literature but can rarely indicate whether a participation program improves water resource management. Resource management outcome evaluation is challenging because resource changes often emerge beyond the typical period covered by the evaluation and because changes cannot always be clearly related to participation activities. Intermediary outcome evaluation has been given less attention than process evaluation but can identify some real achievements and side benefits that emerge through participation. This review suggests that intermediary outcome evaluation should play a more important role in evaluating participation in water resource management.

1. Introduction

[2] Participation in water resource management has gained increasing momentum over the last decades. Key water policy documents such as the European Water Framework Directive [Commission of the European Communities, 2000] and Federal Clean Water Act (http://epw.senate.gov/water.pdf) put great emphasis on the role of stakeholder and public involvement in water management [Bjerregaard, 1998; De Marchi, 2003]. A series of summits reflect the evolution of participation. The World Commission on Environment and Development drew attention to the role of community involvement in decision making [United Nations (UN), 1987]. The Dublin Statement on Water and Sustainable Development (adopted by the UN on 31 January 1992) included participation as one of its guiding principles. In the same year, the Rio Declaration on Environment and Development and Agenda 21 were endorsed, which recognized participation as essential for environmental management [UN, 1992]. The Arhus Convention on Access to Information, Public Participation in Decision-making and Access to Justice in Environmental Matters (UN Economic Commission for Europe, 1998) focused specifically on participation [Hartley and Wood, 2005].

[3] These conventions, statements and declarations identify participation with the objective to improve resource management, and enable individuals and/or groups to participate freely and equally in management. For water resources, an improvement in management is expected because water resource problems are complex and involve many different people with many different interests and opinions. Participation approaches may bring together a range of stakeholders with different interests and enable them to identify their own positions and those of others, leading to a deeper understanding of the issues [Pahl-Wostl, 2002; van den Hove, 2000]. Participation may also lead to decisions being made which are viewed as more legitimate because they have been created through a transparent or democratic process, such as consensus building [van de Kerkhof, 2006]. Decisions viewed as more legitimate may meet less resistance and be easier to implement [Bjerregaard, 1998]. Participation may direct human resources toward an issue which may distribute responsibilities and raise commitment toward resource management [Bjerregaard, 1998; Hemmati, 2002; Pretty, 1995]. Additionally, processes which tap into the knowledge, skills and networks of wide groups of participants could lead to more effective solutions being identified [Newig et al., 2005; van den Hove, 2000].

[4] While participation is often positively associated with resource management, many disadvantages and limitations are also recognized and debate continues over its value and capacity [Koontz and Thomas, 2006; Layzer, 2008; Leach, 2006; Lubell, 2004a; Muro and Jeffrey, 2008]. It is time consuming, delays decision making, may raise financial costs related to personnel and administration [Karl, 2000; Lubell, 2004b], and can be considered an inefficient use of resources if a community group reaches the same decision that would have been identified by a single agency administrator [Irvin and Stansbury, 2004]. Its rhetoric can be used as a rationale to reduce government culpability (remove financial responsibility and shift decision making on politically difficult topics away from government) and participatory decision making can be vulnerable to manipulation by powerful interest groups or cause harm to some stakeholders [Blaikie, 2006; Cooke and Kothari, 2001; Hanemann and Dyckman, 2009; Karl, 2000; Layzer, 2008; Lubell, 2004a]. There is also limited evidence available to support assumptions that it enhances resource management [Coglianese, 1997; Koontz and Thomas, 2006; Reed, 2008]. Despite these acknowledged challenges, participation is a strategy increasingly promoted and even enforced [Innes and Booher, 2004].

[5] There are many terms used to describe participation [Sandström, 2009], and it has been said that participation refers to a principle rather than a rigorous definition [Ker Rault and Jeffrey, 2008; Webler et al., 2001]. In this review, we have defined participation as involvement in a process. Involvement may be passive (for example, receiving information), or active (for example, contributing to plans and decisions in a variety of ways). Since Arnstein [1969]proposed the “ladder of participation,” which categorizes participation according to the level of participant involvement in the decision-making process, researchers have recognized that different levels of participation may lead to both good and bad outcomes [Beierle, 1998; Layzer, 2008; Leach, 2006; Lubell, 2004a]. Therefore we use a broad definition of participation to allow us to include a very wide variety of participation literature. Similarly, in this work, the term participant refers to anyone involved, irrespective of position, power or role.

[6] Considerable water resource management literature exists that describes participation programs and approaches. Some of this work has evaluated the value, outcomes, impacts or lessons learnt, before, during or after the participation activity has taken place. A review of how participation has been evaluated, particularly in water resource management, is needed because evaluation is essential to develop effective participation programs. The aims of this review paper are to organize existing approaches for evaluating participation, to assess their usefulness and provide information and guidance on the methods used. This review is based on published literature and is strongly shaped by the tone of the publications. As will be shown, much literature reports positively on the benefits of participation, perhaps reflecting a trend for participation based on theoretical assumptions, as well as researchers' (often qualitative) findings and experiences. However, this bias may also reflect the difficulty of demonstrating conclusively whether participation does or does not lead to resource management benefits. While few studies show resource benefits, no studies have been identified that have proved a negative link between participation and resource management. Continued evaluation is essential to identify and understand the realities of what participation is and is not capable of achieving.

2. Methods for Evaluating Participation

[7] “Evaluation is the systematic assessment of the worth or merit of an object” [Joint Committee on Standards for Educational Evaluation, 1994, p. 3]. Stufflebeam and Shinkfield [2007] develop this definition to include that evaluation should follow generic steps and produce descriptive and judgmental information. The evaluation's purpose is therefore to provide some form of assessment on how a participation program or project will, is, or has functioned. It can take place at various times, during the preparation of a program (ex ante evaluation), during the program, or after the program (ex post evaluation) [Muro and Jeffrey, 2006].

[8] Evaluation informs managers on program performance and identifies areas for improvement [Stufflebeam and Shinkfield, 2007]. It forms part of an evolving cycle of learning, and is often used to bring closure to a project [Blackstock et al., 2007]. It has also been used to assess the strengths and weaknesses of specific approaches [Beierle, 1998; Chess and Purcell, 1999], to explore the underlying theory that determines the structure of participation [Moote et al., 1997] and to assess the current status of participation [De Stefano, 2010].

[9] Our initial analysis of the literature showed that evaluations of participation in water management can be grouped according to whether they evaluate the process or the outcomes from the program. Process evaluation relates to work that has focused on how participation has taken place [Conley and Moote, 2003] or the quality of the process [Beierle and Konisky, 2000] (Table 1). We have then identified two different forms of outcomes. Intermediary outcomes describe both the development of trust and interaction (social capital) and the achievement of “products” such as agreements [Burgess and Chilvers, 2006], innovation and the creation of shared knowledge or information (Table 1). These less tangible outcomes do not relate to a direct change in resource management at the point in time at which they are evaluated, but may be essential to achieve resource management improvements [Connick and Innes, 2003; Conley and Moote, 2003; Genskow, 2009]. Water resource management outcomes are often the ultimate aim. These outcomes include the implementation of agreements, a measurable improvement in ecological health, improvement in human health or a reduction in conflict between resource users (Table 1). Capturing broader, more ultimate objectives, such as improving human welfare, is likely to be particularly challenging and evaluators generally attempt this through these specific resource management outcomes. The evaluation method selected, and evaluation criteria, is determined by the specific values, strategies and objectives of the evaluator as well as the context in which the program, approach and evaluation operates [Beierle, 1998; Conley and Moote, 2003]. Evaluation findings will be shaped by the criteria selected to conduct the evaluation.

Table 1. Summary of the Three Methods of Evaluation and Their Criteria Identified Through Review of the Literature
Process EvaluationIntermediary Outcome EvaluationResource Management Outcome Evaluation
AccountabilityDevelopment of social capital: interaction and network development and trustEcological improvement
Cost-effectivenessProducts from the process: agreements, end to a stalemate, innovation, institutional change, shared knowledge and informationEconomical improvement
Deadlines and milestones Human health and wellbeing improvement
Facilitation Implementation of an accepted plan
Knowledge inclusion Reduction in conflict/increased harmony

3. Process Evaluation

[10] Process evaluation focuses on how participation has been conducted. Many characteristics of good participation process can be identified in the literature (Table 2). Characteristics are generally formed through theoretical work and evaluation criteria are developed to test empirically whether these theoretical characteristics are present, absent or important in actual case studies. Evaluation criteria are operationalized through instruments (such as interviews, questionnaires and observation) and performance indicators. Indicators are used to simplify and describe a situation and assist in communicating evaluation findings [Wilson and Buller, 2001]. They need to be measurable, usable, sufficiently detailed and structured so that they can be applied in multiple settings and on multiple occasions, they should be reliable (consistent through time and space) and should accurately reflect reality [Leach et al., 2002; Rowe and Frewer, 2004; Stufflebeam and Shinkfield, 2007; Wilson and Buller, 2001].

Table 2. Summary of Desirable Characteristics for Participation Processes and Evaluation Criteriaa
Desirable Characteristics of Good Participation ProcessesbEvaluation Criteria
  • a

    See Table S1 for evidence and rationale for criteria and the data sets collected to conduct evaluation described in the participation literature.

  • b

    Letters in parentheses indicate the following: E = empirically derived preferences; T = theoretically derived preferences.

Accountable discourse (T)Participants' words reflect their values and actions, and participants seek actions that correspond to their arguments [van den Hove, 2006]
Delegation (E)Representatives assume specific tasks or responsibilities as selected by the group based on their personal qualities (trustworthiness and competence) [Kuper et al., 2009].
Responsible leadership (E)Leadership decision making (when required) is built upon a legitimate and fair process and is responsible [Webler et al., 2001].
Cost-Effectiveness and Resources
Cost-effectiveness (T)The process offers value for money (the costs of implementing the program are balanced by the importance of the issue being addressed) [Beierle, 1998; Rowe and Frewer, 2000].
Support (closely connected to access to meetings and representation; E)Support (financial and other) is provided to participants to achieve and maintain participant representativeness [Huitema et al., 2010; Mostert et al., 2007; Moote et al., 1997]
Deadlines and Milestones
Deadlines, milestones, and rewards (E)The process includes a detailed agenda with deadlines and the promise of investment money once agreements are reached [Jiggins et al., 2007].
Use of boundary objects (closely connected to access to information; E)Boundary objects (spreadsheets, presentations, diagrams, etc.) are used to represent real phenomena (e.g., river) and facilitate the process [Fuller, 2009].
 Maps and satellite images are used to facilitate understanding of system interconnections, develop suitable strategies, and improve access to information [Bacic et al., 2006; Mostert et al., 2007].
 Cognitive mapping is used to understand participants' opinions, perspectives, and perceived system interconnections [Mouratiadou and Moran, 2007].
 Field trips take place to facilitate participant understanding of process interactions and develop relationships between participants [Ison and Watson, 2007].
Dialogue (E, T)The process ensures that ideological orientations are not suppressed [Hedelin, 2007].
 Preferences and values of all participants are made explicit and open to the critique of others [van den Hove, 2006].
 A “space of exchange” is created where people feel comfortable sharing their needs, concerns, and values [Jones et al., 2009; Lamers et al., 2010; Moote et al., 1997].
 The process provides opportunity for all to participate and be heard [Webler et al., 2001].
Facilitation (E, T)Facilitation is impartial [Jiggins et al., 2007; Moote et al., 1997; Mostert et al., 2007; Murdock et al. 2005; Reed, 2008; Rowe and Frewer, 2000; Walker et al., 2006].
 Facilitation is dynamic [Ison and Watson, 2007].
 Facilitator is experienced in the issues being addressed [Lamers et al., 2010].
 Facilitator has ability to build trust and establish alliances [Mostert et al., 2007].
 Facilitator is highly skilled [Leach and Pelkey, 2001; Reed, 2008].
 The process strives to find common interests [Webler et al., 2001; van den Hove, 2006].
 Approaches focus on shared values rather than entrenched interests [Beierle, 1998; Jiggins et al., 2007].
Knowledge Inclusion
Knowledge inclusion (T)A variety of knowledge is included to help make informed decisions [Beierle, 2002; Hedelin, 2007; Reed, 2008].
Access to information and meetings (E, T)Adequate scientific and technical resources are provided to participants [Beierle, 2002; Murdock et al., 2005; Rowe and Frewer, 2000].
 Information and meetings are accessible to participants [Chenoweth et al., 2002; Hurlimann and Dolnicar, 2010; Klinke, 2009; Lemos et al., 2010; Mostert et al., 2007; van den Hove, 2006; Walker et al., 2006].
Ground rules and task definition (E, T)Clear ground rules for interactions are set at the start of the process [Moote et al., 1997; Mostert et al., 2007; Rowe and Frewer, 2000].
 Tasks are clearly defined [Rowe and Frewer, 2000].
Legitimate decision making (E, T)The process incorporates consensual decision making [Webler et al., 2001].
 Decision making is based on evidence rather than political motivations [Webler et al., 2001].
 Decision making is structured and clearly displayed [Chilvers, 2009; Rowe and Frewer, 2000].
 Participants' inputs have a genuine impact on policy [Klinke, 2009; Rowe and Frewer, 2000; Walker et al., 2006].
Representation (E, T)Participants represent a broad and cross-cutting section of society and interest groups [Abelson et al., 2003; Blackstock et al., 2007; Chilvers, 2009; Halvorsen, 2001; Hedelin, 2007; McCool and Guthrie, 2001; Mostert et al., 2007; Petts, 2006; Rowe and Frewer, 2000].
 Everyone who might be affected or have an interest is involved [Moote et al., 1997].
 Representativeness is continually sought [Lamers et al., 2010].
 Interactions take place across administrative levels and geographic scales [Newig and Fritsch, 2009a; Plummer and Armitage, 2007; Schlüter et al., 2010].
 Key metaplayers are present to ensure that agreements reached are implemented [Dray et al., 2007].
Timing of involvement (T)Public is involved at a stage when value judgments become important [Rowe and Frewer, 2000].
 Involvement is considered from the onset of the project or program [Hartley and Wood, 2005; Reed, 2008; United Nations Economic Commission for Europe, Convention on Access to Information, Public Participation in Decision-making and Access to Justice in Environmental Matters, 1998].
 Involvement is continuous [Moote et al., 1997].
Promote equal power (E, T)The process is able to deal with power asymmetries [Hedelin, 2007].
 The process allocates power equivalently between participants [Webler et al., 2001].
 Institutional arrangements support or promote power sharing [Ison and Watson, 2007; Reed, 2008].

3.1. Theoretically Derived Process Evaluation Criteria

[11] One of the first theoretically derived process evaluation frameworks, developed by Renn and Webler, was based on Habermas' concepts of ideal speech [see Webler, 1995, 1999]. The Renn and Webler framework identified two essential characteristics of good participation process: fairness (equal distribution of opportunities to participate in the process) and competence (appropriate knowledge, information and procedures to enable participants to protect their interests and take part in the process). Since its inception, evaluators have expanded the basic criteria of fairness and competence by including more specific aspects under these broad criteria. Klinke [2009] applied criteria based on accessibility and capacity to influence (fairness), and dialogue and collective problem solving (competence) to evaluate participation approaches in the Great Lakes region of the USA and Canada. Rowe and Frewer [2000]differentiated between acceptance criteria (democracy and fairness which lead people to accept the process and its outcomes) and process criteria (such as access to resources, structured decision making and cost-effectiveness). Their criteria have been used to evaluate participation in a broad range of water resource management programs in a variety of settings. Examples include the development of a water management plan in the Netherlands [Lamers et al., 2010] and the suitability of participatory modeling for engaging stakeholders in decision making [Zorrilla et al., 2009]. Kuper et al. [2009] used theoretically derived process criteria developed by Rowe and Frewer [2000]to evaluate a program designed to support farmers establish drip irrigation in Morocco. Criteria included representativeness, independence of the process, early involvement, participants' capacity to influence, transparency, access to resources, clear task definition, structured decision making, and cost-effectiveness. Participants were surveyed to determine the degree to which they felt each criterion had been positively achieved during the process. The criteria provided a framework for evaluation that simplified and homogenized the data collection but, importantly, the researchers supplemented the quantitative data with narratives to explain the perspectives revealed by the surveys.

3.2. Empirically Derived Process Evaluation Criteria

[12] Participants, who are actively involved in a process, may have different views to theorists on important characteristics of good participation processes. Evaluators attempt to identify participants' perspectives through instruments such as surveys, questionnaires and interviews. Conducting surveys and holding interviews can be time consuming, case specific and resource intensive. This might not be feasible for every participation program, or for evaluating and comparing multiple programs. More readily available, often quantitative data can be used as proxy indicators for participant satisfaction and attitudes toward the process. Proxy indicators include participation rates, sustained participant involvement and longevity of programs or projects [Genskow, 2009; Murdock et al., 2005; Parker et al., 2009]. Webler et al. [2003. p.108] suggested that the use of such surrogates is theoretically justified as “it is reasonable to expect that people will be more likely to participate if the process meets their normative expectations for good process.” However, proxy indicators are descriptive rather than explanatory, i.e., they describe how the process looks but are not able to assess why it looks the way it does. Evaluators then need to interpret the surrogates in terms of the factors of interest. High participation rate, sustained participant involvement and longevity of the program have been associated with a range of factors. These include availability of resources [Irvine and O'Brien, 2009], motivation and drive of leaders and participants, perceived relevance to participants of the issues being addressed [Ertel, 1979; Forshay et al., 2005], capacity to provide processes which reflect participants' needs and values [Byron and Curtis, 2002; Webler et al., 2001] and capacity to captivate the attention of participants [Lankford and Watson, 2007].

[13] Caution in interpreting proxy indicators is needed. Low participation rates at public meetings for a water reuse program in California, USA, were misinterpreted by program administrators as acceptance or low community interest in the reuse plans. As the program progressed a strong movement against reuse emerged unexpectedly and the program had to be restructured to identify acceptable reuse options [Ingram et al., 2006]. The use of proxy indicators also presents dangers of misrepresentation because active participation tends to be highly correlated to level of education and dominant status in society (in the USA this is male, middle age, married, home owner, high level of income) [Koehler and Koontz, 2008]. Using a nonrepresentative sample to provide information on the ideal characteristics of a participation program may skew the process toward one preferred by such individuals. Process changes to meet the needs of a nonrepresentative group may reduce the willingness, or capacity, of more diverse participants to be involved [Larson and Lach, 2008]. In such a situation, process evaluation and subsequent modification could lead to reduced diversity in representation.

[14] Some very detailed studies attempt to explore both participant perspectives, and the rationale behind perceptions and actions. This is done through a variety of research tools such as ethnographic approaches involving observation [Moote et al., 1997; Tam, 2006], unstructured and semistructured interviews [Clarke, 2008; McCool and Guthrie, 2001], card sorting [Hare and Pahl-Wostl, 2002] and Q methodology [Chess and Johnson, 2006; Danielson et al., 2010; Tuler and Webler, 2010; Webler et al., 2003]. Q methodology is an approach where interviewees sort a range of statements such as, “the process should be cost-effective,” “a fixed deadline is important,” according to the weighting the individual gives to each. This work has been valuable in showing that preferences for good participation processes are driven by underlying factors which vary according to individual characteristics, values and experiences [Tuler and Webler, 2010]. Past experiences of participation play a significant role and dissatisfaction with earlier processes tend to lead to negative perspectives [Webler et al., 2001]. Weible et al. [2004]used postal questionnaires in conjunction with semistructured interviews to compare whether stakeholders' in marine areas in California, USA, preferred a collaborative decision-making approach to a science-based (top-down) approach. They found that federal government stakeholders, environmentalists and scientists preferred a science-based approach while local government and fishing related stakeholders preferred a collaborative approach. Preferences were found to be related to stakeholders' beliefs (i.e., the belief that consensus based negotiations offer the best strategy for resolving environmental issues versus the belief that science driven management strategies offer the best approach).

[15] These evaluation approaches are not only useful for identifying whether programs reach the requirements of theoretically derived criteria, but also for shaping the research enquiry into how and why criteria are not met. For example, Moote et al. [1997]evaluated public involvement in land management planning in the San Pedro River, USA using theoretically derived criteria from the participatory democracy literature. Evaluation criteria included representation and access, information exchange and learning, continuity of participation and sharing of decision-making authority. Through observing meetings and conducting interviews and questionnaires with participants they were able to describe how the program performed against the criteria and identify why it performed as it did. Findings showed that participants felt that representation was low and that this was related to limited time and support to participants for attending meetings. Attendance at meetings declined over the duration of the program and continuity was not achieved. This was attributed to conflicts between participants. The final decision (to acquire riparian lands) was made by the government agency, showing that decision making was not shared.

3.3. Selection of Process Criteria

[16] Review of the literature has identified many desirable characteristics of participation processes. We have split these into seven key themes: accountability, cost-effectiveness, deadlines and milestones, facilitation, knowledge inclusion, legitimacy and power (Table 1). Within each of these groupings there are preferences that relate to Renn and Weblers' criteria of fairness and competence, and Rowe and Frewer's [2000] criteria of acceptance and process. The summary of desirable characteristics and evaluation criteria show that there is, at this stage, no clear agreement on a narrow list of suitable process criteria for evaluating participation (Table 2; see also Table S1 in the auxiliary material). However, those which are theoretically derived and supported through empirical evidence collected through collaborative processes could be considered to have greater weight. This suggests that facilitation, dialogue, access to information and meetings, ground rules and task definition, and representation should form the basis for process evaluations (Table 2).

[17] Processes that promote equal power also seem to be important. This observation is primarily derived through theoretical work, but is also supported by participant perspectives identified through participant surveys and interviews [e.g., Faysse et al., 2010; Park et al., 2009]. However, evaluating whether a participation process actually changes or further concretes existing power structures is likely to be highly complex. Tam [2006] revealed power complexities in her evaluation of participation in a pond construction program in Indonesia. She used ethnographic approaches (living with the communities, observing and recording events and conducting interviews) to identify why the pond was never completed. The evaluation criteria were focused around power, conflict and communication. Findings revealed that power dynamics between participants contributed to low community involvement and project failure. For example, the funding nongovernmental organization (NGO) followed local etiquette of approaching the highest village authorities to gain support for the project, and then allowing these authorities to control who was involved in the project. This meant that the majority of the 25 project members came from just three (comparatively wealthy) families. A culture of harmony meant that excluded villagers did not voice their grievances and instead chose to ignore the project, leading to project abandonment.

[18] Some process factors may be particularly relevant to the water sector. For example, water resources are typically state managed. Agencies involved in participation programs may therefore be particularly concerned about the cost-effectiveness of tax payers' resources, and the publics' perception of the legitimacy of the process (for example, through access and representation). Water resource management frequently involves multiple interest groups and sponsoring agencies may be interested in factors such as facilitation and dialogue that focus on integrating multiple perspectives. Water management decisions might be improved by basing them on the maximum information available. Knowledge inclusion might therefore be considered an important characteristic of good participation process.

[19] Contradiction exists between some criteria and this could lead to complications within an evaluation. Access to information and meetings is an important criterion for a legitimate process. However, a well facilitated process (also an important characteristic) requires that participants express their position honestly. This may require “a space of exchange” where participants feel comfortable speaking openly. During negotiations over agricultural water use efficiency strategies in California, USA, representatives needed to be able to make concessions, but also needed to show their alliance to their constituencies which prevented them from being seen to back down. To overcome this difficulty, closed meetings were scheduled where representatives were able to negotiate away from constituent criticism and agreement could finally be reached [Fuller, 2009]. In this example, facilitation appears to be more important than legitimacy for achieving agreement. This suggests that process evaluation may sometimes need to place criteria in a hierarchical order of importance.

[20] Evaluating accountability appears to present particular challenges. No studies have been found which describe how the criteria of delegation and responsible leadership have been evaluated. As a result, no clear guidance on suitable data sets with which accountability can be evaluated can be provided. This challenge echoes the findings of Rowe and Frewer's [2004] review of 30 public participation evaluation studies. Their work noted that few studies gave sufficient details of the performance indicators used to allow evaluation to be reproduced. However, our work shows that information is provided by evaluators for many of the criteria. Importantly, the diversity in instruments and indicators used by evaluators is not broad. Participant perspectives and satisfaction derived through surveys and interviews are frequently used to evaluate the majority of criteria, along with evaluator descriptions and assessments of processes (Table S1).

4. Evaluation of Intermediary Outcomes

[21] Intermediary outcomes describe outcomes that may not have been the original or ultimate aim from a participation program, project or process. They could be described as important side benefits which can be identified though goal-free evaluation [Scriven, 1991]. We identified two groups of intermediary outcomes (also called actions, outputs, nontangible outcomes and first and second-order outcomes [Connick and Innes, 2003; Innes, 1999; Koontz and Thomas, 2006; Kuper et al., 2009; Layzer, 2008; Plummer and Armitage, 2007]). The first group relates to the development of social capital, such as network development and trust. The second group, based on Connick and Innes [2003], relates to products from the process, such as agreements, an end to a stalemate, innovation, institutional change and development of shared knowledge (Table 3).

Table 3. Summary of Intermediary Outcomes, Evaluation Criteria, and Their Rationalea
Intermediary Outcomes and Evaluation CriteriaRationale for Criteria
  • a

    See Table S2 for the data sets used to operationalize the criteria and key findings in the participation literature.

Social Capital
Interaction and network development 
 Process leads to greater interaction and awareness to other's activities.Connectivity raises capacity for knowledge exchange, engagement and working together [Berkes, 2009].
 Relationships are developed which support continued dialogue. 
 Participants trust one another.Cooperation is more efficient because less time and resources are spent monitoring the actions of others [Lubell, 2007; Pretty and Ward, 2001]
Products of the Process
Agreements are reached and plans are developed 
 Agreement reached between participants.Agreements form the basis for cooperation and concerted action.
End to a stalemate 
 The process leads to discussion between participants who had previously refused to discuss together.Discussion is essential for identifying shared positions or conducting negotiations [van den Hove, 2006]
 Policies and practices are developed which are more creative and context dependent [Connick and Innes, 2003].More effective or suitable solutions are identified.
Institutional change 
 Institutional functions, roles or structures are modified to reflect participants' ideas, values, or requirements.Systems are developed that are more effective or suitable for the setting.
New organizations are created or developed. 
Shared knowledge and information 
 Data, information, and terminology are developed that is accepted and trusted by all participants.Shared understandings facilitate discussion and are essential for identifying shared objectives and ideals or as a basis for conducting negotiations [Jiggins et al., 2007; Steyaert et al., 2007].
 Capacity to manage knowledge and information to undertake collective action is enhanced. 
 Participants gain knowledge and understanding of the issue being addressed [Hatzilacou et al., 2007] 
 The process generates information that would not have been available otherwise.Shared knowledge allows systems to be identified and developed that are effective and suitable for the setting [Petts, 2006]

4.1. Development of Social Capital

[22] Social capital refers to the capacity and willingness of participants to invest in collective activities to achieve shared objectives [Putnam, 1995]. Features to achieve this include trust and connectivity across networks and groups [Pretty, 2003]. Work has shown that social capital is critical for effective resource management [Pretty and Ward, 2001] and for activating participants to engage in environmental policy formation [Jones, 2010]. Research suggests that social capital is also highly correlated with the development of agreements and innovation (products from the process) [Kallis et al., 2009; Lejano and Ingram, 2009]. However, social capital is also linked negatively to resource management [Koontz and Thomas, 2006]. Examples include corruption that undermines environmental regulation [Pelling and High, 2005] or vertical clientelistic networks than obstruct plans designed to enhance environmental conditions [Jones, 2010; Schlüter et al., 2010].

[23] Resource management is information intensive because knowledge of the entire socioecological system is needed which is fragmented across different groups and agencies, operating at different scales [Berkes, 2009]. Information can be mobilized and integrated through networks which cross scales and levels of organizations [Hahn et al., 2006]. Bonding, whereby the relationships between existing connections are strengthened, and bridging, whereby new connections are created between individuals and networks, are two forms of network development associated with social capital [Connick and Innes, 2003; Jones, 2010; Ohno et al., 2010]. Several authors describe how participation processes in water management programs lead to stronger interactions and networks between participants [Collins et al., 2007; Genskow, 2009; Hoverman et al., 2011; Lejano and Ingram, 2009]. Hatzilacou et al. [2007] note that valuable stakeholder alliances and working groups formed through their stakeholder workshops in Greece even though no final plans or agreements were achieved (Table S2).

[24] Trust is associated with enabling free and open dialogue that allows more creative solutions to emerge [Pretty and Ward, 2001]. It has also been associated with leading to greater acceptance of decisions, resulting in more efficient implementation [Newig and Fritsch, 2009b]. Due to the difficulty in identifying trust, indicators of trust have been used in some studies. These include participant confidence in the abilities of the agency, or perception that the agency would consider the values of the participants [Beierle and Konisky, 2000] (Table 3) and the degree to which participants feel they trust each other [Leach and Sabatier, 2005b; Murdock et al., 2005; Ohno et al., 2010] (Table S2). Sultana and Thompson [2004] surveyed participants in floodplain fishery management workshops in Bangladesh. They measured changes in participants' perceptions of trust, cooperation, unity and willingness to work for the common good before and after the workshops. The results showed that 30% of participants perceived that community attitudes had changed and trust, cooperation and unity had increased slightly through the process.

4.2. Products of the Process

[25] Reaching agreements and achieving support for action or management plans are frequently used to indicate a positive intermediary outcome from participation [Bots et al., 2011; Collins et al., 2007; Ison and Watson, 2007; Lamers et al., 2010; Leach et al., 2002; Marttunen and Hämäläinen, 2008]. In the USA, the aim of participation is often associated with achieving consensus [Susskind and Field, 1996; van de Kerkhof, 2006] and reaching an agreement is an indicator that consensus has been achieved.

[26] Overcoming a stalemate or breaking gridlock is described as one of several advantages of participation in decision making [Connick and Innes, 2004; Irvin and Stansbury, 2004]. Fuller [2009]describes how a stalemate was overcome through deliberative processes in the CALFED Bay-Delta Program, California, USA. His evaluation uses comparative analysis to explore the differences between a successful program (one where agriculturalists and environmentalists reached agreement on strategies to raise water use efficiency) to a failed program where no agreement was reached. Through interviews with key stakeholders and a review of program activities he found that a critical challenge was that efficiency had a different meaning to agriculturalists and environmentalists. To agriculturalists efficiency related to the total productivity per unit of water and greater efficiency meant higher yields. To environmentalists efficiency meant reducing total agricultural consumption and allocating sufficient water to the environment. To overcome this a framework that focused on setting goals, such as habitat restoration, and identifying quantifiable objectives to assess goal achievement was developed through collaborative work. The process led stakeholders to engage in joint construction of efficiency strategies to which they were in agreement.

[27] Newig and Fritsch [2009b]conducted a meta-analysis of 40 environmental decision-making case studies to investigate (among other things) creativity in solutions identified through participation. Their analysis found that more creative solutions did seem to emerge from participation. This is supported by descriptive work based on the California Bay-Delta Program, which has shown that collaborative processes can support innovation [Lejano and Ingram, 2009] (Table S2). Intermediary outcome criteria that include innovation and institutional change have been used to evaluate public participation in a major review of the Lake Ontario and St. Lawrence River water level operating system (the LOSL Study) [Carr et al., 2012]. The goal of the LOSL Study was to produce an operating policy that was acceptable to everyone impacted by the water levels and flows in that region. Published information and meeting transcripts were used to show that although the study was not able to identify a consensus option within its 5 year time frame, it did produce many other achievements. Participant involvement in the development of the operating plans led them to be more innovative and raised their legitimacy, particularly when they were perceived to be supported by sound science. Recommendations to improve institutional arrangements for decision making and operations also emerged from the study.

[28] An urban river restoration project in the UK shows how participants were specifically included to capture their ideas and perspectives [Petts, 2006]. A collaborative approach was selected that brought engineering experts and public interest groups together to cocreate an effective restoration plan. Petts [2006]evaluates the program retrospectively, based on her own observations of the process and those of the project team. Criteria that included facilitation and collaborative framing were used to show that facilitation was essential and conducted well, and it was important to use the process to identify values which could be used to determine the ideal urban river environment. For the public, these included emotional characteristics such as tranquil, relaxing, natural and light, while values described as variety of wildlife, safe and flood-free, and varied shape and form were important for the experts. Workshops were used to create a set of 13 community criteria that formed a checklist during plan development to ensure that the restoration would meet the community's wishes. The approach led to a final plan that was both technically sound and socially agreeable, incorporating community values and engineering and environmental needs. This collection of work highlights that even when it is not possible to demonstrate tangible outcomes such as improvements in ecological health, interactions, understandings and processes can be seen to be transformed.

[29] The work of the European Social Learning for Integrated Management (SLIM) project focused on evaluating participation in terms of the achievement of shared meanings, agreements and social spaces. The authors associated these intermediary outcomes with social learning [Ison et al., 2007]. Social learning is a strategy for resource management that is based on the notion that “agreements built on a shared appreciation of reality are more likely to be long lasting” [Steyaert et al., 2007, p. 540]. Problems and solutions are constructed by participants based on co-constructed realities, rather than through negotiation and compromise [Steyaert et al., 2007]. Social learning aims to promote concerted action (bringing different roles together to achieve some common end that emerges during the process) [Collins et al., 2007; Ison et al., 2007]. Under this concept participation is an approach that can be adopted in addition to, rather than instead of, other approaches for water resource management such as market mechanisms and regulations [Ison et al., 2007]. This is important because it suggests that participation operated in conjunction with other approaches will benefit their implementation. For example, Lejano and Ingram [2009] describe how stakeholder collaborative processes led to relatively rapid negotiation of voluntary markets for water transfers between water users in California. This also suggests that intermediary outcomes have the potential to lead to resource management outcomes (discussed further in section 6.1).

4.3. Selection of Intermediary Outcome Criteria

[30] Interaction and network development and trust can be evaluated by participant surveys and case study analysis (Table S2) but these criteria are rarely used in isolation. For example, Genskow [2009] evaluated watershed partnerships in the USA using objectives defined by the government agency who initiated the partnerships (broad stakeholder representation, shared resources among participants and improved ecosystem management). Shared resources were evaluated by determining participants' perspectives on the extent of interaction between participants and the capacity of the partnership to pool and share resources. In a different example, environmental partnerships between the government environmental agency and commercial businesses in the USA were evaluated by assessing trust between stakeholders (Table S2) and the extent to which decisions reflect community values (indicative of achieving consensus; Table 4) [Murdock et al., 2005].

Table 4. Summary of Resource Management Outcomes, Evaluation Criteria, and Their Rationalea
Resource Management Outcomes and Evaluation CriteriaRationale for Criteria
  • a

    See Table S3 for the data sets used to operationalize the criteria and key findings in the participation literature.

Ecological improvement 
 A measurable improvement in ecological condition.“The ultimate measure of success is a partnership's effects on physical, biological, or social aspects of watershed-related problems. Measuring implementation alone is not sufficient because well-executed projects can fail to have the desired consequences due to poor design or unforeseeable events” [Leach et al., 2002, p. 653].
 “Ultimately, we are interested to see whether participation not only improves the knowledge base of the decisions but actually leads to more ecological decisions” [Newig and Fritsch, 2009b, p. 217]
 Critical natural capital (processes and cycles within natural systems that are responsible for sustaining the stability and resilience of ecosystem) should be maintained [Plummer and Armitage, 2007].
Economical improvement 
 More cost-effective solutions are identified.Concern that stakeholder-based decisions will be more politically desirable and therefore unnecessarily expensive [Beierle, 2002].
 Participation achieves outcomes (such as nutrient emission reductions or raised water use efficiency) in the most cost-effective manner [Uysal and Atıs, 2010; Wright and Fritsch, 2011].
 Implementation is achieved at a lower cost.Involvement in decision-making prepares participants for implementation, leading to greater acceptance of policy or regulation, compliance and lower enforcement costs [Newig and Fritsch, 2009b].
 Benefits of participation exceed costs to participants.All stakeholder groups perceive that the benefits from participation exceed the costs (in terms of participant time, training costs, administration, and risks that participation does not lead to intended benefits or even cause harm to some stakeholder groups) [Karl, 2000].
Implementation of an accepted plan 
 Plans are implemented.Plan implementation is identified as a critical indicator of success to scientific, managerial, and public participants interviewed in the USA [McCool and Guthrie, 2001].
 Restoration projects are implemented. 
 Changes in land management practices occur. 
Human health and well-being improvement 
 Human health is improved as a direct result of the program.Direct and indirect health benefits emerge from community-based water supply projects [Eng et al., 1990].
Reduction in conflict 
 Consensus is achieved.All participants need to see a sufficient amount of their own values in the decision to give it their support [Gregory et al., 2001].
 Decisions reflect community values [Murdock et al., 2005].
 Reduced conflict lowers the transaction costs of identifying, implementing, and enforcing a management strategy [Lubell, 2004b].
 Decisions are not appealed against or contended through legal channels.The needs, concerns, and values of participants are addressed in the outcome to the extent that participants will not appeal or contend the outcome through legal channels [Connick and Innes, 2003; Moote et al., 1997].

[31] Agreements are most commonly used as an intermediary outcome criterion due to the relative ease with which these data can be collected. It is also likely that evaluators place greater weight on reaching an agreement because it may precede or be part of a legally recognized or institutionally important management plan. It is important to consider that work has shown that agreements in watershed partnerships emerge over periods greater than several years and correlate to the longevity of the participation program [Leach et al., 2002]. This means that many evaluations that take place during or immediately after a participation activity may not be able to identify agreements.

[32] This section shows that there are many intermediary outcomes that could be valuable for evaluating participation. The literature describes how criteria relating to nontangible intermediary outcomes such as an end to a stalemate, innovation and creation of shared knowledge can be evaluated through well designed participant surveys and case study analyses (Table S2).

5. Resource Management Outcome Evaluation

5.1. Outcome Evaluation Criteria Driven by Interests

[33] Improvements in water resource management are usually the ultimate objective for resource managers. It has been argued that some form of measurement of the outcomes should form the basis of any evaluation [Beierle, 1998; Koontz and Thomas, 2006]. Researchers and water managers are often interested in determining retrospectively how successful a program has been in meeting specific objectives. Their aim is often to identify whether water management benefits emerge from participation and provide recommendations as to whether participation should be continued or adjusted in the future. Five main groups of resource management outcomes have been identified in the literature: ecological improvement, economical improvement, implementation of an accepted plan, human health and well-being improvement, and reduction in conflict (Table 4).

5.1.1. Ecological Improvement

[34] Several water quality management documents advocate an approach to management which involves the public and/or stakeholders (EU Water Framework Directive, US Clean Water Act). Therefore, some evaluators have placed great emphasis on the need to evaluate how participation enhances ecological conditions in the watershed. Available data on catchment conditions prior to, during and after the participatory activity often limit evaluators' capacity to identify ecological changes [Bentrup, 2001; Koontz and Thomas, 2006; Leach et al., 2002]. This is compounded by a need for data at an appropriate temporal and spatial scale to detect changes. To overcome these challenges, participant perspectives on catchment improvement has been used as a surrogate for actual improvements. Leach et al. [2002] asked interviewees to score perceived catchment improvements on a scale from −3 (negative impact on catchment) to +3 (positive impact). The data revealed that the majority of the partnerships were perceived to have had a slightly positive impact (+1.1 was the highest score), though five of the partnerships were perceived to have had a negative impact (−0.5 was the lowest score). In a different study, evaluator perspectives on the relative ecological standard of decisions made by numerous participation programs (on a scale of −4 to +4) resulted in an average score +0.4 [Newig and Fritsch, 2009b].

5.1.2. Economic Improvement

[35] There are a number of hypotheses about the costs and benefits of stakeholder participation in development projects and programs and the impact of participation on the performance and outcomes of these projects and programs. These are based on observations, experiences and case studies of participatory development activities over the years [Karl, 2000]. Several authors have described how participants can be included in and potentially improve an economic analysis, for example, by incorporating social costs and benefits [see Birol et al., 2006; Brouwer, 2008; De Marchi et al., 2000; Messner et al., 2006; Wright and Fritsch, 2011]. However, there is surprisingly little work that attempts to evaluate whether the net benefits of running a program with participation exceed the costs of the participation element of the program (for example, through with-without analysis that undertakes to measure benefits and costs only attributable to the project [seeYoung, 2005]). Work that has attempted to evaluate economic improvements tends to separate out the potential benefits from participation and looks specifically at (i) whether strategies selected through participation are more cost-effective than the status quo or likely alternatives [Beierle, 2002; Wright and Fritsch, 2011], (ii) whether the involvement of participants lowers implementation costs (for example, reduces litigation rates and raises compliance) [Newig and Fritsch, 2009a], and (iii) whether participants perceive that the benefits of participation exceed the costs (Table 4). In addition, authors theorize that a strategy identified and implemented through participation should raise more financial capital than the status quo [Plummer and Armitage, 2007] and it should achieve greater socioeconomic and environmental sustainability (reduce the potential for long-term negative economic impacts) [Almansa and Martínez-Paz, 2011; Plummer and Armitage, 2007].

5.1.3. Implementation of an Accepted Plan

[36] This includes the actual implementation of plans or agreements such as restoration projects, changes to land management practices, monitoring and enforcement of designated protection zones or water allocation arrangements [Koontz and Moore Johnson, 2004; Mostert et al., 2007]. A comprehensive evaluation of project implementation shows a strong correlation between watershed partnership age and achievement of one or more implemented projects [Leach et al., 2002] (Table S3). This suggests that several years are needed from the inception of a participation activity until the implementation of specific projects. To avoid inaccurate outcome evaluation, implementation should perhaps only be used as a criterion when a program has been operating for several years. The type of plan implemented is also important. Education programs or restoration projects are noted to be more easily executed than plans that are contentious such as changes in land use [Leach et al., 2002]. The need for actions from people outside the physical or jurisdictional boundaries covered by the participants may also prevent implementation. This would call for an enlargement of the stakeholder group, and reassessment of whether the process is achieving appropriate representation. It is also important to consider that implementation was rejected as an outcome indicator by Innes [1999] because changes to government or funding policies may prevent an agreed plan being implemented.

5.1.4. Human Health and Well-Being

[37] The health literature includes a number of examples of evaluation work related to participation in water resource management. Impacts of participation on health and well-being can be determined through rates of infection, illness or vector prevalence. For example,Toledo et al. [2007]evaluated the impacts of a community-based mosquito eradication program in Santiago de Cuba using outcome-based criteria that included percentage of houses with uncovered water storage containers and number of households with infected containers. The criteria were assessed in two areas where different programs were implemented: (i) a community program focused on mobilizing the community and promoting healthy behavior and (ii) a top-down control program with larvicide and insecticide spraying. Data collected over 3 years showed little difference between the two areas but the evaluators note that the community participation program may be more sustainable and require lower running costs. Another topic of major interest are the benefits that accompany water provision programs, such as higher rates of child immunizations, which appear to be specific to programs that are community-based [Eng et al., 1990] (see Table S3).

5.1.5. Reduction in Conflict

[38] Studies which attempt to evaluate participation in terms of a reduction in conflict tend to concentrate on its capacity to reach consensus [Duram and Brown, 1999; Lubell, 2004b] (Table 4). Considerable work has been conducted on the value and role of consensus building approaches for resource management [see Susskind et al., 1999]. Consensus suggests that a conflictive situation or issue has been overcome. However, Coglianese [1999] has voiced concerns that focus on reaching consensus means that participants aim to reach an agreeable decision rather than a quality decision. He also argues that the benefits from consensus building approaches (such as a reduction in conflict) can be achieved by engaging and deliberating with participants without focusing on reaching consensus. A reduction in the use of legal channels to contest management decisions has also been used as an indicator to evaluate conflict reduction [Coglianese, 1997] (Table S3). Participants may seek legal channels when they perceive that their interests or concerns have not been addressed through a deliberative process [Connick and Innes, 2003]. As legal processes tend to raise expenses, this outcome is closely related to the criteria of reduced implementation costs.

5.2. Selection of Resource Management Outcome Criteria

[39] Outcome criteria are heavily affected by the agenda, interests or requirements of those involved in the evaluation. For this reason, it has been advocated that participants are involved in developing and selecting evaluation criteria [Ferreyra and Beard, 2007]. Ecological or economic criteria are important for researchers aiming to prove or disprove the value of participation according to their field of specialization (such as environmental science or economics; Table 4). Practitioners may be interested in evaluating whether a program is leading to desired resource management outcomes, such as implementation or reduction in conflict between interest groups. Evaluation findings on economic efficiency could potentially be used to assist decision makers on whether to implement or withdraw support to a program.

[40] The literature suggests that resource management outcome evaluations are sometimes able to show that participation improves resource management (Table S3). For example, health and well-being improvements do seem to correlate with more involvement in water and sanitation. Participation programs are broadly associated with ecological improvements (according to participants' perspectives). However, the findings are limited, which draws attention to the difficulty of applying resource management outcomes as criteria with which to evaluate the impacts of participation. The literature suggests that the most robust outcome evaluations are either narrowly focused or draw on large data sets using meta-analysis.

5.2.1. Narrowly Focused Evaluations

[41] These evaluations require clear objectives identified prior to the evaluation, such as comparison of agricultural productivity before and after water user groups are set up, or disease reduction resulting from participation in community groups. Sustainable development includes three pillars, a quest for social equity, economic growth and environmental protection [UN, 2002]. These pillars suggest that any resource management evaluation should incorporate all three aspects, which may bring into question the relevance of evaluating against narrowly focused objectives. This suggests that resource management outcome criteria need to be carefully considered by evaluators and participants, perhaps early in the program, to ensure that the evaluation criteria reflect the range of objectives as accurately as possible.

5.2.2. Meta-analysis

[42] Large data sets have been used to identify specific objectives that may not be easily identified through analysis of one or two detailed case studies. For example, Schultz et al. [2011] analyzed data from 146 nature reserves to show that participation has no negative effect on conservation. Beierle and Konisky [2000] show that participation reduced conflict in more than 50% of the 19 case studies they evaluated (Table S3). Meta-analysis is also effective for more rigorous statistical assessment of potential impacts from participation programs. Collecting large data sets is challenging due to extensive and complex data requirements and the need for an adequate number of suitable case studies from which a representative data set can be drawn [Bamezai, 1996] but shows considerable value to aiding understanding of participation outcomes [Newig and Fritsch, 2009a].

6. Integrating Process, Intermediary Outcome, and Outcome Evaluations

6.1. Do Good Processes Lead to Good Outcomes?

[43] Process evaluations are often built on the assumption that good processes lead to or are equated with desirable participatory outcomes [Abelson et al., 2003; Chilvers, 2009; De Stefano, 2010; Pahl-Wostl et al., 2007; Rowe and Frewer, 2004]. The logic behind this assumption is that high-quality processes lead to willingness to take part [Webler et al., 2003], and that deliberation, negotiation or consensus building between conflicting interest groups require good processes for plans to be developed, accepted and implemented [Newig and Fritsch, 2009b]. Some work has specifically explored whether good processes do lead to good outcomes. Koontz and Moore Johnson [2004] found that greater participant diversity (representativeness) in watershed groups was associated with participant opinions that plans were developed and issues identified and prioritized (agreements reached; Table S1). A positive correlation was found between finding new and creative solutions and the degree of stakeholder interaction, intensity of communication, information flows, fairness and representativeness for 40 case studies of participation in environmental management [Newig and Fritsch, 2009b]. The authors suggest that new and creative solutions require intensive processes with a high degree of stakeholder interaction and communication that are perceived to be fair and legitimate. Innes and Booher [2004] draw on evidence from a number of case studies to show that collaborative processes (which they identify with dialogue, representation and promotion of equal power) build social capital.

[44] It is also important to consider that many process factors are evaluated by considering participant perspectives (Table S1). Participant satisfaction with the process may be affected by participant satisfaction with the outcome [Murdock et al., 2005], and vice versa. A survey designed to assess participant satisfaction with the processes and the outcomes in community groups in the USA revealed strong correlation between process and outcome satisfaction [McKinney and Field, 2008]. Leach and Sabatier [2005a] describe seeing a “halo effect,” where participants in watershed partnerships with high levels of social capital perceived that their partnership had a greater effect on catchment conditions than those in partnerships with lower levels of social capital. To overcome the potential influence of the participation process on participant perception, Weible [2008] conducted surveys prior to the start of a program to identify participants' initial expectations from a program.

[45] Theoretical work suggests that there are also likely to be strong interactions between process factors, intermediary outcomes related to social capital and resource management outcomes [Connick and Innes, 2003]. Beierle and Konisky's [2000]case study meta-analysis found that conflict was highly correlated to trust.Newig and Fritsch's [2009b] work found that trust (an intermediary outcome) and conflict resolution (a resource management outcome) were highly positively correlated with representativeness, fairness and communication and negatively correlated to structured information extraction. Trust is a complex outcome that is likely to not only emerge from appropriate processes, but also determines how processes takes place. Halvorsen's [2006] review of the literature suggested that trust in the operating agency drives participants' decisions to take part in a process, with low trust either stimulating or inhibiting participation. While Yandle et al. [2011] found that high and low levels of trust between fishery stakeholders were associated with lower participation in management efforts, and medium levels of trust led to the highest rate of participation. Hartley and Wood [2005]asked 22 participants to rank a number of evaluation criteria according to their perceived importance. Trust was ranked as the least important by 16 of the participants because they felt that if participation was effective and well-executed trust would be achieved. This is supported byLeach and Pelkey's [2001] analysis of 37 watershed partnerships in the USA that found neutral facilitation, clear process rules and the sharing of information to be precursors to trust.

[46] Layzer [2008] deliberately searched for intermediary outcomes, which included trust, in seven case studies of collaborative environmental management in the USA because she considered these to explain and enable the achievement of resource management outcomes. The development of trust and communication through regular meetings between fisheries managers and water allocation operators was attributed to enabling water allocations to be more ecologically favorable [Lejano and Ingram, 2009]. Additionally, Uphoff and Wijayaratna [2000] showed that high levels of social capital led to water use efficiency and improved crop yields in Sri Lanka. Similarly, farmers' interaction with agricultural agencies and involvement in management activities such as attending meetings and training classes (sharing information and knowledge) was found to correlate to farmers' decisions to apply best agricultural management plans (so reducing the ecological impact of agriculture) [Lubell and Fulton, 2008]. These findings all suggest that some intermediary outcomes may not only be indicative of good process, but may also be indicative of future resource management outcomes.

6.2. Arguments for Intermediary Outcome Evaluation in the Water Sector

[47] A number of different intermediary outcomes have been described which could all form part of an evaluation (Table 3). Many of these rarely form part of a programs' original objectives and are therefore likely to be overlooked if an objective-based evaluation strategy is used. Resource management outcome evaluation requires that the desirable outcome can be clearly defined and agreed upon. This is often not the case in water management where multiple interests and objectives are present [Conley and Moote, 2003]. For example, Junker et al. [2007] analyzed river restoration projects in Switzerland to show that local public prioritize quality of life and recreation but these preferences are not shared by the environmental, farming, industrial and land owner stakeholder groups who are involved in deliberation activities. Objectives for resource management evaluation tend to be highly influenced by the goals of one interest group. Stufflebeam and Shinkfield [2007, p. 8] argue that “evaluators should avoid judging a program as successful because it achieves its own objectives. The objectives might well be corrupt, dysfunctional, unimportant, not orientated to the needs of the intended beneficiaries, or mainly reflect the motives or other conflicts of interest of those in charge of the program.” Additionally, Tuler and Webler's [2010]research has shown that participants hold different positions on the objectives of a participatory approach. Their work showed that to some participants the objective is to produce and implement a strategy based on best available science combined with stakeholder information. Others see the objective as being to share out decision-making power between participants to find and implement a strategy that meets the needs of the community. While these may not always be incompatible objectives, they do highlight that different people enter into a program with different ambitions, and this will lead to different ideas as to the most suitable criteria to assess whether an outcome has been achieved. This suggests that goal free evaluation, which focuses on finding out what a program is actually achieving rather than what it is supposed to be achieving [Patton, 2002; Scriven, 1991], is particularly suitable for identifying outcomes from participation in water management.

[48] Resource management outcome evaluation is also challenging in the water sector due to the long time scales over which resource management outcomes emerge, the multiple geographical, temporal and institutional scales over which participation activities take place, and nonstationarity of environmental conditions which adds to the complexity of isolating outcomes from their drivers [Ferreyra and Beard, 2007; Koontz and Thomas, 2006]. Leach et al. [2002]evaluated 44 watershed partnerships in California using two intermediary outcome criteria (perceived effect of the partnership on social capital, extent of agreement reached) and four resource management outcome criteria (perceived effect of the partnership on specific problems in the watershed, implementation of restoration projects, conducting monitoring projects, running education and outreach projects). Correlation analysis of the data showed that the criteria were poorly correlated (none of the bivariant correlations exceeded 0.72) therefore each criterion was considered to contribute unique information to the evaluation. Multidimensional scaling analysis of the bivariant correlations showed that the variance between perceived effect on social capital and perceived effect on the watershed could be accounted for by considering a temporal dimension. The authors suggest that these criteria are separate because changes in social capital are short- to medium-term goals, while changes in the watershed are medium- to longer-term goals. This work demonstrates how intermediary and resource management outcomes emerge over different time scales and highlights the potential of intermediary outcomes for short- to medium-term evaluations.

[49] Intermediary outcomes also appear to be measurable and potentially comparable at a variety of geographical and institutional scales. For example, trust can be evaluated locally between members of a community group [Sultana and Thompson, 2004] and regionally or nationally between agencies and citizens [Murdock et al., 2005]. The effect of environmental change on intermediary outcomes is likely to be complex. For example, water stress has been shown to strengthen social capital [Uphoff and Wijayaratna, 2000] and stimulate institutional change [Dinar et al., 2007]. Further work that explores the interactions between processes, intermediary outcomes and resource management outcomes and identifies the factors that turn good processes into good outcomes is essential. One critical challenge is to include the impact of context.

6.3. An Additional Aspect: Context Evaluation

[50] Several workers have drawn attention to the importance of context in determining the processes, intermediary outcomes and resource management outcomes [Newig and Fritsch, 2009b; Ison and Watson, 2007; Steyaert and Jiggins, 2007]. Context attributes have been described as being those which are outside the control of the participants, such as scientific understanding of the issues, geographical complexity and whether participants are confident the process will be beneficial [Beierle and Konisky, 2000]. Other examples include history of the situation, the existence of vested interests, institutions [Plummer and Armitage, 2007] and institutional structure. Ison and Watson [2007] evaluated participation in the development of the Scottish Water Environment and Water Services Act through observation, interviews and document review. Their evaluation covered the context of the situation (institutions and legislative process), the processes which took place (facilitation and representation) and the intermediary outcomes that developed (relational capital and learning). They found that environmental NGO network representatives brought resources, time and energy into the process that were critical for developing and passing the Bill and recent institutional changes (Scottish devolution and the European Water Framework Directive) were vital for enabling this involvement. Governance structure is likely to be particularly important. Empirical work has shown that a history of strong state involvement in water management reduces farmers' willingness to engage in water users associations [Faysse et al., 2010; Theesfeld, 2011]. In contrast, a case study from the Netherlands shows that stakeholders embrace participatory programs more enthusiastically because the Dutch decentralized system gives them greater confidence that the outcomes from their negotiations will be implemented [Jiggins et al., 2007].

[51] To include context, evaluation literature often includes a description of the setting and background for the participation program. Lamers et al. [2010]evaluated the impact of initial conditions by surveying stakeholders engaged in developing a water management plan in the Netherlands. Their evaluation used questionnaires, interviews and workshops which took place before, during and after the program. They used process-based evaluation criteria based onRowe and Frewer [2000] and context and outcome criteria from Beierle and Cayford [2002]. Context was assessed by asking participants the extent to which they agreed with a series of statements on, for example, conflict, relationships, trust and openness. Outcomes were determined by questioning participants on the extent to which they felt their personal and collective goals had been reached, the intensity of conflict at the start and at the end of the process, the degree of trust in the water board, and whether participants felt they had learned anything from the process. Findings showed that although there were conflicting interests at the start, the process overcame them and led to a water management plan that considered different stakeholder interests and was approved by the water board. The evaluators associated the ability to overcome difficult contextual conditions with a well-designed and effective participation process.

[52] Beierle and Cayford [2002] apply an evaluation framework based on context, process and outcomes (achievement of social goals). Under context they describe the type of issue being addressed, the preexisting relationships between the participants, the level of trust in the government and the institutional setting (lead agencies power, position and involvement in the process). Their analysis of 239 cases of participation in environmental decision making found that process factors, such as participants' motivation and quality of deliberation, rather than context factors determined the extent to which participation achieves selected “social goals” defined by Beierle [1999]. Social goals included integrating public values into decisions, building trust, and educating and informing the public (intermediary outcomes) and improving the substantive quality of decisions and resolving conflict (resource management outcomes). Interestingly, context factors seemed to be more significant than process factors for achieving resource management outcomes. This finding is supported by Newig and Fritsch's [2009b]case study analysis that found that context variables, particularly degree of issue complexity, were more significantly correlated with environmental outcomes than process factors. Additionally, several evaluators have attributed process factors with the California Bay-Delta Program's achievement of intermediary outcomes [Connick and Innes, 2003; Fuller, 2009; Lejano and Ingram, 2009], but Hanemann and Dyckman [2009] argue that contextual factors of vague and contentious water rights and reduced political support inhibited the program from achieving a fundamental change in water allocation strategy (a resource management outcome).

[53] Context appears to be a critical factor that should be integrated into any evaluation. This is, and is likely to remain challenging as the process itself will change the context in which the process is operating. Observation and description of initial starting conditions and changes throughout the process are likely to be essential.

7. Conclusions

[54] This review has demonstrated that approaches for evaluating participation can be grouped according to whether they relate to process, intermediary outcomes or resource management outcomes. Evaluation criteria relating to each of these groups have been extensively developed by evaluators over recent decades. Process evaluation forms a major component of the literature. It is critical to ensure that programs are developed that meet the requirements of the participants and determine whether participants are provided with access to, and equity in the decision-making process. This review has identified several process characteristics that may be particularly important for the water sector such as cost-effectiveness, facilitation, legitimacy and knowledge inclusion. This review also suggests that conflicts of interest between criteria may occur in water resource management. Representation may be relinquished to achieve an environment where participants can speak freely and openly. This suggests it may be beneficial to consider forming a hierarchy of process criteria. Any hierarchy will be driven by values, for example, whether facilitation is more important than representation. Therefore participant involvement in designing and shaping the evaluation is essential, particularly in situations where contradictions between process criteria are likely to emerge.

[55] This review is based on the published literature and strongly reflects research findings, the majority of which view participation positively. While only a few studies show resource management benefits from participation, no studies have been identified that have proved a negative link between participation and water management. Many uncertainties remain and it is not yet possible to draw clear and concrete conclusions on the role of participation. Further resource management evaluation is desirable to address these challenges, but difficult for a number of reasons. First, resource changes such as water quality improvements or a rise in biodiversity tend to emerge over medium to long time scales while evaluation typically takes place during or immediately after a participation program. Second, it can be difficult to directly relate resource management changes to an implemented participation program or project. Meta-analyses that use broad and long-range data sets have been shown to offer a good approach for resource management evaluation. Evaluations that are shaped by a focused objective have also led to interesting findings. However, in water resource management, multiple interests are often present, and participants hold many different, and perhaps unknown, objectives. This challenges the rationale for objective-based evaluation and emphasizes the importance of incorporating participants into the design of any evaluation framework.

[56] Intermediary outcomes have been successfully evaluated by a number of authors and this work suggests that they should play a greater role in evaluating participation. Intermediary outcome evaluation reduces objective bias, and describes the actual achievements or side benefits that emerge from a program. They appear to become apparent over a shorter time scale than resource management outcomes making them particularly suitable when evaluating programs or projects that have recently been initiated. Importantly, intermediary outcomes may be indicative of both good processes and future resource management outcomes. More work is essential to understand how processes lead to intermediary outcomes, and the subsequent role of intermediary outcomes in achieving resource management outcomes. By addressing these research needs through consistent evaluation, a deeper understanding will emerge of how participation programs function and conclusions will be reached on their potential role in water resource management.


[57] The authors would like to acknowledge financial support provided by the Austrian Science Funds (FWF) as part of the Vienna Doctoral Programme on Water Resource Systems (DK-plus W1219-N22). We would particularly like to thank the Editor and several anonymous reviewers, who provided extremely valuable suggestions on earlier drafts of this paper.