Towards evaluation criteria in participatory flood risk management

Flood risk consists of complex and dynamic problems, whose management calls for innovative ways of engaging with a wide range of local stakeholders, many of whom lack the technical expertise to engage with traditional flood risk management practices. Participatory approaches offer potential for involving these stakeholders in decision‐making, yet limited advice is available to users in choosing which techniques to employ and what they might expect them to deliver. Assessing the effectiveness of participatory approaches in local flood risk management is a critical step towards better understanding how community resilience is built. This article presents a framework for evaluating participatory approaches to flood risk management that covers four evaluation elements (context, process, substantive, and social outcomes). Practical success criteria are provided for evaluation, with references indicating where further advice and guidance can be sought. Criteria are tailored to the requirements of flood risk management, and aim to be sufficiently flexible for the framework to be easily transferable.


| INTRODUCTION
Shifts in the governance of flood risk in England, from statecentred management towards "greater reliance on horizontal, hybrid, and associated forms of government" (Hill & Lynn, 2005, p. 173) have paralleled increased emphasis on stakeholder participation.
Following the 2007 Pitt Review into UK flooding (Pitt, 2008), several policy changes were introduced by the Flood and Water Management Act (2010), strengthening and clarifying the responsibilities of local authorities as "Lead Local Flood Authorities," tasking them with the development of flood risk management strategy at the local level (Twigger-Ross et al., 2014). These responsibilities, alongside a new partnership funding approach, which shares the costs of flood defences between national and local funding sources (DEFRA, 2011), necessitates greater public participation in flood risk management.
Public participation had been gaining prominence in flood risk management following recognition that pure structural engineering interventions and short-term response strategies to crises are unsatisfactory (cf. European Commission, 2007). Attention is being given to long-term, nonstructural mitigation strategies including a wide range of interventions involving the public. Increased public involvement in risk management raises both risk awareness and event preparedness, demonstrated by the appropriate actions taken by the public before and during flood events; second, the local population may provide knowledge that is fruitful for risk prevention efforts; third, the involvement of the public legitimises processes and enhances the acceptance of prevention measures; and, finally, the coping and adaptive capacity 1. The Collaboration Challenge: flood risk management should be conducted locally and developed in collaboration with local stakeholders, where those at risk become active participants in risk governance; and 2. The Capacity-Building Challenge: capacity must be built at the local level, acknowledging new sources of knowledge and expertise, particularly with regard to nonstructural responses, the management of surface water, flood recovery, and insurance.
Although participatory approaches are often viewed as more legitimate, democratic, and inclusive, several studies warn against taking this romanticised view (cf. Haughton, Bankoff, & Coultard, 2015). There exists a need to ensure that the approach is not used as a political tool to push the agendas of the powerful, who attempt to legitimise their arguments by using participation as a front for activities that are in reality far short of participatory. Given intentions are good, the need remains to understand the trade-offs between the considerable resources that participation may require and its potential benefits. Many studies report on the benefits of participation from a normative perspective, while their actual gains (from a substantive and instrumental perspective) may be those most valuable to flood risk management. This implies a need for holistic evaluation of participatory processes and their outcomes.
This article engages with theoretical literature to present a framework for the holistic evaluation of participation in flood risk management. Traditionally, flood risk management is a field which employs complex techniques to model and understand the spatial distribution of risk. To contribute to local flood risk management, participatory approaches need to employ techniques which are accessible to stakeholders with limited or no technical expertise, while remaining sufficiently robust to capture complexity. Many participatory modelling techniques (e.g., Bayesian networks, system dynamics, fuzzy cognitive mapping) have yet to be widely applied in flood risk management; thus a flexible evaluation framework will additionally support the selection of the most appropriate technique for a given context. The objectives of this paper are twofold: 1. outline a framework for evaluating participation in flood risk management by exploring the dimensions and determinants of effectiveness; and 2. populate that framework with practical criteria that allow users to design and undertake effective participation in a range of different contexts.
The framework proposes an evaluation of four connected elements ( Figure 2).
Process criteria, while insufficient in isolation, facilitate ongoing improvement of participatory processes through constructive feedback. The aim is to assess characteristics that are common to all participatory processes, which collectively impact the efficacy of a process to achieve the collaboration and capacity-building challenges identified earlier. Context criteria indicate an area's predisposition to participation, guiding technique selection and implementation, such that acceptance and engagement potential are maximised. Substantive outcome criteria assess a process against user objectives and expectations, which are identified both before a process and immediately afterwards. Finally, social outcome criteria assess the ability of a process to enhance community flood resilience. As many social effects take time, this is achieved by exploring aspects of the participatory process that directly promote three components of social capacity: knowledge exchange, stakeholder motivation to proactively reduce flood risk, and stakeholder networking (Buchecker, Menzel, & Home, 2013). Taken in combination these components indicate the ability of a community to prepare for, resist the impact of, respond to, and recover from a flood event.
Throughout this article, the word participant refers to all persons taking part in the participatory process; the word facilitator refers to those delivering participatory activities; and the word user refers to those that will use outcomes of the process (i.e., those driving the process and undertaking  Investigation into individual and institutional response to flooding events, with a particular focus on managing flood risk by enhancing the social factors related to resilience Cashman (2009Cashman ( , 2011 Gloucester, Gloucestershire Exploring how local knowledges were collated, shared, and utilised to build local resilience; using stakeholder interviews taken before and after the summer 2007 floods McEwen and Jones (2012) evaluation), who may or may not also be participants. In some cases the user may be a single easily identified organisation (e.g., the Environment Agency) and in others there may be multiple users who are interested in different outcomes.

| EVALUATING THE EFFECTIVENESS OF PARTICIPATORY PROCESSES
While early attention was given to the benefits of participation and conceptual frameworks for public involvement (Chambers, 1994;Pretty, 1995); recent research has shifted focus to the designing of more effective, legitimate participatory processes (Abelson et al., 2003;Rowe & Frewer, 2000).

| Dimensions of effectiveness
Effective participatory processes are essential to building community flood resilience. Young (1994) identifies various dimensions of effectiveness, which we draw on here to frame our evaluation framework for participation in flood risk management: 1. Effectiveness as problem solving; 2. Effectiveness as goal attainment; 3. Behavioural effectiveness; 4. Constitutive effectiveness.
The first dimension, effectiveness as problem solving, concerns the extent to which processes solve the problems that motivated them to be created in the first place, such as changes to legislation that prevent new developments in the 1:100 year floodplain. The second dimension, effectiveness as goal attainment, measures the extent to which users goals are achieved over time, such as a reduction in the number of properties at risk from a 1:100 year flood event.
In each process, user goals will vary according to local needs and priorities, and this dimension does not assume that by meeting all user goals, the broader challenges of participation in flood risk management (Section 1) will be addressed. Both of these dimensions map onto the substantive outcomes evaluation element of this framework (Figure 3), where the aim is to assess the extent to which the process has achieved user goals, and address the Collaboration Challenge.
The third dimension, behavioural effectiveness, looks at whether a process causes one or more of its participants to change their behaviour as a result of participating, such as the development of an emergency plan to prioritise actions before and during a flood event; while the fourth, constitutive effectiveness, concerns the extent to which processes give rise to increased social practice on the part of its participants, such as an increased feeling of responsibility in the population to actively manage their own flood risk. Both of these dimensions can be mapped onto the social outcomes evaluation element of the framework (Figure 3), and address the Capacity-Building Challenge.

| Determinants of effectiveness
To design effective participatory processes, one must identify the determinants of effectiveness. Young (1994) argues that variables driving effectiveness can be endogenous (attributes or properties of the process itself ) or exogenous (driving forces that influence the course of the process and its outcomes). Endogenous variables can be manipulated by facilitators, and might include the levels of representation of local flood risk stakeholders and the accessibility of participatory activities. These variables map onto the process  Exogenous variables are usually outside of conscious control within the process lifespan, but exert an influence over the effectiveness of both the process and its outcomes (National Research Council, 1996). They might include the complexity of local flooding issues and the existing level of consensus on how these should be managed. Facilitators understanding influential exogenous variables are clearly at an advantage when designing processes that are welladapted to the context in which they are expected to be effective. These variables are explored using context evaluation, which supports the pragmatic tasks of technique selection, objective identification, and stakeholder analysis; by allowing the user to assess the potential for participatory modelling to be successfully employed. It is conducted before the participatory elements of the process, such that the process can be adapted as required.
By framing participation in terms of effectiveness one can identify what flood risk management requires of each evaluation element. However, to translate this understanding into an effective process in practice, criteria are required that are bespoke to flood risk management, yet sufficiently flexible for use in a range of contexts.

| SELECTING EVALUATION CRITERIA
Both theory-based and user-based evaluations are used to select evaluation criteria (Chess, 2000).
A theory-based approach is used for context, process, and social outcomes. This approach develops criteria from the literature, and applies them universally to participatory efforts (cf. Fiorino, 1990;Webler, 1995). By reviewing participation literature, criteria grounded in best practice can be standardised. This is particularly important in flood risk management, where many participatory modelling techniques have not been widely applied (Voinov & Bousquet, 2010). In order to compare and contrast different techniques, standard process evaluation criteria are suggested. These assess the desirable characteristics of the process as opposed to the efficacy of the technique used. Similarly, theory-based evaluation criteria are used for both context and social outcomes, where the literature on community resilience enables systematic evaluation that captures contextual variability.
A user-based approach is adopted for substantive outcomes. Users develop their own criteria, allowing different users to have different goals. Permitting users to generate their own criteria provides insight into what the flood risk community are hoping to gain from adopting a participatory approach. User-based evaluation adds credibility and usefulness to the overall evaluation by helping to ensure the diverse range of user needs are captured (Greene, 1987;Syme & Sadler, 1994).

| CRITERIA FOR CONTEXT EVALUATION
While context criteria evaluate exogenous variables largely outside the control of facilitators (Figure 3), these often have a significant impact upon the success or failure of a process. The importance of context on the effectiveness of participatory processes is reinforced in the National Research Council's Understanding Risk report which states: "…results depend less on the tool and more on its users and the setting in which it is used.
[…] The history of an issue, level of conflict, scientific data, and existing power dynamics may also influence the outcome as much as the method" (National Research Council, 1996, p. 96).
The importance of context is exemplified with reference to Thirlby, North Yorkshire (Twigger-Ross et al., There is consensus on how the issue should be managed 2011). Thirlby is a small rural community (population 110), which experienced flash flooding in June 2005 after a period of intense rainfall. Water levels rose quickly and velocities were high, making roads impassable, and leaving several residents cut off from external aid for several hours. Local understanding of flood risk prior to 2005 had been poor, as previous floods were very minor, leading several residents to view the event as a "one-off." Since the event, efforts by the Environment Agency to create an emergency plan to improve efficiency have met little support, with the majority of residents feeling that the community coped effectively. The history of the settlement as a relatively isolated community resulted in a culture where the residents see themselves as self-sufficient and resilient, and although there is not a lack of trust in outside agencies, they are seen as slow and inefficient. In terms of social complexity, the bonding networks are very strong within the community, but the linking networks with external agencies are relatively weak (see Figure 6). Thoughtful and continued engagement with community leaders will allow everyone involved to learn from the lessons of the 2005 flood, and create a community which is more amenable to future participation.

| CRITERIA FOR PROCESS EVALUATION
Process evaluation is aimed at iterative improvement, providing feedback to facilitators during participatory activities (Chess, 2000;Tuler & Webler, 1995). It may be conducted using participant surveys; event or activity logs; key participant interviews; focus groups; meeting observation (including debriefing sessions); and reviewing of key documents (Beierle, 1999). In this framework, criteria are drawn from examples of good practice in the participation literature. Process criteria are derived using Fiorino's four principles of effective participation ( Figure 4 and Table 3) (Fiorino, 1990), which are particularly suited to flood risk management, where governance and decision-making is being devolved to local government, and community resilience is essential (cf. Steinführer et al., 2009; also Section 6). While these principles are strongly linked to arguments for increasing stakeholder participation, they fall short of evaluating the process design qualities that support both recruitment and retention of participants. Including quality as a fifth criterion helps to ensure that goals are defined early, clearly communicated and measureable; and highlights the importance of managing participant expectations. Recent findings from participatory research in the Great Lakes region, shows us that without measurable progress towards clear goals and objectives, stakeholders can become disillusioned with the process, and may lose motivation or withdraw their support (Hartig, Zarull, Heidtke, & Shah, 2010).

| Accessibility
An accessible process is one where participants can understand and use all the information available to them, enabling them to participate equally and in an informed manner (Webler, Tuler, & Krueger, 2001). When processes involve elements of participatory modelling, any activities conducted "behind the scenes" should be communicated to the participants in a clear and timely fashion. Without transparency, there is a risk that models become a black box, where the assumptions and shortcomings of the model are neither understood nor appreciated by all participants (Prell et   The participants are representative of the affected community and the full range of views All participants were given opportunity to make a substantive contribution

Responsiveness
The participatory process was flexible to change, with the agenda and activities shaped by the needs and goals of the participants Participatory tools were chosen according to local objectives, resources and available data Quality Participatory activities were facilitated in a professional manner The participants knew what was expected of them and what they could expect to gain from participating The process had clear purpose, objectives, and direction 2007). If participants are able to contribute to and challenge models as they are being built, they are more likely to understand the assumptions of those models, the extent to which they can be used reliably, and the level of uncertainty in the results (Voinov & Gaddis, 2008). It is widely accepted that the technical and statistical language used in flood risk, particularly around uncertainty and likelihood, is a barrier to engagement (Cotton et al., 2015). This is an issue that is currently being addressed by the Environment Agency (the national regulator in England) who are seeking to avoid such language in their training materials, instead opting for a focus on visuals, stories, qualitative language and nonspecific indicators of severity (Environment Agency, 2012). Ensuring that language and activities are accessible facilitates participation by individuals from socioeconomic groups with low levels of education, which have been identified as among those most vulnerable to flooding (Burningham, Fielding, & Thrush, 2008;Twigger-Ross et al., 2014). A study into public dialogue in Newcastle-upon-Tyne, Tyne and Wear further revealed that the translation of language and statistics into "impact information" was required if participatory activities were going to be meaningful to those living in flood risk areas (Cotton et al., 2015).

| Deliberation
A deliberative process is measured by the quality of communication between participants (Beierle & Konisky, 2000) striking a balance between providing all with the opportunity to have their views considered, and arriving at agreement on goals and actions (Halvorsen, 2001). Creating a space for deliberation and dialogue affords a process greater legitimacy, while supporting the achievement of other process criteria (quality and responsiveness) (Webler et al., 2001).
Until recently, flood risk management in the United Kingdom rarely involved the public, drawing instead on expertise from local government, regulatory agencies, and commercial consultancies (Haughton et al., 2015).

| Representation
A representative process requires the participant group to be a microcosm of the community at risk, such that the full range of flood risk knowledges can be gathered, evaluated, and used to support recommendations (Beierle, 1999;Beierle & Konisky, 2000;Webler et al., 2001). It can be argued that those spending their lives interacting with a system will have privileged knowledge of physical and social systems (cf. Wynne, 1996), including flow paths taken by water in the early stages of a flood event; choke-points on local watercourses where debris accumulates and causes blockages; and the spatial distribution of highly vulnerable members of the community (Tapsell, Burton, Oakes, & Parker, 2005). This local knowledge allows them to contribute the critical fine detail to broader strategies developed by experts (Twigger-Ross et al., 2014), and provide an insight into how decisions are made locally during a flood event (O"Sullivan et al., 2012).
A dichotomy between local and expert knowledge is of limited usefulness, and risks the privileging of one knowledge type over another (see Somerset Levels example in Section 6.2.1). Recent research argues for "hybrid knowledge formations" (Haughton et al., 2015) where different knowledges are viewed as complementary, and processes are encouraged to strike a balance between expert practitioners and local stakeholders (Smith & Wales, 1999). Emergent practice shows the coproduction of flood knowledge(s) is becoming more widespread (Haughton et al., 2015), led by several high-profile attempts to integrate scientific and local perspectives Lane et al., 2011;Ryedale Flood Research Group, 2008).
The localism agenda in the United Kingdom has started to influence flood risk policy in England and Wales, with a focus on the sharing of responsibilities for flood risk management among newly created Lead Local Flood Authorities, the introduction of a partnership approach to funding, and a reemphasis on community engagement and local action (Nye et al., 2011;Thaler & Priest, 2014;Twigger-Ross et al., 2014). Cashman's (2009Cashman's ( , 2011 work in Bradford, United Kingdom (which flooded in both 2000 and 2003) highlights many of the issues caused by the dispersal of responsibilities for flood risk management between a multiplicity of agencies; including, inter alia, the isolation of key actors and the creation of institutional barriers to participation. Identifying and recruiting a representative group of stakeholders can help to clarify the roles and responsibilities of different agencies, and facilitate shared dialogue and networking (Cotton et al., 2015;Twigger-Ross et al., 2014).

| Responsiveness
Lacking specific guidance on how processes should be structured, those managing flood risk often turn to professional consultancies for advice (Haughton et al., 2015). This introduces the risk that experts arrive with a preferred methodology that they then shape to fit; a mentality summed up as "when you have a hammer, everything looks like a nail" (Prell et al., 2007). This top-down, one-size-fits-all approach ignores the local context and the range of risk perceptions that exist in a community. Voinov and Gaddis (2008) propose general guidelines for responsive process design, arguing that the structure should be set by participant goals, available data, and local time and resource constraints; thus emphasising the importance of context (Section 4). Often simple outcomes that can be easily communicated (see Section 5.1) are more useful than complex models with limited applicability, high set-up costs, and less available data for calibration and validation (Voinov & Bousquet, 2010). In their Pickering, North Yorkshire study, Lane et al. (2011, p. 24) were careful to offer participants "an opportunity to make something together", without being specific about what the outcomes would be. This purposeful vagueness emphasised that the process was designed to coproduce knowledge, shifting the burden from trying to make an off-the-shelf model work, to developing a model directly suited to the local context. In terms of its substantive outcomes, the study developed a "collective competence" (Lane et al., 2011, p. 32) within the participatory group, such that members could actively reengage with the flood risk decisionmaking process, which had previously arrived at an impasse between agencies and local stakeholders.

| Quality
Given the time commitments required by participation, it is important to evaluate the features of a process (e.g., structure, leadership, and organisation) that maximise satisfaction, supporting the recruitment and retention of a motivated group of participants (Hartig et al., 2010). While the responsiveness of a process is important (see Section 5.4), participant expectations need to remain realistic (Webler et al., 2001). This is accomplished through clear direction, goals and milestones, which manage participant expectations: both what is expected of them, and what they can expect from participating (Barreteau, Bots, & Daniell, 2010;Chess & Purcell, 1999). Clear expectations help develop the trust that is fundamental to facilitating continued use of any outputs after the participatory process (Jakeman, Letcher, & Norton, 2006;Voinov & Bousquet, 2010).
Ideally, participants should be involved from the beginning of the process to help define a common vision for managing flood risk, providing input into the process aims, scope, and methodology (Chess & Purcell, 1999;Hartig et al., 2010). Building a shared vision can help understand the complexity of the problem and lead to the emergence of innovative solutions (Ashley et al., 2012); improve the usefulness and credibility of the resulting outcomes; and foster ownership of local flood risk management interventions (Lachapelle & McCool, 2005;Voinov & Gaddis, 2008). This is exemplified in the case of Appleby-in-Westmorland, Cumbria which was supported by the Environment Agency in the development of a flood action plan, including a procedure for distributing sandbags in a flood event (Harries, 2010). By working with, rather than imposing themselves on the community (which already considered itself highly resilient), the two parties were able to overcome previous distrust.

| CRITERIA FOR OUTCOME EVALUATION
The outcomes of participatory processes include both coproduced substantive outcomes and longer-term social outcomes. Substantive outcomes include conceptual models of the flood risk system; recommendations on which interventions residents Interacting social capacities effecting a change in resilience (modified after Cutter et al., 2008;DFID, 2011;Buchecker et al., 2013) wish to explore through traditional modelling methods; identification of misconceptions or knowledge gaps; and consensus around what individuals and the community can do to help (cf. Chess & Purcell, 1999). Longer-term social outcomes are those that build social capacity (knowledge exchange, motivation to proactively reduce flood risk, and networking between stakeholders), acting as an indicator of community resilience, and its potential to be enhanced (Höppner, Buchecker, & Bründl, 2010;Kuhlicke et al., 2011;Pahl-Wostl et al., 2007). Several studies have shown that those responding favourably to a participatory process, may not respond favourably to outcomes (cf. Mazmanian & Nienaber, 1979), highlighting the need for evaluation of both process and outcome criteria, including longer-term influences of participatory efforts on social capacity, appreciating that "exploring only immediately apparent programmatic outcomes may be short-sighted" (Chess & Purcell, 1999, p. 2691.

| Substantive outcomes
While applying normative criteria to process and social outcome evaluations make studies easier to compare and improve upon, it is important that evaluation extends to those elements and results that have salience to those who are responsible for using the outcomes of the process (Chess, 2000). Outcome criteria therefore vary depending on culture, environmental problem, social, and political history, and other contextspecific factors (Chess & Purcell, 1999).
Understanding and monitoring the achievement of user goals can support the selection of the most appropriate techniques for achieving them. It also provides critical information about how closely goals are shared amongst stakeholders; identifying potential areas of disagreement and conflict, as well as any common ground that can be used to build trust. It is inevitable that user goals will be continually adjusted in response to a realisation of what the technique can realistically achieve given time, resource, data, expertise, and other constraints. A strong evaluation of substantive outcomes will give participants the opportunity to comment on any unexpected outcomes, as well as providing information on any shortcomings.

| Social outcomes
Increasing frequency and severity of flooding in the United Kingdom has led to concern surrounding societal exposure and vulnerability (Brown & Damery, 2002). Where much effort has been focussed on understanding current and future trends in hazard exposure, less has been spent on addressing the factors that make people more or less vulnerable to losses in their well-being (Lindley et al., 2011). Vulnerability is defined here as the "pre-event, inherent characteristics or qualities of social systems that create the potential for harm" (Cutter et al., 2008) and arises from an individual's lack of resistance (ability to counteract the immediate effects of flooding and not be adversely affected), and/or resilience (capacity to function, recover and adapt following a flooding event).
Although the social factors affecting vulnerability are well documented (cf. Morrow, 1999), vulnerability is primarily addressed using resistance strategies, which are most appropriate for hazards that are easily predictable and occur with some frequency, and unlikely to be fully effective against unprecedented or surprise events (Longstaff, 2005). In these events, such as flooding from intense rainfall, it is generally accepted that the system will undergo a period of transient distress, followed by a return to pre-event functioning (Flynn, 1994). Resilience strategies are slowly reflecting this thinking, advocating adaptation (not mere stability) in response to change (see Figure 5), thus ensuring that any pre-  existing vulnerabilities are not reproduced through the recovery process (Norris, Stevens, Pfefferbaum, Wyche, & Pfefferbaum, 2008;Twigger-Ross et al., 2014). It is particularly important in flood risk that we consider how participation might be used to build community resilience throughout the flood event cycle (see Figure 5). We are also interested in whether resilience is more or less likely to increase in the future as a result of participation. It is proposed that this be estimated by evaluating social capacity, defined as "the features of social organisation, such as networks, norms, and trust that facilitate coordination and cooperation for mutual benefit" (Putnam, 1993, p. 35), which are built through an iterative long-term process of re-discovering, enhancing and developing community resources and abilities (Kuhlicke et al., 2011;Wenger, 1999). We suggest three interacting social capacities are evaluated: knowledge exchange, motivation to proactively reduce flood risk, and networking between stakeholders ( Figure 5) (Buchecker et al., 2013;cf. Norris et al., 2008). By evaluating the form and quantity of social capital in a community, conclusions can be drawn about the existing level of community resilience, and the potential for it to increase over time.
Building resilience is a primary goal of increased participation and often the justification for opening up the decision-making process to local stakeholders (cf. Beierle, 1999). The aim is for participatory efforts to foster an environment in which social capacities are maximised, such that when faced with challenge or shock, as in the case of a flood event, social capital can be utilised by the community to reduce exposure and vulnerability. Through a review of the social capacity literature, success statements are offered as indicators that each of the three social capacities (Buchecker et al., 2013) are present and/or being developed (Table 4). These statements are designed to be used before and after a process, to assess any change as a result of participation.

| Knowledge capacity
Flood risk management is increasingly moving away from the primacy of scientific knowledge and the "technical fix" for managing flood risk (Brown & Damery, 2002); appreciating the tacit knowledge held by local stakeholders (Berkes, Colding, & Folke, 2000;Folke, Colding, & Berkes, 2003). To facilitate this shift, the role of participation changes from knowledge delivery to knowledge coproduction (Folke et al., 2003), a change that requires participation to facilitate the unlocking of knowledge capacity at individual and community levels (Höppner et al., 2010).
The consequences of failing to understand public perception of flood risk were made clear in the flooding of the Somerset Levels (a large, flat area of reclaimed agricultural land) in winter 2013. During the event, a dichotomy emerged between local and expert knowledge, centred on whether the flooding has been exacerbated by a reduced drain clearance regime imposed by national agencies. Management of flood risk was simplified by the media to the single question of "to dredge or not to dredge?" (Fitzpatrick, 2014), and pressure was placed on central government to promise renewed drain clearance despite lack of clear scientific evidence on its effectiveness (Fitzpatrick, 2014;Haughton et al., 2015). Examples such as this support the argument that privileging of one type of flood risk knowledge over another leads to poor policy decisions (Lidskog, 2008).
Novel approaches in Pickering, North Yorkshire are introducing ways local stakeholders can engage in the coproduction of flood risk knowledge, drawn from both local and expert sources; and are at the leading edge of creating and utilising a "hybrid knowledge" of flood risk (Haughton et al., 2015;Landström et al., 2011;Lane et al., 2011;Ryedale Flood Research Group, 2008).

| Motivation capacity
Building social capacity requires members of the community to engage with activities and initiatives designed to reduce flood risk. In the event of flooding, motivated individuals create an "informed, capable, critical mass" and are those most likely to act as community champions or peer educators (Deeming, 2008;O'Neill, 2004). While motivation can be attributed to a range of psychological and social drivers (cf. Miles, Sullivan, & Kuo, 1998), the highest levels of motivation capacity are found in communities where sharing knowledge and supporting one another are viewed as moral obligations.
The benefits and barriers to participating in community activities aimed at reducing flood risk are exemplified in the case of Southwell, Nottinghamshire. Following major flooding in summer 2013, a local action group formed out of residents' motivations to reduce flood risk in the town. The group quickly affiliated itself to the National Flood Forum, a national charity set up to support and guide the activities of local flood action groups across the United Kingdom, becoming Southwell Flood Forum (SFF). SFF coordinates community activities such as clearing local watercourses and managing emergency road closure schemes, as well as liaising with local flood risk agencies. Motivational barriers to participating in the activities of the SFF include: persisting attitudes among some residents that responsibility for managing flood risk lies with local and national agencies; a lack of appreciation of the benefits of their own actions towards reducing flood risk and increasing resilience, especially among those who have not experienced flooding; and a hesitancy to contribute if residents doubt the value of their nontechnical, local, experiential, and historical knowledge (J. Huson, personal communication).

| Network capacity
The networks that foster the formation of social capacity were popularly categorised by Putnam (2000) into bonding, bridging and linking ties ( Figure 6 and Table 5). It is argued that communities dense in all three of these ties will also contain the full range of individuals required to form strong, effective and long-lasting community groups capable of delivering resilience during and after a flood event (Folke et al., 2003).
Participatory processes have the potential to enhance network capacity. During flooding events, the critical factor determining the success of the emergency response is the way a community works with and alongside national agencies, local government, emergency services and the media (Brown & Damery, 2002). This can be exemplified using the case of Wraysbury, Berkshire. Wraysbury is a small community (population 3,500) by the River Thames, 18 miles west of London. Following the onset of flooding on February 7, 2014, it was not until four days later that outside agencies and the armed forces arrived to provide emergency assistance to the village. In the intervening time, 84 of 103 homes in Wraysbury had been evacuated, and the worst of the damage had been done. A small, dedicated group of local individuals relied on their bonding and bridging ties within the community to provide feelings of emotional connectedness and support (Berkman & Glass, 2000;Putnam, 1993); maintain information flows (Pelling & High, 2005); and facilitate collective action (Putnam, 2000). Those coordinating the emergency response efforts later reflected on the need to enhance linking ties in order to better structure their response and allocate resources (including information). For example, improved knowledge of how to officially request "Military Aid to the Civil Community" which mobilises intervention by the armed forces, could have sped up the provision of external emergency assistance to the community (S. Burrows, personal communication).

| CONCLUSIONS AND IMPLICATIONS FOR PRACTICE
While recent legislation, such as the Floods Directive (European Commission, 2007), advocates the building of flood resilience, the procedural nature of many legislative instruments make it difficult to move towards a more participatory approach. Notably, no process or criteria for evaluation is given in the participation element of the Directive. This paper is a first step towards addressing that deficit.
Participation in flood risk management forms an iterative cycle (Figure 2), in which the assessment of different elements is required in order for an evaluation to be complete and holistic. Four distinct evaluation elements are identified: context, process, substantive outcomes, and social outcomes. This paper presents criteria for each of these elements, and examples of success statements that could be used in a practical assessment. While these statements are by no means exhaustive, and will be developed as experience is gained, they provide a starting point for framing the scripts used to elicit participant opinion through mechanisms such as participant survey, semistructured interview or workshop evaluation forms. The framework further ensures evaluation remains a focus of participation before, during, and after the process; appreciating that the outcomes of any participatory Definition Example (see Figure 5)

Bonding
Relationships between individuals with a shared social identity. They are often observed immediately following a flooding event, when individuals withdraw from wider society and rely on close-nit groups, thus reducing exposure to perceived external risks Person A, with strong bonding ties represents an altruistic individual, with strong social skills, whose closeness to those around them has accumulated an understanding of local needs and aspirations that constitutes a "community history" Bridging Relationships of continued exchange, whereby individuals are connected more by common interests or goals than by their social identity Person B, with strong bridging ties, is an individual with a great diversity of acquaintances, who can use their connections outside of the immediate group to facilitate the sharing of information, knowledge and skills Linking Relationships typified by an explicit vertical power differential. Linking ties with organisations and national institutions promote participation by transferring management rights, and therefore power, downwards Person C, with strong linking ties, is the individual who can "make things happen" by maintaining partnerships with regional and national institutions who often act as sources of experience, funding and certified expertise Bonding, bridging, and linking ties as a model for building network capacity process, both social and substantive, will shape the context of the next (Figure 2). Process evaluation provides a standard assessment of all participatory activities, based on five characteristics: accessibility, deliberation, representation, responsiveness, and quality. Context evaluation captures information about the landscape in which these participatory activities are set, and whether that landscape promotes or discourages participation. It is carried out by the facilitator before the participatory process, as the results can be used to support the design of a process that best meets local needs, resources and existing capacities. It further identifies user goals, which can be later revisited. Substantive outcome evaluation develops success statements in collaboration with the users of these outcomes, which may include conceptual or numerical models, recommendations, formal emergency plans and/or risk communication materials, providing an ongoing assessment of whether participant goals are being met.
Finally, social outcome evaluation proposes that an assessment of community resilience can be attained by evaluating three interacting social capacities: knowledge, motivation, and networking. Taken in combination, these capacities interact to determine the way in which a community can prepare for, resist the impact of, respond to and recover from a flood event, in other words, predicting their level of community resilience and the potential for enhancement.
This article moves towards a framework for evaluating the use of participatory approaches in flood risk management. Detailed and consistent evaluation of applications can support the choice of the most suitable technique to apply in a given context; manage expectations of what that technique can deliver; and provide a clear indication of whether or not it has been successful. With the plethora of techniques available to practitioners, and the complex and context-specific nature of flood risk, providing advice of this nature is critical to ensuring participatory processes are legitimate, democratic and inclusive.