SEARCH

SEARCH BY CITATION

Keywords:

  • Influence diagrams;
  • Bayesian belief networks;
  • Decision analysis;
  • Pesticide risk management;
  • Ecological risk assessment

Abstract

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. CONCLUSIONS
  8. Acknowledgements
  9. REFERENCES

The pesticide policy arena is filled with discussion of probabilistic approaches to assess ecological risk, however, similar discussions about implementing formal probabilistic methods in pesticide risk decision making are less common. An influence diagram approach is proposed for ecological risk-based decisions about pesticide usage. Aside from technical data, pesticide risk management relies on diverse sources, such as stakeholder opinions, to make decisions about what, how, where, and when to spray. Bayesian influence diagrams allow multiple lines of evidence, including process related information from existing data and expert judgment, in 1 inclusive decision model. In ecological risk assessments, data informally incorporated for pesticide usage decisions, such as field and laboratory effect studies along with chemical monitoring and modeling data, can be formally incorporated and expressed in linked causal diagrams. A case study is presented from the perspective of an environmental manager wishing to efficiently control pests while minimizing risk to local aquatic receptors. Exposure modeling results and toxicity studies were incorporated, and an ecological risk assessment was carried out but combined with hypothetical information on spraying efficacy and valuation of outcomes that would be necessary for making risk management decisions. The variables and their links in the influence diagram are ones that are important to a manager and can be manipulated to optimally control pests while protecting nontarget resources. Integr Environ Assess Manag 2012; 8: 339–350. © 2011 SETAC


INTRODUCTION

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. CONCLUSIONS
  8. Acknowledgements
  9. REFERENCES

This article provides guidance on using decision analysis with influence diagrams (IDs) for pesticide risk management decisions. The ID created in this article illustrates how a quantitative tool can be constructed for causal inference and to increase understanding in pest control management. The ID will focus on what is valuable and how to obtain value in decision outcomes through the generation and selection of alternatives and their measures. Pesticide management can be largely alternative-driven, with a focus on regulatory actions (e.g., modifying or prohibiting uses, finding an alternate means of pest control). The approach used to generate the ID will be value-focused (Keeney 1992) to prioritize what is important and generate or select alternatives that realize outcomes of value. The IDs will also be used to frame and model the uncertainties in the decision process.

Having a decision-making system that can weigh evidence and adapt to change becomes necessary as one examines current decision-making trends in environmental management. Collaborations in environmental projects can span federal, state, local, and tribal governments and include industry, academic, and citizen groups. Many lines of evidence need to be considered in an environmental management task, along with differing viewpoints and interpretations (Burger 2002). The environmental decision process can take many years, requiring provisional goals and careful planning to arrive at valid conclusions. Thus, the process that goes into making integrated environmental management decisions should be carefully analyzed and understood. Because of the magnitude of some environmental problems, weighing the costs and benefits of management actions might seem to be a formidable task. However, considering the many impacts for a decision can prevent undesirable consequences from knowable circumstances.

Complexity of environmental risk management problems

The risk assessment is a significant factor in the decision-making process for pesticide use (USEPA 1995). Scientists, regulators and courts of law accept the US Environmental Protection Agency's practice of considering factors beyond risk assessment results during decision making (USEPA 2004). Risk management does not rely solely on technical data for a decision. Scientific input should be regarded as an important asset in the decision process but not the entire foundation for decisions (Cortner 2000). Like other environmental management problems (e.g., penalty and incentive development for greenhouse gas emissions, industrial chemical safety management, invasive species eradication, contaminated habitat cleanup), many sources of information must be drawn upon to reach a decision. Economic, social, political, and legal considerations go into pesticide management decisions (USEPA 1995). These considerations include economic costs and benefits, the susceptibility and inclination of populations at risk, technological capabilities, legal issues, and societal values (USEPA 2004). At the scope of the United States federal government, as well as a state regulatory body, risk management decisions with pesticides undoubtedly affect a number of stakeholders and must consider a variety of sources of values and technically based information.

Figure 1 illustrates some factors that go into a pesticide risk management decision, the types of decisions made, and the groups affected by the decision. Understanding what is important in making a pesticide risk management decision is a multidisciplinary task requiring expert input and analysis from many different fields. With the difficulty of understanding and weighing all factors effectively before choosing an action plan, application of an appropriate formal decision analysis to pesticide risk management might provide better solutions.

thumbnail image

Figure 1. A schematic of pesticide risk management decision making. The ovals on the left indicate technical data that must go into the decision. The middle portion indicates the types of decisions made and the ovals on the right indicate the stakeholders affected by the decisions.

Download figure to PowerPoint

Introduction to Bayesian belief networks and influence diagrams

Bayesian statistics apply prior probabilities and likelihoods to produce posterior probabilities. The prior probability expresses what is believed to be true about a state for a variable and/or can be built on past experimentally derived evidence. The likelihood comes into play with new evidence and is the probability of that evidence given a potential value for the variable in question. Bayes' theorem is used to adjust the prior probabilities with the likelihood function to obtain a posterior probability distribution (Morgan 1968). A Bayesian network uses Bayesian probabilistic thinking to describe relationships among variables. In a Bayesian network, the probability of a cause (B) can be diagnosed from an observed effect (A) using conditional probabilities and Bayes' rule to obtain a posterior probability for p(B|A = a) (Kjaerulff and Madsen 2010). If the network only contains variables related to the system processes, it is known as a Bayesian belief network (BBN). However, such a network is known as an influence diagram (ID) if it also includes variables for decisions and the costs and/or benefits of those decisions. Thus, IDs not only model the processes of concern in a decision problem but also how decisions affect those processes and how the expected changes in the processes affect the loss or payoff from making certain decisions. Additional information on the development of IDs can be found in Pearl (2005) and Howard et al. (2006).

A BBN or ID is represented as an acyclic graph with nodes and arcs. In a BBN, nodes represent random variables with an associated range of states. Influence diagrams can be developed from BBNs by adding decision and utility (cost or profit) nodes (Figure 2). Arcs are visually represented as arrows connecting nodes and represent a probabilistic dependency or interaction in the case of arcs that connect chance nodes or nodes representing processes, forcing functions, and state variables. Chance nodes can either be continuous or discrete variables, and their relationships can be learnt from data sets, entered as equations, or built using a variety of other probabilistic methods. When an arc enters a decision node, it indicates an informational relationship and that the state of the predecessor node is known at the time the decision is made. An arc entering a value or utility node indicates that valued outcomes are directly dependent on the state of the variables or decisions.

thumbnail image

Figure 2. A basic influence diagram. A, B1, B2, and B3 are chance nodes; D is a decision node; and V is a value node. A conditional probability table is displayed for whether A is true or false given the states of its parents, B1 and B2.

Download figure to PowerPoint

A conditional probability table may be used to represent the potential states of a node, A, given the states of its parent nodes (B1, B2,…Bn). In this case, A would be considered a child node due to its direct dependence on B1, B2,…Bn, or the parent nodes. In Figure 2, chance node A is a child node to chance nodes B1 and B2 whereas the chance node B3 is independent to A. The decision and value nodes are represented by D and V, respectively. A conditional probability table is illustrated for the states of A, which are true and false, given whether each of the parents is in a true or false state. Thus, nodes A, B1, and B2 are Boolean nodes with their only states being true or false. Other common node discretizations are binary (e.g., high, low), ordered (e.g., high, medium, low) and integral (e.g., 0–10, 10–20, >20) (Korb and Nicholson 2004). Conditional probability tables are used in chance nodes to represent the probability of node states occurring due to decisions being made or other processes occurring. Utility nodes have utility functions attached to them indicating the expected utility from the outcome states of its parent nodes (Korb and Nicholson 2004).

Utility functions provide subjective models of preferences for attributes. The shape of utility functions expresses the risk attitude of a stakeholder or decision maker for preferences in conditions of uncertainty (Clemen and Reilly 2001). Multi-attribute utility functions index the marginal utilities to an overall preference model for a multiple objective problem. Utilities can be hard to formally elicit from stakeholders or decision-makers and can make the application of IDs over BBNs more difficult (Bielza et al. 2010). The structuring of objectives and their attributes or measures has a strong influence on the nature of the utility functions and should be done using appropriate procedures such as those described in Keeney (1992). Within an ID representation of a problem domain, decisions that provide the highest expected utilities are recognized as being the optimal choices (Kjaerulff and Madsen 2010). A multi attribute utility function can be incorporated with conditional probabilities in Bayesian networks to represent the uncertainty of value in outcomes: equation image (Korb and Nicholson 2004), where E is available evidence, i is a state for an outcome variable, a is an action with outcome Oi, U(Oi | a) is the utility of all potential outcomes when a is done, and P(Oi | E,a) is the probability distribution over the outcome states conditional on observations of E and a.

Coupled with a formal decision analysis, an ID is a powerful tool for integrating preference information and scientific evidence. The structure of an ID or BBN shows the lack of correlations among variables (Smith 1988), or sometimes, causal inferences among variables (Bromley et al. 2005). Causation descriptions can be useful for calculating potential outcomes based on available information. Mapping the decision problem with a causal diagram is an intuitive way of expressing understanding about relevant aspects of decision making. Within IDs, decisions do not have to be isolated tasks and awareness of the impacts or effects of individual decisions on variables, valued outcomes, and future decisions can be displayed and modeled.

Assessing the qualitative and quantitative aspects of a management problem with Bayesian networks

In the past, measuring uncertainty was difficult when there were many variables involved with the consequences (Morgan 1968). Bayesian networks provide a transparent way of evaluating uncertainty if management problems contain many variables from different fields (Varis and Lahtela 2002). Although the structure of a BBN or ID represents qualitative linkages, the likelihood of node states represent quantitative information (Kleiter 1996). The qualitative structure of IDs can convey the logical connections whereas the quantitative structure provides a means of representing the uncertainties in the decisions and processes (Bacon et al. 2002). The quantitative aspects of the ID also allow one to consider evidence weight in decision and process outcomes.

The graphical and quantitative structure of IDs in decision-making makes them adaptable to many fields and levels of expertise (Howard and Matheson 2005). Nodes representing variables of any kind, e.g., physical, economic, or biological, may be included in an ID. Although qualitative, the structure of the ID is important in real-world decision problems where variables are often complex and linked to one another (Bromley et al. 2005). A diagram in a decision problem can help decompose the potential events in a manner that is informative and allows practitioners to use joint probabilities as expressions of causation (Smith 1988).

Through the use of probabilistic functional relationships, IDs can represent previous data or subjective opinion quantitatively, and can be all-inclusive models for a project incorporating algorithms and evidence from many areas. Conditional probabilities linking child and parent nodes can be developed from previously collected data sets or model outputs. Expert opinion may be used as the best existing information in cases where knowledge of conditional relationships is limited. For instance, someone with knowledge about processes at a contaminated site might be able to give expert opinion that fills data gaps (Day et al. 1997, O'Hagan et al. 2006). Expert judgment will not be gathered for the current IDs and BBNs, but its role in generating information when adequate data are not available will be further discussed.

The quantitative aspects of IDs can be used to assess the certainty of the best data available to a decision-maker (Bromley et al. 2005). Managers might be motivated by formal decision analysis to fill data gaps that are required for the decision modeling, e.g., the marketing decision work by Gensch (2001). Thus, even if there are difficulties in determining probabilities for various relationships, the ID can be a communication tool about data needs. Influence diagrams can also be used to highlight technical disputes among stakeholders. Essentially, a node with states representing different hypotheses might be placed in the diagram to indicate that when one of the states (i.e., beliefs in a hypothesis) is true, a potential outcome will be more or less likely (Conroy et al. 2008).

Pesticide risk assessment and management tasks require characterizing the uncertainties of complex processes for a diverse array of pesticides and uses and, therefore, are error prone. Influence diagrams can improve or minimally make more explicit the risk management process. An example is provided here that demonstrates these features. The structure of the ID will be set up to emulate how a decision-maker contemplates a problem and will illustrate how IDs can be useful decision-making tools for enhancing understanding, transparency, and communication in a decision process.

MATERIALS AND METHODS

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. CONCLUSIONS
  8. Acknowledgements
  9. REFERENCES

Problem scenario

The ID in this example will be developed from the perspective of a manager who would like to chemically control adult mosquitoes while reducing risk to nontarget receptors. Although there are several adulticides to kill adult mosquitoes available, the manager decides that a particular insecticide is most effective for the mosquito populations she wishes to eradicate at a particular time of year. To set up the problem, the manager wishes to differentiate the objectives of the problem, the attributes to quantify the objectives, and the various alternatives to achieve positive results given the objectives. Differentiating the problem features facilitates decision problem organization and specifies in a transparent manner why decisions were made to stakeholders. The manager has already received risk assessment results for the problem. The models used to generate these results are discussed below.

Ecological risk assessment of a mosquito adulticide

An ecological risk assessment was conducted for a mosquito adulticide, malathion, and aquatic organisms to quantitatively examine potential risks. Conventional risk assessment tools generated the information necessary to characterize risks. Risk assessment data were generated using Agdrift® version 2.0.05 (SDTF, 2002) and PERPEST version 1.1.0.1 (Van Nes and van den Brink 2003) whereas the risk management IDs and BBNs were constructed in Netica™ version 3.24 (Norsys 2006). The IDs and BBNs were used to integrate the Agdrift and PERPEST outputs to examine risk management scenarios.

Agdrift uses a Lagrangian approach that allows it to consider the effects of spray equipment, droplet size, and climatological variables on deposition of chemicals (Teske et al. 2002). With Agdrift, one can vary such spraying characteristics as application rates, drop size distribution, wind speeds, release heights, wind direction, evaporation, aircraft type, meteorology, distance spraying occurs from a body of water (buffer area), flight lines, and topography of the field (Teske et al. 2002). First an initial pulse concentration realized on a water body was evaluated. The model was set to the agriculture aerial Tier II level to access more parameters than would be found in Tier I. The active rate, drop size distribution (ASAE Fine, ASAE Very Fine, and Very Fine to Aerosol types), distance from edge of pond, and wind speed were varied in each model run (as decisions and external factors). Variations of each of these factors for each model run were input in the root node states of the ID (Figure 3). A Wasp helicopter was chosen for application and a temperature (77°F), relative humidity (75%), and pond width and depth (816 feet and 5.5 feet) were chosen to reflect spray conditions during summertime in a New York City park. Default assumptions were used for the rest of the model parameters that were held constant over all Agdrift runs. A detailed analysis would examine the accuracy of the chosen spray characteristics including aerial release heights, ultra low volume (ULV) droplet spectra, nozzle settings, field and climate variables, and the uncertainty of inputs and outputs. In addition, Agdrift and other common spray drift models do not explicitly examine ULV spray scenarios (Davis et al. 2007, Schleier III et al. 2008) so more realistic modeling of aerosol particle diameters, beyond the coarser default drop size distributions applied in this article, would be an important consideration.

thumbnail image

Figure 3. Mosquito adulticide influence diagrams exhibiting the effects of 2 spray scenarios to effects categories for fish and arthropods and effectiveness in killing adult mosquitoes. The chance nodes are gray and brown and the decision nodes are blue. The utility node is a pink hexagon labeled “U.” The expected utility values are displayed in the blue decision nodes.

Download figure to PowerPoint

The data from Agdrift (initial average concentration in a water body) were placed into the PERPEST model to obtain metrics of ecological risk. PERPEST relies on a technique called case-based reasoning (CBR) to provide estimates of risk. To implement a CBR approach, PERPEST contains a database with cases (Van den Brink et al. 2002). The database has information about the laboratory toxicity and fate characteristics of a pesticide or information about the effects of a pesticide on freshwater microcosm species at various concentrations. The latter data set also contains information on whether the pesticide was applied as single or multiple applications, and whether water was stagnant or flowing. The effects observed in the freshwater microcosm experiments are placed in 5 different categories ranging from changes to community structure to no effects. Various ecological endpoints are assessed based on the outcome of the individual microcosm experiments for each effect class. These stored experiences are then compared to a user-entered scenario and used to derive a probability of each of the effects classes occurring for an endpoint based on the information contained in the cases. In PERPEST, cases were weighted using the toxic unit, the mode of action, the molecule group, and the substance. These were used to weigh the cases when selecting analogous scenarios to the current input parameters for the chemical and the loadings. Cases were selected using the nearby toxic unit, that is, chemicals in the case database were chosen with a similar toxic unit that is based on loadings and the toxicity of the chemical. Malathion was selected as the input chemical, and its structure and toxicity characteristics were used as the weighing and selecting basis for PERPEST. The initial average concentration in Agdrift was assumed to correspond with the exposure concentration used for toxic unit calculations in PERPEST'S cases (normally the peak nominal concentration).

The USEPA (2002) stated that a pesticide toxicity assessment can be initiated with information on pesticide mode of action and effects related to its toxic mode. This information is used by PERPEST for establishing similarity of cases. Models like PERPEST that rely on probabilistically relating accumulated evidence are ideal for pesticide risk management tasks and can assist in establishing conditional probabilities for risk-based IDs. Moreover, ecological risk assessments normally focus on direct effects on population or community-level parameters from pesticide exposure (Pastorok 2003). From ecotoxicological studies with multiple species, PERPEST includes ecologically relevant endpoints built from microcosm studies related to different structural and functional components of ecosystems and uses information from studies that include indirect effects from pesticide exposures.

Table 1 displays the source model inputs that were varied and selected outputs with corresponding ID and/or BBN categorical labels. For more details, see the manual and technical documentation for Agdrift and PERPEST. The input variables were selected from Agdrift for this example to examine a range of spraying scenarios. In practice, the states for each of the input variables should encompass a full range of real-world scenarios that could influence the objective nodes of minimizing adverse ecological effects and effectively killing adult mosquitoes.

Table 1. Descriptions of nodes used in the IDs
Node typeNode labelStatesState definitionsSource
  1. ASAE = American Society of Agricultural Engineers; ID = influence diagram; mph = miles per hour.

ChancePond downwind distanceNear100 feet of buffer area from a pondAgdrift input
  Moderate150 feet of buffer area from a pond 
  Far200 feet of buffer area from a pond 
DecisionApplication rateHigh0.24 pounds of active ingredient/acreAgdrift input
  Moderate0.12 pounds of active ingredient/acre 
  Low0.06 pounds of active ingredient/acre 
DecisionDrop size distributionFineASAE Aerosol to Very Fine distributionAgdrift input
  ModerateASAE Very Fine distribution 
  CoarseASAE Fine distribution 
ChanceWind speedHigh10 mphAgdrift input
  Moderate5 mph 
  Low2 mph 
ChanceEffects: fishNoneNo significant adverse effects to fishPerpest output
  SlightLimited effects to fish of short duration 
  RecoveryClear effects to fish but recovery occurs in <8 weeks 
  Unknown recoveryClear effects to fish with uncertain recovery 
  ExtirpationClear effects to fish and no recovery after 8 weeks 
ChanceEffects: arthropodsNoneNo significant adverse effects to arthropodsPerpest output
  SlightLimited effects to arthropods of short duration 
  RecoveryClear effects to arthropods but recovery occurs in <8 weeks 
  Unknown recoveryClear effects to arthropods with uncertain recovery 
  ExtirpationClear effects to arthropods and no recovery after 8 weeks 
ChanceEfficacyEffectiveSuccessful adult mosquito abatementHypothetical scale
  Not effectiveUnsuccessful adult mosquito abatement 

Construction of Bayesian belief networks and influence diagrams

Probabilities of ecological effects to various aquatic taxa generated with PERPEST were used to construct the quantitative node relationships in Netica (Norsys 2006). The Netica software provides a platform for building BBNs or IDs from users input nodes and arcs, or conditional probability tables. Inferences from relationships can be examined with a BBN. The effects of uncertain outcomes on potential rewards or costs from decision can be viewed through utility and decision nodes in a Netica-generated ID. Users can also establish probabilistic relationships from mining monitoring data using Netica algorithms.

Decision, chance, and utility nodes were set up within Netica to represent the objectives, attributes, states of nature, and value in the problem. The root nodes (nodes without predecessors and marginal distributions) contain uniform or flat probability distributions across their states. A multi-attribute utility scale was created to reflect how a hypothetical group of stakeholders might adjudicate about the trade-off in values between ecological resource integrity and the effectiveness of the spraying procedure. The utility scale used in this article will be a tabulation of the function based simply on numerical values assigned to represent the strength of positive or negative outcomes. In practice, value models should be constructed with stakeholder elicitations, and proper assumptions should be met when constructing a utility function.

RESULTS

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. CONCLUSIONS
  8. Acknowledgements
  9. REFERENCES

Influence diagrams for environmental management

To establish the ecological objectives, the manager contemplates the populations that might be exposed. Several water bodies used by fisherfolk and others for recreation exist near the locations where adult mosquitoes congregate. The manager formulates a fundamental objective to make the best possible decisions about how to effectively spray to control West Nile Virus vectors while minimizing adverse ecological effects to nontarget organisms. The lower-level objectives for minimizing adverse ecological effects to nontarget organisms would be to minimize adverse ecological effects to fish and nontarget arthropods. Arthropods can be assumed to be susceptible to the adulticides because the target organisms (mosquitoes) fall into this taxonomic group. However, the manager is concerned about the generation time for fish in comparison to arthropods if a mortality event occurs. The manager realizes that these 2 attributes (fish and arthropods) are not independent in real-world scenarios. However, the probabilities of effects for each are independently specified from the output of PERPEST; the manager considers this pragmatic specification of independence to be acceptable.

The structure of the ID for the current example is given in Figure 3. The network has 5 chance nodes, 2 decision nodes, and 1 utility node. The chance nodes are based on circumstances important to minimizing adverse ecological effects and mosquito control. The important circumstances represented by the chance nodes are the downwind distance of the pond from the spraying location (access availability or pond distance from plane lines) and the wind speed during spraying. Each of these is a process outside of the decision-maker's control when a date, time, and location are selected. The influence of the latter variables could be modeled in the current ID if the decision-maker thought it would be valuable to capture this information. However, once the decision-maker knows where or when spraying should occur, she can input the parameters in the current decision model without losing information. The other 3 chance nodes specify potential outcomes from spraying. Adverse effects might occur to nontarget organisms and mortality to target organisms also might occur. There would be other outcomes after a spraying event, but these were deemed to be most important for management and stakeholders and so are the ones displayed for assessment. The outcome of each of these 3 variables influences the utility of the decision that is expressed with a pink utility node.

Risk managers are charged with confining risk estimates to levels that are needed for the purposes of their decisions and these risk estimates should not be too cautious nor too incautious (USEPA 2004). The Federal Insecticide Fungicide and Rodenticide Act stipulates that a pesticide use will not cause “unreasonable adverse effects on the environment” (USEPA 2003). Following this mandate, the standard USEPA consideration in evaluating pesticides is unreasonable risk to humans or ecology (USEPA 2000). Risk considerations in the current example will focus on probabilities of local extinction of a taxonomic group, given previous evidence from field or mesocosm scenarios.

Different choices for the spraying event are examined in Figure 3. Although evaluating risks and effectiveness is important, the value of the IDs in Figure 3 lies in the ability to examine the trade-offs in light of utilities. For the purpose of comparison, 2 IDs are displayed in Figure 3 with different spray characteristics highlighted. In Figure 3A, the pond downwind distance is far, the application rate is moderate, the drop size distribution is set to its coarsest value, and the wind speed is high. The resulting utilities (displayed in the decision nodes) are higher than the one in Figure 3B, indicating that this spray scenario is better. Examining the ecological effects and efficacy variables indicates that the risk is much lower in Figure 3A but the efficacy is less. This is interesting given that the efficacy outcome was weighted higher in the utility function than either of the ecological outcomes. In terms of trade-offs, the risk was low enough to compensate for having a lower probability of controlling adult mosquitoes. Changing some basic assumptions about the distance downwind from the pond, the application rate, and the drop size distribution causes a change in the utility values for our decisions. Some decisions might mitigate several issues and should be examined closer (Bierbaum 2002).

For each of the decisions to be made, we can estimate the probabilities of adverse effects for fish and arthropods as well as the efficacy of the spraying event. Table 2 presents the results based on the decisions of application rate and drop size distribution when spraying for a case scenario in which the pond downwind distance is known to be near and the wind speed is high. The potential risk of interest to fish is for the sum of the unknown recovery and the extirpation categories (adverse effects). Summing of the none, slight, and recovery node states would give a probability of low or negligible effects. Alternatively, they could be deduced from the adverse effects probabilities by subtracting them from 1 or 100 depending on whether the probabilities are expressed in decimal numbers or percents. As can be seen from Table 2, a higher application rate and smaller drop sizes will potentially give a more effective spraying. However, the same decisions increase potential risks to fish. The utility values were constructed as a way of measuring these trade-offs based on the value that is placed by stakeholders on chemically controlling adult mosquitoes or loss of fish. The highest expected utilities are generally found by deciding on moderate and low application rates with a coarse drop size distribution. The magnitude of changes in utility should be interpreted with caution. As observed in a decision analysis for a fisheries management context, conversions from their multiple attribute utility functions on a scale of 100 equated 1 utility score increase to $4.5 million in benefits (McDaniels 1995). The interpretation of expected utilities should consider the trade-off information as well as the risk attitudes incorporated into the utility function.

Table 2. Potential risk to fish (adverse effects probabilities), effectiveness, and expected utility of adulticide management decisions (application rate, drop size distribution) when pond downwind distance is near and the wind speed is high
Application rateDrop size distributionp (fish adverse effects)p (effective)Expected utility
HighFine0.150.9374
HighModerate0.1050.9074
HighCoarse0.040.8477
ModerateFine0.070.8975
ModerateModerate0.040.8676
ModerateCoarse0.010.8082
LowFine0.040.8175
LowModerate0.040.7874
LowCoarse0.010.7280

Another advantage of placing probabilities of risk into an ID or BBN is the ability to project backwards to spraying factors that will lead to certain risk levels. Within the ID, we can specify likelihoods for ecological outcomes and observe what field characteristics are favored for these outcomes. Also, if the decision nodes are chance nodes or random variables in an inference problem, we might examine what application rate or drop size distribution would be most likely given high or low-risk levels. Figure 4 shows the 2 decision nodes (application rate and drop size distribution) as chance nodes (effectively turning the ID in Figure 3 to a BBN). By entering a finding of 100% extirpation to fish and arthropods in Figure 4A, we observe that the shortest pond downwind distance has the highest probability (39%) of causing this while wind speed is most likely to be high. In Figure 4B, evidence of no adverse ecological effects is entered into the BBN and the far pond downwind distance has the highest probability of causing no adverse ecological effects (35.5%). The low wind speed also has the greatest chance of causing no adverse ecological effects (38%). Also note from both figures that there is a lower probability of having an effective spraying when no adverse ecological effects are observed (82% vs 68%). Observations can be placed within a BBN for causes to compare and contrast probabilities of adverse ecological effects. Findings can also be specified for each of the effects categories and probabilities of various causes examined.

thumbnail image

Figure 4. Bayesian belief network displaying probability distributions for spraying conditions when evidence is entered for (A) local extirpation to fish and arthropods and (B) no adverse effects to fish and arthropods.

Download figure to PowerPoint

From Figure 4A, one can see that the 2 chance nodes related to decisions have states that are more likely given clear ecological effects than the other 2 root nodes, wind speed and pond downwind distance. The high application rate presents a much higher probability of having clear ecological effects on fish and arthropods than the other application rates. Also, the finest drop size distribution is more likely to produce adverse ecological effects. The opposite trend from Figure 4A was found for a scenario with no adverse effects to either fish or arthropods in Figure 4B.

All of these steps can also be done with the efficacy node. Thus, although the spraying outcomes were constructed from modeled data and expert opinion, the BBN allows us to clearly view what inputs are more likely to give certain outputs. This is an invaluable inference tool for risk management, but the ID should be used to ascertain the options that might bring the highest reward or lowest adverse effects under certain scenarios. To simplify the decision selection process, Netica can recommend policy options by optimizing decisions. Drawing arcs from wind speed and pond downwind distance to the 2 decision nodes would give recommended decisions for any combination of the site conditions that might be encountered. In addition, Netica can find optimal decisions via node absorption using methods described in Shachter (1986, 1988, 1990) and Norsys (1997).

DISCUSSION

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. CONCLUSIONS
  8. Acknowledgements
  9. REFERENCES

Evaluating uncertainty with experts

One of the challenges of applying IDs to environmental decisions is the common lack of information with which to estimate required probabilities. This can be partially circumvented using expert opinion to fill in the data gaps and elicit probabilities. As with the present example, the adverse effects nodes for fish or arthropods were derived from the output of several models, however, the efficacy node had unclear estimates, so expert judgment would be useful to derive the associated probabilities.

To establish effective spraying scenarios, clearly defined schemes would be created and information given to a group of experts to consider. Pairwise comparisons might be elicited like Merrick et al. (2000) did in a risk assessment for Prince William Sound oil transport, quantiles might be elicited for efficacy probabilities and/or the values of internal and external factors (e.g., wind speed), or odds of an effective spraying might be elicited given various spray application scenarios. For information on elicitation protocols for individual experts, see Hora (2007).

Restrictions would be established to delimit the information needed from the experts. For example, the experts would be told ULV applications (e.g., 3 fl oz of chemical per acre) will be used to ensure that the adulticide lingers long enough to kill as many mosquitoes as possible. Thermal fogging may also need to be considered by the expert. Aerial applications will be done with high-pressure pumps and over larger distances than ground applications for more effective removals. Other constraints would be placed on atmospheric stability, spray equipment, release heights, aerial swath widths, wind speed ranges, droplet size calibrations, and allowable application rates depending on policies followed by the abatement agency.

Checks would be made on the elicited effectiveness ratings. For example, higher application rates and the lower end of the droplet sizes should be more effective for mosquito control. Also, the upper end of the recommended wind speeds is predicted to be more effective due to a high amount of vegetation in the region where the mosquitoes congregate. The expert's performance can be assessed individually or weighted and combined with other experts using calibration and information scores on variables with known outcomes (Cooke 2009). Calibration would evaluate the expert's predictions with experimental results and information would relate to the specificity of predictions. Thus, the expert would be penalized for giving results that are broad and uninformative.

Using an individual's beliefs to derive probabilities is a Bayesian task, as it emphasizes the lack of knowledge of a situation using uncertainties and can be an advantageous way to fill in data gaps (Smith and von Winterfeldt 2004). Influence diagrams can assist in collecting data by presenting the issues and the known and unknown variables in a clear manner. Additional information on expert elicitation can be found in O'Hagan et al. (2006) and Roman et al. (2008). Clemen and Winkler (1999) also discuss procedures for combining probability distributions from multiple experts.

Evaluating uncertainty with stakeholders

Stakeholders can also provide information on the probability of outcomes with IDs. Bayesian networks provide an outlet for representing stakeholder beliefs in structural and quantitative aspects of problems including how their perceptions can change with different types of information (Welp et al. 2006). Stakeholders involved with model building at the outset of a project might develop greater trust in the model due to their knowledge of the modeling components and their understanding of its capabilities (Voinov and Bousquet 2010). Arentze et al. (2008) developed and tested a methodology for eliciting structure, probabilities, and utilities for an ID with stakeholders. For a regional water usage problem, Molina et al. (2010) involved stakeholders early in the development of BBNs that allowed the construction of models that reflected diverse concerns and perceptions. Despite some difficulties in the workshops, the ID in Henriksen et al. (2007) included varying expert and citizen stakeholder group beliefs on groundwater quality protection and discussed how important variables not readily available were identified by including stakeholders in the model building process. A node was added to identify a divergence in belief about outcomes in the Henriksen et al. (2007) study. Conroy et al. (2008) also demonstrated how differing views among stakeholders on uncertainty can be examined with IDs. Giordano et al. (2010) noted how developing BBNs with different stakeholders allowed them to learn about their differences in perceptions. However, methods for ID development with stakeholders should be constructed with care as the process of properly eliciting probabilities can be difficult for some stakeholders (Zorilla et al. 2010).

Adaptive management and monitoring

The development of IDs includes tractable ways of incorporating value judgments and technical judgments to assess the expected value of perfect or imperfect information in reducing uncertainties important to decision making (see Clemen and Reilly [2001] for examples of this approach in IDs). Adaptive management (AM) can be useful in an ID by testing whether the impact of decisions perform as anticipated (Cain 2001). This was demonstrated in a simple example by Conroy et al. (2008), where evidence input in the ID for scenarios updated knowledge about whether certain beliefs of stakeholders were more or less valid. The ID presented in this article could be extended for this purpose by adding a parent to the Efficacy node, with states indicating different expert opinions about whether an effective spraying occurred for the varying site characteristics and decision variables. After a spraying, monitored efficacy would indicate whether the spraying was effective, and this along with the decisions and site characteristics would be input to the ID. Beliefs about which expert models are more likely would then be updated and examined given this new evidence. Conroy et al. (2008) discuss additional considerations such as measurement uncertainty that would be optimally accounted for in an ID that is useful for such a task.

An ID should be used to represent knowledge available during a reasonable time frame (Cain 2001). As AM is undertaken, the decision process might improve and new IDs would be constructed to reflect this better understanding. Bayesian networks, developed even when little information is available, can be evaluated and improved using AM (Ames et al. 2005). Nyberg et al. (2006) describes a complete AM approach for BBNs and IDs that include stakeholder and/or expert participation along with monitoring and updating of the model structure and underlying probability tables. After developing an ID through targeted monitoring and experimentation, the ID can be used to test decision scenarios and repercussions of informational variables on decisions in the Nyberg et al. (2006) approach. Furthermore, Pollino et al. (2007) discuss how AM can be usefully considered with BBNs over traditional ecological risk approaches for updating and learning as developments occur in the management and monitoring process.

Disadvantages of influence diagrams

Some of the factors that contribute to difficulties in constructing and using IDs are prevalent in many decision analysis implementations such as deriving probabilities and utilities. Bielza et al. (2000) discuss some of the difficulties in constructing IDs for complex decision problems. According to them, this would include tracking time-dependent decisions in the “no forgetting” sequence required by IDs, asymmetry from numerous constraints on decisions, the difficulties in eliciting utility functions, and communicating recommendations from utilities in a manner that is appropriate and proper. Asymmetry results when taking one action causes a different set of options to appear than if another option is taken (Owens et al. 1997). Its representation can be better displayed and communicated in decision trees (Owens et al. 1997). Decisions are also discrete actions available to a decision-maker at a certain time and not having continuous representations can be difficult when decision intensities must be considered. Some problems also shared by BBNs are the difficulties modeling temporally continuous relationships, restrictions on spatial and temporal scale variations in relationships throughout the diagram, and the inevitable lack of data to establish strong relationships at the outset of a problem (Liedloff and Smith 2010). However, when applied properly, IDs can be a very beneficial tool. For example, Gómez et al. (2007) significantly improved a hospital's neonatal treatment system for jaundice cases with the development of a complex ID and application of other decision analysis tools such as an objectives hierarchy. Successful applications of IDs that improved real-world decisions such as Gómez et al. (2007), along with past failures, should be considered in future work that develops IDs for environmental management.

CONCLUSIONS

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. CONCLUSIONS
  8. Acknowledgements
  9. REFERENCES

In the current article, we illustrated how a manager could set up an ID to evaluate unintended risks and compare these risks to beneficial consequences from a series of decisions. The objectives, alternatives, and attributes were chosen and placed in an ID so that stakeholders can understand the thinking behind management decisions and the trade-offs that were evaluated before taking a course of action. Also discussed was the application of IDs for a risk management program with a temporal or a monitoring component, for example, adaptive risk management tasks. Incorporating diverse information from varied model output and monitoring tasks is possible in IDs making them especially conducive to weighing tradeoffs from the usage of pesticides.

Updating Bayesian networks with new data to match changing scenarios can be a straightforward process. For example, the status of parent nodes in a BBN can be easily manipulated if new evidence becomes available. The effects of these changes on the model output can then be witnessed quite easily (Wooldridge and Done 2004). In addition, BBNs can be back transformed to explore how a child node's outcome can be influenced by its parents (Wooldridge and Done 2004). As Henderson and Burn (2004) stated, BBNs or IDs “model complex causal inter-relationships in a flexible way.” This flexibility along with the ability to include many types of information makes IDs ideal candidates to logically determine courses of action in risk management.

An ecological risk-based decision is difficult to make without a formal process or guidance to weigh evidence and trade-offs. Yet, the decision-making process can often be a loose, unstructured process due to misinformed traditions and guidance. Many hidden factors then are included during decision-making. For instance, people have a tendency to fixate on initial conditions or select dominant processes when informally implementing solutions to problems (Tversky et al. 1988). A decision analysis framework with IDs could be a useful tool in pesticide risk management to circumvent the tendency to unconsciously become fixated by dominant processes. The quantitative aspects of the ID can assess some of the uncertainty in complicated risk decisions and how these decisions influence the objectives in the problem.

As demands for accountability from the public increase, future decision-making tasks could become more transparent and traceable. If this occurs, erroneous assumptions that go into decision making will be made more explicit and new ways of making risk-based decisions should be inevitable. The IDs displayed above allow risk managers to communicate with stakeholders about the various aspects of the decision problem and the potential value from implementing decisions. The structure of an ID encourages constructive thinking and openness. When this structure and the quantitative aspects are considered together, the ID can be regarded as a manageable tool for organizing data in a decision problem.

Just as in science, assessments for environmental management can benefit from peer review to further give indication that the process was integral, useful, and fair (Bierbaum 2002). Following the trends of ecological risk assessment (Russell and Gruber 1987), formal guidelines for risk management will help decisions be more defensible and justifiable when they undergo public scrutiny. These guidelines should be accommodating enough to allow the decision-maker to match the conditions of the problem she is assessing. For peers or stakeholders to understand the factors that went into a decision, a causal network or ID with associated notes on its setup can be given so the analyses can be evaluated in an expedient and tractable manner. Whatever framework is chosen for a decision analysis, it should encourage observation, questioning, and reframing. Problems should be conceptualized through a framework, not bound by it. Combined with other tools in a multicriteria decision analysis, the ability of the ID to evaluate trade-offs can be examined in light of other methodological requirements and output (Yatsalo et al. 2007).

Influence diagrams can be computational models with intuitive graphical user interfaces designed to implement and model the requirements needed for decisions. Over the last several years, ID use has increased in many management fields. This increase follows the advent of high computational power that allows ID construction including many variables and complex algorithms. The current availability of IDs is a boon to managers tackling complex decision problems. As such, its ability to represent the important management factors in a decision problem and quantitatively describe uncertainties in decision outcomes should not be overlooked in future risk management tasks.

Acknowledgements

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. CONCLUSIONS
  8. Acknowledgements
  9. REFERENCES

This work was done in partial fulfillment of J Carriger's PhD degree requirements. This is VIMS contribution no. 3158.

REFERENCES

  1. Top of page
  2. Abstract
  3. INTRODUCTION
  4. MATERIALS AND METHODS
  5. RESULTS
  6. DISCUSSION
  7. CONCLUSIONS
  8. Acknowledgements
  9. REFERENCES
  • Ames DP, Neilson BT, Stevens DK, Lall U. 2005. Using Bayesian networks to model watershed management decisions: An East Canyon Creek case study. J Hydroinf 7: 267282.
  • Arentze TA, Dellaert BGC, Timmermans HJP. 2008. Modeling and measuring individuals' mental representations of complex spatio-temporal decision problems. Environ Behav 40: 843869.
  • Bacon PJ, Cain JD, Howard DC. 2002. Belief network models of land manager decisions and land use change. J Environ Manag 65: 123.
  • Bierbaum R. 2002. The role of science in federal policy development on a regional to global scale: Personal commentary. Estuaries 25: 878885.
  • Bielza C, Gómez M, Ríos-Insua S, Fernández del Pozo JA. 2000. Structural, elicitation and computational issues faced when solving complex decision making problems with influence diagrams. Comput Oper Res 27: 725740.
  • Bielza C, Gómez M, Shenoy PP. 2010. Modeling challenges with influence diagrams: Constructing probability and utility models. Decis Support Syst 49: 354364.
  • Bromley J, Jackson NA, Clymer OJ, Giacomello AM, Jensen FV. 2005. The use of Hugin® to develop Bayesian networks as an aid to integrated water resource planning. Environ Model Softw 20: 231242.
  • Burger J. 2002. Incorporating ecology and ecological risk into long-term stewardship on contaminated sites. Remediation 13: 107119.
  • Cain J. 2001. Planning Improvements in natural resources management: Guidelines for using Bayesian networks to support the planning and management of development programmes in the water sector and beyond. Oxon (UK): Centre for Ecology and Hydrology. 116 p.
  • Clemen RT, Reilly T. 2001. Making hard decisions with DecisionTools®. Pacific Grove (CA): Duxbury. 733 p.
  • Clemen RT, Winkler RL. 1999. Combining probability distributions from experts in risk analysis. Risk Anal 19: 187203.
  • Conroy MJ, Barker RJ, Dillingham PW, Fletcher D, Gormley AM, Westbrooke IM. 2008. Application of decision theory to conservation management: Recovery of Hector's dolphin. Wildl Res 35: 93102.
  • Cooke R. 2009. Obtaining distributions from groups for decisions under uncertainty. In: Williams TM, Samset K, Sunnevåg KJ, editors. Making essential choices with scant information: Front-end decision making for major projects. Basingstoke (UK): Palgrave MacMillan. p 257276.
  • Cortner HJ. 2000. Making science relevant to environmental policy. Environ Sci Policy 3: 2130.
  • Davis RS, Peterson RKD, Macedo PA. 2007. An ecological risk assessment for insecticides used in adult mosquito management. Integr Environ Assess Manag 3: 373382.
  • Day SJ, Morse GK, Lester JN. 1997. The cost effectiveness of contaminated land remediation strategies. Sci Total Environ 201: 125136.
  • Gensch D. 2001. A marketing-decision-support model for evaluating and selecting concepts for new products. Interfaces 31: S166S183.
  • Giordano R, Preziosi E, Romano E. 2010. An integration between cognitive map and Bayesian belief network for conflicts analysis in drought management. In: Swayne DA, Yang W, Voinov AA, Rizzoli A, Filatova A, editors. iEMSs 2010 International Congress on Environmental Modelling and Software Modelling for Environment's sake, Fifth Biennial Meeting. Ottawa (CA): International Environmental Modelling and Software Society.
  • Gómez M, Bielza C, Fernández del Pozo JA, Ríos-Insua S. 2007. A graphical decision-theoretic model for neonatal jaundice. Med Decis Making 27: 250265.
  • Henderson JS, Burn RW. 2004. Uptake pathways: The potential of Bayesian belief networks to assist the management, monitoring and evaluation of development-orientated research. Agric Syst 79: 315.
  • Henriksen HJ, Rasmussen P, Brandt G, von Bülow D, Jensen FV. 2007. Public participation modeling using Bayesian networks in management of groundwater contamination. Environ Model Softw 22: 11011113.
  • Hora SC. 2007. Eliciting probabilities from experts. In: Edwards W, Miles RF Jr, von Winterfeldt D, editors. Advances in decision analysis: From foundations to applications. Cambridge (UK): Cambridge University Press. p 129153.
  • Howard RA, Matheson JE. 2005. Influence diagram retrospective. Decis Anal 2: 144147.
  • Howard RA, Matheson JE, Merkhofer MW, Miller AC, Warner North D. 2006. Comment on influence diagram retrospective. Decis Anal 3: 117119.
  • Keeney RL. 1992. Value-focused thinking: A path to creative decision making. Cambridge, MA: Harvard University Press. 432 p.
  • Kjaerulff UB, Madsen AL. 2010. Bayesian networks and influence diagrams: A guide to construction and analysis. New York (NY): Springer Science+Business Media. 318 p.
  • Kleiter GD. 1996. Propagating imprecise probabilities in Bayesian networks. Artif Intell 88: 143161.
  • Korb KE, Nicholson AE. 2004. Bayesian artificial intelligence. London, UK: Chapman and Hall/CRC. 364 p.
  • Liedloff AC, Smith CS. 2010. Predicting a ‘tree change’ in Australia's tropical savannas: Combining different types of models to understand complex ecosystem behaviour. Ecol Model 221: 25652575.
  • McDaniels TL. 1995. Using judgment in resource management: A multiple objective analysis of a fisheries management decision. Oper Res 43: 415426.
  • Merrick JRW, Rene van Dorp J, Harrald J, Mazzuchi T, Spahn JE, Grabowski M. 2000. A systems approach to managing oil transportation risk in Prince William Sound. Syst Eng 3: 128142.
  • Molina JL, Bromley J, García-Aróstegui JL, Sullivan C, Benavente J. 2010. Integrated water resources management of overexploited hydrogeological systems using object-oriented Bayesian networks. Environ Model Softw 25: 383397.
  • Morgan BW. 1968. An introduction to Bayesian statistical decision processes. Englewood Cliffs (NJ): Prentice-Hall. 116 p.
  • Norsys. 2006. Netica Version 3. Vancouver (BC): Norsys Software.
  • Norsys. 1997. Netica application: User's guide. Version 1.05. Vancouver (BC): Norsys Software.
  • Nyberg JB, Marcot BG, Sulyma R. 2006. Using Bayesian belief networks in adaptive management. Can J For Res 36: 31043116.
  • O'Hagan A, Buck CE, Daneshkhah A, Eiser JR, Garthwaite PH, Jenkinson DJ, Oakley JE, Rakow T. 2006. Uncertain judgments: Eliciting expert's probabilities. Chichester, West Sussex (UK): John Wiley and Sons. 321 p.
  • Owens DK, Shachter RD, Nease RF. 1997. Representation and analysis of medical decision problems with influence diagrams. Med Decis Making 17: 241262.
  • Pastorok RA. 2003. Introduction: Improving chemical risk assessments through ecological modeling. Hum Ecol Risk Assess 9: 885888.
  • Pearl J. 2005. Influence diagrams: Historical and personal perspectives. Decis Anal 2: 232234.
  • Pollino CA, Woodberry O, Nicholson A, Korb K, Hart BT. 2007. Parameterisation and evaluation of a Bayesian network for use in an ecological risk assessment. Environ Model Softw 22: 11401152.
  • Roman HA, Walker KD, Walsh T, Conner L, Richmond HM, Hubbell BJ, Kinney PL. 2008. Expert judgment assessment of the mortality impact of changes in ambient fine particulate matter in the U.S. Environ Sci Technol 42: 22682274.
  • Russell M, Gruber M. 1987. Risk assessment in environmental policy-making. Science 236: 286290.
  • Schleier JJ III, Davis RS, Shama LM, Macedo PA, Peterson RKD. 2008. Equine risk assessment for insecticides used in adult mosquito management. Hum Ecol Risk Assess 14: 392407.
  • [SDTF] Spray Drift Task Force. 2002. Agdrift® Version 2.0.05. Spray Drift Task Force Spray Software. http://www.agdrift.com/index.htm
  • Shachter RD. 1986. Evaluating influence diagrams. Oper Res 34: 871882.
  • Shachter RD. 1988. Probabilistic inference and influence diagrams. Oper Res 36: 589604.
  • Shachter RD. 1990. An ordered examination of influence diagrams. Networks 20: 535563.
  • Smith JQ. 1988. Decision analysis: A Bayesian approach. London (UK): Chapman and Hall. 138 p.
  • Smith JE, von Winterfeldt D. 2004. Decision analysis in management science. Manag Sci 50: 561574.
  • Teske ME, Bird SL, Esterly DM, Curbishley TB, Ray SL, Perry SG. 2002. AgDRIFT®: A model for estimating near-field spray drift from aerial applications. Environ Toxicol Chem 21: 659671.
  • Tversky A, Slovic P, Sattath S. 1988. Contingent weighting in judgment and choice. Psychol Rev 95: 371384.
  • [USEPA] US Environmental Protection Agency. 1995. Guidance for risk characterization. Washington (DC): Science Policy Council.
  • [USEPA] US Environmental Protection Agency. 2000. The role of use-related information in pesticide risk assessment and risk management. Washington (DC): Office of Pesticide Programs.
  • [USEPA] US Environmental Protection Agency. 2002. Guidance on cumulative risk assessment of pesticide chemicals that have a common mechanism of toxicity. Washington (DC): Office of Pesticide Programs.
  • [USEPA] US Environmental Protection Agency. 2003. Label review manual: 3rd edition. Office of Prevention, Pesticides, & Toxic Substances. EPA 735/B/03/001.
  • [USEPA] US Environmental Protection Agency. 2004. An examination of EPA risk assessment principles and practices. Washington (DC): Office of the Science Advisor. EPA 100/B-04/001.
  • Van den Brink PJ, Roelsma J, Van Nes EH, Scheffer M, Brock TCM. 2002. PERPEST model: A case-based reasoning approach to predict ecological risks of pesticides. Environ Toxicol Chem 21: 25002506.
  • Van Nes EH, van den Brink PJ. 2003. PERPEST version 1.0, manual and technical description: A model that Predicts the Ecological Risks of PESTicides in freshwater ecosystems. Wageningen University (NL): Alterra-rapport 787.
  • Varis O, Lahtela V. 2002. Integrated water resources management along the Senegal River: Introducing an analytical framework. Water Res Dev 18: 501521.
  • Voinov A, Bousquet F. 2010. Modeling with stakeholders. Environ Model Softw 25: 12681281.
  • Welp M, de la Vega-Leinert A, Stoll-Kleemann S, Jaeger CC. 2006. Science-based stakeholder dialogues: Theories and tools. Global Environ Change 16: 170181.
  • Wooldridge S, Done T. 2004. Learning to predict large-scale coral bleaching from past events: A Bayesian approach using remotely sensed data, in-situ data, and environmental proxies. Coral Reefs 23: 96108.
  • Yatsalo BI, Kiker GA, Kim J, Bridges TS, Seager TP, Gardner K, Satterstrom FK, Linkov I. 2007. Application of multicriteria decision analysis tools to two contaminated sediment case studies. Integr Environ Assess Manag 3: 223233.
  • Zorilla P, Carmona G, De la Hera á, Varela-Ortega C, Martínez-Santos P, Bromley J, Henriksen HJ. 2010. Evaluation of Bayesian networks in participatory water resource management, Upper Guadiana Basin, Spain. Ecol Soc 15: 12.