SEARCH

SEARCH BY CITATION

Keywords:

  • Public participation;
  • stakeholder involvement;
  • collaborative decision making;
  • negotiation;
  • mediation;
  • evaluation

Abstract

  1. Top of page
  2. Abstract
  3. 1. INTRODUCTION
  4. 2. THE POLITICS AND SCIENCE OF STAKEHOLDER-BASED DECISION MAKING
  5. 3. METHODOLOGY
  6. 4. RESULTS AND DISCUSSION
  7. 5. CONCLUSION
  8. ACKNOWLEDGMENTS
  9. REFERENCES

The increased use of stakeholder processes in environmental decision making has raised concerns about the quality of decisions these processes produce. Some claim that stakeholders make inadequate use of scientific information and analysis and are all too ready to sacrifice technical quality for political expediency. This article looks to the case study record to examine the quality of the decisions from stakeholder-based processes. The data for the analysis come from a “case survey,” in which researchers coded information from 239 published case studies of stakeholder involvement in environmental decision making. These cases reflect a diversity of planning, management, and implementation activities carried out by environmental and natural resource agencies at many levels of government. Overall, the case-study record suggests that there should be little concern that stakeholder processes are resulting in low-quality decisions. The majority of cases contain evidence of stakeholders improving decisions over the status quo; adding new information, ideas, and analysis; and having adequate access to technical and scientific resources. Indeed, data suggest that it is the more intensive stakeholder processes—precisely those that have aroused recent concern—that are more likely to result in higher-quality decisions.


1. INTRODUCTION

  1. Top of page
  2. Abstract
  3. 1. INTRODUCTION
  4. 2. THE POLITICS AND SCIENCE OF STAKEHOLDER-BASED DECISION MAKING
  5. 3. METHODOLOGY
  6. 4. RESULTS AND DISCUSSION
  7. 5. CONCLUSION
  8. ACKNOWLEDGMENTS
  9. REFERENCES

Stakeholder participation in environmental decision making has increased at all levels of government in the last decade. Among federal agencies, the Environmental Protection Agency (EPA), the Department of Energy (DOE), and the Department of Defense have initiated more than 200 citizen advisory groups at contaminated sites around the country.(1) State environmental agencies conducting comparative-risk projects have convened interest-group representatives and the general public to help make decisions about environmental priorities.(2,3) Local governments have increasingly engaged citizens in watershed-management efforts, sustainability projects, and a myriad of other planning and management activities.

The increased use of stakeholder processes signals a shift in how the public becomes involved in environmental decision making and the influence that the public can wield. Stakeholder processes are moving beyond the traditional public-meeting format to methods that involve relatively small groups of people in intensive, and often consensus-based, collaborative processes. Even the term “stakeholder involvement” denotes a deeper, more personalized stake in decision making than the more general and impersonal term “public participation.” Stakeholder processes are also gaining greater influence over public decisions than their predecessors. The most obvious examples are regulatory negotiations, where interest groups formally negotiate the content of proposed rules.

Underlying the trend toward more frequent, more intensive, and more influential stakeholder involvement is a recognition that environmental decisions are “political” as well as scientific; that is, resolving environmental problems requires addressing the interests and values of the public in ways that cannot be resolved with technical tools only, such as risk assessment and cost-benefit analysis.(4)

Some analysts have raised the concern, however, that stakeholder processes are shifting the emphasis of environmental decision making too far in the political direction.(5–8) They worry that stakeholder processes may sacrifice the quality of decisions in pursuit of political expediency.

The research literature on stakeholder processes contains surprisingly little analysis on the quality of the decisions such processes produce.(9) Indeed, assessing the quality of stakeholder-based decisions raises difficult issues about the purpose and appropriate evaluation of these decision processes. When confronted with the myriad motivations for bringing stakeholders to the table, examining just the quality of decisions is an overly narrow yardstick. Stakeholder processes have many and varied purposes beyond making decisions:(10–12) capacity building and social learning, conflict resolution, and networking are among them. However, ignorance about the quality of stakeholder-based decisions has left a large void in our knowledge of how the trend toward increased stakeholder-based decision making is affecting environmental policy.

This article describes a systematic analysis of how stakeholder processes have affected the quality of environmental decisions. The analysis uses data on 239 case studies in environmental decision making conducted in the United States over the last 30 years. Information collected from the cases is used to answer four rather conventional questions about the quality of decisions influenced or made by stakeholders:

  • Are decisions more cost effective than likely alternatives?
  • Do decisions increase joint gains among parties over likely alternatives?
  • Do participants contribute innovative ideas, useful analysis, or new information?
  • Do participants have access to scientific information and expertise?

Considerable variety in the topics and types of participation described in the 239 case studies allows answers to these questions to be drawn across a wide range of experience. The cases are nearly evenly split between state and local efforts on the one hand and federal efforts on the other. They are similarly split between pollution-related cases and those concerning natural resources. Slightly more than 80% of the cases deal with decisions that are specific to a single site or geographic feature and the remainder concern broader policy issues.

More importantly, the cases describe a variety of different kinds of participatory processes, ranging in intensity from public meetings to formal stakeholder negotiations. Such a range allows comparison between traditional public participation processes and the more intensive processes that have brought concerns about the quality of stakeholder-based decisions to the fore. If the “political” element of the newer, more intensive, stakeholder processes is indeed leading to a sacrifice in quality, then such a sacrifice should be apparent here.

The results of the analysis indicate that there should be little concern that stakeholder processes are resulting in low-quality decisions. The majority of cases contain evidence that stakeholders are making better decisions, contributing new information and ideas, and utilizing technical resources in their decision processes. Interestingly, more intensive stakeholder processes are more likely to produce high-quality decisions than traditional public participation processes.

2. THE POLITICS AND SCIENCE OF STAKEHOLDER-BASED DECISION MAKING

  1. Top of page
  2. Abstract
  3. 1. INTRODUCTION
  4. 2. THE POLITICS AND SCIENCE OF STAKEHOLDER-BASED DECISION MAKING
  5. 3. METHODOLOGY
  6. 4. RESULTS AND DISCUSSION
  7. 5. CONCLUSION
  8. ACKNOWLEDGMENTS
  9. REFERENCES

The theoretical and practical literature on public participation reflects a deep understanding that environmental decision making has both political and technical dimensions. Some have characterized this duality as a tradeoff between the need for public acceptance and the need for decision quality.(13) Others recognize public participation's instrumental role in making decisions acceptable and decision processes more normatively appealing, and also recognize that the public, not just agency experts, can bring relevant knowledge to decisions.(11,12) Many evaluation frameworks, even those that focus on the more “political” dimensions of public participation, include requirements for good scientific information and technical analysis, as well as an implicit recognition of the importance of decision quality.(14–16) The politics and science duality is most clear in a framework drawn from Habermas's theory of communicative action, which sees good public participation as both “fair” and “competent.” Fairness is achieved through broad representation and equalization of participants' power, while competence often involves the use of scientific information and technical analysis to settle factual claims.(17,18)

In 1996, the National Research Council (NRC) acknowledged the political and technical dimensions of environmental decision making by proposing an “analytic-deliberative” process.(4) Public deliberation would accommodate the political dimension, and various types of analysis would accommodate the technical dimension. An iteration between deliberation and analysis would avoid a sharp dichotomy between facts and values. The NRC included few specifics, however, on how to actually carry out such a process.

Dietz provides a useful starting point for thinking about the design of such an analytical-deliberative process. In the context of social-impact assessment, he decomposes environmental decision making into two questions. The first asks “What will happen?” The second asks “Is this a good idea?”(19)

Even analysts who promote intensely democratic approaches to decision making recognize the legitimate role of science and technical analysis in answering the “What will happen?” question.(20–22) Environmental data, analysis of cause and effect, modeling, forecasting, and other scientific and technical tools rightly underpin most efforts to characterize environmental problems and assess the likely outcomes of alternative choices. Sophisticated approaches exist for incorporating expertise and technical analysis into decision-making processes otherwise rooted in public deliberation.(20–23)

Debate is mainly over settling the question “Is this a good idea?” Answering this question—whether done via politics, economics, engineering, public deliberation, or some other approach—is inevitably a normative process because it assigns values to various alternatives and arrives at a choice of the best way to proceed. Advocates of a technocratic approach argue that the public interest is best served through the use of tools, such as risk assessment and cost-benefit analysis, that identify which outcome maximizes the welfare of economically rational agents.(24) Advocates of stakeholder processes argue for a more participatory and deliberative approach to identifying the public interest, one that recognizes a more complex basis for human motivation and a broader set of public values.(19)

Critiques of stakeholder-based decision making have challenged the ability of such processes to adequately answer both “What will happen?” and “Is this a good idea?” On the first question, critics charge that participation processes do not make adequate use of existing scientific information and technical resources. In a 1998 study of stakeholder processes, Yosie and Herbst concluded that “many stakeholder processes do not obtain sufficient knowledge and insight from scientists and, consequently, are less informed.”(5) In 2001, EPA's Science Advisory Board (SAB) arrived at a similar conclusion, writing that absent “substantial financial resources, adequate time, and high-quality staff…stakeholder decision processes…frequently do not do an adequate job of addressing and dealing with relevant science.”(6)

On the “Is this a good idea?” question, critics identify two principal failings of decision making by the public. First is lay people's susceptibility to various psychological processes that lead them to overestimate some risks and underestimate others.(25) When the public overestimates risks, the argument goes, they recommend remedies that are too expensive; when they underestimate risks, they create unnecessary cases of death and disease.(7,12)

The second perceived failing of public decision making is a political one, arising from the distribution of costs and benefits of policy choices. In many cases, the participating public (for example, members of a local community surrounding a contaminated site) will directly receive the benefits from environmental improvement without having to pay the costs, or, at most, paying them indirectly. There is every incentive, then, to call for more protection than a technocratic weighing of costs and benefits would support.(8) Whether due to the distribution of costs and benefits, or to psychological biases, the public's sense of a “good idea” is considered to be anything but by advocates of a more technocratic approach.

There would be considerably less argument over stakeholder processes if we knew whether they routinely produced decisions that were cost ineffective, rooted in ignorance of scientific facts, or devoid of any value-added information and ideas. Unfortunately, neither the supporters nor the detractors of stakeholder processes have compiled much data on decision quality. Much of the evaluation work on stakeholder processes has focused on the participation process, not its decision outcomes.(10,16,17) Those who have questioned the adequacy of scientific and technical inputs in stakeholder processes have based their conclusions on expert interviews and a smattering of cases.(5,6) Those whose critiques rest on the public's psychological failings and the political dynamics of decision making extrapolate their criticisms from more general critiques of the public's influence on decision making in the representative democratic system.(7) They largely ignore the possibility that stakeholder processes—with sufficient attention to education, deft choice of participants, and a thoughtful structuring of the process—may well be a cure for these perceived psychological and political failings rather than a manifestation of them.

3. METHODOLOGY

  1. Top of page
  2. Abstract
  3. 1. INTRODUCTION
  4. 2. THE POLITICS AND SCIENCE OF STAKEHOLDER-BASED DECISION MAKING
  5. 3. METHODOLOGY
  6. 4. RESULTS AND DISCUSSION
  7. 5. CONCLUSION
  8. ACKNOWLEDGMENTS
  9. REFERENCES

To analyze the quality of stakeholder-based decisions, data were compiled from a “case survey” of 239 cases of public participation in environmental decision making. A case survey is analogous to a normal closed-ended survey, except that a “reader-analyst”“asks” a standard set of questions of a written case study rather than of a person.(26–29) It is a formal process for systematically coding relevant data from a large number of qualitative sources for quantitative analysis. Derived data can support data analysis even if the questions addressed in the analysis are different from those posed in the original case study.(30)

Researchers screened more than 1,800 case studies—drawn from journals, books, dissertations, conference proceedings, and government reports—ultimately identifying the 239 cases making up the data set.1 Each case was coded for more than 100 attributes covering the type of environmental issue, characteristics of the people who participated, important features of the participatory process, and the outcomes achieved. Each attribute was assigned a score—usually low, medium, or high—based on a standard template. Each score was given one of three weight-of-evidence measures, ranging from “solid evidence” to “best informed guess.” Data with the lowest weight of evidence were not used in the analysis. The scores were accompanied by a written entry describing the attribute in qualitative terms.

Each case was coded by one of three researchers or by pairs of them. To ensure consistent coding among researchers, a process of intercoder reliability testing was used. This involved pairs of researchers reading and coding the same subset of case studies independently and then comparing codes. Where codes conflicted, researchers reached consensus on the correct code and clarified the coding template as necessary. Pairs of researchers continued to code sets of cases in parallel until they consistently achieved two-thirds agreement, a level of reliability regarded as satisfactory in the literature.(29) At that point, each case was then coded by only one researcher. As the coding progressed, researchers would periodically code a set of cases in parallel to assure that intercoder reliability was being maintained. Overall, around 10% of the cases were used in the intercoder reliability process.

Data analysis consisted mainly of counts of scores and a review of the qualitative information accompanying them. Relatively simple comparative statistics were used to develop correlation coefficients and to identify statistically significant differences between sets of data. The statistical analysis used a Kendall's tau-b correlation coefficient and a chi-squared test of significance.2

Although it has been used in the policy analysis and business literature, the case-survey methodology is still somewhat experimental and a few important caveats should be mentioned. The quality of the data used in a case survey is only as good as the quality of the case studies from which the data come. Moreover, cases by different authors and for different purposes will report on different aspects of a process, leaving some data gaps. The analysis accounted for these problems somewhat by the assignment of weight-of-evidence scores and by drawing on enough cases to overcome problems with data gaps. In describing the results of the analysis below, the number of cases with relevant information is always mentioned.

Any case survey has to contend with issues of possible bias. Of particular concern is a success bias—that only successful cases are written up and that authors have a tendency to overemphasize the good things that stakeholders accomplished. There are two reasons to think, however, that a success bias may not be as prevalent in the published literature on stakeholder involvement as it is in the experimental research literature. First, authors don't necessarily have an incentive to write up only successful cases. Many of the cases in this study come from doctoral dissertations or other studies where multiple cases are compared and unsuccessful cases provide as much, or more, insight as successful ones. Second, different authors define the “success” of stakeholder processes differently, and very few define it in terms of decision quality. Thus, even if there is an overall success bias, it is unlikely to extend to the quality criteria used here.3

4. RESULTS AND DISCUSSION

  1. Top of page
  2. Abstract
  3. 1. INTRODUCTION
  4. 2. THE POLITICS AND SCIENCE OF STAKEHOLDER-BASED DECISION MAKING
  5. 3. METHODOLOGY
  6. 4. RESULTS AND DISCUSSION
  7. 5. CONCLUSION
  8. ACKNOWLEDGMENTS
  9. REFERENCES

Before turning to the results of the evaluation questions, it is useful to describe the types of cases involved and the kinds of stakeholder processes used.

Each of the 239 cases fell into one of six topical categories. Thirty-one percent of cases concerned natural resources planning and management, such as the development of a habitat-conservation plan or a management plan for a park. Investigation and cleanup of hazardous-waste sites, many of them in the Superfund program, made up 22% of the cases. Cases of policy development (including comparative risk assessment) made up 17% of the cases. Fifteen percent of cases concerned the siting of new facilities, and 10% of cases involved the design of permits and operating requirements for existing facilities. The remaining 5% of cases involved regulatory design and standard setting.

There was considerable variety within each of the six categories. In facility-siting cases, for example, members of the public debated over incinerators, power plants, dams, dikes, canals, mines, highways, landfills, ferry terminals, and radioactive-waste facilities. In cases of regulatory design, participants negotiated the fine points of policies concerning asbestos, water pollutants, industrial equipment leaks, bear hunting, worker safety, transportation, gasoline additives, wood-burning stoves, and hog farming.

The cases took place in more than 40 states over the last 30 years. Most of the cases—84%—concerned a single site or geographic feature. Those involving issues of overarching policy made up the remaining 16% of the data set. Local agencies took the lead in 17% of the cases, and state governments took the lead in 38%. Federal agencies took the lead in 38% of the cases, and a mixture of federal and state government institutions shared the lead in the remaining 7%.

The kinds of participatory processes used ranged from public meetings to intensive negotiations, with roughly one-quarter of the 239 cases falling into each of four categories:

  • 21% involved public hearings, meetings, and workshops in which access was generally open to any interested citizen.
  • 25% involved advisory committees that were not explicitly trying to seek consensus. These committees typically had a defined and consistent membership, with participants selected to represent various interest groups or points of view.
  • 30% involved advisory committees that were seeking consensus. In these cases, decision making took on aspects of internal negotiations among participants, with opposing interests forced to work together to come to a common solution.
  • 23% involved negotiations and mediations in which participants—usually formally representing interest groups—formulated consensual agreements that would bind their organizations to particular courses of action.

In this article, processes will be regarded as “increasing in intensity” as they go down the above list. As they increase in intensity, they become less focused on gathering information from a broad range of “the general public” and more focused on reaching agreement among a small group of “stakeholders.” What participants contribute to decisions differs according to the type of participatory process. In most public hearings, meetings, and workshops, participants often simply contribute ideas and information that agencies use to inform decisions. In more intensive processes, such as negotiation and mediation, participants actually make decisions themselves. Part of the reason for using a variety of evaluation questions here is to accommodate the different ways the public contributes to decision making.

The results for each of the evaluation questions—and how the results relate to the type of participation process used—are discussed separately below.

4.1. Are Decisions More Cost Effective than Likely Alternatives?

The cost-effectiveness question analyzes whether decisions developed through stakeholder processes are more or less cost effective than probable alternatives. It does not measure whether the choice to use a stakeholder process is a more cost-effective way to make a decision than some other approach (e.g., an experts-only process). Asking whether stakeholder-based decisions are more cost effective than likely alternatives addresses the concern that stakeholders will recommend plans that are politically expedient, but unnecessarily expensive.

Given that one of the principal rivals to stakeholder-based decision making is cost-benefit analysis, surprisingly few case studies presented any information about the cost effectiveness of decisions. Only 17 out of 239 cases could be scored for this criteria. In 50% of these, stakeholder processes were credited with increasing the cost effectiveness of decisions. A good example is the Fernald Citizens Task Force—a stakeholder advisory committee established to advise the Department of Energy (DOE) on the remediation of its Fernald, Ohio nuclear-weapons facility. After working intensively for two years, the task force recommended a clean-up plan that DOE says was $2 billion cheaper than what DOE had originally planned.(31)

In 29% of the cases, the opposite result occurred—stakeholder-based decisions were less cost effective than likely alternatives (in 18% of cases cost effectiveness stayed about the same). In these low-scoring cases, one of the principal criticisms of stakeholder processes rang true: more expensive solutions were required to satisfy the interests of the parties involved. One example is the Army Corps of Engineers' (the Corps) stakeholder process to develop a disposal plan for waste from a water-treatment plant in Ohio's Three Rivers Watershed. Unresolved differences between urban residents, who would benefit from the proposed disposal plan, and rural citizens, who would bear the risks, led stakeholders to recommend a plan that was “less impressive, less efficient, and more costly but also more politically acceptable” than what the Corps originally proposed.(32)

There are too few cases to make a quantitative connection between the cost effectiveness of decisions and the type of stakeholder process used to reach them. The Fernald and the Three Rivers Watershed cases, however, do provide some qualitative insights. In the Fernald case, the task force was a consensus-based advisory committee composed of representatives of a variety of different interests who worked together for two years. It is highly unlikely that they would have developed their money-saving plan without that high level of engagement. In the Three Rivers Watershed case, participation consisted of a series of public meetings and informal consultations between the Corps and various interest groups. The process never allowed those who would benefit from the project and those who would bear its risks to come together and try to come up with plan that was more cost effective and equitable.

4.2. Do Decisions Increase Joint Gains Among Parties Over Likely Alternatives?

The joint-gains question examines whether some participants are made better off through stakeholder-based decisions (relative to a likely alternative) without any participant becoming worse off. It is a standard measure in the negotiation literature that traces its roots to “Pareto optimality” in game theory and the economics of Coasian bargaining. Agreement among parties in a negotiation is often used as a proxy that joint gains have increased because parties would not be expected to agree to a solution that left them worse off than the status quo.(33)

Seventy case studies provided information on joint gains. In 69% of these, joint gains increased; participants were able to “expand the pie” and find solutions that were not obvious when the process began. For example, in a dispute over water supply for snowmaking at Sugarbush Mountain in Vermont, joint gains were achieved through the realization that switching the water source from one stream to another would increase overall environmental protection and supply adequate water for snowmaking. The authors of the case write that as the process progressed and participants discussed various options and the technical data, “each party realized there might be options available which might allow them to keep their objective and settle the case.”(34)

In 26% of the 70 cases with relevant data, joint gains remained about the same, and in 6% they actually decreased. In cases where joint gains remained the same or decreased, enthusiasm over the final agreement was not shared among participants. For example, in the case of the Environmental Protection Agency's Project XL permitting process for Intel Corporation's facility in Chandler, Arizona, parties came to agreement on a flexible facility permit. The permit gave Intel much of what it wanted, and gave EPA a successful “win” for its pilot program. However, only 40% of participants surveyed felt that the benefits of the agreement were widely distributed.(35) Eighty percent felt that Intel got most of what it wanted, while only half said that local stakeholders got what they wanted. National environmental groups argued that not enough gains were made in environmental improvement.

Not surprisingly, most of the data on joint gains comes from cases of negotiation and mediation. Also not surprisingly is that it was in these types of processes that increases in joint gains were most likely to occur. In the 37 negotiation and mediation cases with adequate data, joint gains increased in 84% and stayed the same or decreased in only 16%. In the remaining 33 cases, which used other types of less intensive participation processes, joint gains increased in 52% and stayed the same or decreased in 48%. The difference between the two kinds of cases is highly statistically significant.4

That negotiations and mediations would be much better at increasing joint gains than less intensive forms of participation is not surprising; they are specifically designed to do so. Parties would not be expected to participate if they could not reasonably expect to be better off through a negotiated agreement. More subtly, however, negotiation and mediation cases are more likely to be robust enough to allow participants to reframe problems and explore solutions in order to identify creative, win-win solutions, as in the Sugarbush case.

4.3. Do Participants Contribute Innovative Ideas, Useful Analysis, or New Information?

The third evaluation question focuses on the extent to which stakeholders contribute new ideas, analysis, and/or information to decision making. This criterion is more appropriate than the previous two for cases in which the public is not actually making decisions, but is contributing to the knowledge base that government agencies then use to determine a course of action.

Answering the question involves averaging four separate, but highly related, criteria that record whether participants:

  • Contributed information that would not otherwise have been available,
  • Undertook technical analyses to improve the foundations for decision making,
  • Came up with innovative ideas, and/or
  • Fostered a more holistic and integrated way of looking at a problem. 5

Out of the 121 cases with adequate data, 76% demonstrated some contribution of innovative ideas, useful analysis, and/or new information. The Buffalo River Citizens Committee, for example, was a major force behind better data collection on the water quality of the Buffalo River, as part of an effort to clean up the area where the river joins Lake Erie.(36) In another case, which concerned a plan for navigation and flood control on the Missouri River between Kansas City and St. Louis, stakeholders performed analyses on the economic, hydraulic, recreation, environmental, and land-use aspects of various levee alternatives.(32) In the case of a mediation regarding the damming of the Snoqualmie River in the 1970s, stakeholders took a yes/no question about building the dam and turned it into the more holistic question of “How do we provide some level of flood control, ensure the continued economic viability of the farmers and the towns, and build the kind of land-use plans and controls that maintain the valley as a greenbelt with broad recreational value?”(37) In these and many other cases, stakeholders contributed ideas, analysis, and information that increased the quality of decisions.

In 24% of the 121 cases with adequate information on this criterion, participants did not add much in the way of ideas, analysis, and information. The principal reason was not that they were unable to—or even that lead agencies ignored what input was provided. In most cases, participants did not contribute to the substantive content of decisions because processes were not designed for them to do so. Process designers, and often the participants themselves, saw the stakeholders' roles as “values consultants” or consumers of technical material, but not as a source of knowledge, ideas, and analysis.

It was in the more intensive participatory processes—where stakeholders themselves had more substantial control over the process and outcomes—that they provided more input in the way of ideas, information, and analysis. Indeed, as shown in Fig. 1, there is a strong tendency for participants to contribute ideas, analysis, and information in the more intensive participatory processes. The trend shown in the figure is statistically significant.6

image

Figure 1. Comparing contribution of ideas, information, and analysis across process type.

Download figure to PowerPoint

4.4. Do Participants Have Access to Scientific Information and Expertise?

A final perspective on the substantive quality of decisions comes not from looking at outcomes but from looking at the decision-making process. This question examines the degree to which the process provided adequate scientific and technical resources, either in terms of the technical training and experience of participants (internal capacity) or in terms of their access to external technical resources and expertise (external resources).

Discourses about stakeholder participation often rest on an assumption that citizens participating in environmental policy decisions are lay people rather than experts. Yet the capacity that participants bring to the table often is quite impressive, both in terms of scientific and technical training and in terms of in-depth knowledge of the issues under discussion. In addition to bringing their own expertise to the table, stakeholders often access external resources through a variety of methods, such as hiring consultants, interacting with technical advisory committees, or otherwise querying outside experts.

In 74% of the 149 cases with adequate information, stakeholders had a relatively high level of internal capacity and external resources.7 An example of a stakeholder group with much internal capacity is the Northern States Power Advisory Task Force, which included two physicists, a university biologist, other scientists and engineers, and many people with long histories of involvement in energy issues.(38) Interesting models for processes where stakeholders did not have such a high degree of technical expertise but effectively tapped into external expertise were citizen juries. In such processes, randomly selected members of the public (the jury) listened to testimony and asked questions of a series of experts (the witnesses) in order to render informed judgment on a particular policy topic.(22)

In 14% of the 149 cases, the process was deficient in both internal and external technical resources (in 12% of cases there was a moderate amount of internal and external resources or a mix of the two). For example, in a stakeholder process concerning the cleanup of the highly contaminated Lipari Landfill in New Jersey, participants had little internal technical capacity and few external technical resources, which effectively shut them out of decision making and left the community feeling “ignorant and overwhelmed.”(39) This sort of process is not a reflection of the public's inability to understand and become involved in decision making; such cases typically illustrate how limiting access to technical resources can be used as a strategy for excluding the public from decision making altogether.

In what is by now a familiar pattern, access to internal and technical resources improved as stakeholder processes became more intense. As shown in Fig. 2, in 92% of cases involving negotiations and mediations, participants had a high degree of internal capacity and access to external technical resources. That same figure for public meetings and hearings is 39%. The trend toward increasing technical resources as processes intensify is statistically significant.8

image

Figure 2. Comparing internal and external technical resources across process type.

Download figure to PowerPoint

The trend in Fig. 2 can largely be explained by two things. First, the process for selecting participants for the more intensive stakeholder processes favors those with more training and experience with the topics at hand; interests represented in these processes want to make sure that they have their most effective staff involved. Second, the amount of time and resources typically available for more intensive processes favors hiring technical experts or otherwise educating stakeholders on technical material.

5. CONCLUSION

  1. Top of page
  2. Abstract
  3. 1. INTRODUCTION
  4. 2. THE POLITICS AND SCIENCE OF STAKEHOLDER-BASED DECISION MAKING
  5. 3. METHODOLOGY
  6. 4. RESULTS AND DISCUSSION
  7. 5. CONCLUSION
  8. ACKNOWLEDGMENTS
  9. REFERENCES

Based on an examination of 239 case studies, we should be rather optimistic about the quality of stakeholder-based decisions. Across a diversity of cases, most of the evidence points toward quality decision making from stakeholder processes.

And it is a broad and varied conception of quality. Cost effectiveness and the contribution of ideas, information, and analysis measure quality in a “problem-solving” sense, challenging stakeholders to think of the most ingenious solution. Joint gains is more a measure of “political quality,” asking participants to devise a solution that is to everyone's satisfaction. Measures of internal and external technical resources evaluate, at least in part, “technical quality,” rating processes by the degree to which they utilize the best information and analysis.

Across all the conceptions of quality, one result is consistent—more intensive forms of stakeholder involvement are more likely to produce higher-quality decisions. The result runs counter to fears that politics is trumping decision quality. These same intensive processes are the most explicitly “political” forms of public involvement.

Of course, the intensive processes are often longer, better funded, and usually attract more committed participants than less intensive processes. But it may also be that the political features of more intensive stakeholder processes create positive synergies with the quality of their outcomes. Resolving conflict often requires dealing with scientific uncertainty through appeals to independent expertise, joint fact finding on the part of all participants, or new research altogether. Arguments are generally won or lost based on the quality of “the facts,” creating an incentive to gather more information and improve understanding of information that exists. Mistrust among stakeholders and between stakeholders and government may uncover questionable science and bad ideas. Building trust may require tapping into independent sources of expertise or generating new knowledge.

There may be many ways to produce decisions of high technical quality, but there are relatively few methods that do so while also educating the public, eliciting public values, resolving conflict, and building trust in agencies, as many stakeholder processes do. That society can make some headway on these more “political” features of decision making and not sacrifice quality is indeed a positive endorsement for engaging stakeholders in environmental decision making.

ACKNOWLEDGMENTS

  1. Top of page
  2. Abstract
  3. 1. INTRODUCTION
  4. 2. THE POLITICS AND SCIENCE OF STAKEHOLDER-BASED DECISION MAKING
  5. 3. METHODOLOGY
  6. 4. RESULTS AND DISCUSSION
  7. 5. CONCLUSION
  8. ACKNOWLEDGMENTS
  9. REFERENCES

Funding for this research was provided by the National Science Foundation under Grant Number 9818728. The author would like to thank Resources for the Future colleagues Jerry Cayford, Terry Davies, Allen Blackman, and Mike Toman, as well as Allan Mazur at Syracuse University, Gail Charnley at HealthRisk Strategies, Caron Chess at Rugers University, and two anonymous reviewers for comments on earlier drafts of this article.

Footnotes
  • 1

    Case studies were screened based on the following criteria:

    • dealt with public involvement in environmental decision making, generally at the administrative level;

    • occurred in the United States;

    • occurred since 1970;

    • had an identifiable lead (or otherwise interested) government agency;

    • described a discrete mechanism (or set of mechanisms) used to engage the public;

    • described participation of nongovernmental citizens other than regulated parties; and

    • contained sufficient information on context, process, and outcomes.

  • 2

    The Kendall's tau b correlation coefficient is based on the number of concordant and discordant pairs of observations in a contingency table, using a correction for ties. It is an appropriate nonparametric measure of correlation for ordinal data. A rule of thumb for using the chi-squared test of association is that the expected count of each cell in the contingency table should be greater than five (and preferably greater than 10), which was met in all cases here.

  • 3

    In addition to qualitative insights into bias, the author examined two sources of possible bias quantitatively. The first—possible bias originating from motivations of case-study authors to emphasize success—was found to contribute a small and statistically insignificant bias toward success. The second form of bias was less amenable to analysis. It concerned the measure of whether stakeholders contributed ideas, analysis, and information. It may be that case-study authors were more likely to report when such contributions occurred than when they did not. Unfortunately, we have no way of knowing whether this is true or not without more information on the cases. What we can do, however, is compare whether this criteria makes the case-study pool look more successful on the whole than the other criteria (which are not subject to such possible bias). The criteria for measuring stakeholders' contributions of ideas, analysis, or information does not, however, make the case-study pool look anymore successful in any statistically significant way.

  • 4

    A chi-2 test resulted in a p value of 0.015 with 70 observations.

  • 5

    The correlations among these measures were greater than 0.84 for each matched pair. The aggregate criteria was a rounded average of the scores on each individual measure, with ties rounded up.

  • 6

    A chi-2 test resulted in a p value of 0.009. The Kendall's tau-b correlation coefficient was 0.28. There were 121 observations.

  • 7

    High-scoring cases were those in which high-quality internal or external resources were present without an off-setting lack of one or the other (e.g., high/low combinations of internal capacity and external resources). Medium-scoring cases were those in which either internal capacity and external resources were moderate or a high level of one of the two was offset by a low level of the other. Low-scoring cases were those in which the process was deficient in internal capacity or external resources and the deficiency was not compensated by either a high degree of internal capacity or external resources.

  • 8

    A chi-2 test resulted in a p vaue <0.001. The Kendall's tau-b correlation coefficient was 0.37. There were 148 observations.

REFERENCES

  1. Top of page
  2. Abstract
  3. 1. INTRODUCTION
  4. 2. THE POLITICS AND SCIENCE OF STAKEHOLDER-BASED DECISION MAKING
  5. 3. METHODOLOGY
  6. 4. RESULTS AND DISCUSSION
  7. 5. CONCLUSION
  8. ACKNOWLEDGMENTS
  9. REFERENCES
  • 1
    Federal Facilities Environmental Restoration Dialogue Committee (FFERDC). (1996). Consensus principles and recommendations for improving federal facilities cleanup. Washington, DC: Environmental Protection Agency.
  • 2
    Perhac, R. (1998). Comparative risk assessment: Where does the public fit in? Science, Technology & Human Values, 23(2), 221242.
  • 3
    Western Center for Environmental Decision-Making (WCED). (1997). Public involvement in comparative risk projects. Boulder, CO: Western Center for Environmental Decision-Making.
  • 4
    Stern, P. C., & Fineberg, H. V. (Eds.). (1996). Understanding risk: Informing decisions in a democratic society. Washington, DC: National Academy Press.
  • 5
    Yosie, T., & Herbst, T. (1998). Using stakeholder processes in environmental decisionmaking: An evaluation of lessons learned, key issues, and future challenges. Washington, DC: Ruder Finn Washington.
  • 6
    Environmental Protection Agency Science Advisory Board (SAB). (2001). Improved science-based environmental stakeholder processes. A commentary by the EPA Science Advisory Board, EPA-SAB-EC-COM-01-006. Washington, DC: United States Environmental Protection Agency.
  • 7
    Viscusi, W. K., & Hamilton, J. T. (1999). Are risk regulators rational? Evidence from hazardous waste cleanup decisions. Working Paper 99-22. Washington, DC: AEI-Brookings Joint Center for Regulatory Studies.
  • 8
    Hamilton, J. T., & Viscusi, W. K. (1999). “How costly is clean?” An analysis of the benefits and costs of Superfund site remediations. Journal of Policy Analysis and Management, 18(1), 227.
  • 9
    Webler, T. (1999). The craft and theory of public participation: A dialectical process. Journal of Risk Research, 2(1), 5571.
  • 10
    Beierle, T. (1999). Using social goals to evaluate public participation in environmental decisions. Policy Studies Review, 16(3/4), 75103.
  • 11
    Fiorino, D. (1990). Citizen participation and environmental risk: A survey of institutional mechanisms. Science, Technology, and Human Values, 15(2), 226243.
  • 12
    Perhac, R. (1996). Defining risk: Normative considerations. Human and Ecological Risk Assessment, 2(2), 381392.
  • 13
    Thomas, J. (1993). Public involvement and governmental effectiveness: A decision-making model for public managers. Administration and Society, 24(4), 444469.
  • 14
    Chess, C., Dietz, T., & Shannon, M. (1998). Who should deliberate when? Human Ecology Review, 5(1), 4548.
  • 15
    Chess, C., & Purcell, K. (1999). Public participation and the environment: Do we know what works? Environmental Science and Technology, 33, 26852692.
  • 16
    Rowe, G., & Frewer, L. (2000). Public participation methods: A framework for evaluation. Science, Technology, and Human Values, 25(1), 329.
  • 17
    Renn, O., Webler, T., & Wiedemann, P. (1995). Fairness and competence in citizen participation: Evaluating models for environmental discourse. Dordrecht, the Netherlands: Kluwer Academic Publishers.
  • 18
    Webler, T., & Tuler, L. (2000). Fairness and competence in citizen participation: Theoretical reflections from a case study. Administration and Society, 32(5), 566595.
  • 19
    Dietz, T. (1994). “What should we do?” Human ecology and collective decision making. Human Ecology Review, Summer/Autumn (1), 301309.
  • 20
    Dietz, T. (1984). Social impact assessment as a tool for rangelands management. In National Research Council (Ed.), Developing strategies for rangelands management. Boulder, CO: Westview.
  • 21
    Renn, O., Webler, T., Rakel, H., Dienel, P., & Johnson, B. (1993). Public participation in decision making: A three-step procedure. Policy Sciences, 26, 189214.
  • 22
    Crosby, N., Kelly, J. M., & Schaefer, P. (1986). Citizens panels: A new approach to citizen participation. Public Administration Review, March/April, 170178.
  • 23
    Gregory, R. (2000). Using stakeholder values to make smarter environmental decisions. Environment, 24(5), 3444.
  • 24
    Breyer, S. (1995). Breaking the vicious circle: Toward effective risk regulation. Cambridge, MA: Harvard University Press.
  • 25
    Slovic, P., Fischhoff, B., & Lichtenstein, S. (1982). Facts versus fears: Understanding perceived risk. In D.Kahneman, P.Slovic, & A.Tversky (Eds.), Judgment under uncertainty: Heuristics and biases. Cambridge, UK: Cambridge University Press.
  • 26
    Lucas, W. (1974). The case survey method: Aggregating case experience. Rand Report No. R-1515-RC. Washington, DC: Rand Corporation.
  • 27
    Yin, R., & Heald, K. (1975). Using the case survey method to analyze policy studies. Administrative Science Quarterly, 20, 371381.
  • 28
    Bullock, R., & Tubbs, M. (1987). The case meta-analysis method for OD. Research in Organizational Change and Development, 1, 171228.
  • 29
    Larrson, R. (1993). Case survey methodology: Quantitative analysis of patterns across case studies. Academy of Management Journal, 36(6), 15151546.
  • 30
    General Accounting Office (GAO). (1991). Designing evaluations. Washington, DC: General Accounting Office.
  • 31
    Applegate, J. (1998). Beyond the usual suspects: The use of citizens advisory boards in environmental decisionmaking. Indiana Law Journal, 73(3), 143.
  • 32
    Mazmanian, D., & Nienaber, J. (1979). Can organizations change? Environmental protection, citizen participation, and the Army Corps of Engineers. Washington, DC: Brookings Institution.
  • 33
    Bingham, G. (1986). Resolving environmental disputes: A decade of experience. Washington, DC: Conservation Foundation.
  • 34
    Fitzhugh, J., & Dozier, D. (1996). Finding the common good: Sugarbush water withdrawal. Unpublished manuscript on file with the author. Available at http://emis.com/repository/articles/suqarbush.pdf.
  • 35
    Orenstein, S. (1998). Evaluation of Project XL stakeholder processes. Washington, DC: RESOLVE, Inc.
  • 36
    Kellogg, W. (1993). Ecology and community in the Great Lakes Basin: The role of stakeholders and advisory committees in the environmental planning process. Doctoral dissertation. New York: Cornell University.
  • 37
    Cormick, G., & Patton, L. (1980). Environmental mediation: Defining the process through experience. In L.Lake (Ed.), Environmental mediation. Boulder, CO: Westview Press.
  • 38
    Ducsik, D., & Austin, T. (1986). Open power plant siting: The pioneering (and successful) experience of Northern States Power public involvement. In D.Ducsik (Ed.), Energy facility planning: The electric utility experience. Boulder, CO: Westview Press.
  • 39
    Kaminstein, D. S. (1996). Persuasion in a toxic community: Rhetorical aspects of public meetings. Human Organization, 55(4), 458464.