On the value of formal assessment of uncertainty in regulatory analysis


Professor Robert N. Stavins, John F. Kennedy School of Government, Harvard University, 79 John F. Kennedy Street, Cambridge, MA 02138, USA. Email: robert_stavins@harvard.edu


The US Office of Management and Budget introduced in 2003 a new requirement for the treatment of uncertainty in Regulatory Impact Analyses (RIAs) of proposed regulations, requiring agencies to carry out a formal quantitative uncertainty assessment regarding a regulation’s benefits and costs if either is expected to reach $1 billion annually. Despite previous use in other contexts, such formal assessments of uncertainty have rarely been employed in RIAs or other regulatory analyses. We describe how formal quantitative assessments of uncertainty – in particular, Monte Carlo analyses – can be conducted, we examine the challenges and limitations of such analyses in the context of RIAs, and we assess how the resulting information can affect the evaluation of regulations. For illustrative purposes, we compare Monte Carlo analysis with methods typically used in RIAs to evaluate uncertainty in the context of economic analyses carried out for the US Environmental Protection Agency’s Nonroad Diesel Rule, which became effective in 2004.

1. Introduction

Benefit–cost analysis is a core component of a Regulatory Impact Analysis (RIA), which Presidential Executive Order 12866 requires for all “economically significant” Federal regulatory actions.1 Similar requirements for RIAs have been in place since the Reagan administration2 and prior procedures for regulatory review date back to the administration of Richard Nixon.3 In carrying out analyses to estimate proposed regulations’ benefits and costs, analysts must frequently rely on inputs to those analyses that are uncertain – sometimes substantially so.4 Such uncertainties in underlying inputs are propagated through analyses, leading to uncertainty in ultimate benefit and cost estimates. But despite such uncertainty, the most prominently displayed results in RIAs are typically single, apparently precise estimates of benefits, costs, and net benefits,5 masking uncertainties inherent in their calculation and possibly obscuring tradeoffs among competing policy options (Arrow et al. 1996).6

Historically, efforts to address uncertainty in RIAs have been very limited, but new guidance set forth in the US Office of Management and Budget’s Circular A-4 on Regulatory Analysis has the potential to enhance dramatically the information provided in RIAs regarding uncertainty in benefit and cost estimates (OMB 2003). Circular A-4 requires the development of a formal quantitative analysis of uncertainty regarding a regulation’s economic impact if either annual benefits or costs are expected to reach $1 billion.

Over the years, the use of formal quantitative uncertainty assessments – in particular, Monte Carlo analysis7– has become common in a variety of fields, including engineering, finance, and several scientific disciplines.8 It has also been found to be useful in certain regulatory contexts. For example, the US Environmental Protection Agency has recognized that it can be an important element of risk assessments (EPA 1997). But efforts to quantify uncertainties formally have rarely been made in the context of RIAs. Instead, uncertainty typically has been addressed qualitatively or through sensitivity analysis.

In this paper, we examine the role that formal quantitative assessment of uncertainty can play in regulatory analysis. In Section 2, we provide a brief overview of how such assessments of uncertainty can be developed, focusing on the execution of Monte Carlo analyses. In Section 3, we assess the potential benefits of uncertainty assessments for policy-making and in Section 4 we compare formal quantitative assessments with typical means of addressing uncertainty in RIAs, drawing on the results of three Monte Carlo analyses of uncertainty in the economic analyses of EPA’s Nonroad Diesel Rule, which became effective in 2004. We carried out one of these Monte Carlo analyses (Stavins et al. 2004) and EPA, two others as a part of the RIA that accompanied its final rulemaking. In Section 5, we consider the challenges and limitations of public agencies carrying out Monte Carlo analyses and in Section 6 we examine the potential value of making assessments of uncertainty more prominent in RIAs. We conclude in Section 7.

2. Implementation of Monte Carlo analysis in an RIA

The approach most commonly used by public agencies to characterize uncertainty in RIAs and other benefit–cost analyses is sensitivity analysis, which typically involves examining the effects on estimated net benefits of changing the values chosen for particular inputs to their highest and lowest possible values within some plausible range. For example, in a “base case,” the net benefits of a regulation might be computed with the expected (or the best-guess of the) number of lives saved per year by the regulation, say 15; and the sensitivity analysis might then carry out the same calculation of net benefits, sequentially using the analyst’s estimates of the lowest and highest possible number of lives that the regulation might save each year, say 5 and 25.

Such sensitivity analysis, particularly this sort of common, extreme-case sensitivity analysis, shows three critical problems.9 First, it fails to take account of important available information regarding assumed values of parameters. Typically, it is believed that values near the base-case assumptions are much more likely to occur than are values near the extremes of the range of plausible values. Therefore, the worst case and the best case are highly unlikely to occur (because they actually require the joint occurrence of a number of different low-probability events). In the example above, life-saving outcomes of 5 and 25 lives per year may both be exceptionally unlikely, whereas, outcomes between 10 and 20 may be much more likely to occur.

A second major problem with conventional sensitivity analysis is that it does not provide information about the dispersion (variance) of the distribution of net benefits.10 If, for example, there are two policies that cannot be distinguished in terms of the expected values of their net benefits, then a policy-maker might wish to choose the policy with the smaller variance (because that policy would have a greater likelihood of producing net benefits similar to the predicted expected value).11 With conventional sensitivity analysis, the policy-maker will not know which alternative has the lesser or greater variance, and thus which policy is more likely to achieve the desired result.

A third limitation of conventional sensitivity analysis is that such analysis typically involves the perturbation of one or perhaps two parameters at a time in isolation. But in the real world, uncertainty in net benefits is driven by the interaction of many variables and corresponding parameters. Only detailed simulations – in which many, if not all, parameters are allowed to vary simultaneously – can provide analysts with a reliable sense of potential outcomes, that is, resulting uncertainty in net benefits.12

Monte Carlo analysis provides a means of overcoming all three of these problems through a formal quantitative uncertainty assessment. The first step involves the development of probability distributions of uncertain inputs to an analysis.13 These probability distributions reflect the implications of uncertainty regarding an input for the range of its possible values and the likelihood that each value is the true value.

There are multiple sources of uncertainty in any regulatory impact analysis. Several taxonomies of such uncertainty have been developed (Finkel 1990; Morgan & Henrion 1990; Linkov & Burmistrov 2003). A useful composite taxonomy is offered by Krupnick et al. (2006), which recognizes four categories of uncertainty: variability, parameter uncertainty, model uncertainty, and decision uncertainty. Variability refers to the inherent diversity of an empirical quantity across space, time, or individuals. Such uncertainty cannot be reduced by further research (although there can be uncertainty about the distribution of variability). Parameter uncertainty refers to lack of knowledge about some empirical quantity, due to measurement error (including both random error and statistical variation and systematic bias), unpredictability, lack of data, or extrapolation errors. Model uncertainty, like parameter uncertainty, is due to lack of knowledge, but about fundamental, systematic behavior. Whereas parameter uncertainty is due to the practical limits of available data, model uncertainty is due to limitations in the ability of modelers to represent real-world systems in mathematical models. Finally, decision uncertainty refers to uncertainty regarding aspects of an analysis that reflect social valuation of regulatory outcomes, such as the choice of discount rate, for example.

These diverse sources of uncertainty can be accounted for by using a variety of methods that rely on existing data, expert judgment, or a combination of the two.14 But it should be recognized that only those sources of uncertainty that the analyst can characterize quantitatively can find their way into a formal quantitative analysis.15

Once probability distributions of inputs to a benefit–cost analysis are established, a Monte Carlo analysis is used to estimate the resulting probability distribution of the regulation’s net benefits by carrying out the calculation of benefits and costs thousands of times. With each iteration of the calculations, new values are randomly drawn from each input’s probability distribution and used in those calculations. Over the course of these repeated calculations, the frequency with which any given value is drawn for a particular input is governed by that input’s probability distribution. For example, if an input has a 30% chance of being 10, then, on average, 10 will be used as the value for that input in 30 out of every 100 calculations. If a sufficient number of calculations are carried out, the range of resulting net benefit estimates and the frequency of particular estimates within that range characterizes the probability distribution of net benefits.

The probability distribution of net benefits can be effectively approximated by dividing the range of net benefit estimates into a number of equal increments and then counting the frequency with which the repeated calculations produce net benefits falling into each increment. These frequencies can be represented pictorially by a histogram, providing a graphic image of the estimated distribution of net benefits (Fig. 1). The more random the draws that make up the Monte Carlo simulation, the more likely is it that the resulting histogram is a good approximation of the “true” distribution of net benefits.16 Such a histogram can provide an effective visual display of the distribution, so that a decision-maker can readily perceive not only the expected value, but also the spread and skewness of net benefits.17 Equally important, the numerical results of the Monte Carlo analysis can be used directly to calculate the expected value, the variance, and other important summary statistics characterizing uncertain net benefits.

Figure 1.

A histogram of estimated net benefits from a Monte Carlo simulation and the associated true probability distribution of net benefits, based on uncertainty in inputs to the net benefits calculation.

Note: The probability distribution show n here is symmetric, but relevant empirical distributions are frequently highly skewed (asymmetric).

3. Value of uncertainty assessments for policy-making

Assessments of uncertainty in regulatory analysis can lead to both short-term and long-term gains for policy-making. In the short term, such assessments allow more informed evaluations of proposed regulations and comparisons among regulatory alternatives. In the long term, such assessments can help establish research priorities.

Assessments of uncertainty can provide valuable information for policy-makers evaluating proposed regulations in at least two respects. First, findings regarding uncertainty in a net benefits estimate can provide a context for interpreting an estimate of the expected value of a regulation’s net benefits. Second, consideration of uncertainties in underlying inputs, and how those uncertainties interact, can lead to an estimate of the expected value of a regulation’s net benefits that differs from the estimate that would be produced if uncertainty in underlying inputs was not accounted for and single values were used for each input in calculating net benefits.

Uncertainty assessments provide insights regarding the distribution of possible net benefits associated with a regulation. Similar point estimates can be associated with vastly different distributions of possible outcomes around those estimates and the distribution of possible outcomes can significantly affect perceptions of an estimate of a regulation’s net benefits. For example, Figure 2 depicts the probability distributions of annualized net benefits for a hypothetical proposed regulation and for an alternative to that proposed regulation. The expected value of the annualized net benefits for both the proposed regulation and the alternative regulation are $1 billion. But there is a 20% probability that the proposed regulation will have negative net benefits, whereas there is less than a 1% probability that the alternative regulation will have negative net benefits.

Figure 2.

Probability distributions of annualized net benefits of a hypothetical proposed regulation and of an alternative regulation with the same expected value of net benefits.

There may even be occasions where the distribution of possible outcomes associated with a regulation should be given equal or greater consideration than the most likely or average outcome. Such occasions could arise when there is a low probability of an outcome involving either extraordinarily high costs or benefits. An obvious example would have been an analysis of investment in levee maintenance in New Orleans before Hurricane Katrina, as a thorough analysis of uncertainty would have shown that there was a low probability of a catastrophic outcome (due to the confluence of a number of factors). This potentially important characteristic of a regulation could not be discerned without going beyond an estimate of the expected value of net benefits to consider the full distribution of possible outcomes.

Assessments of uncertainty in benefit and cost estimates also can aid the evaluation of alternative approaches to proposed regulatory actions. As differences in the benefits or costs of alternatives may be one of many factors that influence which alternative is chosen, it is important that policy-makers understand the degree of confidence that can be placed in estimates of those differences. If two alternatives cost the same amount, and the first alternative is estimated to yield greater benefits than the second, the weight that policy-makers give to this difference, relative to other factors, should be influenced by the probability that the second alternative’s benefits could, in fact, be no less than those of the first.18

In addition to providing important context for interpreting point estimates from a benefit–cost analysis, there are circumstances in which consideration of uncertainty can lead to different point estimates than would be developed in a deterministic analysis (that does not account for uncertainty). First, careful consideration of uncertainty in inputs to benefit–cost analyses, which is necessary in the context of Monte Carlo analysis, can lead to different point estimates of the expected values for those inputs than may be used in a deterministic analysis. For example, the point estimate of an input used in a deterministic analysis may represent that input’s expected value using one particular estimation model. But, consideration of model uncertainty may show that other plausible models would yield significantly different point estimates. Therefore, a point estimate of the input’s expected value that accounts for the distribution of possible values resulting from model uncertainty would differ from an estimate based on just one model, which might be used in a deterministic analysis.

Second, an input may factor into an analysis in a manner in which the ultimate net benefit estimate will differ depending on whether only a point estimate is used for the input or the distribution of that input’s possible values is explicitly incorporated in the analysis. This is because for some mathematical functions applied to an uncertain input, an estimate of the function’s expected value that accounts for the distribution of the input’s possible values will differ from an estimate that results from applying the function to the expected value of the input.19 For example, if an uncertain input has an equal probability of being 2 or 6, the square of that input would have an equal probability of being 4 and 36, leading to an expected value of 20 (the average of the two possible outcomes). However, if one simply developed an estimate of the square of that input by determining the variable’s expected value, which is 4, and then squaring it, the result would be 16.

Third, correlations among uncertain inputs can cause the expected value of net benefits resulting from those uncertain inputs to differ from a net benefits estimate developed by using point estimates for the inputs.20 For example, if the values of two uncertain inputs are positively correlated, then the expected value of their product – accounting for uncertainties and this correlation – would be greater than the product of point estimates of each input’s expected value. Thus, if that product were a component of benefits, an analysis that does not account for uncertainties in the inputs and correlations between them would underestimate the expected value of benefits.

Uncertainty assessments also can help identify the most significant determinants of uncertainty regarding a regulation’s net benefits and thereby the potential for future research to reduce that uncertainty. In the short term, this information can be used to determine whether there may be benefits to delaying a rulemaking, in anticipation of future improvements in knowledge that will allow for more informed decision-making. In the long term, the cumulative information gained from uncertainty assessments in RIAs can be used to identify sources of uncertainty that have the greatest effect on estimates of the economic influence of regulations. This knowledge can be used to help establish research priorities.

4. Comparison of formal quantitative uncertainty assessments with conventional means of addressing uncertainty in RIAs

Sensitivity analysis is one of the most common methods used to characterize uncertainty in regulatory analysis. To compare sensitivity analysis with formal quantitative assessments of uncertainty, we examine the application of these two approaches in a recent regulatory analysis.

During the same period that OMB was finalizing its new guidance for regulatory analysis contained in Circular A-4, EPA was finalizing its Nonroad Diesel Rule. This Rule establishes new emissions standards for land-based, non-road diesel engines, such as those used in construction and in agricultural equipment.21 To facilitate the use of technologies needed to meet these standards, which are phased in beginning in 2008, the Rule requires significant reductions in the sulfur content of diesel fuel used in non-road engines. The Rule also requires reductions in the sulfur content of locomotive and marine diesel fuels.

With other investigators, we carried out a Monte Carlo analysis of uncertainty in the net benefits of the proposed Nonroad Diesel Rule (Stavins et al. 2004).22 In the RIA that accompanied EPA’s final rulemaking, EPA itself presented the results of two Monte Carlo analyses that address uncertainty in particular determinants of that Rule’s benefits (EPA 2004, appendix 9B). In the RIAs for the proposed and final rulemaking, EPA also used methods of assessing uncertainty that are more typical of RIAs. Thus, the Nonroad Diesel Rule provides a useful basis for assessing the merits of Monte Carlo analysis, relative to typical means of addressing uncertainty in RIAs.23

Our Monte Carlo analysis of EPA’s proposed Rule incorporated probability distributions for more than 60 inputs to EPA’s analysis, including inputs to both cost and benefit estimates.24 These probability distributions were largely limited to characterizations of statistical variation, using readily available data that EPA relied on in developing the RIA, and so we did not address all sources of uncertainty. Therefore, the resulting probability distributions of the Rule’s costs and benefits likely understate the extent of uncertainty in those estimates. The EPA’s Monte Carlo analyses only address uncertainty in the Rule’s benefits resulting from reductions in ambient concentrations of particulate matter (PM). In particular, EPA analyzes the implications for its benefits estimate of uncertainty in concentration–response (C–R) functions for certain PM-related health effects – most importantly premature mortality – and uncertainty in economic values of reductions in those effects.25

There is one important difference between EPA’s two Monte Carlo analyses. In one of EPA’s analyses (and in our analysis), only statistical variation is considered in characterizing uncertainty in the C–R function for PM-related premature mortality. These characterizations of uncertainty are based on the results of a single, observational epidemiological study, and thereby do not account for uncertainty as to whether the model used in the study is the correct model for estimating the C–R function. Model uncertainty, however, can be a major source of overall uncertainty.26 In EPA’s other Monte Carlo analysis, its characterization of uncertainty in the C–R function for premature mortality draws on expert judgments that account for some aspects of model uncertainty (EPA 2004, pp. 9–218).

4.1. Estimation of the range of possible economic impacts associated with a regulation

Because of the number of sources of uncertainty in a regulatory analysis and the complexity of their interactions, assessments of the extent of uncertainty in a regulation’s net benefits – or components thereof – that are conducted with ordinary sensitivity analysis are unlikely to represent the true extent of uncertainty accurately. For example, in the draft RIA for the proposed Nonroad Diesel Rule, EPA’s quantitative consideration of uncertainty regarding the Rule’s cost is limited to an assessment of the effect on net benefits of actual costs being 20% lower and 20% higher than its primary estimate (EPA 2003, pp. 9–49). But our Monte Carlo analysis indicates that there is more than a 5% probability that costs could differ from EPA’s primary estimate by more than 20%.27

The EPA also presents an assessment of the implications for its benefits estimate of scenarios in which reductions in fine PM2.5 emissions resulting from the Rule are 5% below and 5% above its primary estimate. Our Monte Carlo analysis finds, however, that there is more than a 75% probability that emission reductions will be outside EPA’s defined range of plus or minus 5%.

It is unlikely that a Monte Carlo analysis will comprehensively address all sources of uncertainty in the estimation of a regulation’s economic impact. Therefore, the results of such analyses will likely understate the range of possible outcomes. Nevertheless, ranges produced by such Monte Carlo analyses provide substantially more reliable information than do less rigorous means of addressing uncertainty.

4.2. Evaluation of the likelihood of particular outcomes within a range

By providing a full characterization of the distribution of possible outcomes, Monte Carlo analysis provides information on the probability of particular outcomes within a range. Such understanding of the likelihood of values within a range is essential for meaningful interpretation of the range. For example, when a range of possible net benefits is provided, some may assume that all values within that range are equally likely to be the ultimate outcome. But this is rarely the case. Others may assume that the distribution of possible net benefits is symmetric.28 This, too, is typically not the case. EPA’s second Monte Carlo analysis shows that, given the sources of uncertainty that it assesses, there is a 90% probability that actual benefits in 2030 will fall within the first half of its estimated range of possible benefits.29

Conventional methods of addressing uncertainty in RIAs, such as sensitivity analysis, are less likely to provide meaningful guidance regarding the probability that a regulation’s net benefits will fall above or below certain values.30 In investigating uncertainty in its estimate of the proposed Nonroad Diesel Rule’s benefits, EPA examines the implications of uncertainty in three income elasticities for the economic values of avoiding particular health effects.31 To do this, EPA calculates the benefit estimate that results when all three elasticities are set to their lowest value within the range that EPA believes to be plausible for each. Likewise, EPA examines the consequences of setting all elasticities at the high end of their respective ranges.

In conducting this exercise, the analysts are examining extremely unlikely scenarios. Even if, for each income elasticity, there were as much as a 20% probability that its true value is no more than the low estimate that EPA uses, there would be less than a 1% probability that the true values of all three elasticities are at or below these low values.32 But the fact that the specific scenarios examined are highly improbable does not imply that there is a low probability that benefits could be outside the established range. Although the probability of the particular scenarios may be small, uncertainties in many other inputs not considered in the exercise can lead to a high probability of benefits being outside the range. Indeed, the range that EPA establishes in this exercise is from 82 to 141% of its primary point estimate for 2030 benefits, but our Monte Carlo analysis indicates that there is at least a 73% probability that 2030 benefits will be outside that range.33

These are striking examples of how ordinary sensitivity analysis can offer misleading information regarding the range of net benefits that could result from a regulation, and the likelihood of various outcomes within that range. However, sensitivity analysis can be useful for indicating the extent to which uncertainty in particular inputs contributes to overall uncertainty in net benefits. But the implications of uncertainty in one input cannot be understood fully without the use of a formal quantitative assessment to characterize overall uncertainty.

4.3. Effect of uncertainty assessments on point estimates

In some circumstances, uncertainty assessments can change point estimates. EPA’s second Monte Carlo analysis provides an example. In that analysis, EPA develops a characterization of uncertainty in the C–R function for premature mortality that goes beyond uncertainty arising from statistical variation and incorporates other sources of uncertainty, such as model uncertainty. This leads to a probability distribution for anticipated reductions in premature mortality whose expected value is nearly 30% less than the point estimate EPA calculates in an analysis that does not account for these broader sources of uncertainty (Fig. 3).34

Figure 3.

Effects on the probability distribution of reductions in premature mortality in 2030 from EPA’s non-road rule when sources of uncertainty in the concentration–response function for premature mortality beyond statistical variation are considered.

Note: For each distribution, the box defines the range within which 50% of the Monte Carlo simulation’s estimates of premature mortality reduction fell; 25% of the estimates fell at or below the bottom of the box, and the remaining 25% of the estimates fell at or above the top of the box. The lines define the range within which 90% of the Monte Carlo simulation’s estimates of premature mortality reduction fell; 5% of the estimates fell at or below the bottom of the line and 5% fell at or above the top of the line.

Source: Adapted from US EPA (2004, pp. 9–240, figure 9B-4). These estimates are based on a preliminary EPA analysis of the Nonroad rule and are therefore not directly comparable to EPA’s final estimate of the Rule’s benefits.

As this shows, Monte Carlo analysis can indicate when uncertainties in inputs to a benefit–cost analysis cause the expected value of a regulation’s net benefits to differ from what would be suggested by a deterministic analysis. Sensitivity analysis may examine the implications of uncertainties that can bring about such results, but it cannot address all possible outcomes resulting from those uncertainties or indicate the probability associated with any one outcome. Therefore, unlike Monte Carlo analysis, sensitivity analysis cannot provide policy-makers with insight regarding whether uncertainties in inputs cause the expected value of a regulation’s net benefits to differ substantially from the primary net benefits estimate that would result from a deterministic analysis.

5. Challenges and limitations of Monte Carlo analysis

Several concerns have been raised regarding the potential use of Monte Carlo analysis in the context of Regulatory Impact Analysis, as well as within the broader context of risk assessment (Ferson 1996; Poulter 1998).

5.1. Is additional computational effort required?

Implementation of a Monte Carlo analysis imposes two requirements that are not strictly necessary to develop point estimates of benefits and costs. First, instead of requiring a single point estimate for each input to a benefit–cost analysis, Monte Carlo analysis requires the development of probability distributions for key uncertain inputs. Second, numerous repetitions of the calculations in a benefit–cost analysis must be carried out. These requirements may appear burdensome (Poulter 1998), but a reasonable balance can be struck that accounts for the additional information that can be gained from increasingly comprehensive uncertainty assessments, and the additional effort that is required to develop such assessments, relative to what is already expended on RIAs.

First, as with benefit–cost analysis, a Monte Carlo analysis does not need to be exhaustive to offer valuable insights. For example, EPA’s Monte Carlo analyses for its Nonroad Diesel Rule addressed the implications of limited types of uncertainty in just a few of the numerous uncertain inputs in its regulatory analysis. Nonetheless, those Monte Carlo analyses offer a much more informative assessment of uncertainty than is offered by sensitivity analysis alone.

Second, in developing probability distributions for uncertain inputs in a regulatory analysis, uncertainty from statistical variation can often be characterized with little additional effort relative to that needed to develop point estimates. Much of the data necessary for such characterizations already will have been collected for the development of point estimates.

Characterizing other sources of uncertainty in inputs, however, can require more effort. For example, EPA’s second Monte Carlo analysis required an expert elicitation to characterize uncertainty in the C–R function for premature mortality beyond that attributable to statistical variation.35 The amount of additional effort necessary to develop a Monte Carlo analysis can be minimized through careful consideration of which input uncertainties are worthwhile addressing in the analysis. The importance of more fully characterizing uncertainty in the C–R function for premature mortality in EPA’s analysis of the Nonroad Diesel Rule is made clear by the fact that reductions in premature mortality account for ninety percent of the Rule’s quantified benefits (EPA 2004, pp. 9–28).

Although a Monte Carlo analysis can require additional effort to characterize uncertainty in inputs to a benefit–cost analysis, that effort often may be warranted even in the absence of the needs of a Monte Carlo analysis. Such characterizations of uncertainty may be necessary just to develop an accurate point estimate for an input.

Developments in computer performance and software over the years have dramatically reduced the amount of effort required to conduct calculations for a Monte Carlo analysis, once input uncertainties have been characterized. Widely available software allows the execution of Monte Carlo analysis in common spreadsheet programs on a desktop computer.36 Also, modern programming techniques allow the writing of Monte Carlo computer programs with minimal additional effort, relative to that needed to produce point estimates. Nevertheless, Monte Carlo analysis may be resource intensive in some contexts, for example, in the case of EPA’s environmental systems models. Such resource demands call for striking a balance between thoroughness of analysis, on the one hand, and implementation costs, on the other.

5.2. Will greater use of Monte Carlo analysis bring about “paralysis by analysis”?

Will OMB Circular A-4’s requirement that formal quantitative uncertainty analyses be conducted for proposed regulations with annual benefits or costs expected to reach $1 billion tie up the regulatory system, leading to the so-called “paralysis by analysis?” In contrast, is the number of rules that exceed the $1 billion threshold so small as to render the requirement meaningless? Here are the facts. In a recent 1-year period, between October 2004 and September 2005, 3,980 final rules were published in the Federal Register. Of these, 292 were reviewed by OMB, 45 of which were deemed “major” final rules, given their anticipated economic impacts.37 Of these 45 major rules, only six had estimates of annual benefits or costs that exceeded $1 billion (OMB 2006).

Given that analysts can strike a balance between how comprehensive and resource intensive a Monte Carlo analysis is, it is unlikely that a requirement affecting so few regulations per year need significantly delay, let alone paralyze the regulatory process. However, because these analyses are targeted at the most significant rules – those exceeding a threshold 10 times greater than the threshold for an RIA – the insights they bring to the regulatory process can be of considerable value.

5.3. Is Monte Carlo analysis strictly appropriate only for certain types of uncertainty?

It is generally recognized, even by its critics, that Monte Carlo analysis is “relatively straightforward” for assessing the effects of parameter uncertainty and natural variability, characterizations of which are frequently available in useful statistical form (Ferson 1996). But model uncertainty, from this perspective, can be a greater challenge. This should not be denied. But model uncertainty can be addressed quantitatively, such as through expert elicitation methods, as shown by EPA’s own work in this regard. More importantly, it is crucial to keep in mind that failure to be comprehensive in including all sources of uncertainty by no means renders invalid a Monte Carlo analysis, which includes some important sources of uncertainty.

5.4. Does Monte Carlo analysis favor less stringent regulation?

A share of the general distrust that some advocates have for benefit–cost analysis appears to have spilled over to uncertainty quantification in this realm. It seems that some fear that simply acknowledging explicitly the degree of uncertainty that underlies the findings in RIAs will cause policy-makers and others to be less willing to impose regulations (Poulter 1998). If this is the case, it hardly seems that the appropriate answer would be to hide such uncertainty from regulators and the public. More to the point, however, is the reality that there is no reason a priori to expect an uncertainty analysis to favor benefits or costs. Indeed, formal quantification of uncertainty can be and has been supportive of greater regulatory stringency.38

Too much should not be claimed for Monte Carlo analysis, however. It should go without saying that “Monte Carlo methods cannot do everything, and they cannot solve all problems” (Ferson 1996). As with any analytical tool, formal quantification of uncertainty can yield results that are incorrect or unjustifiable if based on assumptions that are problematic or if the data employed are not empirically justified. Furthermore, if a Monte Carlo analysis is not appropriately explained, it can reduce the transparency of regulatory analysis. However, if properly executed and described, such analysis can be fully supportive of appropriate stakeholder participation (Farrow et al. 2001).

6. Prominence of uncertainty assessments in RIAs

Point estimates of regulations’ net benefits have been given far greater prominence in RIAs than discussions of uncertainty associated with them. Uncertainty assessments are frequently relegated to appendices and discussed in a manner that makes it difficult for readers to discern their significance. This may be inevitable, given that single point estimates can be communicated more easily than lengthy qualitative assessments of uncertainty or a series of sensitivity analyses. A Monte Carlo analysis produces a quantitative probability distribution of a regulation’s net benefits, a concise and effective summary of uncertainty that can be communicated almost as briefly as a point estimate.39 If a summary of uncertainty in an estimate is not given equal prominence with the estimate itself, essential context for interpreting the estimate is lost.

Some resistance to the use of Monte Carlo analysis and prominent presentation of its results may be due to the perception that such analysis requires more judgment and therefore makes results more speculative (National Research Council 2002). Also, some have expressed concern that, given the inevitably incomplete nature of any Monte Carlo analysis, prominently presenting its results would incorrectly lead readers to conclude that results are more certain than they are. Both concerns seem unfounded. First, developing characterizations of uncertainty in inputs for a Monte Carlo analysis often involves making explicit and transparent judgments that must be made in any event. Moreover, to the extent that a Monte Carlo analysis is incomplete in its characterization of uncertainty regarding a regulation’s net benefits, that fact can surely be communicated qualitatively.

7. Conclusions

Uncertainty is inevitable in estimates of regulations’ economic impacts and assessments of the extent and nature of such uncertainty can provide important information for policy-makers evaluating proposed regulations. Such information offers a context for interpreting benefit and cost estimates and can lead to point estimates of regulations’ benefits and costs that differ from what would be produced by purely deterministic analyses. In addition, these assessments can help establish priorities for research.

Due to the complexity of interactions among uncertainties in inputs to RIAs, an accurate assessment of uncertainty can be gained only through the use of formal quantitative methods. Such analysis involves relatively straightforward extensions to benefit–cost analysis. Its use can offer significant insights, while requiring only limited additional effort relative to that already expended on RIAs. Many of these insights cannot be gained using methods of addressing uncertainty that have typically been employed in RIAs, such as extreme-case sensitivity analysis. The analysis of EPA’s Nonroad Diesel Rule provides an example of the differences between the information that can be offered by Monte Carlo analysis and that offered by conventional means of addressing uncertainty in RIAs. We compared findings from the two methods and described how formal quantitative methods offer policy-makers a richer and more reliable characterization of uncertainty regarding a regulation’s economic impact even if such formal methods are not completely exhaustive in their treatment of uncertainty. The differences in information provided by the two methods result, in part, from the capacity of Monte Carlo analysis to account appropriately for interactions among the numerous uncertain inputs to a benefit–cost analysis.

The new requirement for formal quantitative assessments of uncertainty that is incorporated in OMB’s Circular A-4 can mark a significant step forward in enhancing regulatory analysis under Executive Order 12866. It has the potential to improve substantially our understanding of uncertainty regarding the economic influence of regulations and thereby to lead to more informed policy-making.


Helpful comments were provided by Cary Coglianese, Jody Freeman, Jason Johnston, Sheila Olmstead, the editors, and several anonymous referees. Financial support for our previous work on this subject was provided by ExxonMobil and the American Chemistry Council. The authors alone are responsible for any errors.


  • 1

    Executive Order 12866 requires RIAs for all rules that are likely to “have an annual effect on the economy of $100 million or more or adversely affect in a material way the economy, a sector of the economy, productivity, competition, jobs, the environment, public health or safety, or State, local, or tribal governments or communities.” See Executive Order 12866, 58 Fed. Reg. 51,735 (30 September 1993). If a regulation meets this criterion, or any of three other criteria, it is deemed a “significant regulatory action” under Executive Order 12866. However, Executive Order 12866 only requires an RIA for those significant regulatory actions that meet the first criterion, described above. In a 20 September 2001 memorandum, Dr. John Graham, the Administrator of the US Office of Management and Budget’s Office of Information and Regulatory Affairs, defined an “economically significant” rule as one that meets that first criterion, leading to the requirement for an RIA (Graham 2001). The $100 million threshold is not indexed for inflation and has not been modified over time. Although Executive Order 12866 was originally issued by the Clinton administration (Hahn et al. 2003), its RIA requirements have continued to be enforced by the administration of President George W. Bush, which chose not to issue a new Executive Order (Graham 2001).

  • 2

    Throughout the Reagan and George H. W. Bush administrations, RIAs were required under Reagan Executive Orders 12291 and 12498. Under Executive Order 12291, 46 Fed. Reg. 13,193 (17 February 1981), agencies were required to conduct RIAs for all proposed and final rules that were anticipated to have an effect on the national economy in excess of $100 million. This Executive Order has been called the “foremost development in administrative law of the 1980s” (Morgenstern 1997).

  • 3

    President Richard Nixon required a “Quality of Life” review of selected regulations, beginning in 1971 (Hahn 2000) and President Gerald Ford formalized this process in 1974 with Executive Order 11821, 39 Fed. Reg. 41,501 (29 November 1974). Subsequently, President Jimmy Carter’s Executive Order 12044 required analysis of proposed rules and centralized review by his Regulatory Review Group (Hahn 2000).

  • 4

    In cases where benefits cannot be monetized adequately, cost-effectiveness analysis can provide an economic framework for evaluating a regulation (OMB 2003, pp. 9–10).

  • 5

    Net benefits are benefits minus costs, all of which are expressed either as the present discounted value of future streams or as annualized amounts. For a general discussion of discounting in this context, see Goulder and Stavins (2002). Normative issues regarding the use of benefit–cost analysis as part of the policy process are beyond the scope of this essay. For a broad, recent review, see Adler and Posner (2001).

  • 6

    It should also be noted that some RIAs do not provide even single quantitative estimates of net benefits, because they fail to quantify or monetize impacts (Hahn et al. 2000).

  • 7

    Monte Carlo analysis is a means of statistical evaluation of mathematical functions using random samples. The probability of certain outcomes is approximated by running multiple trial runs, called iterations, using random variables. The “Monte Carlo” designation is a reference to the casino in Monaco. The method’s use of randomness and the repetitive nature of the process are analogous to the activities conducted in a casino. The phrase “Monte Carlo analysis” is often used (as it is here) to refer generally to probabilistic modeling that is conducted using sampling-based simulation approaches. However, among specialists it is sometimes used to refer to a specific method of drawing random values in the context of a sampling-based simulation.

  • 8

    Monte Carlo analysis may be best known for its use during World War II in the design of the atomic bomb. It has since been used in a wide range of applications, including the analysis of traffic flow on highways, the development of models for the evolution of stars, and attempts to predict fluctuations in the stock market (Hammersley & Handscomb 1964). For more recent texts on the use of Monte Carlo methods in engineering and finance see Ayyub (1997, 2003), Glasserman (2003), and McLeish (2005).

  • 9

    Despite the severe problems documented here that are inherent in sensitivity analysis (in contrast with formal quantitative assessments of uncertainty through Monte Carlo analysis), public agencies ought not be condemned for having relied in the past on this approach. The vast majority of textbooks and practical guides to carrying out benefit–cost analysis make no mention of Monte Carlo analysis, instead recommending conventional sensitivity analysis. See, for example, Brent (1996), Gramlich (1998), and Loomis and Helfand (2001). A key exception is the text by Boardman et al. (2001). EPA’s own revised guidelines for carrying out economic analysis mention the possibility of using Monte Carlo analysis, but in the most recent version (EPA 2000) do not provide practical guidance. Whereas Monte Carlo analysis is superior to sensitivity analysis when both are viable options, sensitivity analysis still has value in addressing the implications of uncertainty in inputs for which a reasonable probability distribution cannot be developed. As Circular A-4 indicates “Sensitivity analysis is especially valuable when the information is lacking to carry out a formal probabilistic simulation” (p. 41).

  • 10

    The variance of a random variable is a measure of its dispersion, indicating how far the possible values of that variable typically are from its expected value.

  • 11

    When the true value of a variable such as net benefits is uncertain, the “mean” or “expected value” is the weighted average of all possible values for that variable, where the weight assigned to each possible value reflects the probability that that value is the true value. The “most likely” value of that variable is the value that has the highest probability of being the true value.

  • 12

    For example, the severe loss of life in New Orleans due to Hurricane Katrina in 2005 was a function of interactions among several disparate factors: (i) a low-probability storm intensity; (ii) a low-probability storm trajectory; (iii) low-probability failures of the levee system; and (iv) difficulties in achieving high evacuation rates. Simulations focused on a scenario that incorporated just one or even a pair of these factors would have masked the full range of possible outcomes associated with a hurricane making landfall on the Gulf Coast.

  • 13

    A probability distribution is the mathematical expression (function) that gives the probabilities that a particular variable equals each of a sequence of possible values.

  • 14

    For a discussion of methods of characterizing these uncertainties, see Morgan and Henrion (1990).

  • 15

    Also, the types of uncertainty that should be accounted for in a Monte Carlo analysis depend, to some extent, on the objectives of the analysis. For example, although there may be uncertainty regarding the influence of a regulation on individual households and the variability across households, analysts may wish to exclude consideration of variability across households if the goal of the analysis is to understand uncertainty regarding general impacts, such as the median household.

  • 16

    By the law of large numbers, the frequencies in the histogram will converge to the “true” underlying probabilities as the number of random draws (iterations) approaches infinity.

  • 17

    Skewness refers to the degree to which a distribution departs from symmetry around its mean value. Referring to Figure 2, the probability distribution of annualized net benefits of the proposed regulation is highly skewed to the right (with some relatively small probability of very high values), whereas the probability distribution of annualized net benefits of the alternative regulation is symmetric.

  • 18

    In some areas of regulation, such as the environmental realm, there is frequently greater uncertainty surrounding estimates of benefits than estimates of costs. Whereas a Monte Carlo analysis may show this difference, it does not favor either side of the ledger. In other words, it does not bias the resulting analysis in either direction.

  • 19

    See Borodovsky and Lore (2000).

  • 20

    Two uncertain inputs are correlated if the probability that one input’s true value falls within a particular range of its possible values is related to, rather than independent of, another uncertain input’s true value. Correlations can result from either the direct effect of one input’s value on that of the other input, or from indirect relationships. Two inputs are positively (negatively) correlated if “high” values of one tend to be associated with “high” (“low”) values of the other.

  • 21

    For further details on the final rule, see 69 Fed. Reg. 38,958 (29 June 2004).

  • 22

    See Jaffe and Stavins (2004).

  • 23

    The EPA’s use of Monte Carlo analyses in evaluating this rule was intended to be illustrative only. Thus, the analyses did not lead to different regulatory decisions than EPA would have reached absent the Monte Carlo illustration. EPA states that the Monte Carlo analyses were designed to “illustrate how some aspects of … uncertainty … can be handled in a benefits analysis” (EPA 2004, pp. 9–29), and subsequently states that these analyses were not used to inform any regulatory decisions in the rulemaking (pp. 9–203).

  • 24

    In our analysis, no changes were made to the underlying benefit–cost analysis conducted by EPA. Therefore, the results from our Monte Carlo analysis can be compared with the results from other methods used by EPA to address uncertainty.

  • 25

    The C–R functions describe the relations between changes in ambient PM levels and changes in the incidence of particular health effects, such as premature mortality. They are used to estimate reductions in adverse health effects that can be expected from emissions reductions following the Rule’s implementation.

  • 26

    See, for example, Koop and Tole (2004).

  • 27

    Given that our analysis omits several significant sources of uncertainty in the cost estimate, it is likely that our analysis understates the probability that costs will be outside the range defined by EPA.

  • 28

    That is, some may assume that while various outcomes may have different likelihoods of occurring, actual net benefits would be just as likely to fall somewhere in the first half of the range of possible values as in the second half.

  • 29

    US Environmental Protection Agency (2004, pp. 9–245, figure 9B-9). EPA’s analysis of the Nonroad Diesel Rule’s benefits focused on benefits in 2 years, 2020 and 2030. Because it will take several decades for the fleet of existing non-road engines to be completely replaced with new engines incorporating the required emissions controls, benefits in 2030 are more representative of the Rule’s long-run annual benefits.

  • 30

    There are circumstances in which sensitivity analysis may provide insights of this type, particularly when there are very few uncertain inputs and the sensitivity analysis examines the implications of uncertainties in all inputs simultaneously. In practice, however, benefit–cost analyses in RIAs are rarely this simple.

  • 31

    US Environmental Protection Agency (2003, pp. 9-196–9-197). These income elasticities are used to estimate how the value that individuals place on avoiding PM-related health effects will change with changes in per-capita income.

  • 32

    This assumes that these income elasticities are uncorrelated.

  • 33

    US Environmental Protection Agency (2003, pp. 9–197, table 9B-6).

  • 34

    US Environmental Protection Agency (2004, pp. 9-213–9-245). In particular, see figure 9B-4.

  • 35

    Expert elicitation involves a “formal, highly structured and well documented process whereby expert judgments … are obtained” (EPA 2004, pp. 9–214).

  • 36

    Examples of such software include Crystal Ball and @Risk, which function as companions to spreadsheet programs, such as Excel.

  • 37

    To be deemed a major final rule by OMB, a rule typically must be expected to have annual costs or benefits of at least $100 million.

  • 38

    Poulter (1998) provides an example of how probabilistic analysis was used by the Sierra Club to oppose licensing of the Diablo nuclear power plant.

  • 39

    See Figure 1 for an illustrative example. The means of communicating uncertainty in such a fashion have been addressed in published work. See, for example, Morgan and Henrion (1990, Ch. 9); US EPA (1997); and National Research Council (2002, Ch. 5).