Severe uncertainty and info-gap decision theory

Authors


Correspondence author. E-mail: keith.hayes@csiro.au

Summary

  1. Info-gap decision theory (IGDT) seeks to provide a framework for rational decision-making in situations of severe uncertainty. The theory proposes non-probabilistic models of uncertainty and requires relatively small information inputs when compared to alternative theories of uncertainty.
  2. Info-gap decision theory has been criticised because it is based upon models that do not guarantee good decisions in situations of severe uncertainty, where severe means a ‘very large’ uncertainty space and very poor initial estimates of the unknown elements in this space.
  3. This paper reviews the use of this method in ecology where it is receiving interest in applied environmental management applications. Paradoxically, ecological applications of IGDT focus almost exclusively on only one source of uncertainty in ecological problems, model parameter uncertainty, and typically ignore other sources, particularly model structure uncertainty and dependence between parameters, that can be just as severe.
  4. Ecologists and managers contemplating the use of IGDT should carefully consider its strengths and weaknesses, reviewed here, and not turn to it as a default approach in situations of severe uncertainty, irrespective of how this term is defined. We identify four areas of concern for IGDT in practice: sensitivity to initial estimates, localised nature of the analysis, arbitrary error model parameterisation and the ad hoc introduction of notions of plausibility.

Introduction

The discipline of ecology is central to responding to a range of major challenges facing the world community over the coming years. For example, how do we predict climate impacts and respond to them effectively to protect biodiversity and human amenity? How do we design efficient agricultural systems to feed a growing population while not eroding the natural resource base? All of these issues will require tactical and strategic decision-making by governments and communities who will need to trade off different interests and philosophies. One challenge in doing this is the uncertainty that exists in our ability to understand systems and predict future outcomes.

Uncertainty is pervasive in ecology. Variability, observation error, model structure uncertainty and other sources of uncertainty are to be found in virtually all applied ecological problems. Environmental systems are complex and sometimes poorly observed. As a result, environmental managers are often compelled to make decisions in situations with incomplete information. This has resulted in a significant literature around decision-making under uncertainty. The vast majority of approaches use probability to describe uncertainty, and when relevant data are extensive, and questions are essentially empirical, a wide range of statistical approaches to inference are available. When data are sparse, or the system complex, Bayesian probabilistic methods are still well defined but this process relies on expert opinions expressed through priors and can require costly or difficult elicitation processes. When data or knowledge is very sparse, as in the case of ‘complete ignorance’ (if this in fact exists), fundamental questions about the suitability of probability arise (Colyvan 2008).

Given these issues, any new method that purports to provide a mechanism for making good decisions with sparse information is likely to attract the attention of ecologists and environmental managers. In particular, a method and theory that purports to make robust decisions in the face of ‘severe uncertainty’ is a major achievement for science. Info-gap decision theory (IGDT) makes these claims. Ben-Haim (2001, p.x) describes IGDT as radically different from all current theories of decisions under uncertainty. IGDT is different because it offers a non-probabilistic approach to decision-making under uncertainty. It claims to provide a quantitative representation of Knight's concept of true uncertainty for which ‘there is no objective measure of probability’ (Ben-Haim 2004). It has been identified as a method that is suitable when the information base is so depauperate that the analyst cannot parameterise a probability distribution, decide on an appropriate distribution or even identify the lower or upper bounds on possible parameter values (Halpern et al. 2006).

Since the first comprehensive description of the theory in 2001, IGDT has attracted many proponents, principally in the fields of ecology, engineering and economics. Counting citations on http://info-gap.com/ suggests that there are three books and at least 140 journal articles on this subject, which are becoming increasingly well cited (Fig. 1). It has had a significant presence in ecology being applied to a wide range of important issues, including the allocation of resources to mitigate biodiversity impacts caused by global change and the design of effective marine reserves.

Figure 1.

Annual number of info-gap decision theory publications, and their citations, retrieved by the search (‘Info-Gap’ OR ‘Information Gap Decision Theory’) applied to the topic field, across all years and all databases, in Thomson Reuters Web of Science. Note that records for 2013 are incomplete.

Info-gap decision theory, however, has also been criticised on two grounds: (i) it is not a radically new theory but rather a reformulation of minimax analysis that has been known in the operations research literature for over 60 years and (ii) its robustness functions are ‘radius of stability’ models that only measure the robustness of a decision in the local neighbourhood of an initial estimate and not over the global uncertainty space. Hence, IGDT is utterly unsuitable to situations of severe uncertainty (Sniedovich 2008, 2008, 2010a, 2012). Ben-Haim maintains that this criticism misses the point (Ben-Haim 2012). Sniedovich (2008) bases his arguments on mathematical proofs that may not be accessible to many ecologists but the impact of his analysis is profound. It states that IGDT provides no protection against severe uncertainty and that the use of the method to provide this protection is therefore invalid.

The meaning of severe uncertainty lies at the heart of Sniedovich's second criticism and the subsequent debate within the academic community that has ensued. The term ‘severe uncertainty’ is not formally defined in IGDT. By inspection of IGDT's models of uncertainty, working assumptions and assertions, however, Sniedovich (2010b, 2012) identifies three defining characteristics: (i) the uncertainty space can be unbounded (Ben-Haim 2006, p. 210), (ii) any initial estimate of the elements within this space may be substantially wrong (Ben-Haim 2006, p. 281), and no better than a wild guess (Ben-Haim 2010, p. 2), and (iii) there are no grounds for believing that the truth is more or less likely to be in the neighbourhood of the initial estimate than in the neighbourhood of any other point in the uncertainty space.

While these characteristics provide a definition of severe uncertainty, it is not universally accepted in public debates. Burgman (2008) stipulates that Sniedovich's definition of severe uncertainty is too narrow to be useful and maintains that in the applied world of environmental decision-making, identifying decisions that are robust to the uncertainty in an initial estimate is vital. In particular, Burgman (2008) argues that IGDT is useful because there comes a time when decisions have to be made, and when this occurs, it is useful to know how much deviation around an initial estimate can be accommodated before critical performance requirements are threatened, irrespective of how one arrives at the initial estimate. Furthermore, during this process, it is understood that there is no absolute guarantee that the truth lies in the local neighbourhood of the initial estimate.

There is therefore a tension among (i) the proposed method, IGDT, (ii) a set of mathematical proofs that relate it to known techniques and clarify that it cannot deal with ‘severe uncertainty’, (iii) a rhetorical argument justifying its practical use and (iv) a community of practice that is evolving around its application. To further discussion of IGDT, we review the use of this method in ecology and provide an accessible description and analysis of the method. We also present a discussion of the issues that a potential user needs to consider in applying this approach. In Section ‘'Info-gap decision theory'’, we give a concise synopsis of IGDT. While the notation is necessarily complex, this synopsis involves no abstract mathematics or proofs and concludes with a simple example to illustrate the approach. In Section ‘'Review of ecological practice'’, we examine the most important sources of uncertainty in model-based decision-support systems and the manner in which these are addressed in ecological applications of IGDT. Section ‘'Practical utility'’ identifies practical issues that can occur with IGDT under strict or more relaxed definitions of severe uncertainty, together with a synthesis of issues identified by other authors and original insights we have gained in the process of completing this review. Section ‘'Discussion'’ discusses the IGDT method from an applied ecological perspective and concludes with recommendations for ecologists who are contemplating the use of IGDT.

Info-gap decision theory

The basics

Info-gap decision theory seeks to find decisions that are robust to uncertainty. It measures the robustness of a decision by showing how wrong an initial estimate of the uncertain elements of a problem can be before the rewards associated with this decision fall below a critical user-defined threshold. It encourages the analyst to compare alternative decisions and switch decisions or amend their aspirations for critical requirements, in order to buy immunity against uncertainty.

The theory is based on the following elements:

  1. A decision space Q that includes a number of alternative decisions, actions or choices (q ∈ Q) available to a manager. The decision space can lie on the real line ℜ, such as the distance between marine reserves, or it may be a discrete set of choices in a set, such as alternative management strategies for an endangered species;
  2. An uncertainty space S that includes all the uncertain elements of a problem. These elements may be scalars, vectors, functions, or sets of these things;
  3. A reward function R that measures how successful the decision is. The reward function is a mapping from the domains of the decision space and uncertainty space to the real numbers ℜ. In ecological applications, the reward function is typically a utility function or a process-based model, and the uncertain elements are usually some or all of the model's parameters. The range of the reward function may be bounded, such as the probability of extinction, or unbounded, such as the net present value of a management strategy. Associated with the reward function is a critical value inline image that the manager specifies and aims to meet or exceed; and
  4. A non-probabilistic model inline image for the uncertain quantities u in the reward function, parameterised by the non-negative parameter α that measures uncertainty in terms of the disparity between an initial estimate of the uncertain quantities inline image and other possible values. Formally, inline image is a mapping from inline image into the power set of S, where D is a subset of S (Ben-Haim 1999). Put more simply, inline image defines neighbourhoods in the uncertainty space S (see below).

Info-gap decision theory uses these constructs to identify decisions based on their robustness and opportunity. The opportunity of a decision is the minimum amount of uncertainty that enables the possibility of outcomes that exceed the critical reward inline image. We focus here, however, on the robustness of decisions for brevity, but also because the opportunity function is rarely used in applied ecological problems.

The robustness function, denoted inline image, defines the robustness of a decision q to be the maximum amount of uncertainty α such that the minimum reward (influenced by uncertain quantities u) associated with the decision, min R(q,u), is greater than the critical reward inline image.

display math(eqn 1)

The aim of IGDT is to help guide the decision maker in choosing the best decision where best is defined by the criteria inline image – that is, the decision with the greatest robustness for a given uncertainty model, reward function and initial estimates inline image of the uncertain elements in this function.

Info-gap decision theory prescribes a variety of non-probabilistic models of uncertainty. The most popular is the fractional error model that creates an expanding interval around the initial estimates of uncertain parameters inline image

display math(eqn 2)

with weight parameter inline image such that inline image. This model creates a family of nested intervals inline image, inline image, in n dimensions, that become broader as the value of α, known colloquially as the ‘horizon of uncertainty’, increases (Ben-Haim 2006).

The weight parameter inline image allows the analyst to moderate the influence of individual parameters on the horizon of uncertainty, for example setting low values (inline image) for parameters that are known to be less variable or less uncertain than others. Ben-Haim (1994, 2006) describes a variety of other non-probabilistic uncertainty models, including examples that have been subsequently used in ecological applications, such as the envelope-bound model (inline image), where ψ is a function determining the shape of the envelope, and ellipsoid-bound models (inline image), where V is a positive definite real symmetric matrix that describes the association between the elements of u.

A simple illustrative example

While the mathematics of IGDT can be intimidating, the concepts are in fact quite straightforward. In this section, we present a highly stylised, conservation management problem to demonstrate the key issues. Assume we have an area that will be set aside to conserve a single species. Our decision space Q consists of two possible management regimes. Regime 1 involves culling feral predators in the reserve, while regime 2 involves constructing nest boxes to increase the number of nesting sites. We assume that there is a probability inline image that the species persists over a particular period under regime 1 and inline image the corresponding probability under regime 2. The reward in this case inline image is the probability that a species does not go extinct given the choice of regime, so inline image. We use a fractional error model for inline image, q = 1,2 with the bounds inline image denote inline image as the initial estimate required by IGDT, and set the weight parameters inline image such that

display math(eqn 3)

A key point to note is that the choice of the weight parameters assumes, in some undefined way, that we understand how the uncertain parameters vary together. In this case, we assume that they in some sense (undefined) scale fractionally equivalently.

Our objective is to maximise the probability that the species will persist in the reserve. Given the uncertainty model [3], the minimum reward is

display math(eqn 4)

Equating the minimum reward to the critical value inline image, and assuming that inline image, we find that inline image. Solving for α results in the robustness function

display math(eqn 5)

Hence the robustness is a simple piecewise linear function with slope and intercept determined by the initial estimate inline image and a lower bound of zero. The left panel of Fig. 2 plots inline image on the x-axis and R(q,u) on the y-axis. This plot is referred to as a robustness plot. In this simple example, reward is measured by the probability of persistence, and the robustness curves do not intersect (unless the initial estimates are equal). In more complicated settings, however, they may intersect so the most robust strategy, as defined by IGDT, will depend on α and the critical reward. We note that the core concept of the IGDT method is that given a critical reward we choose the most robust strategy. The critical reward (we assume inline image) in this example is shown by the dotted line in the left panel of Fig. 2. By examining this panel, it is clear that according to IGDT, regime 1 is the best because its robustness is higher than that of regime 2 at the critical reward.

The right panel of Fig. 2 plots the robustness curve in the two-dimensional parameter space (inline image). The panel shows the neighbourhood inline image as a nested set of rectangles whose area increases with α, and how the minimum reward (Fig. 2, left; eqn (eqn 4)) is the lower left corner of these rectangles. Thus, the robustness scales linearly with α as the robustness curve tracks the lower left corner of each rectangle (Fig. 2, right).

The decision space in this example is delineated into two regimes by the line inline image (solid line of Fig. 2b). The robustness curve lies entirely in regime 1, hence option 1 is unambiguously identified as the preferred decision choice. Note that different choices of inline image can lead to different preferred decisions. In particular, arbitrarily small deviations in the initial estimate along the boundary inline image will lead to changes in the preferred management decision. In more realistic higher dimensional examples, the decision space will be delineated by more complex planes or surfaces. The effect of small deviations along these planes or surfaces, however, will remain the same.

Figure 2.

(Left) Robustness functions for a stylised conservation problem. The dotted horizontal line shows a hypothetical critical reward (minimum probability of persistence) of inline image. (Right) Parameter space plot showing the division of the decision space into two regimes, the robustness function (long-dash line), horizon of uncertainty (short-dash rectangles) and critical reward (dotted lines).

Review of ecological practice

There are at least 23 IGDT studies published within the ecological literature. The principal application problems revolve around:

  1. Surveillance and control programs for invasive species. The objective here is to test the robustness of different surveillance strategies, inspection protocols or eradication strategies, to the uncertainty associated with the introduction, establishment and spread of non-native species (Burgman et al. 2010; Carrasco et al. 2010; Davidovitch et al. 2009; Moffitt et al. 2007, 2008; Rout et al. 2009; Yemshanov et al. 2010a, b);
  2. Conservation strategies for endangered species. These studies examine the robustness of different conservation methods, such as translocation, captive breeding and poaching control, to the uncertainty associated with the efficacy of these methods and the biological processes (mortality, reproduction) that control the extinction risk (van der Burg & Tyre 2011; Crone et al. 2007; McDonald-Madden et al. 2008; Regan et al. 2005); and
  3. The design of nature reserves. These studies examine the effects of uncertainty associated with habitat quality, presence, persistence and/or dispersal of species on decisions about the size and connectivity of reserve networks (Halpern et al. 2006; Moilanen & Wintle 2006; Moilanen et al. 2006; Nicholson & Possingham 2007).

Other ecological issues that have been examined with IGDT include calculating offset ratios for impacted habitat, managing biodiversity and wetlands under different climate impact scenarios, price-based vs. quantity-based pollution control strategies, the effects of fire risk on forest management strategies, the effects of uncertainty on multicriteria decision analysis and an analysis of animal foraging behaviour (Carmel & Ben-Haim 2005; Levy et al. 2000; McCarthy & Lindenmayer 2007; Moilanen et al. 2009; Stranlund & Ben-Haim 2008; Walshe & Massenbauer 2008; Wintle et al. 2011).

The reward function, context, uncertainty model and parameter bounds associated with 20 of the 23 environmental studies published to date are summarised in Tables S1–S4 (Supporting information). We have excluded from this summary the studies by Moffitt et al. (2007, 2008), and Wintle et al. (2011) because these authors do not describe the uncertainty model used in their analysis.

Severe uncertainty in ecology

The ecological literature typically presents IGDT as a new technique that can better accommodate severe uncertainty. There are, however, many sources of uncertainty in applied ecological problems. In quantitative, model-based decision-support systems, the principal sources are observation error, model structure uncertainty, variability in, and limited understanding of, the true values of model parameters and finally the association or dependence between model parameters. It is against these sources of uncertainty that the credentials of a robust decision support system for ecological problems should be judged.

Model structure uncertainty

Model structure uncertainty occurs in the processes that are included or excluded from the model (such as density dependence, competition, age structure), the relationships between variables that represent processes that are included in the model, the scale and resolution of the model, and whether or not stochastic forces are included. There is ample evidence in the literature that decisions regarding these issues have a significant impact on the predictions and accuracy of ecological models (Arhonditisis & Brett 2004; Fulton et al. 2004; Hill et al. 2007; Los & Blaas 2010; McCarthy et al. 1995; Pascual et al. 1997; Wood & Thomas 1999).

Model structure uncertainty qualifies as ‘severe’ in many practical situations: the uncertainty space is often very large, if not infinite, except perhaps for classes of nested models, and while ecological models are not usually made of wild guesses, they often are a poor representation of reality and may be substantially wrong. Furthermore, in the absence of observations, it can be difficult to a priori identify which model, among a set of competing models, is likely to be the most plausible representation of reality.

Model structure uncertainty in IGDT occurs in the reward function and uncertainty model. In all but three of the current ecological applications of IGDT, however, the solution is predicated on a single reward function – that is, a single model that describes the process and/or utilities associated with the problem in hand, although alternative possible models are available. Furthermore, in all but one example (Rout et al. 2009), all of these solutions are predicated on a single uncertainty model. The fractional error model is by far the most widely employed – 14 of the 20 studies summarised here use it. Three of the remaining six examples use an envelope-bound model, while the other examples use a proportional error model and ellipsoid models.

There are four general strategies for tackling model structure uncertainty – ignore it, compare alternative structures, envelope the effects of alternative structures and average across them (Hayes 2011). IGDT can accommodate the first three approaches but the most prevalent approach in current ecological applications is to simply ignore it. Moilanen et al. (2006) and Moilanen et al. (2009) provide examples that compare alternative model structures, while McDonald-Madden et al. (2008) envelope the effects of alternative structures.

Parametric uncertainty

The parameters of ecological models are usually subject to two sources of uncertainty: ecological parameters often represent processes that are variable (growth rates, fecundity, dispersal distances, carrying capacity, etc.) but we may also be unsure of how to accurately characterise this variability. In our experience, however, parametric uncertainty in ecological problems is not severe, at least not in the sense of IGDT's criteria. The domain of the variables of an ecological model are typically not ‘large’ or unbounded because biological and physical parameters almost always have feasible constraints. Point estimates for these parameters are often informed by direct observation, information from similar species and environments, or even allometric relationships, and only occasionally are these estimates of such extremely poor quality that they are no better than a wild guess. Furthermore, ecologists are often able to identify values for uncertain parameter that are more likely than others.

Our assertions in this regard are supported by the ecological applications of IGDT. The uncertain parameters of 14 of the 20 cases reviewed here are bounded by an interval with pre-determined upper and lower limits. Ten of the ecological applications ‘info-gap’ proportions, probability or a set of probabilities, for example the extinction probabilities of endangered species under different management regimes. Here, the range of α is implicitly constrained as in the simple example described above. In three cases, constraints are explicitly placed on the uncertain parameters in the problem – namely the utility of action–state pairs in a Bayesian network, the spread velocity and radius of an invaded area (Table S1, Supporting information) and the performance level of policy alternatives (Table S4, Supporting information). Finally, in two cases, the range of α is explicitly restricted to lie on a particular range but with no clear justification (Table S1, Supporting information).

Several applications also base their initial parameter estimates on extant information. van der Burg & Tyre (2011), for example, use field surveys in Colorado as a basis for nest success estimates in Nebraska. Halpern et al. (2006) use estimates of dispersal distances from a log-normal distribution fitted to estimates from many different species. Nicholson & Possingham (2007) are able to quote dispersal distance estimates for the actual species of interest in their study area. In these instances, the initial estimates of uncertain parameters are clearly not wild guesses but rather estimates of a much better quality.

Dependence

There are at least two reasons why the uncertainty about the dependence between a model's parameters is likely to be more severe than the uncertainty that surrounds the marginal knowledge of the parameters themselves. In the first instance, the uncertainty space can be large: as noted above, ecological parameters usually represent variable quantities or processes, and these quantities may be related to one another in a linear or nonlinear fashion. Moreover, nonlinear forms of association can take the form of central dependence, upper tail dependence, lower tail dependence or indeed combinations of these (Denuit et al. 2005). Secondly, dependence between ecological parameters arises through the direct and indirect interactions and feedback that occur naturally in ecological systems or as a result of anthropogenic impacts (Dambacher & Ramos-Jiliberto 2007; Vinebrooke et al. 2004). Many of these interactions are subtle and sometimes non-intuitive. It is therefore quite possible that an initial dependence model will be wrong and it may be difficult to assess whether or not the true dependency model lies in the neighbourhood of an initial estimate.

As a worst-case analysis, IGDT implicitly accommodates all forms of dependence for a given horizon of uncertainty and the info-gap error models chosen. In reality, the parameter space is often constrained by dependence between ecological components and processes that are represented by parameters in the reward function, and this should be reflected in the choice and parameterisation of the error model. The fractional error model, prevalent in ecological applications, implies that the horizon of uncertainty increases in each dimension with perfectly positive linear dependence. This model does not allow the analyst to incorporate information about more complicated relationships between parameters and the constraints that this may impose on the feasible parameter space. In extreme situations this could lead to rewards that are impossible in the real world.

Practical utility

Sensitivity to the initial estimates

Info-gap decision theory's non-probabilistic uncertainty models create a family of nested sets that are centred on, and emanate out from, the vector of initial estimates (inline image; Ben-Haim 1999). The quality of this initial estimate is important because IGDT conclusions are contingent upon, and vary with, this estimate. The example in Section ‘'Info-gap decision theory'’ demonstrates this sensitivity. The result of the analysis changes depending on the initial guess, and the preferred strategy is directly decided by it.

Figure 3 replicates two published examples and demonstrates further the sensitivity of IGDT recommendations to the initial estimate. The example in the top panel is a biosecurity surveillance problem. The decision space Q consists of two surveillance strategies, and the uncertain parameter is the probability p of detecting a pest species in a single sample (Davidovitch et al. 2009). The recommended strategy is the one with the greatest robustness for a critical detection probability  ≥ 0·8, but this changes between the two strategies when the initial estimate of the detection probability halves from 0·22 to 0·11. The example in the bottom panel is marine reserve design problem. The decision space consists of four discrete choices of reserve spacing d = [25,50,100,500] (in kms), and the uncertain parameter is the inverse of the species' mean dispersal distance β (Halpern et al. 2006). In this case, the authors do not identify a critical persistence probability but suggest that d = 25 kms is the best choice because this achieves the greatest persistence probability across most levels of uncertainty. This conclusion, however, is based on an initial estimate inline image set to the mean of a log-normal distribution for single-generation fish dispersal distances. Repeating this analysis with the initial estimate set to the mode of this distribution suggests that d = 100 kms is a better strategy over the range of uncertainty considered by the authors, and that d = 25 is only preferred when uncertainty becomes an order of magnitude higher.

Figure 3.

Replicated results of two ecological studies showing how info-gap solutions can be sensitive to initial estimates. (Top) Biosecurity surveillance problem where the probability of detecting a pest species in a single sample (p) is uncertain. The original recommended strategy (Strategy 1) cannot meet the critical performance requirement (detection probability  ≥ 0·8, vertical dashed line) if the initial estimate of detection probability (inline image) is halved from 0·22 to 0·11. (Bottom) Marine reserve problem showing how the robustness of different strategies [distance (d) between reserves in kms] varies with the initial estimate of the inverse of a species' mean dispersal distance (inline image). Note scale change on the robustness axis.

Clearly, the effect of the initial estimate on IGDT conclusions will vary on a case-by-case basis. These examples simply highlight the possibility of a better management strategy under different initial conditions. This might seem obvious but this facet of the theory is not explored in the ecological literature. Rout et al. (2009) is a rare exception in this regard, warning that IGDT should not be mindlessly applied and should not be applied where the uncertainty is so severe that a ‘reasonable’ initial estimate cannot be selected. What constitutes a reasonable estimate, however, remains unclear and implicitly evokes concepts of likelihood, which the theory specifically precludes.

Local nature of the analysis

The worst-case nature of the robustness defined in eqn (eqn 4) has implications for the scope of the analysis. Even though the uncertainty model is unbounded in some cases, the extent of meaningful analysis concludes as soon as the horizon of uncertainty includes a possibility with the minimum possible reward. This occurs because the uncertainty sets are nested, hence all further horizons will have this minimum reward. This is explained clearly by Sniedovich (2012, p. 1635) and is at the core of his critique.

Sniedovich's analysis is a rigorous mathematical treatment and we do not repeat this analysis here. We illustrate the issue using the simple example presented in Section ‘'Info-gap decision theory'’. For illustration, assume that history in other locations suggests that even if a strategy is successful in a specified time period, if the probability of persistence is <0·45, there is no long-term hope of the species' survival. In this case, the reward is zero as soon as the probability of persistence goes below 0·45. This occurs when α = 0·44 for strategy one and α = 0·18 for strategy two. Thus, no consideration of possibilities outside these horizons occurs. The effect of this cut-off on the robustness calculations is shown in Fig. 4. This contrived example highlights the limited consideration of the parameter space in most IGDT analysis.

Figure 4.

Robustness functions for the stylised conservation problem assuming no reward if the probability of persistence is <0·45. The bold line delimits the extent of the parameter space explored.

Parameterisation of error models

Info-gap decision theory uses a single parameter α to represent the level of uncertainty in its error. While mathematically convenient for the theory, and necessary to produce simple robustness plots, this introduces a range of practical problems in an actual analysis. Real applications typically have multiple uncertain parameters and questions about dependency and relative uncertainty of parameters becomes important. The IGDT method attempts to address relative uncertainty by incorporating a weight into the error models. An example is given in eqn (eqn 2). We have repeated our simple example with a weight of 0·3 for the uncertainty model for inline image. We show this analysis in Fig. 5. The interpretation of this weight, however, is not clearly defined, and the choice of the best management regime can be sensitive to it as demonstrated in the figure. In the absence of any notion of plausibility, likelihood or probability, it is difficult to explain what the weight 0·3 means. In ecological practice, at least to date, the weight parameter is always set at 1, implying that none of the uncertain parameters in the problem are better known or characterised than any of the others. Ferson & Tucker (2008) identify this as a basic weakness that limits practical application because in most practical situations, some part of the problem is better known than others.

Figure 5.

Robustness functions for the stylised conservation problem assuming weighting of 0·3 for the info-gap uncertainty model for inline image.

Plausibility, probability and the horizon of uncertainty

Rout et al. (2009) argue that IGDT 'challenges us to question our belief in the nominal estimate, so that we evaluate whether differences within the horizon of uncertainty are plausible'. Wintle et al. (2011) assert that IGDT can ‘identify the decision option that is most robust to a range of plausible levels of uncertainty’. Burgman et al. (2010) notes that if the robustness curves of different decision strategies cross, then the analyst may need to choose which side of the crossing values is more plausible in order to decide between strategies. IGDT, however, asserts that we need no concept of chance, frequency of recurrence, likelihood, plausibility or belief in order to speak of uncertainty (Ben-Haim 2001, p. 18).

This discrepancy between the evolving practise and IGDT's original assertions arises because IGDT conclusions are only unambiguous when the robustness curves do not cross. For a given critical reward, the strategy with the highest robustness is chosen. As argued previously, there are no guarantees that this is a good strategy, but there is no confusion about its choice. If the robustness curves cross, however, the most robust decision depends on the critical reward and the level of uncertainty, and this invites people to consider what level of uncertainty is plausible.

If a decision maker can specify the plausibility of the level of uncertainty in a parameter, then they have gone a long way towards specifying a probability. We stress that IGDT theory does not recommend this approach. The theory asserts that you choose the strategy with the greatest robustness for a given critical reward. We highlight this point, however, because the form of the robustness curve invites this analysis and the ecological literature reflects this view.

Info-gap decision theory provides no measure or mechanism to express plausibility unless the initial estimates inline image represent ‘best’ estimates, where ‘best’ means the estimates that an expert believes are most likely to be true. In this case, deviations from the best estimates can be considered increasingly unlikely, such that plausibility scales, in some monotonic fashion, with α. Some authors, such as Hall & Harvey (2009) for example, have already adopted this interpretation and assumed that the values of u become increasingly unlikely as they diverge from inline image.

In situations of severe or mild uncertainty, one might argue that an expert can have a belief in a best guess, but have no feeling for the relative plausibility of other possibilities. We do not believe, however, that this is a sensible proposition. It would seem particularly fortuitous that an expert can assess that their best guess is more likely than all other possibilities, but that they could not make another judgement about the likelihood of the other possibilities.

Discussion

Promoting IGDT as a method for severe uncertainty has helped generate the interest, and controversy, that currently surrounds the theory. It is not, however, the only approach to uncertainty when one cannot reliably identify a precise probability distribution, and it is not the only non-probabilistic theory of uncertainty. It is not therefore a theory that ecologists should feel compelled to use in data-poor situations but rather a choice based on significant reflection about the context of the analysis, and the strengths and weaknesses of the theory.

We believe, however, that most ecological applications of IGDT have taken the claims of Ben-Haim (2006) at face value. A typical example of this is the analysis presented in Halpern et al. (2006). This paper reviews uncertainty techniques applicable to marine reserve design and provides a taxonomy presented in Fig. 6. This proposed taxonomy presents info-gap as providing a technique that can accommodate the ‘largest’ amount of uncertainty and asserts that this uncertainty is unbounded.

Figure 6.

Figure from Halpern et al. (2006) providing a taxonomy of uncertainty techniques. (Reproduced by permission of John Wiley and Sons Inc © 2006).

The literature and discussion presented in this paper demonstrate that the results of Ben-Haim (2006) are not uncontested. Mathematical work by Sniedovich (2008, 2010a) identifies significant limitations to the analysis. Our analysis highlights a number of other important practical problems that can arise. It is important that future applications of the technique do not simply claim that it deals with severe and unbounded uncertainty but provide logical arguments addressing why the technique would be expected to provide insightful solutions in their particular situation.

To assist in this process, we identify four issues that should be considered carefully before applying IGDT to real problems:

  1. Sensitivity to initial estimates,
  2. Localised nature of the analysis,
  3. Error model choice and parameterisation,
  4. Notions of plausibility.

Viewing the robustness curve as a sensitivity analysis is problematic. Sensitivity in the neighbourhood of a poor estimate gives no insight into sensitivities elsewhere in the parameter space. If we wish to make an assumption that the true value is in the neighbourhood of our best guess, we are working in a context that is inconsistent with the stated motivations of the IGDT method. In particular, this would imply that there is considerable information about plausible values of parameters and in this case a probabilistic analysis should be considered.

The current treatment of plausibility in ecological applications of IGDT is also unsatisfactory. Historically, concepts of plausibility have led to several alternate theories. Of these, probability theory and statistical inference has emerged as the most enduring and widely practised method for coherently defining and manipulating uncertain quantities. Plausibility is being evoked within IGDT in an ad hoc manner, and it is incompatible with the theory's core premise, hence any subsequent claims about the wisdom of a particular analysis have no logical foundation. It is therefore difficult to see how they could survive significant scrutiny in real-world problems. In addition, cluttering the discussion of uncertainty analysis techniques with ad hoc methods should be resisted.

Debates about the nature of the IGDT method mask a deeper issue. Ecological applications of IGDT have, almost without exception, ignored one of the most severe sources of uncertainty in ecological problems, model structure uncertainty, and paradoxically concentrated on what is often one of the least severe sources; the uncertainty associated with model parameters. This problem is compounded by the rhetoric that sometimes accompanies these applications. Ignoring severe sources of uncertainty in any problem does not, of course, provide any assurance that good decisions have been identified, irrespective of what decision theory is applied. No decision theory, IGDT or otherwise, can ensure against bad decisions driven by implausible or illogical models, models that fail to represent the known unknowns or indeed the unknown unknowns. The best chance of making a good decision is to identify the critical temporal and spatial scales and key processes of the problem in hand and construct logical models that reflect these scales and processes, with testable assumptions clearly stated, and respect relevant theories and observations.

Acknowledgements

All authors confirm no conflict of interest.

Ancillary