### Introduction

- Top of page
- Summary
- Introduction
- Info-gap decision theory
- Review of ecological practice
- Practical utility
- Discussion
- Acknowledgements
- References
- Supporting Information

The discipline of ecology is central to responding to a range of major challenges facing the world community over the coming years. For example, how do we predict climate impacts and respond to them effectively to protect biodiversity and human amenity? How do we design efficient agricultural systems to feed a growing population while not eroding the natural resource base? All of these issues will require tactical and strategic decision-making by governments and communities who will need to trade off different interests and philosophies. One challenge in doing this is the uncertainty that exists in our ability to understand systems and predict future outcomes.

Uncertainty is pervasive in ecology. Variability, observation error, model structure uncertainty and other sources of uncertainty are to be found in virtually all applied ecological problems. Environmental systems are complex and sometimes poorly observed. As a result, environmental managers are often compelled to make decisions in situations with incomplete information. This has resulted in a significant literature around decision-making under uncertainty. The vast majority of approaches use probability to describe uncertainty, and when relevant data are extensive, and questions are essentially empirical, a wide range of statistical approaches to inference are available. When data are sparse, or the system complex, Bayesian probabilistic methods are still well defined but this process relies on expert opinions expressed through priors and can require costly or difficult elicitation processes. When data or knowledge is very sparse, as in the case of ‘complete ignorance’ (if this in fact exists), fundamental questions about the suitability of probability arise (Colyvan 2008).

Given these issues, any new method that purports to provide a mechanism for making good decisions with sparse information is likely to attract the attention of ecologists and environmental managers. In particular, a method and theory that purports to make robust decisions in the face of ‘severe uncertainty’ is a major achievement for science. Info-gap decision theory (IGDT) makes these claims. Ben-Haim (2001, p.x) describes IGDT as radically different from all current theories of decisions under uncertainty. IGDT is different because it offers a non-probabilistic approach to decision-making under uncertainty. It claims to provide a quantitative representation of Knight's concept of true uncertainty for which ‘there is no objective measure of probability’ (Ben-Haim 2004). It has been identified as a method that is suitable when the information base is so depauperate that the analyst cannot parameterise a probability distribution, decide on an appropriate distribution or even identify the lower or upper bounds on possible parameter values (Halpern *et al*. 2006).

Since the first comprehensive description of the theory in 2001, IGDT has attracted many proponents, principally in the fields of ecology, engineering and economics. Counting citations on http://info-gap.com/ suggests that there are three books and at least 140 journal articles on this subject, which are becoming increasingly well cited (Fig. 1). It has had a significant presence in ecology being applied to a wide range of important issues, including the allocation of resources to mitigate biodiversity impacts caused by global change and the design of effective marine reserves.

Info-gap decision theory, however, has also been criticised on two grounds: (i) it is not a radically new theory but rather a reformulation of minimax analysis that has been known in the operations research literature for over 60 years and (ii) its robustness functions are ‘radius of stability’ models that only measure the robustness of a decision in the local neighbourhood of an initial estimate and not over the global uncertainty space. Hence, IGDT is utterly unsuitable to situations of severe uncertainty (Sniedovich 2008, 2008, 2010a, 2012). Ben-Haim maintains that this criticism misses the point (Ben-Haim 2012). Sniedovich (2008) bases his arguments on mathematical proofs that may not be accessible to many ecologists but the impact of his analysis is profound. It states that IGDT provides no protection against severe uncertainty and that the use of the method to provide this protection is therefore invalid.

The meaning of severe uncertainty lies at the heart of Sniedovich's second criticism and the subsequent debate within the academic community that has ensued. The term ‘severe uncertainty’ is not formally defined in IGDT. By inspection of IGDT's models of uncertainty, working assumptions and assertions, however, Sniedovich (2010b, 2012) identifies three defining characteristics: (i) the uncertainty space can be unbounded (Ben-Haim 2006, p. 210), (ii) any initial estimate of the elements within this space may be substantially wrong (Ben-Haim 2006, p. 281), and no better than a wild guess (Ben-Haim 2010, p. 2), and (iii) there are no grounds for believing that the truth is more or less likely to be in the neighbourhood of the initial estimate than in the neighbourhood of any other point in the uncertainty space.

While these characteristics provide a definition of severe uncertainty, it is not universally accepted in public debates. Burgman (2008) stipulates that Sniedovich's definition of severe uncertainty is too narrow to be useful and maintains that in the applied world of environmental decision-making, identifying decisions that are robust to the uncertainty in an initial estimate is vital. In particular, Burgman (2008) argues that IGDT is useful because there comes a time when decisions have to be made, and when this occurs, it is useful to know how much deviation around an initial estimate can be accommodated before critical performance requirements are threatened, irrespective of how one arrives at the initial estimate. Furthermore, during this process, it is understood that there is no absolute guarantee that the truth lies in the local neighbourhood of the initial estimate.

There is therefore a tension among (i) the proposed method, IGDT, (ii) a set of mathematical proofs that relate it to known techniques and clarify that it cannot deal with ‘severe uncertainty’, (iii) a rhetorical argument justifying its practical use and (iv) a community of practice that is evolving around its application. To further discussion of IGDT, we review the use of this method in ecology and provide an accessible description and analysis of the method. We also present a discussion of the issues that a potential user needs to consider in applying this approach. In Section ‘'Info-gap decision theory'’, we give a concise synopsis of IGDT. While the notation is necessarily complex, this synopsis involves no abstract mathematics or proofs and concludes with a simple example to illustrate the approach. In Section ‘'Review of ecological practice'’, we examine the most important sources of uncertainty in model-based decision-support systems and the manner in which these are addressed in ecological applications of IGDT. Section ‘'Practical utility'’ identifies practical issues that can occur with IGDT under strict or more relaxed definitions of severe uncertainty, together with a synthesis of issues identified by other authors and original insights we have gained in the process of completing this review. Section ‘'Discussion'’ discusses the IGDT method from an applied ecological perspective and concludes with recommendations for ecologists who are contemplating the use of IGDT.

### Review of ecological practice

- Top of page
- Summary
- Introduction
- Info-gap decision theory
- Review of ecological practice
- Practical utility
- Discussion
- Acknowledgements
- References
- Supporting Information

There are at least 23 IGDT studies published within the ecological literature. The principal application problems revolve around:

- Surveillance and control programs for invasive species. The objective here is to test the robustness of different surveillance strategies, inspection protocols or eradication strategies, to the uncertainty associated with the introduction, establishment and spread of non-native species (Burgman
*et al*. 2010; Carrasco *et al*. 2010; Davidovitch *et al*. 2009; Moffitt *et al*. 2007, 2008; Rout *et al*. 2009; Yemshanov *et al*. 2010a, b); - Conservation strategies for endangered species. These studies examine the robustness of different conservation methods, such as translocation, captive breeding and poaching control, to the uncertainty associated with the efficacy of these methods and the biological processes (mortality, reproduction) that control the extinction risk (van der Burg & Tyre 2011; Crone
*et al*. 2007; McDonald-Madden *et al*. 2008; Regan *et al*. 2005); and - The design of nature reserves. These studies examine the effects of uncertainty associated with habitat quality, presence, persistence and/or dispersal of species on decisions about the size and connectivity of reserve networks (Halpern
*et al*. 2006; Moilanen & Wintle 2006; Moilanen *et al*. 2006; Nicholson & Possingham 2007).

Other ecological issues that have been examined with IGDT include calculating offset ratios for impacted habitat, managing biodiversity and wetlands under different climate impact scenarios, price-based vs. quantity-based pollution control strategies, the effects of fire risk on forest management strategies, the effects of uncertainty on multicriteria decision analysis and an analysis of animal foraging behaviour (Carmel & Ben-Haim 2005; Levy *et al*. 2000; McCarthy & Lindenmayer 2007; Moilanen *et al*. 2009; Stranlund & Ben-Haim 2008; Walshe & Massenbauer 2008; Wintle *et al*. 2011).

The reward function, context, uncertainty model and parameter bounds associated with 20 of the 23 environmental studies published to date are summarised in Tables S1–S4 (Supporting information). We have excluded from this summary the studies by Moffitt *et al*. (2007, 2008), and Wintle *et al*. (2011) because these authors do not describe the uncertainty model used in their analysis.

#### Severe uncertainty in ecology

The ecological literature typically presents IGDT as a new technique that can better accommodate severe uncertainty. There are, however, many sources of uncertainty in applied ecological problems. In quantitative, model-based decision-support systems, the principal sources are observation error, model structure uncertainty, variability in, and limited understanding of, the true values of model parameters and finally the association or dependence between model parameters. It is against these sources of uncertainty that the credentials of a robust decision support system for ecological problems should be judged.

##### Model structure uncertainty

Model structure uncertainty occurs in the processes that are included or excluded from the model (such as density dependence, competition, age structure), the relationships between variables that represent processes that are included in the model, the scale and resolution of the model, and whether or not stochastic forces are included. There is ample evidence in the literature that decisions regarding these issues have a significant impact on the predictions and accuracy of ecological models (Arhonditisis & Brett 2004; Fulton *et al*. 2004; Hill *et al*. 2007; Los & Blaas 2010; McCarthy *et al*. 1995; Pascual *et al*. 1997; Wood & Thomas 1999).

Model structure uncertainty qualifies as ‘severe’ in many practical situations: the uncertainty space is often very large, if not infinite, except perhaps for classes of nested models, and while ecological models are not usually made of wild guesses, they often are a poor representation of reality and may be substantially wrong. Furthermore, in the absence of observations, it can be difficult to *a priori* identify which model, among a set of competing models, is likely to be the most plausible representation of reality.

Model structure uncertainty in IGDT occurs in the reward function and uncertainty model. In all but three of the current ecological applications of IGDT, however, the solution is predicated on a single reward function – that is, a single model that describes the process and/or utilities associated with the problem in hand, although alternative possible models are available. Furthermore, in all but one example (Rout *et al*. 2009), all of these solutions are predicated on a single uncertainty model. The fractional error model is by far the most widely employed – 14 of the 20 studies summarised here use it. Three of the remaining six examples use an envelope-bound model, while the other examples use a proportional error model and ellipsoid models.

There are four general strategies for tackling model structure uncertainty – ignore it, compare alternative structures, envelope the effects of alternative structures and average across them (Hayes 2011). IGDT can accommodate the first three approaches but the most prevalent approach in current ecological applications is to simply ignore it. Moilanen *et al*. (2006) and Moilanen *et al*. (2009) provide examples that compare alternative model structures, while McDonald-Madden *et al*. (2008) envelope the effects of alternative structures.

##### Parametric uncertainty

The parameters of ecological models are usually subject to two sources of uncertainty: ecological parameters often represent processes that are variable (growth rates, fecundity, dispersal distances, carrying capacity, etc.) but we may also be unsure of how to accurately characterise this variability. In our experience, however, parametric uncertainty in ecological problems is not severe, at least not in the sense of IGDT's criteria. The domain of the variables of an ecological model are typically not ‘large’ or unbounded because biological and physical parameters almost always have feasible constraints. Point estimates for these parameters are often informed by direct observation, information from similar species and environments, or even allometric relationships, and only occasionally are these estimates of such extremely poor quality that they are no better than a wild guess. Furthermore, ecologists are often able to identify values for uncertain parameter that are more likely than others.

Our assertions in this regard are supported by the ecological applications of IGDT. The uncertain parameters of 14 of the 20 cases reviewed here are bounded by an interval with pre-determined upper and lower limits. Ten of the ecological applications ‘info-gap’ proportions, probability or a set of probabilities, for example the extinction probabilities of endangered species under different management regimes. Here, the range of *α* is implicitly constrained as in the simple example described above. In three cases, constraints are explicitly placed on the uncertain parameters in the problem – namely the utility of action–state pairs in a Bayesian network, the spread velocity and radius of an invaded area (Table S1, Supporting information) and the performance level of policy alternatives (Table S4, Supporting information). Finally, in two cases, the range of *α* is explicitly restricted to lie on a particular range but with no clear justification (Table S1, Supporting information).

Several applications also base their initial parameter estimates on extant information. van der Burg & Tyre (2011), for example, use field surveys in Colorado as a basis for nest success estimates in Nebraska. Halpern *et al*. (2006) use estimates of dispersal distances from a log-normal distribution fitted to estimates from many different species. Nicholson & Possingham (2007) are able to quote dispersal distance estimates for the actual species of interest in their study area. In these instances, the initial estimates of uncertain parameters are clearly not wild guesses but rather estimates of a much better quality.

##### Dependence

There are at least two reasons why the uncertainty about the dependence between a model's parameters is likely to be more severe than the uncertainty that surrounds the marginal knowledge of the parameters themselves. In the first instance, the uncertainty space can be large: as noted above, ecological parameters usually represent variable quantities or processes, and these quantities may be related to one another in a linear or nonlinear fashion. Moreover, nonlinear forms of association can take the form of central dependence, upper tail dependence, lower tail dependence or indeed combinations of these (Denuit *et al*. 2005). Secondly, dependence between ecological parameters arises through the direct and indirect interactions and feedback that occur naturally in ecological systems or as a result of anthropogenic impacts (Dambacher & Ramos-Jiliberto 2007; Vinebrooke *et al*. 2004). Many of these interactions are subtle and sometimes non-intuitive. It is therefore quite possible that an initial dependence model will be wrong and it may be difficult to assess whether or not the true dependency model lies in the neighbourhood of an initial estimate.

As a worst-case analysis, IGDT implicitly accommodates all forms of dependence for a given horizon of uncertainty and the info-gap error models chosen. In reality, the parameter space is often constrained by dependence between ecological components and processes that are represented by parameters in the reward function, and this should be reflected in the choice and parameterisation of the error model. The fractional error model, prevalent in ecological applications, implies that the horizon of uncertainty increases in each dimension with perfectly positive linear dependence. This model does not allow the analyst to incorporate information about more complicated relationships between parameters and the constraints that this may impose on the feasible parameter space. In extreme situations this could lead to rewards that are impossible in the real world.

### Discussion

- Top of page
- Summary
- Introduction
- Info-gap decision theory
- Review of ecological practice
- Practical utility
- Discussion
- Acknowledgements
- References
- Supporting Information

Promoting IGDT as a method for severe uncertainty has helped generate the interest, and controversy, that currently surrounds the theory. It is not, however, the only approach to uncertainty when one cannot reliably identify a precise probability distribution, and it is not the only non-probabilistic theory of uncertainty. It is not therefore a theory that ecologists should feel compelled to use in data-poor situations but rather a choice based on significant reflection about the context of the analysis, and the strengths and weaknesses of the theory.

We believe, however, that most ecological applications of IGDT have taken the claims of Ben-Haim (2006) at face value. A typical example of this is the analysis presented in Halpern *et al*. (2006). This paper reviews uncertainty techniques applicable to marine reserve design and provides a taxonomy presented in Fig. 6. This proposed taxonomy presents info-gap as providing a technique that can accommodate the ‘largest’ amount of uncertainty and asserts that this uncertainty is unbounded.

The literature and discussion presented in this paper demonstrate that the results of Ben-Haim (2006) are not uncontested. Mathematical work by Sniedovich (2008, 2010a) identifies significant limitations to the analysis. Our analysis highlights a number of other important practical problems that can arise. It is important that future applications of the technique do not simply claim that it deals with severe and unbounded uncertainty but provide logical arguments addressing why the technique would be expected to provide insightful solutions in their particular situation.

To assist in this process, we identify four issues that should be considered carefully before applying IGDT to real problems:

- Sensitivity to initial estimates,
- Localised nature of the analysis,
- Error model choice and parameterisation,
- Notions of plausibility.

Viewing the robustness curve as a sensitivity analysis is problematic. Sensitivity in the neighbourhood of a poor estimate gives no insight into sensitivities elsewhere in the parameter space. If we wish to make an assumption that the true value is in the neighbourhood of our best guess, we are working in a context that is inconsistent with the stated motivations of the IGDT method. In particular, this would imply that there is considerable information about plausible values of parameters and in this case a probabilistic analysis should be considered.

The current treatment of plausibility in ecological applications of IGDT is also unsatisfactory. Historically, concepts of plausibility have led to several alternate theories. Of these, probability theory and statistical inference has emerged as the most enduring and widely practised method for coherently defining and manipulating uncertain quantities. Plausibility is being evoked within IGDT in an ad hoc manner, and it is incompatible with the theory's core premise, hence any subsequent claims about the wisdom of a particular analysis have no logical foundation. It is therefore difficult to see how they could survive significant scrutiny in real-world problems. In addition, cluttering the discussion of uncertainty analysis techniques with ad hoc methods should be resisted.

Debates about the nature of the IGDT method mask a deeper issue. Ecological applications of IGDT have, almost without exception, ignored one of the most severe sources of uncertainty in ecological problems, model structure uncertainty, and paradoxically concentrated on what is often one of the least severe sources; the uncertainty associated with model parameters. This problem is compounded by the rhetoric that sometimes accompanies these applications. Ignoring severe sources of uncertainty in any problem does not, of course, provide any assurance that good decisions have been identified, irrespective of what decision theory is applied. No decision theory, IGDT or otherwise, can ensure against bad decisions driven by implausible or illogical models, models that fail to represent the known unknowns or indeed the unknown unknowns. The best chance of making a good decision is to identify the critical temporal and spatial scales and key processes of the problem in hand and construct logical models that reflect these scales and processes, with testable assumptions clearly stated, and respect relevant theories and observations.