An Evaluation of the Treatment of Risk and Uncertainties in the IPCC Reports on Climate Change

Few global threats rival global climate change in scale and potential consequence. The principal international authority assessing climate risk is the Intergovernmental Panel on Climate Change (IPCC). Through repeated assessments the IPCC has devoted considerable effort and interdisciplinary competence to articulating a common characterization of climate risk and uncertainties. We have reviewed the assessment and its foundation for the Fifth Assessment Reports published in 2013 and 2014, in particular the guidance note for lead authors of the fifth IPCC assessment report on consistent treatment of uncertainties. Our analysis shows that the work carried out by the ICPP is short of providing a theoretically and conceptually convincing foundation on the treatment of risk and uncertainties. The main reasons for our assessment are: (i) the concept of risk is given a too narrow definition (a function of consequences and probability/likelihood); and (ii) the reports lack precision in delineating their concepts and methods. The goal of this article is to contribute to improving the handling of uncertainty and risk in future IPCC studies, thereby obtaining a more theoretically substantiated characterization as well as enhanced scientific quality for risk analysis in this area. Several suggestions for how to improve the risk and uncertainty treatment are provided.


INTRODUCTION
The climate change research issue is mainly about understanding how the climate and major climate parameters will change over time due to human interventions. Important parameters include global atmospheric temperature, ocean temperatures, rising sea levels, melting of snow and ice, and an intensification of natural hazards. (1) We cannot know for sure how the future will be-there are risks and uncertainties-but using scientific assessment methods and simulation tools the goal is to obtain the necessary insights that will clarify the boundaries of what we can predict with some degree of certainty and what we cannot predict. Much research on climate change is also about basic scientific claims related to complex cause-effect chains, leading to models that can be tested against events and developments of the past and then applied to the future. (2) Making such predictions and investigating such claims are challenging as the physical climate system is highly complex, has aspects that are inherently chaotic, and involves nonlinear feedback loops operating on a wide variety of time scales. (3,4) We have some empirical knowledge about the effects of interactions between human interventions and climate reactions but studying such processes encounters many challenges in terms of not well understood causal structures, intervening variables, feedback loops, and dynamic inconsistencies. How to adequately carry out the scientific analyses and, in particular, the way to treat risks and uncertainties, is not straightforward. Clearly, any serious attempt made to perform such research requires a solid methodological platform for how to conduct the analyses and how to treat risks and uncertainties.
In this article, we have taken a closer look at the methodological platform for the work carried out by the Intergovernmental Panel on Climate Change (IPCC), in relation to the Fifth Assessment Reports published in 2013 and 2014, and in particular the guidance note for lead authors of the fifth IPCC assessment report on consistent treatment of uncertainties. (5)(6)(7)(8) Our focus is on the concepts and methods of how risk and uncertainties are understood and treated.
A number of articles have addressed the IPCC approach to risk and uncertainties, some with an aim of presenting and justifying the approach chosen, others with a more critical view. The work of Manning (3) belongs to the former category-it is written by a member of one of the IPCC Working Groups for the Fourth Assessment Reports (AR4) (9) and provides an overview of the treatment of uncertainties in these reports, with links to the approach established for the Third Assessment Report (AR3). Key features of the evolution in perspectives are discussed, with special attention to the terms "likelihood" and "confidence" as alternative ways of expressing uncertainty. According to Manning (3) these two terms are interpreted as follows in AR4: likelihood expresses the chance of a defined outcome in the physical world and is estimated using expert judgment. Confidence expresses the degree of understanding and/or consensus among experts and is a statement about expert judgment. Following the guidance note on how to treat uncertainties in AR4 (9) we interpret these definitions to mean that likelihood refers to a probabilistic assessment of some welldefined outcomes having occurred or occurring in the future, and a confidence level is used to characterize uncertainty that is based on expert judgment as to the correctness of a model, an analysis, or a statement.
Jonassen and Pielke (10) point to some problems of the uncertainty treatment in the AR4 reports. These authors perform an analysis of the treatment of uncertainty by the AR4 with a focus on differences between the IPCC guidance note and actual practice (for example, related to the terms likelihood and confidence). They detected many inconsistencies. Other studies have reached similar conclusions. (11) Many of these inconsistencies can be traced back to differences between the three working groups of the AR4, although an overall common guidance note on how to treat uncertainties was also developed for the AR4. (9) For further critical reviews, see Morgan et al., (12) Swart et al., (13) von Storch and Krauß, (4) and Budescu et al. (14) The challenges related to the concepts "likelihood" and "confidence" are discussed by a number of authors. (10,15,16) The meaning of these concepts is subject to different interpretations. In the guidance note for lead authors of the Fifth IPCC Assessment Report (AR5) on consistent treatment of uncertainties, (8) which provides the basis for the Fifth IPCC Assessment Reports, (5)(6)(7) the meaning and distinction of these concepts are given considerable treatment. Yet, we will argue in this article that all the IPCC documents still suffer from a lack a precision-the scientific bases for these concepts are unclear, the consequence being that the inconsistencies seen through AR4 are not significantly reduced in the AR5. More important than the lack of consistency, though, is the fact that these guidance notes prescribe perspectives on risk and uncertainties that could lead to a distorted communication of the results of the scientific studies and seriously misguide the public and relevant decisionmakers. An example is the reference to risk as a function of consequences and probability. It is a main aim of this article to point to the need for moving beyond the probability-based perspectives on risk.
In recent years several new risk perspectives have been developed, that are based on uncertainties and not probability. We will demonstrate that these new approaches provide a stronger and more appropriate basis for climate change analysis than those adopted by IPCC so far. A key feature of these perspectives is the sharp distinction between risk and uncertainty and how these two are measured. Much of the IPCC terminology on risk and uncertainty lacks this dichotomy.
Compared to other critical comments to the IPCC foundations, this article goes beyond a critical assessment of the consistency and cohesiveness of the IPCC approaches attempts to provide new insights in terms of conceptual and analytic clarifications of the key concepts "risk" and "uncertainty." We will concentrate our analysis to sharpening the meaning of probability and relate the IPCC treatment of risks to new developments in the field of risk conceptualization and characterization.
The remaining sections of this article are organized as follows. First, in Section 2 we review and discuss the relevant IPCC documents on risk and uncertainty. Then in Section 3 we present some suggestions for improvement, highlighting the issues indicated above. We will cover alternative and broader risk perspectives as well as clarifications concerning some of the fundamental building blocks of the risk and uncertainty assessments, including the definitions of terms such as likelihood, probability, chance, and confidence. The final Section 4 provides some conclusions.

REVIEW AND DISCUSSION OF THE IPCC DOCUMENTS
In this section, we take a closer look at the perspective on risk and uncertainties adopted in the IPCC work. We will focus on two aspects: (1) How risk is understood.
(2) How uncertainties are represented/expressed and treated.
In order to better understand and assess the IPCC perspectives we will refer first to earlier reports, in particular to the AR4 from 2007.

Risk
A search for the term "risk" in the IPCC reports yields a high number of matches. Here are three examples.
(1) . . . that it is more likely than not that anthropogenic influence has contributed to an increased risk of drought in the second half of the 20th century, . . . (5, p.72) (2) . . . hence there is a risk that this carbon may return to the atmosphere. (5, p.551) (3) Sea-level rise and increases in extreme rainfall are projected to further increase coastal and river flood risks and without adaptive measures will substantially increase flood damages. (6, p. 27) Clearly, the term risk is understood differently in these examples. In case (1) risk is used as a term for probability (likelihood, chance) in a wide sense. Case (2) is similar but here an aspect of magnitude of the probability is added: for sure, the probability is not negligible. In case (3) risk is obviously more than a probability; flood risk captures also the consequence aspect of flooding. The term risk in the third example is interpreted similarly to the definition given in the guidance note (8) where we read that risk is a function of probability and consequences.
In the 2007 IPCC reports (9, p. 64) risk was generally understood to be the product of the likelihood of an event and its consequences (the expected value), but this interpretation of risk is not used in the latest reports. The concept of expected values to represent risk in situations such as climate change has proven to be inadequate, as emphasized by many analysts and researchers. (17,18) However, we will argue that the current more comprehensive risk perspective used by IPCC in its 2013 and 2014 reports is too narrow for adequately assessing climate change risk. Probability is a powerful tool for representing/describing uncertainties but it is not a perfect tool. One may, for example, assess that two different situations have probabilities equal to 0.2, but in one case the assignment is supported by a substantial amount of relevant data, whereas in the other by effectively no data at all. (19) The number alone does not reveal this discrepancy. When linking risk as a concept to a specific measuring device (probability) special attention and care are warranted. The appearance of numbers could hide the nature of uncertainty that is connected with such an allegedly precise assessment. To discuss this further we need to clarify what the meaning of the term probability (and related terms such as likelihood and chance) is.
In the IPCC documents some scales are introduced to express the level of likelihood and confidence, but precise definitions of these terms are mostly lacking. We will take a closer look at this in the coming section.

Uncertainties
As was commented on in the introduction section, climate change research is very much about uncertainties, and the approach taken for the uncertainty analysis-how uncertainty is to be represented and treated-is obviously critical to the overall results of the research. The IPCC frameworks for uncertainty treatment are well documented. (8,9) We begin with the AR4 (8) framework. This framework comprises three different ways of describing the uncertainties, one is qualitative and two are quantitative. We refer to the quantitative approaches as the confidence and the likelihood approaches, respectively. The qualitative approach is adopted by WG III (Mitigation of Climate Change), whereas WG II (Impacts, Adaptation, and Vulnerability) has used a combination of the two quantitative approaches. WG I (the Physical Science Basis) has predominantly used the likelihood assessments. The difference between the confidence and likelihood approaches is important and has been thoroughly discussed in the literature, as was mentioned above.
Let us first take a look at the confidence concept. Following the guidance note on how to treat uncertainties in AR4 (20) a confidence level is used to characterize uncertainty that is based on expert judgment as to the correctness of a model, an analysis, or a statement. A scale of confidence levels is used to express the assessed numerical chance for a finding to be judged as being correct: very high confidence at least 9 out of 10; high confidence about 8 out of 10; medium confidence about 5 out of 10; low confidence about 2 out of 10; and very low confidence less than 1 out of 10.
These explanations are vague and difficult to understand, and make the concept of confidence not very meaningful and useful for practical purposes. And in the fifth reports, (5)(6)(7)(8) the quantification of confidence has been omitted. Now it is stated that IPCC relies on two metrics for communicating the degree of certainty in key findings: (8) (1) Confidence in the validity of a finding, based on the type, amount, quality, and consistency of evidence (e.g., mechanistic understanding, theory, data, models, expert judgment) and the degree of agreement. Confidence is expressed qualitatively.
(2) Quantified measures of uncertainty in a finding expressed probabilistically (based on statistical analysis of observations or model results, or expert judgment).
Furthermore, it is said that a level of confidence is expressed using five qualifiers: "very low," "low," "medium," "high," and "very high." It synthesizes the author teams' judgments about the validity of findings as determined through evaluation of evidence and agreement. In contrast to the fourth assessment documents, (20) the new guidance states that confidence should not be interpreted probabilistically.
As observed above, the numerical confidence levels used in the fourth assessment reports are difficult to interpret. As we will show in the coming section, it is, however, possible to provide an easily understandable definition of such a numerical confidence scale in the context of subjective (knowledgebased, judgmental) probabilities. As a matter of fact there is a well-established framework that gives a precise and meaningful basis to the concept of confidence. Our solution differs from the recommendation of the fifth assessment reports (5)(6)(7)(8) to reject the quantification of confidence but to give it a more meaningful interpretation. We agree that there is a need for characterization beyond quantification, but we think it is unfortunate that the confidence concept has no scale for assessing the confidence measure. This is likely to result in a lack of consistency and more arbitrary assignments.
Then let us look into the likelihood concept. According to the guidance note from 2010, (8) likelihood is a quantified measure of uncertainty in a finding expressed probabilistically (based on statistical analysis of observations or model results, or expert judgment). It is also said that likelihood provides a calibrated language for describing quantified uncertainty: It can be used to express a probabilistic estimate of the occurrence of a single event or of an outcome (e.g., a climate parameter, observed trend, or projected change lying in a given range). Likelihood may be based on statistical or modeling analyses, elicitation of expert views, or other quantitative analyses.
Again there is a lack of precision in the definition and analysis. Based on these explanations, the researchers are not adequately guided on how to understand and treat uncertainties. The term probability is not defined. Since there are different meanings of the term probability we cannot obtain the precision required without a clarification on this point. There is no clear distinction made between the underlying concept (probability) and how this concept is measured, estimated, or operationalized. Is likelihood the underlying concept or the measurement of it? The current text is not precise on this point and this leads to confusion.
As far as we understand the IPCC documents the idea that the likelihood concept is trying to reflect is stochastic variation (also referred to as aleatory uncertainties (21) ), expressed through probability models, for example, the Poisson distribution model describing the variation in the number of events occurring in a specified period of time.
Probabilities defined through the probability models are referred to as frequentist probabilities (or chances in a Bayesian context (22) ). An example is the frequentist probability that the number of events is zero for the time period considered, understood as the fraction of periods having zero event occurrences (it is equal to exp{-λt}, where λ is the expected number of events occurring per unit of time (the occurrence rate parameter) and t is the length of the period considered). The probability models are fundamental in statistical analysis and are based on the existence of a large (in theory infinite) number of similar situations to the one studied. For many physical phenomena such models are easy to justify, in other cases the models are based on more speculative assumptions. Given that a probability model is justified, we can estimate its parameters and perform related uncertainty assessments. Hence uncertainty comes into play in two ways: (1) Through differences between the model output and the actual value of the quantity considered.
Expert judgment is commonly used in relation to item 2, to estimate the parameters of the probability models. If probability models have been justified, we thus face two levels of uncertainty, the probability model reflecting variation (stochastic/aleatory uncertainties) and epistemic uncertainties about the "true" values of the parameters of the model. (21,23) Now returning to the IPCC likelihood concept, it is not clear what the concept refers to. Take the Poisson distribution and the frequentist probability p of at least one event occurring. If we know the value of p, there are no uncertainties; we can conclude that the probability is, for example, likely or very likely according to the scale introduced. But if p is not known, we cannot make such statements, unless we move to the "confidence sphere" and state our (the analyst's) confidence or degree of belief about where the "true" p is. Of course, if a large amount of relevant data are available, the estimated probability p* of p could be close to p, but we need to be very careful when replacing judgments about p* with judgments about p. Beliefs about the estimated likelihood of events constitute something different from beliefs about the "true" underlying likelihood. The IPCC framework on uncertainty treatment is not sufficiently precise on this point and does not provide adequate guidance. When it is said that likelihood may be based on statistical or modeling analyses, elicitation of expert views, or other quantitative analyses, it is referring to estimation of probabilities (p*). Then we are, however, into the confidence sphere where we describe our confidence or degree of beliefs. There are uncertainties due to both category 1 and 2 (model, parameter).
We could go on with examples from the IPCC documents that create more confusion than clarity on this issue. Take, for example, the guidance note (14, p. 4) that expresses: A range can be given for a variable, based on quantitative analysis or expert judgment: Assign likelihood or probability for that range when possible; otherwise only assign confidence. Explain the basis for the range given, noting factors that determine the outer bounds. State any assumptions made and estimate the role of structural uncertainties. Report likelihood or probability for values or changes outside the range, if appropriate.
To illustrate our point, suppose that the analyst can give a range of the number of events occurring in a specific period of time, say {0, 1, 2}. A probability model can in this case be specified (the Poisson model), but we may have large uncertainties in the parameter of this model. Are we then able to assign likelihood and probability according to the IPCC guidance note? We cannot know, as we simply do not understand what the likelihood concept expresses. Clearly, we can use a confidence measure in this case, for example, a 90% uncertainty interval for λ, but the link to likelihood and probability is not clear when using the IPCC guidance (8) as a reference.
In our view, what is required is clarity about the conceptual foundations for the analysis, and that includes as a minimum that all basic concepts are precisely defined and coherently related to each other. There is certainly a need for improvement in this respect when looking at the guidance given by IPCC. (7)

SUGGESTIONS FOR IMPROVEMENT
In this section we will present a risk and uncertainty framework suitable for studying climate change that meets the basic criteria of precision and coherence. It is based on the following conceptual and methodological pillars: (24,25) (1) Risk exists when considering an activity (even if this risk is not described or measured). Concrete alternative definitions that can be used are: (A1) "Risk = Event or consequences" r Risk is a situation or event where something of human value (including humans themselves) is at stake and where the outcome is uncertain. (26,27) r Risk is an uncertain consequence of an event or an activity with respect to something that humans value. (28) (A2) "Risk = Consequences/severity of these + Uncertainty" r Risk is equal to the two-dimensional combination of the consequences (of an activity) and associated uncertainties. (29,30) r Risk is uncertainty about and severity of the consequences (or outcomes) of an activity with respect to something that humans value. (31) Although the definitions in A1 point in the right direction, only the more recent specifications in A2 have the full set of properties necessary to provide a consistent and coherent risk perspective, (31) and they should therefore be used for the underlying concept of risk.
(2) Risk can be described or measured in different ways. A risk description is obtained by specifying the events/consequences C and using a description (measure) of uncertainty Q. Specifying the events/consequences means to identify a set of events/quantities of interest C' that characterize the events/consequences C. An example of a C' related to climate change is sea level rise. Depending on the principles laid down for specifying C and the choice of Q we obtain different perspectives on how to describe/measure risk. As a general description of risk we can write (C',Q, K), where K is the knowledge that Q and C' are based on.
A judgment about what is a high or a low risk needs to reflect the totality of the elements of the risk description, not only Q. For example, if the consequences could be severe and the related probabilities are relatively small, the risk could still be judged as high if the background knowledge is poor.
Some common examples of Q: r Q = P, a subjective (knowledge-based, judgmental) probability. This probability is a subjective measure of uncertainty conditional on some background knowledge K (the Bayesian perspective). The probability is interpreted with reference to an uncertainty standard, for example, an urn: if the assessor assigns a probability of an event A equal to, say, 0.1, it means that the assessor compares his/her uncertainty about the occurrence of the event A with drawing at random a specific ball from an urn that contains 10 balls. To show the dependency of the background knowledge K (data, models, assumptions) that the probabilities are based on, we write P(A|K). (22) r Q = P, an imprecise (interval) proba- and say that the analyst states that his/her assigned degree of belief is greater than the urn chance of 0.10 (the degree of belief of drawing one particular ball out of an urn comprising 10 balls) and less than the urn chance of 0.5. The analyst is not willing to make any further judgments. Then the interval [0.1, 0.5] can be considered an imprecision interval for the probability P(A). Considerable research has been conducted in recent years to establish theories and calculation rules for dealing with imprecise probabilities, (19,32,33) two main directions being imprecision intervals based on the theories of possibility and evidence (formally, possibility theory can be seen as a special case of the evidence theory). Note that the intervals used by IPCC, for example, that the likelihood ࣙ 66%, can be viewed as an imprecision interval if likelihood refers to a knowledge-based probability. However, if likelihood refers to a frequentist probability, the interval, for example, [0.66, 1] just refers to a set of possible values for the unknown frequentist probability. Also in the case that the likelihood is to be interpreted as an estimated frequentist probability, we can think about the interval as a type of imprecision interval, in the sense that the analyst is not willing to specify the estimate more precisely than the interval prescribed.
r Q = CI, a confidence interval for a parameter θ in a traditional statistical setting. This perspective presumes the establishment of a probability model with parameter θ . Formally, a risk description based on the traditional statistical thinking then takes the form: (θ *,CI, K), where θ * is the estimate of θ .
The risk is normally linked to future activities, but the above framework is comprehensive and can also be applied to study unknown Cs linked to the past. The suggested approach that we outlined above is referred to as a risk-uncertainty framework to highlight that it is not limited to assess events and associated consequences occurring in the future.
Uncertainty seen in isolation from the consequences and the severity of the consequences cannot be used as a general definition of risk. Large uncertainties need attention only if the potential outcomes are large/severe in some respect. Look at the following example: (18, p. 52) the activity considered can result in only two outcomes, 0 and 1, corresponding to 0 and 1 fatality, and the decision alternatives are A and B, having uncertainty (probability) distributions (0.5, 0.5), and (0.0001, 0.9999), respectively. Hence, for alternative A there is a higher degree of uncertainty than for alternative B, meaning that risk according to this definition is higher for alternative A than for B. However, considering both dimensions, both uncertainty and the consequences, we would, of course, judge alternative B to have the highest risk as the negative outcome 1 is nearly certain to occur.
The same conclusion we would make in the degenerate case where the distribution is (0.0, 1.0).
Probability models with parameters can also be introduced when knowledge-based and imprecise probabilities are used, but these perspectives do not presume the existence of probability models to allow for uncertainty assessment as the traditional statistical setting does. A CI in a traditional statistical setting does not reflect uncertainties other than variations in the data sampled. If a probability model with parameters can be justified, it leads to the common dichotomy between aleatory uncertainties and epistemic uncertainties as discussed in Section 2.2.
There exists no simple answer on how to best represent and express the uncertainties U about the consequences C. Any uncertainty assessment is dependent on the analysts. This needs to be taken into account in the follow-up of the risk and uncertainty assessments. For all types of uncertainty measures Q, a humble attitude is required, as the measure is just a tool and it has its limitations. Subjective probabilities are more analyst dependent than, for example, interval probabilities, but may also include valuable expert and analyst judgments for the decisionmaker.
Let us look at some practical procedures for presenting the risks and uncertainties in line with this perspective. Consider first a case where we study the possible occurrences of some events A 1 , A 2 , . . . Then the exact or interval probabilities for A i are assigned, given some background knowledge K. These probabilities are knowledge based but could be partly based on probability models and assessments of uncertain parameters. Then we may establish best estimates of the consequences given the occurrence of the events, for example, by computing the conditional expected consequences E(C|A i ). However, E(C|A i ) could deviate strongly from the actual consequences C. In this case, we are led to consider more closely the uncertainty of C given the occurrence of A i . A probability distribution could be established, but in practice it may be more attractive to simply produce an uncertainty interval for the unknown consequences of the event. We may, for example, establish a 90% uncertainty interval [a,b], meaning that the assessor is 90% confident that the true value of C is in this interval, where "confidence" refers to a subjective probability assignment.
The probability for A i and the uncertainty interval produced do not express the strength of knowledge that supports them. In addition to the probability-based measures, we need to provide a way of expressing that the numbers are based on a rather strong or weak knowledge base. A simple way of doing this is to use some broad categories, as shown in Fig. 1. The figure illustrates three events A, with assigned probabilities for specified categories of consequences, and with corresponding assignments of the strength of knowledge. Here a crude direct grading of the strength of knowledge that supports the probabilistic analysis is carried out, in line with the scoring used by Flage and Aven. (34) The knowledge is weak if one or more of these conditions are true: (1) The assumptions made represent strong simplifications.
(2) Data/information are/is nonexistent or highly unreliable/irrelevant.  If, on the other hand, all (whenever they are relevant) of the following conditions are met, the knowledge is considered strong: (1) The assumptions made are seen as very reasonable. the models used are known to give predictions with the required accuracy.
Cases in between are classified as having medium strength of knowledge.
Another approach for characterizing the strength of background knowledge is presented in Aven. (35) It is based on a crude assessment of the "assumption deviation risk," reflecting risks related to the deviations from the conditions/states defined by the assumption made.
To summarize, if C is the quantity of interest (for example, the sea level rise for the next 20 years; it could also be a parameter of a probability model), a subjective probability or interval probability can be established, the basis normally being various types of models and expert judgments. In addition we need to make some type of judgment of the strength of the knowledge K.
Next, we would like to address potential surprises relative to the knowledge (black swans (36)(37)(38) ). Looking at the above risk description, what type of events, not covered by the assessment, could occur? Three categories of such events are: (38) (1) Events that were completely unknown to the scientific environment (unknown unknowns).
(2) Known events that were not on the list of events from the perspective of those who carried out a risk analysis (unknown knowns). (3) Events on the list of known events in the risk analysis but judged to have negligible probability of occurrence, and thus not believed to occur.
The term "black swan" is used to express any of these types of events, tacitly assuming that it carries an extreme impact.
The first category of black swan type of events (1) is the extreme, the type of event is unknown to the scientific community, for example, a new type of virus. The second type of black swans (2) is events that are not captured by the relevant risk assessments, either because we do not know them, or we have not made a sufficiently thorough consideration. If the event then occurs, it was not foreseen. If a more thorough risk analysis had been conducted, the event could have been identified. The September 11 attack is a good example of this type of black swans.
The third category of black swans (3) comprises events that occur despite the fact that the occurrence probability was judged to be negligible. The events are known, but considered so unlikely that they are ignored-they are not believed to occur and cautionary measures are not implemented. An example is the event that an underwater volcano eruption occurs in the Atlantic Sea, resulting in a tsunami affecting, for example, Norway. The events are on the list of risk sources and hazards but were then removed as their probability is judged as negligible. Their occurrence will come as a surprise.
To confront this type of risk knowledge is the key. Risk assessment is about producing this knowledge and hence we need to look into these assessments and improve them to better incorporate the black swan type of risk. Compared to current practice many improvements can be made. Here is an example of a checklist for those aspects to consider in a risk assessment to ensure that this type of risk is given adequate attention. (39) (1) Is there an overview of the assumptions made?
(2) Has a risk assessment of the deviations from assumptions been conducted (an assumption deviation risk assessment)? (3) Is the strength of knowledge on which the assigned probabilities are based assessed? (4) Is this strength included in the risk description? (5) Have special efforts been made to uncover the black swans of the type unknown knowns? (6) Have special efforts been made to assess the validity of the judgments made where events are considered not to occur due to negligible probability? (7) Have people and expertise who do not belong to the initial analysis group been used to detect such conditions? (8) If the expected values of a quantity are specified, has the uncertainty of this quantity been assessed (for example, using a 90% uncertainty interval for this quantity)?
Here is a concrete example how an adjusted risk analysis can be carried out to highlight such issues: (39) First, a list of all types of risk events having low risk by reference to the three dimensions, assigned probability, consequences, and strength of knowledge, is established. Then a review of all possible arguments and evidence for the occurrence of these events should be provided, for example, by pointing to historical events and experts' judgments not in line with common beliefs. To perform these assessments, a group of people other than those performing the risk assessment is needed. The idea is to allow for and stimulate different views and perspectives, in order to break free from prevailing beliefs and obtain creative processes. A useful means in doing such assessments is the construction of scenarios that provide plausible but still unlikely developments that can lead to the revelation of unknown unknowns or unknown knowns. This list of events, with associated risk descriptions, and this type of argument and evidence is presented along with the risk events having the highest risk scores according to assigned probability, consequences, and strength of knowledge.
The above analysis provides an example of how to present risk according to the (C',Q, K) framework. There is an ongoing discussion in the literature concerning this issue of how to represent and express the risks and uncertainties. (19,32,33,(40)(41)(42) In its general form this discussion is beyond the scope of this article. What we focus on here are the potential implications of our risk uncertainty perspective on the climate change research methodology and in particular the treatment of risk and uncertainties in such a context.

Implications for the Assessment Approach Concerning Climate Change Risk and Uncertainties
The above perspective on risk and uncertainty provides a more solid theoretical and also practical foundation for dealing with risks and uncertainty in climate change research. A conceptual basis has been suggested in this article, in which all key concepts are well-defined and have meaningful interpretations. Consistency can thus be achieved. Furthermore, the framework is flexible in the sense that different approaches and methods can be integrated in the measurement of risk and uncertainties, reflecting the fact that there are different views and perspectives on how to best represent and express uncertainties.
Let us see how the procedures adopted in the IPCC reports fit into our framework and how some of the issues raised above are clarified by inserting our framework into the IPCC mandate.
First, let us consider the qualitative confidence system described in IPCC (8) and briefly reported in Section 2.2. Here a level of confidence is expressed using five qualifiers: "very low," "low," "medium," "high," and "very high," and it synthesizes the analysis teams' judgments about the validity of findings as determined through evaluation of evidence and agreement. Say that the statement of concern is "sea level rises by more than 0.2 m the next 20 years." Then we may use the IPCC (8) guidance to express a confidence in this statement (for example, "high") to be true based on an evaluation of the two dimensions evidence and agreement among experts. We acknowledge the suitability of such an approach in practice, but question why knowledge-based probabilities are not introduced to ensure consistency in interpretations and an easy understandable scale of reference for improved uncertainty description and communication. We have shown that such a scale exists. If a statement has confidence "high" it may express that the analysts are 80% sure that the statement is true, i.e., they compare their uncertainty and degree of belief by drawing a red ball out of an urn comprising 5 balls where 4 are red. We have no problems in understanding that using numbers without a rationale (as we would say the use of confidence numbers were in AR4) should be avoided, but this is not the case here-knowledge-based probabilities can be defined and they are easy to define and communicate. Imprecise probabilities represent an alternative to using a precise probability number. The analysts may, for example, say that their confidence is expressed by the probability interval 50-90%, the interpretation given above for Q = P. For any probabilistic analysis, exact or using intervals, the strength of knowledge that the probabilities are based on, need to be assessed, as well as the potential for surprises, as discussed in the previous section.
What we suggest is to replace the vague qualitative confidence statements by IPCC (8) by knowledgebased probabilities (with the above interpretation), and judgments of the strength of knowledge supporting the probabilities. Specific surprise assessments should also be conducted.
Confidence is a way of reflecting the U in the above framework. The likelihood dimension is closely linked to probability models and frequentist probabilities but, as discussed above, the guidance notes (8) lack precision on this point. Our view is that we do not need the likelihood concept as introduced by the IPCC documents. All that we need is included in the concepts probability model and associated parameters, including frequentist probabilities. We thus appreciate the need for distinguishing between epistemic and aleatory uncertainties when the latter can be meaningfully defined.
This makes it clear that uncertainties are both related to parameter uncertainties and the difference between the model output and the actual value of the quantity studied. A clear distinction between the model concepts and their estimations is required to ensure structure and scientific quality of the uncertainty analysis. Such a dichotomy is not seen in the IPCC documents, and in our view much of the issues discussed in this article can be traced back to this shortcoming.
Conceptualizing and treating risk and uncertainties in relation to climate change are challenging tasks, and critical reflections on which ideas, principles, and methods to use are essential for achieving sound scientific quality of the assessments performed. Considerable work has been conducted to this end, some contributions being highlighted in this article. However, there is always more room for improvement considering that the climate change models are under high scrutiny by stakeholders and politicians. For some other interesting perspectives related to this discussion, see Cooke, (43,44) who reviews the use of uncertainty analysis and provides arguments against the adoption of fuzzy logic.

CONCLUSIONS
In this article, we have argued that the IPCC assessment reports fall short of a theoretically and conceptually convincing foundation when it comes to the treatment of risk and uncertainties. The risk perspectives adopted do not adequately reflect extreme outcomes and a poor knowledge base. The important concepts of confidence and likelihood used in the IPCC documents remain too vague to be used consistently and meaningfully in practice.
An alternative risk uncertainty framework for climate change research has been presented in this article. We have argued that this framework leads to more theoretically substantiated characterizations of the risks and uncertainties, as well as an improved conceptual and methodological foundation of the climate change research-particularly for future IPCC studies. We have made several suggestions for how to improve the risk and uncertainty treatment. Specifically, we argue for the use of knowledge-based probabilities to guide the assignments of confidence and remove the term likelihood from the methodological terminology. Instead, future reports should focus on the fundamental concepts such as probability models and frequentist probabilities (which reflect aleatory uncertainties), model parameters, knowledge-based probabilities, and imprecise probabilities, which we were able to show can be clearly defined and coherently linked to each other. In particular, the distinction between underlying parameters (for example, frequentist probabilities) and their estimation methods needs to be highlighted. In addition, it is essential that the strength of knowledge that the probabilities (exact or interval) are based on is adequately assessed, as well as the potential for surprises, as discussed in Section 3.
The critical remarks on the treatment of uncertainty and risk should not be interpreted as an expression of skepticism toward the main insights from IPCC climate change reports or even misjudged as a critical position toward the major assumption of anthropogenic climate change. This is also not our main field of expertise. (45) The main purpose of this article is to contribute to strengthening the quality of the complex and ambiguous assessment of future climate change by suggesting improvements in the analytic handling of risks and uncertainty. This can assist IPCC in making its messages and recommendations more robust.