An error estimation method for precipitation and temperature projections for future climates


  • F. M. Woldemeskel,

    1. School of Civil and Environmental Engineering, University of New South Wales, Sydney, New South Wales, Australia
    Search for more papers by this author
  • A. Sharma,

    Corresponding author
    1. School of Civil and Environmental Engineering, University of New South Wales, Sydney, New South Wales, Australia
      Corresponding author: A. Sharma, School of Civil and Environmental Engineering, University of New South Wales, Sydney, NSW 2052, Australia. (
    Search for more papers by this author
  • B. Sivakumar,

    1. School of Civil and Environmental Engineering, University of New South Wales, Sydney, New South Wales, Australia
    2. Department of Land, Air and Water Resources, University of California, Davis, California, USA
    Search for more papers by this author
  • R. Mehrotra

    1. School of Civil and Environmental Engineering, University of New South Wales, Sydney, New South Wales, Australia
    Search for more papers by this author

Corresponding author: A. Sharma, School of Civil and Environmental Engineering, University of New South Wales, Sydney, NSW 2052, Australia. (


[1] Projections of precipitation and temperature from Global Climate Models (GCMs) are generally the basis for assessment of the impact of climate change on water resources. The reliability of such assessments, however, is questionable, since GCM projections are subject to uncertainties arising from inaccuracies in the models, greenhouse gas emission scenarios, and initial conditions (or ensemble runs) used. The purpose of the present study is to quantify these sources of uncertainties in future precipitation and temperature projections from GCMs. To this end, we propose a method to estimate a measure of the associated uncertainty (or error), the square root of error variance (SREV), that varies with space and time as a function of the GCM being assessed. The method is applied to estimate uncertainty in monthly precipitation and temperature outputs from six GCMs for the period 2001–2099. The results indicate that, for both precipitation and temperature, uncertainty due to model structure is the largest source of uncertainty. Scenario uncertainly increases, especially for temperature, in future due to divergence of the three emission scenarios analyzed. It is also found that ensemble run uncertainty is more important in precipitation simulation than in temperature simulation. Estimation of uncertainty in both space and time sheds lights on the spatial and temporal patterns of uncertainties in GCM outputs. The generality of this error estimation method also allows its use for uncertainty estimation in any other output from GCMs, providing an effective platform for risk-based assessments of any alternate plans or decisions that may be formulated using GCM simulations.

1. Introduction

[2] Global climate change is anticipated to have enormous impacts on our water resources. Although it is hard to make exact predictions of these impacts, a majority of studies suggests intensification of the global hydrologic cycle and occurrence of more-frequent and greater-magnitude extremes, such as floods and droughts [Intergovernmental Panel on Climate Change (IPCC), 2007; Kundzewicz et al., 2008; Milly et al., 2002]. Recent increases in abnormal floods and droughts around the world seem to have only strengthened such findings. As a result, study of climate change impacts on water resources is at the forefront of scientific research today (see Cleugh et al. [2011], Fowler et al. [2007], IPCC [2007], Obeysekera et al. [2011], Piao et al. [2010], Sahoo et al. [2011], Sivakumar [2011], and Towler et al. [2010] for some recent accounts).

[3] Toward assessing the impacts of climate change on water resources, the following steps are typically adopted: (1) projection of future climate data (e.g., precipitation, temperature) using Global Climate Models (GCMs); (2) downscaling of coarse-scale GCM outputs to fine-scale data appropriate for hydrology and water resources studies; and (3) estimation of river flow and groundwater levels using hydrologic models. Although this procedure is considered reasonable, there are also important questions about its reliability because of the various uncertainties involved in GCM projections, downscaling methods, and hydrologic models [Sivakumar, 2011; Xu, 1999]. The present study focuses on the quantification of uncertainties associated with outputs from GCMs.

[4] Uncertainties in GCM outputs arise due to many factors, including uncertainty in the representation of the climate system in models, uncertainty in greenhouse gas emissions (GGE) scenarios, and the internal variability of the climate system itself. Yip et al. [2011] describe these as follows: “Model uncertainty arises because of an incomplete understanding of the physical processes and the limitation of implementation of the understanding. Scenario uncertainty arises because of incomplete information about future emissions. Internal variability is the natural unforced fluctuation of the climate system.” Extensive research has already been carried out toward understanding of the overall uncertainties in climate change impact assessment using multiple GCMs/RCMs (Regional Climate Models), GGE scenarios, ensemble runs, downscaling methods, and hydrologic models [e.g., Chen et al., 2011; Déqué et al., 2007; Kay et al., 2009]. Such assessments give a wide range of values for a given variable of interest (e.g., flow), which can then be used for possible alternative planning and designing needed.

[5] A major limitation of the above approach, however, is that the single or multiple GCM/RCM simulations are assumed to be a good representative of what will happen in the future. This is not a reasonable assumption, given the known/unknown uncertainties in GCM simulations, especially if only a single model and scenario are used. In view of this problem, an important question is if it is possible to explicitly quantify the uncertainty for any GCM output variable in space and time, by making use of estimates of simulations from multiple GCMs? Reliable quantification of these uncertainties indeed allows one to ask more sensible questions, general and specific, such as: (1) Are GCM estimates of precipitation over high altitudes less uncertain compared to those over coasts? (2) Is the precipitation uncertainty associated with El-Niño events higher than that associated with other large-scale climatic events? (3) Where and how should the uncertainty associated with rainfall inputs be taken into account in planning, design, and management of water resources structures? Furthermore, one can also apply such information to investigate the propagation of GCM uncertainty to impact assessment models (e.g., hydrologic models) and to potentially reduce bias in model parameter estimation using methods such as simulation extrapolation [Chowdhury and Sharma, 2007] or Bayesian total error analysis [Kavetski et al., 2006] that would otherwise occur due to GCM output uncertainty (see Wilby [2005] for details).

[6] Quantification of uncertainties in GCM simulations requires utilization of many ensemble runs for each model and scenario. However, the climate modeling groups around the world produce, as of now, at the most only a few ensemble runs for each scenario. Using these limited number of ensemble runs, studies generally quantify the uncertainties in model, scenario, and internal variability in GCM projections and then add up these individual contributions to obtain the total uncertainty [Déqué et al., 2007; Hawkins and Sutton, 2009; Hodson and Sutton, 2008; Yip et al., 2011]. An important step in such studies is the use of multiple GCMs/RCMs, GGE scenarios, and ensemble runs as well as the application of the analysis of variance (ANOVA), which is a statistical method to partition variances between and within groups [e.g., Harris, 1994]. For instance, Déqué et al. [2007] use ten RCMs, two GGE scenarios, three GCMs for boundary forcing, and three ensemble runs to evaluate the uncertainty for mean change of precipitation and temperature, and report that uncertainty due to GCM is greater than other uncertainties, especially for temperature. Hawkins and Sutton [2009], fitting polynomial functions to temperature data, show that the relative importance of the above three sources of uncertainty varies in different regions and for different forecast lead times; the results are also shown to be comparable with those from the ANOVA analysis [Yip et al., 2011].

[7] Despite their usefulness, the above studies possess an important limitation, which is that the uncertainties in GCM simulations are quantified for long-term means of climate data, such as precipitation and temperature (here ‘long-term’ refers to five years or more). Although quantification of uncertainties for long-term mean offers insights on its magnitude, it does not offer any information about the variability of the uncertainty at shorter timescales (e.g., monthly, annual). The main reason behind the analysis for long-term mean is the disagreement of GCM simulations at shorter timescales. In the present study, we attempt to overcome this problem by estimating the uncertainties across space and time.

[8] The main purpose of the present study is to develop an error estimation method that yields approximate quantification of the main sources of uncertainty in future GCM projections in both space and time. To this end, we analyze uncertainties in spatial and temporal patterns of GCM simulations at global and regional scales. We also discuss the issues of independence and choice of GCM(s) with regard to uncertainty estimation. More specifically, we formulate a method that estimates an uncertainty metric, which we call “Square Root Error Variance” (SREV), for future climate projections.

[9] In this study, we estimate uncertainty for GCM precipitation and temperature simulations at a monthly time step across the world for the period 2001–2099. Our focus on precipitation and temperature is based on our specific interest in water resources assessment: precipitation is the most important input for hydrologic models (e.g., rainfall-runoff), whereas temperature forms a key input for estimation of evaporation and evapotranspiration. Nevertheless, our error estimation method is general and can be used for estimation of uncertainty in other GCM output variables (e.g., wind velocity, atmospheric pressure) as well.

[10] The rest of this paper is organized as follows. Section 2 describes the GCM data sets, and section 3 presents details of the proposed error estimation method. Section 4 presents the uncertainty estimation results. A discussion of these results is made in section 5, and conclusions are given in section 6.

2. Data

[11] Monthly precipitation and temperature outputs from six GCMs of the World Climate Research Programme (WCRP) Coupled Model Inter-comparison Project phase 3 (CMIP3) multimodel data sets are considered for analysis in the present study. The multimodel data sets are downloaded from the Earth System Grid (ESG) website ( We use the CMIP3 data sets as an example (as they are already established well) to demonstrate the applicability of the error estimation method; however, the method can be applied to CMIP5 or other data sets as well. The GCMs are selected on the basis of availability of at least three ensemble runs for three Special Report on Emission Scenarios (SRES) emission scenarios (B1, A1B, and A2), so as to allow estimation of all three sources of uncertainty (i.e., model, scenario, and ensemble runs) as well as to be confident of interpretations of results and conclusions.

[12] The above three scenarios (B1, A1B, and A2) are carefully chosen to represent a wide range of emission scenarios, i.e., low, medium, and high forcing effects, respectively, as they are based on different assumptions about population growth, economic development, energy use, and globalization [IPCC, 2007; Knutti et al., 2008]. The scenarios are more specifically characterized as follows: B1 represents a convergent world with low population growth, rapid changes in economic structures toward a service and information economy, reduction in materials intensity, and the introduction of clean and resource efficient technologies; A1B represents a future world of very rapid economic growth and rapid introduction of new and more efficient technology; A2 represents a very heterogeneous world with economic development primarily regionally oriented and per capita economic growth and technological change more fragmented.

[13] Overall, six GCMs, three scenarios, and three ensemble runs for the period 2001–2099 are considered, resulting in a total of 54 (3 × 3 × 6) monthly time series. Table 1 presents some basic information about the groups that have developed these GCMs and the spatial resolutions of these models. Figure 1 shows an example of the projections of global mean precipitation (Figure 1, left) and temperature (Figure 1, right) for the three scenarios (B1, A1B, and A2) with a single ensemble run; the mean values are obtained by smoothing values over five years using lowess smoother. The figure reveals that the projections for both precipitation and temperature corresponding to the three scenarios diverge in the future (especially after 2040). Global mean and standard deviation of precipitation and temperature for the six GCMs during two different time periods (i.e., 2011–2030 and 2071–2090) are given in Table 2. The mean precipitation increases during 2071–2090 in comparison to 2011–2030 for all the scenarios considered (B1, A1B, and A2); however, B1 scenario shows larger standard deviation than A1B and A2 scenarios. Similarly, mean temperature also shows an increase during 2071–2090 in comparison to 2011–2030. Further, A2 scenario, which is the most extreme scenario considered, gives the largest temperature increase during 2071–2090.

Table 1. List of GCMs and Their Atmospheric Horizontal Resolutionsa
GCMModeling Group(s), CountryAtmospheric Horizontal Resolution
  • a

    See IPCC [2007]. The horizontal resolutions are expressed in triangular spectral truncation as well as degrees of latitude/longitude.

PCM (Parallel Climate Model)National Center for Atmospheric Research (NCAR), USAT42 (∼2.8° × 2.8°)
CCSM3 (the Community Climate System Model, version 3)National Center for Atmospheric Research (NCAR), USAT85 (∼1.4° × 1.4°)
MIROC3.2 (medres) (a Model for Interdisciplinary Research On Climate, version 3.2)Centre for Climate System Research (The University of Tokyo), National Institute for Environmental Studies, and Frontier Research Centre for Global Change (JAMSTEC), JapanT42 (∼2.8° × 2.8°)
ECHO-GMeteorological Institute of the University of Bonn, Meteorological Research Institute of KMA, and Model and Data group, Germany/KoreaT30 (∼3.9° × 3.9°)
ECHAM5/MPI-OMMax Planck Institute for Meteorology, GermanyT63 (∼1.9° × 1.9°)
CGCM3.1 (T47) (Coupled Global Climate Model, version 3.1)Canadian Centre for Climate Modeling & Analysis, CanadaT47 (∼2.8° × 2.8°)
Figure 1.

Global mean (left) precipitation and (right) temperature smoothed over five years using lowess smoother. The light colors show precipitation and temperature for six GCMs and three SRES scenarios (B1, A1B, and A2). The bold colors show mean of precipitation and temperature for six GCMs for each scenario. A single ensemble run (run 1) is shown for each SRES scenario.

Table 2. Global Mean (μ) and Standard Deviation (σ) of Precipitation and Temperature for Six GCMs Under B1, A1B, and A2 Scenarios for Two Future Time Periods (2011–2030 and 2071–2090)
Precipitation (mm/month)
Temperature (K)

3. Methodology

[14] The proposed method for uncertainty estimation involves four important steps: (1) data interpolation to common grid; (2) data conversion to percentiles; (3) uncertainty estimation; and (4) translation of the estimated uncertainty to time series. The procedure for conversion of data to percentiles and estimation of uncertainty for each quantile is somewhat similar to the quantile regression approach [Koenker and Bassett, 1978], which estimates functional relationships of variables at any quantile of a distribution. However, unlike quantile regression, our approach estimates uncertainty of GCMs simulations at any and every percentile. The above four steps are discussed in more detail below.

[15] Step 1Data interpolation to a common grid. Precipitation and temperature data gathered from the above six GCMs (at different spatial resolutions) are interpolated to a common grid, i.e., at 3° × 3° latitude/longitude grid. To achieve this, an inverse distance weight interpolation method using four nearest grid cells is applied, after Nawaz and Adeloye [2006].

[16] Step 2Data conversion to percentiles.The common-gridded data are ranked in ascending order from the beginning to the last time step (which is 1188, corresponding to the number of values in the time series = 99 × 12).Figure 2, for instance, shows the percentile plots for precipitation (Figure 2, left) and temperature (Figure 2, right) for all the six GCMs, three scenarios, and three ensemble runs at a grid cell in Southeast Australia (−32.5° latitude and 147.5° longitude). The figure clearly reveals the variability of precipitation and temperature at different percentiles. However, greater variability across simulations can be seen particularly at small percentiles for temperature than for precipitation.

Figure 2.

Percentile plots of (left) precipitation and (right) temperature at a point in Southeast Australia (latitude = −32.5°, longitude = 147.5°). Each color shows different GCMs and consists of nine precipitation and temperature values (For three SRES scenario and three ensemble runs). The names of the GCMs are indicated by P (PCM), CC (CCSM3), M (MIROC3.2 (medres)), EG (ECHO-G), EM (ECHAM5/MPI-OM), and CG (CGCM3.1 (T47)).

[17] Step 3Calculation of uncertainty. Uncertainty in GCM simulations can be assessed by either analyzing the consistency between different GCM projections or comparing historical GCM simulations with observed data [Dessai et al., 2005; Räisänen, 2007]. The former approach is chosen in this study, as our interest herein is to assess the uncertainty of future climate projections. Standard deviation at a particular percentile is used as a measure of uncertainty, as it calculates variability between equally possible climate projections of multiple GCMs. Here we apply the standard deviation in a novel way, which we call “Square Root Error Variance” (SREV), to estimate model, scenario, and ensemble run uncertainty individually as well as their total. Equations (1) to (3) are used to calculate the model, scenario, and ensemble run uncertainty at each percentile (p), denoted as SREVpM, SREVpS, and SREVpE, respectively (M – model, S – scenario, and E – ensemble run):

display math
display math
display math

where var is variance, Mp | Sp, Ep is precipitation/temperature values for a given scenario and ensemble run at p, S p | M p, E p is precipitation/temperature values for a given model and ensemble run at p, and E p | M p, S p is precipitation/temperature values for a given model and scenario at p. For clarity of presentation, the notations of grid cell indexes are excluded. Further, the units for the SREV values are similar to those for precipitation and temperature, as the case may be (i.e., ‘mm’ for precipitation and degree ‘K’ for temperature).

[18] As mentioned earlier, M = 6, S = 3, and E = 3 (representing number of GCMs, scenarios, and ensemble runs) are considered in the present study. The parameter Er is an ensemble run chosen randomly from 1 to 3 (see Table 3) at about the median percentile (i.e., rank = 594 out of 1188); the superscript (r) is to indicate that Er is a randomly chosen ensemble run. The table shows such ensemble runs used for calculating model and scenario SREV. For example, for calculating model SREV, ensemble runs 1, 2, 3, 1, 2, and 2 are used. The variability at this percentile among the different GCMs, scenarios, and ensemble runs is shown in Figure 3 for precipitation and temperature. At this percentile, the variance of precipitation is equal to 3295, 47, and 4 mm2 and of temperature is equal to 10, 1, and 0.01 K2 for model, scenario, and ensemble runs, respectively. The reason behind using only a single combination of ensemble runs for the estimation of model and scenario uncertainty is this: since the variability across ensemble runs is much smaller than the model and scenario SREV, the choice of different combinations of ensemble runs does not really have any effect in the estimated values of the model and scenario SREV.

Table 3. Randomly Selected Ensemble Runs (From 1 to 3) Used to Calculate Model and Scenario Square Root Error Variance (SREV) at About Median Percentilea
  • a

    The numbers in the parentheses are used for scenario SREV estimation. Model SREV is calculated for each scenario, whereas scenario SREV is calculated for each model, and then mean uncertainty for either case is determined. The symbols used for GCMs are defined as P (PCM), CC (CCSM3), M (MIROC3.2 (medres)), EG (ECHO-G), EM (ECHAM5/MPI-OM), and CG (CGCM3.1 (T47)).

B11 (3)2 (3)3 (3)1 (3)2 (3)2 (3)
A1B1 (3)2 (3)3 (3)1 (3)2 (3)2 (3)
A21 (3)2 (3)3 (3)1 (3)2 (3)2 (3)
Figure 3.

Variability of model, scenario, and ensemble at rank about 50th percentile for (a, b, and c) precipitation and (d, e, and f) temperature at grid cell similar to Figure 2. Whiskers show range from minimum to maximum values. Figures 3a and 3d show model variability for three scenarios (B1, A1B, and A2) and ensemble run 1; Figures 3b and 3e show scenario variability for six models and ensemble run 1; and Figures 3c and 3f show ensemble run variability for six GCMs and A2 scenario. The names of GCMs are same as given in Figure 2.

[19] The symbol V is a variable representing precipitation/temperature, with VMSE being the Eth observation for model M and scenario S. The symbols inline image, inline image, and inline image are precipitation/temperature values averaged over models, scenarios, and ensemble runs, respectively, and are given by:

display math
display math
display math

[20] Finally, total SREV (SREVpT) is obtained by taking square root of sum of squares of individual SREV, as follows:

display math

[21] It is important to note that the SREV metric used in this study is equivalent to the conditional total standard deviation of the variables of interest conditional to specific percentiles and, thus, can be interpreted likewise (here ‘standard deviation’ (SD) refers to the standard deviation of all data points estimated by mixing data from different models, scenarios, and ensemble runs). Hence, it is indeed a reasonable and useful statistic for inferring uncertainty associated with the GCM outputs. The following short example helps explain how the SREV metric can be interpreted. Let us assume that the precipitation and total SREV for a given GCM output are 100 and 20 mm/month, respectively. With the assumptions that the precipitation data follow a Normal distribution and that the sample standard deviation represents the standard deviation of the population, one can say that the precipitation data may fall in the range 60–140 mm/month (=100 ± 2 × 20) with a 95% probability.

[22] An important assumption involved in this method of estimating uncertainty is that the non-exceedance probabilities of different GCMs are consistent. This is also closely related to the assumption in the quantile-based bias correction approach ofLi et al. [2010]that matches GCM simulations with observations at the same percentile. However, unlike the quantile-based bias correction method, in the present study, we estimate SREV of model, scenario, and ensemble runs matching percentiles of GCM projections. We also assume that each of the six GCMs analyzed is independent of the others, an assumption usually made in climate studies [Pirtle et al., 2010]. A further assumption in this method is that all GCM projections analyzed have equal uncertainty at any percentile, as is made in equations (1) to (3).

[23] At this point, it is also relevant to note that the method followed here to calculate the individual sources of uncertainty and the total uncertainty is similar in intent to the one adopted in the ANOVA approach [Hodson and Sutton, 2008; Holtanová et al., 2010]. However, there are also notable differences between the two approaches in terms of the assumptions involved, as pointed out as follows. According to the ANOVA approach, variables are averaged across models, scenarios, and ensemble runs in the evaluation of model, scenario, and ensemble run variability. For instance, variables are averaged across scenarios and ensemble runs to obtain values (six values in the present case) for each GCM at a particular percentile, which are then used to calculate model variability. However, instead of averaging out variables across models and scenarios, we select, in our approach, the ensemble runs randomly (from 1 to 3). As a result of this assumption, equation (7) does not include the interaction term, which supposedly accounts for different model responses to the same forcing [Hodson and Sutton, 2008]. In our case, the interaction term is partially shared between model and scenario uncertainty.

[24] Step 4Translation of uncertainty to time series. The estimated individual and total SREV conditional on the percentiles of simulations are converted to actual time series. The month and year of GCM simulations at any given percentile are used to obtain SREV value for that particular month and year of the time series.

4. Results

[25] The proposed error estimation method is now applied to the monthly precipitation and temperature data from the six GCMs described above. Here, for the sake of brevity, we present results for only two of these six GCMs: ECHO-G and ECHAM5/MPI-OM. These two GCMs are selected mainly based on information and recommendations available from past studies, especially for Australian conditions: ECHO-G has been shown to have better skills in representing probability density function of precipitation and temperature [Perkins et al., 2007] and ECHAM5/MPI-OM in representing persistence [Johnson et al., 2011], compared to a host of other GCMs analyzed. The spatial uncertainties are discussed in section 4.1 for two future time spans (2020s, i.e., mean of 2011 to 2030; and 2080s, i.e., mean of 2071 to 2090), whereas temporal uncertainties are discussed in section 4.2 for global and selected regional means.

4.1. Spatial Uncertainty

[26] Figures 4 and 5show the square root of error variances (SREV) of precipitation and temperature for ECHO-G and ECHAM5/MPI-OM models, respectively; these results correspond to scenario A2 and ensemble run 1.

Figure 4.

Maps of square root error variance (SREV) values for total, model, scenario, and ensemble uncertainty for (a) precipitation and (b) temperature for 2020s (2011–2030 mean) and 2080s (2071–2090 mean). The SREV values are shown for model ECHO-G, with A2 scenario and ensemble run 1.

Figure 5.

Same as Figure 4but for ECHAM5/MPI-OM.

[27] For precipitation, the SREV values for 2020s show that the total uncertainty of ECHO-G is larger in midlatitudes and reduces toward high and low latitudes (Figure 4a). This is in accordance with previous studies, which also report considerable uncertainty for precipitation change at midlatitudes than at high and low latitudes [IPCC, 2007; Miller and Yates, 2006]. The results also indicate that model uncertainty is the main contributor to the total uncertainty in all regions and is much larger than scenario and ensemble run uncertainties (see first row and first through fourth columns of Figure 4a). However, scenario and ensemble run uncertainties are also considerable in midlatitudes, with scenario uncertainty being generally larger than ensemble run uncertainty. The results for 2080s are similar to those for 2020s, except that a slight increase in uncertainty is estimated in midlatitudes, especially for scenario uncertainty (see second row and first through fourth columns of Figure 4a).

[28] As for temperature, unlike precipitation, uncertainty is large in high and low latitudes, with the maximum values obtained over the Arctic Ocean (see first row and first column of Figure 4b), which show large warming in the future due to a decrease in ice cover and thickness [Räisänen, 2007]. With regard to contributions of uncertainties, model uncertainty is still the main contributor to the total uncertainty, similar to that observed for precipitation; however, scenario uncertainty is more pronounced for temperature than for precipitation (see first row and second through fourth columns of Figure 4b). Further, scenario uncertainty increases for 2080s when compared to that for 2020s, which is likely due to the divergence of the three scenarios for 2080s than for 2020s (see second row and first through fourth columns of Figure 4b). It is important to note that the total uncertainty does not show any noticeable increase for 2080s compared to 2020s, since model uncertainty, the main contributor, is almost constant. The results of ECHAM5/MPI-OM (Figures 5a and 5b) are comparable with ECHO-G, except that a decrease is observed for uncertainty in temperature in the Polar regions (see second row and first column ofFigure 5b). The results are generally consistent with those observed for the other four GCMs as well (not shown).

4.2. Temporal Uncertainty

4.2.1. Global Mean

[29] Figures 6a and 6bpresent the temporal SREV values for precipitation for global five-year moving average for ECHO-G and ECHAM5/MPI-OM, respectively; these results correspond to scenario A2 and ensemble run 1. The results indicate that model uncertainty is the largest contributor to the total uncertainty, consistent with the results obtained for the spatial uncertainty case (section 4.1). Further, the SREV values of ECHO-G and ECHAM5/MPI-OM are also comparable, with the only exception being that the model uncertainty for precipitation from ECHAM5/MPI-OM is larger than that from ECHO-G (Figure 6b).

Figure 6.

Global mean of total, model, scenario, and ensemble square root error variances for (a and b) precipitation and (c and d) temperature. Figures 6a and 6c are for ECHO-G and Figures 6b and 6d are for ECHAM5/MPI-OM with A2 scenario, and ensemble run 1. Five-year moving average using lowess smoother is shown. The vertical axis is in logarithmic scale.

[30] For temperature, scenario uncertainty is found to be significantly greater than ensemble run uncertainty, and it clearly shows an increasing trend due to the divergences of the different scenarios in the future (Figures 6c and 6d). The ensemble run uncertainties estimated here are also comparable with the results obtained by Hawkins and Sutton [2009], who report an overall uncertainty (i.e., mean of uncertainty in space and time) of 0.12 K. The scenario uncertainty estimates between the two studies are also generally comparable; however, the model uncertainty is underrepresented in Hawkins and Sutton [2009] compared to our results. Since uncertainties vary in different regions depending on precipitation and temperature magnitudes as well as on patterns of precipitation and temperature projections, spatial mean uncertainties of some selected regions are also estimated, as detailed next. The global mean monthly uncertainty estimates for the six GCMs are made available online (

4.2.2. Regional Means

[31] For studying uncertainties in regional means, three regions located at different geographic regions, and also with vastly different climatic conditions, land use characteristics, and socio-economic development are considered: Western Australia (WA), the Amazon (A), and Greenland (GL). The extent of each of these regions and the number of grid cells considered are shown inTable 4.

Table 4. Temporal Uncertainty Analysis for Regional Means: Regions and Their Basic Details
Region NameSymbolLatitude ExtentLongitude ExtentNumber of Cells
West AustraliaWA13°S – 34°S113°E – 131°E49
AmazonA8°S −13°S49°W – 73°W64
GreenlandGL68°N – 86°N19°W – 58°W98

[32] Figures 7a–7cshow the precipitation SREV values for ECHO-G, scenario A2, and ensemble run 1. The results show that model uncertainty is largest in the Amazon (Figure 7b) and smallest in Greenland (Figure 7c). However, this is not the case for temperature, with the largest model uncertainty observed for Greenland (Figure 7f) followed by that for Western Australia (Figure 7d) and the Amazon (Figure 7e). These observations are even clearer in Figure 8, which shows that the relative contributions (i.e., proportions) of model, scenario, and ensemble run uncertainties to the total uncertainty for the above regions as well as for the global (G) means. The relative contribution of ensemble run uncertainty for global mean is approximately 8% and 5% for precipitation and temperature, respectively. This is a clear indication that ensemble run uncertainty is more important in precipitation estimation than in temperature estimation, an observation also made by Räisänen [2001]. Unlike for precipitation, the relative contribution of scenario uncertainty for temperature increases for 2080s when compared to that for 2020s. Similar regional variations in the accuracy of GCMs for different variables have also been reported by Johnson and Sharma [2009], although their study uses a variable convergence score approach for assessment of agreement between/among GCM projections.

Figure 7.

Regional mean of total, model, scenario, and ensemble square root error variances for (a, b, and c) precipitation and (d, e, and f) temperature with ECHO-G, A2 scenario, and first ensemble run. Five-year moving average using lowess smoother is shown. The vertical axis is in logarithmic scale.

Figure 8.

Percentage of contribution of model, scenario, and ensemble run SREV to the total SREV for (a) precipitation and (b) temperature for two time spans (2020s and 2080s). EG (ECHO-G) and EM (ECHAM5/MPI-OM) with A2 scenario and first ensemble run are shown. The symbols used for they axis label are defined as follows: G (Global), A (the Amazon), WA (West Australia), and GL (Greenland).

5. Discussion

[33] Precipitation uncertainty is considerably large in midlatitudes, which is also in accordance with the generally large magnitude of rainfall occurrence in these regions due to air masses that converge from both northern and southern hemispheres (Figures 4 and 5). Conversely, precipitation uncertainty is very small in latitudes close to the north and south poles as well as in arid and semi-arid regions that generally receive only a meager amount of rainfall throughout the year, such as Sahara and the Middle East. It is possible, therefore, that large uncertainty in wet regions and small uncertainty in dry regions could simply be an indication of the direct relationship between uncertainty in precipitation estimation and magnitude of precipitation. This can indeed be seen from the percentile plots of GCM simulations presented inFigure 2; i.e., the variability between different projections is small at small percentiles, but it increases as the percentile increases.

[34] To further examine the effects of precipitation and temperature magnitude on uncertainty, the SREV ratio is calculated by dividing the SREV values for 2020s by the mean monthly simulated precipitation and temperature (see first row and first and second columns of Figure 9). The results generally reveal that the spatial variability of SREV diminishes when divided by precipitation and temperature. There are a few exceptions to this observation, however, such as some areas in the Pacific Ocean, Atlantic Ocean, and Sahara, which have noticeably larger uncertainty with respect to the magnitude of precipitation; the observation of large uncertainty relative to precipitation magnitude for Sahara is also consistent with that reported by Johnson and Sharma [2009]. For temperature, however, the spatial pattern of SREV values is not affected when divided by the magnitude of temperature.

Figure 9.

(top) Ratio of SREV to mean monthly precipitation and temperature for 2020s and (bottom) mean monthly 2020s precipitation and temperature for ECHO-G under A2 scenario and ensemble run 1.

[35] The reason behind the small uncertainty for temperature against large uncertainty in precipitation in midlatitudes could be twofold. First, the range of (or difference between) maximum and minimum values of temperature is significantly smaller when compared to that of precipitation. Second, there is considerable variability in the minimum temperature values (including below zero ones), unlike precipitation where it is generally zero. The mean SREV estimates of the regional 5-year moving average for Greenland are less variable in time than those for Western Australia and the Amazon for precipitation, whereas the reverse is true for temperature (Figure 7). Possible reasons for this are: (1) differences between the numbers of grid cells used for calculating mean SREV for different regions (Table 4), since a smoother mean is normally expected when the SREV is calculated for a large area; and (2) differences in the magnitude of precipitation and temperature between these regions. Further analysis is needed to substantiate the extent to which these two possibilities affect the temporal variability of SREV in different regions.

[36] In this study, the evaluation of uncertainty in GCM precipitation and temperature projections is done at regional and global scales at a monthly time scale. Since none of the past studies, to our knowledge, have considered similar spatial and temporal scales for uncertainty estimation of GCM outputs, direct comparisons of the present results with others are not possible. However, some reliable comparisons between the results over a long-term timescale (i.e., five years or more) can be made. For instance, our study indicates that the uncertainty due to model is equal to about 80% and 75% for precipitation and temperature, respectively. This is clearly in accordance with the studies byDéqué et al. [2007], Hawkins and Sutton [2009], and Yip et al. [2011], which report that the largest source of uncertainty is due to model, especially for projections until 2050. In our study, scenario uncertainty is the second largest followed by ensemble runs for both precipitation and temperature. Although scenario uncertainty slightly increases after 2050, the order of percentage contribution from the three sources stays the same throughout the 21st century. However, the studies by Yip et al. [2011] and Hawkins and Sutton [2009] report that the scenario uncertainty becomes greater than model uncertainty for temperature projection after about 2050. Hawkins and Sutton [2011] also report that ensemble runs is the largest uncertainty for precipitation projections until 2030 and model uncertainty dominates thereafter. These differences in the percentage contribution of different sources of uncertainty between our study and other studies could be due to the methods adopted. More specifically, our study ascertains uncertainty by focusing on a specific quantile in contrary to past studies which estimate uncertainty for temporal mean of the variable. As for spatial distribution of uncertainty, our results indicate that precipitation uncertainty is large in midlatitudes and temperature uncertainty is large in the north and south poles. These results generally agree well with previous findings by Hawkins and Sutton [2011] and Miller and Yates [2006] for precipitation and by Hawkins and Sutton [2009] for temperature.

[37] The large percentage contribution of model uncertainty suggests that climate processes are inadequately represented in the GCMs. This problem is expected, since GCM simulations have systemic deviations from observations that needs to be corrected. Applying statistical bias correction methods, such as Equidistant quantile mapping [Li et al., 2010] or Nested bias correction [Johnson and Sharma, 2012], prior to estimating SREV values could significantly reduce model uncertainty. However, such uncertainty reduction is due neither to an advancement of our understanding of the climate processes nor to an improvement of the implementation of this understanding, but it is simply through post-processing of the statistical properties of GCM simulations to match with observations. We will investigate the extent of SREV reduction through bias correction in a future study.

[38] Another important assumption in the proposed error estimation method for assessment of uncertainty in future climate projections is that the GCMs are independent. However, this assumption is imprecise, since the models (and modeling groups) share theoretical concepts, data information, literature, and even codes [Pirtle et al., 2010]. Justification for the independence of GCMs is a challenging topic in climate studies, and there have indeed been some efforts so far [Abramowitz and Gupta, 2008; Masson and Knutti, 2011; Pennell and Reichler, 2011; Power et al., 2012]. These studies evaluate independence mainly based on whether different GCM simulations differ from multimodel mean (or observations) significantly or not. However, as reported by Pirtle et al. [2010], this does not necessarily prove independence, as models could still share similar biases. Nonetheless, it is common to assume that GCMs are independent in climate studies [Pirtle et al., 2010], and thus the independence assumption for the six models is used in our study. It is relevant to note that the findings of Masson and Knutti [2011] and Pennell and Reichler [2011], albeit their limitations, are also in favor of this assumption, as each of the six GCMs analyzed is less dependent on the other GCMs in their studies. Further, both these studies assess independence for the 20th century climate, but it is difficult to guarantee whether the same is true for future projections [Power et al., 2012].

[39] To examine the effects of choice and independence of GCM on uncertainty estimation, a preliminary sensitivity analysis is carried out here. To this end, the SREV values are compared for four different combinations of GCMs, as follows: (1) Group-1 – comprises the six GCMs analyzed in this study; (2) Group-2 – comprises 22 GCMs, each having at least a single ensemble run for the A1B scenario; (3) Group-3 – comprises 14 GCMs that are supposed to be independent according toPennell and Reichler [2011]; and (4) Group-4 – comprises 14 GCMs, each having at least a single ensemble run for the B1, A1B, and A2 scenarios.

[40] Figure 10apresents the estimated SREV results for precipitation simulations, for each grid cell along latitude averaged over all longitudes. The four groups generally show similar patterns; however, some differences are also seen at the peak values close to the equator for the case of uncertainty in terms of model. The GCMs in Group-3, which are considered to be independent, result in larger SREV peak when compared to that in Group-4, although both groups use equal number of GCMs (i.e., 14). This could be due to the interdependence of some of the GCMs in Group-4, which results in an underestimation of SREV. The six GCMs in Group-1 also underestimate the peak SREV compared to Group-2 and Group-3.

Figure 10.

Sensitivity of SREV for different combinations of GCMs for each grid cell along latitude averaged over all longitudes for (a) precipitation and (b) temperature. Group 1 – comprises all the six GCMs analyzed in this study; Group 2 – comprises 22 GCMs, each having at least a single ensemble run for the A1B scenario; Group 3 – comprises 14 GCMs that are presumed to be less dependent on each other, according to Pennell and Reichler [2011]; and Group 4 – comprises 14 GCMs, each having at least a single ensemble run for B1, A1B, and A2 scenarios. Model SREV sensitivity is shown for the four groups; however, scenario SREV sensitivity computation is possible only for Group 1 and Group 4.

[41] The SREV results for the temperature simulations are presented in Figure 10b. Group-2 gives higher SREV than all other groups, especially at low and high latitudes. Group-3 and Group-4 have similar SREV in all latitudes, which is an indication that SREV is less sensitive to the independent GCMs categorized in Group-3. There is also no noticeable sensitivity of scenario SREV to model choice, regardless of whether the data is precipitation or temperature. These observations leads to an interpretation that the interdependence of GCMs could affect the estimation of SREV and, therefore, further investigation is required to obtain a good sub-set of independent GCMs for SREV estimation. However, at this stage, evaluation of the interdependence of GCMs is premature, due to lack of a solid assessment framework as well as detailed information about internal structure of GCMs. Study of this is an interesting topic, especially for climate researchers, if detailed information about the models is made publicly available.

[42] In this study, all the six GCMs analyzed are assumed to have equal uncertainty at a particular percentile (Step 3 of the error estimation method). Several studies have suggested that the skill of the GCMs varies in reproducing the 20th century climate [e.g., Johnson et al., 2011; Perkins et al., 2007]. To address this issue, Hawkins and Sutton [2009] used multiplicative weights to account for varying accuracy of GCMs in their estimation of uncertainty for future projections. Introducing weights to account for accuracy of GCMs is meaningful only if the past accuracy of GCMs would be repeated for future projections as well. However, this is not the case, as reported, for instance, by Power et al. [2012] and Jun et al. [2008], where they discussed weak relationship between the accuracy of current and future GCMs simulations. In addition, lack of an accurate GCM skill measurement framework as well as limitation of length of verification data also complicate any attempt at finding reasonable weights [Irving et al., 2011; Weigel et al., 2010]. These issues have essentially led us to assume an equal uncertainty for all GCMs at a particular percentile.

6. Conclusions

[43] This study developed a new method for estimation of uncertainty in precipitation and temperature GCM simulations across space and time. The basis for this was six GCMs from CMIP3, selected so as to have at least 3 ensemble runs for the three emission scenarios considered. Monthly precipitation and temperature simulations for the period 2001–2099 from each of these models served as the basis for ascertaining the square root error variance (SREV), a measure equivalent to the standard deviation of the error conditional to the rainfall or temperature simulated. The SREV was calculated empirically, matching percentiles of different GCM simulations with an assumption that non-exceedance probabilities across models were consistent. Three main sources of uncertainty, namely model, scenario, and ensemble run, as well as the associated total uncertainty were evaluated for each of the six GCMs, three GGE scenarios, and three ensemble runs.

[44] The results show that model uncertainty is the largest contributor to the total uncertainty followed by scenario and ensemble run uncertainty for both precipitation and temperature. Unlike uncertainty due to model and ensemble runs (which is almost constant), scenario uncertainty shows a significant increase in the far future due to divergence of the three emission scenarios. This increase is particularly more pronounced for temperature than for precipitation. The results also reveal that the patterns of precipitation and temperature uncertainties in space are different: for precipitation, large uncertainties are estimated in midlatitudes close to the equator (which receive large amount of rainfall), whereas for temperature, large uncertainty is estimated in high and low latitudes. This suggests that the accuracy of GCMs in space varies on the type of variable analyzed. The ensemble uncertainty is more pronounced for precipitation than for temperature globally as well as in certain regions. The estimated SREV values are generally less affected by GCM independence and model choice, although considerable sensitivity is observed at the peak values. Nevertheless, further investigations are required to categorically establish, and possibly confirm, the effects of GCM interdependence in uncertainty estimation.

[45] The proposed error estimation method is an important contribution for improving climate models and climate change impact studies, since it is useful to improve climate models in locations where large uncertainty is revealed. The uncertainties at the monthly time step, together with GCM precipitation and temperature simulations, can also be used to mitigate parameter bias due to input uncertainty in climate change impact studies on water resources. Finally, the generality of the error estimation method allows its use for estimation of uncertainty for any other variable simulated from GCMs.


[46] This research was supported by an Australian Research Council funded project. The authors gratefully acknowledge the CMIP3 data contributions that served as the basis of the results reported here.