Alternative ways to evaluate a seasonal dynamical downscaling system


  • D. W. Shin,

    Corresponding author
    1. Center for Ocean-Atmospheric Prediction Studies, Florida State University, Tallahassee, Florida, USA
    • Corresponding author: D. W. Shin, Center for Ocean-Atmospheric Prediction Studies, Florida State University, Tallahassee, FL 32306–2840, USA. (

    Search for more papers by this author
  • Steve Cocke

    1. Center for Ocean-Atmospheric Prediction Studies, Florida State University, Tallahassee, Florida, USA
    Search for more papers by this author


[1] The value added by dynamically downscaling using a regional climate model, compared to using only a global climate model, is explicitly unveiled by using nontraditional skill evaluation statistics. The conventional model evaluation methods, such as temporal correlation of seasonal average rainfall, cannot always demonstrate the value of dynamically downscaled data. One of our primary alternative metrics for evaluating downscaling methodologies is comparing crop yields simulated using the downscaled data, assuming nonirrigated conditions, to yields simulated using observations. Rainfed crops in the southeast United States are very sensitive to water stress. In fact, these crops are often more sensitive to periods of wet/dry spells than to seasonal rainfall totals. Thus, using crop models as a performance metric provides an alternative to simply evaluating the prediction/simulation of the seasonal mean of some particular meteorological variables and has more practical relevance. Furthermore, while the discovery of climate signals contributing to total summertime precipitation variability remains elusive, our results suggest that dynamical regional models may better simulate intraseasonal variability than simply the anomalies of the seasonal mean, which further justify their usefulness in application models.

1 Introduction

[2] A number of regional climate models have been developed and extensively used to better resolve regional-scale seasonal climate variability from global model data [e.g., Gao et al., 2012; Chan and Misra, 2011; Nunes and Roads, 2007; Cocke et al., 2007; Sun et al., 2006; Roads, 2004; Fennessy and Shukla, 2000; Ji and Vernekar, 1997; Giorgi, 1990]. If a regional model captures the anomalies of the seasonal mean well at a higher resolution, it is generally considered as a successful dynamical downscaling system. However, most regional models tend not to resolve those anomalies better if its global model counterpart does not have the correct, or detectable, climate signals. Hence, not everyone agrees on the usefulness of downscaling approaches [e.g., Pielke and Wilby, 2012]. Some contend that the only improvement a regional model can provide is more detailed spatial representation of weather systems. Is there any value hidden in using a regional model even if it cannot improve the simulation or prediction of anomalies of the seasonal mean?

[3] In order to evaluate weather and/or climate model performance, many evaluation metrics have been used (e.g., anomaly correlations, biases, root-mean-square error, Heidke skill score, Brier skill score) and/or developed [e.g., Moise and Delage, 2011; Gleckler et al., 2008; Taylor, 2001]. While many of them are aimed at large-scale diagnostics (e.g., hemispheric modes of variability, assessment of the annual or diurnal cycle), there are other metrics which could more readily reveal the usefulness of a downscaling system. In this study, the authors attempt to discover the added value in a dynamical downscaling system by applying global and regional model data to an agricultural model.

2 Models

[4] Seasonal climate simulations are performed for the period 1987–2005 using the Center for Ocean-Atmospheric Prediction Studies (COAPS) global and regional modeling system [Shin et al., 2005, 2006; Cocke and LaRow, 2000]. There are two seasonal ensemble simulations per year: one for the summer crop-growing season (March–September) and another for the winter season (October–March). Each ensemble has a set of 10 members, developed using slightly different atmospheric initial conditions. All simulations use observed weekly sea surface temperatures. Season-long daily climate data (specifically, rainfall, maximum and minimum temperatures, and incoming solar radiation) are archived for analysis and used in a crop model. Whereas the horizontal resolution of the global model employed is T63 (~1.875°), that of the nested regional model is ~20 km, roughly resolving the county scale. The regional model domain is 23°N–37°N, 92°W–76°W, which covers the southeast United States where weather/climate has major effects on agricultural yields.

[5] The Decision Support System for Agrotechnology Transfer (DSSAT) version 4.0 [Jones et al., 2003] is used to perform maize yield simulations. Season-long daily climate data from in situ observations and the COAPS global and regional models are used as inputs for the DSSAT crop model for the period 1987–2005 at Tifton, Georgia (31.5°N, 83.5°W). Tifton is chosen as a representative location partly because it is located in the middle of the regional model domain and partly because it is one of a few stations where observed daily surface solar radiation data are available. The crop simulations are performed with site-specific soil profiles, fixed fertilizer applications, and rainfed conditions; the planting date for each year is 1 April. The crop model uses the same initial conditions (e.g., soil moisture) for each run. Therefore, the weather/climate input is the only parameter that can change crop yields in a given year.

3 Evaluation Methods

3.1 A Conventional Model Evaluation Method

[6] The skill of seasonal simulation of precipitation of the climate model using one of the most commonly used conventional model evaluation methods is shown in Figure 1. The global model skill scores in terms of temporal correlation for JJA and DJF average precipitation are shown in Figures 1a and 1b, respectively. It is well known that many global models have some degree of predictability over the southeast United States during winter because of the strong teleconnection to tropical Pacific Ocean sea surface temperatures [e.g., Cocke et al., 2007; Saha et al., 2006; Higgins et al., 2000]. But, this is not the case for the summer season. There is no detectable skill over the southeast United States in terms of the anomalies of JJA mean.

Figure 1.

Temporal correlation of the ensemble mean seasonal forecasts of precipitation for (a) JJA and (b) DJF using a global model and for (c) JJA and (d) DJF using a regional model for the period 1987–2005. Colored regions indicate statistical significance at approximately 5% level.

[7] If the same skill evaluation metric is employed for the regional model simulations, we obtain a similar result for the dynamically downscaled seasonal average rainfall fields (see Figures 1c and 1d), which provide more detailed spatial information of the signal at a county scale, but nothing more than that. Hence, we might be tempted to conclude that there is no value added by using a regional model, except for reducing some of the biases originating from the global model.

3.2 Alternative Evaluation Methods

[8] An alternative way to expose the value of a regional model might be through the use of application-oriented models, such as a crop model used in this study [Baigorria et al., 2007; Shin et al., 2010]. Although a crop model provides a single yield value per year, it uses season-long daily climate data, not seasonal average (or total) climate data. This means that the crop yield values implicitly include the high-frequency variability of seasonal climate information (e.g., dry/wet spell sequences).

[9] Figure 2 shows the simulated maize yields using observed, global, and regional model data and their corresponding rainfall totals for Tifton, Georgia (31.5°N, 83.5°W), for the period 1987–2005. Figure 2b is a kind of one-dimensional version of Figure 1. Since there is almost no temporal correlation for either the global (r = −0.032) or regional (r = −0.037) model rainfall totals with observed total, we could conclude that there is no interannual predictability over this location, especially during the summer season. However, we reach a different conclusion if we evaluate temporal correlations in terms of maize yield amounts (Figure 2a). Whereas the global-model-based maize yields are less correlated with the observed yields (r = 0.128), the regional-model-based maize yields are well correlated with the observed (r = 0.405, statistically significant at the 5% level). If the year 2005 is excluded, the correlation becomes 0.72. The year 2005 was unique in that tropical cyclone activity began very early that year and was a record year for Atlantic hurricane activity overall. There were at least three tropical storms which passed near Tifton, GA and produced significant amount of rain. Currently, while some climate models can predict overall seasonal Atlantic activity [e.g., LaRow et al., 2010; Vitart, 2006], they cannot accurately forecast landfalling tropical cyclone activity (let alone associated precipitation) months in advance. Thus, we cannot expect these models to predict the variance of crop yields that could be related to tropical cyclone rainfall.

Figure 2.

Simulated (a) maize yields for 1987–2005 and (b) their corresponding rainfall totals at Tifton, Georgia (31.5°N, 83.5°W). The observed weather-driven values are denoted by black dots, the global model-driven values by open circles and whiskers, and the regional model-driven values by closed circles and whiskers. Symbol E denotes El Niño; L, La Niña; and N, Neutral year.

[10] Climatological average yield amounts are also better produced by the regional model than by the global model data. The underestimation of yield amounts by the global model is most likely because of the greater frequency of rain days and the lesser amount of daily rainfall produced by the model. Further analyses are shown below to determine why the regional model performs better than the global model in the yield simulations.

[11] To reveal the hidden value of seasonal dynamical downscaling, we examine an interesting finding shown in the maize yields and rainfall totals for an El Niño year (1988) and a La Niña year (1989). Even though the observed rainfall totals are similar (Figure 2b, ~550 mm) for both years, the maize yields (Figure 2a) are substantially different between year 1988 (4548 kg/ha) and year 1989 (9644 kg/ha). The difference can be explained by examining the water stresses on the maize. Water stress in a crop model is commonly computed by combining many variables such as rainfall, evaporation, soil moisture, runoff, and crop physiology. The crop model provides the daily values of water stress during the crop-growing seasons. The time series of simulated maize water stresses for 1988 and 1989 are shown in Figure 3: the maize experienced long-lasting water stress in 1988, but a brief spell of water stress in 1989. This explains why the maize yields in 1989 are almost twice those in 1988. A similar argument can be made for the years 1998 and 1999 (not shown).

Figure 3.

Simulated water stresses for maize during 1988 (El Niño) and 1989 (La Niña) crop-growing seasons.

[12] Can we also justify the above conclusion using rainfall data only and not using a crop model? This is partly possible by evaluating the time series of accumulated rainfall. Figure 4a shows the observed accumulated rainfall amounts in Tifton, Georgia, for 1988 and 1989: the higher the slope, the greater the amount of rainfall. We can also infer water availability from the slopes of accumulated rainfall during the crop growing season. Season-long rainfall totals are similar for both years, but the rainfall histories from planting days to harvest days are entirely different.

Figure 4.

Accumulated rainfall amounts for 1988 (El Niño) and 1989 (La Niña) crop-growing seasons from (a) observation, (b) a global model, and (c) a regional model.

[13] The corresponding time series of accumulated rainfall from the global and the regional models are shown in Figures 4b and 4c, respectively. A simple comparison of these two figures clearly reveals the added value of the regional model relative to the global model. It is evident that the global model rainfall amounts are too small to provide enough water to the crop for the first 100 days after planting in 1989 and for the whole season in 1988. This explains why the global-model-based maize yields are much lower than the observed-based yields (Figure 2a). Meanwhile, the slopes (or rainfall histories) of 1988 and 1989 are relatively well reproduced from the regional model, but the seasonal total rainfall amounts are relatively not. These similar slope patterns explain the better maize yield simulations by the regional model in Figure 2a.

[14] There are potentially more useful and relevant metrics that one can use for predicted rainfall data to evaluate model performance, especially for agricultural applications. The Lawn-and-Garden Moisture Index (LGMI), one of several drought indices, can be used to express the water availability (soil moisture) during the crop-growing season. The LGMI was developed by the Alabama Office of State Climatology to estimate the capacity of current soil moisture to sustain healthy lawns and gardens. The LGMI is a function of rainfall only. The current index is computed on the basis of how much rainfall occurred in the past 21 days in a specific location. All rainfall during the previous seven-day period is considered to be equally important, but rainfall before that time is discounted according to a linear sliding scale. A detailed computational procedure can be found in Christy [2004]. It should be noted that if the LGMI is less than 50 mm, crops begin to experience water stress (dry conditions), but if it is more than 50 mm, crops have sufficient soil moisture to grow. Figure 5 shows the LGMI time series for the 1988 and 1989 crop-growing seasons from (a) observations, (b) the global model, and (c) the regional model, respectively. The problem of the global model rainfall is clearly presented in terms of LGMI (Figure 5b): conditions are too dry for crops to grow. This problem is substantially reduced by the regional model. Although the detailed time evolution of LGMI is quite different from the observation, the regional model captures the overall observed signal reasonably well. Therefore, we can conclude that this derived statistic is a useful tool that can reveal the hidden value of the regional model for downscaling purposes.

Figure 5.

Same as Figure 4 but for the Lawn and Garden Moisture Indies (LGMI).

4 Discussion

[15] As noted above, the total seasonal precipitation was a poor predictor of yield. To examine this issue further, we used a brute force search method and found that the accumulated precipitation for days 50 through 85 after planting (i.e., the reproductive period for maize) had the highest correlation against yield, around 0.80. Thus, if a model can accurately predict the precipitation during this time period, there is a chance that the model may predict the correct yield. In Figure 6, we show a scatterplot of the model correlation of predicted Day 50–85 precipitation (against observed) versus the correlation of model predicted yield (against observed) for 10 ensemble members each of the global and regional model simulations. These results show that the skill of an ensemble member's prediction of yield is commensurate with the member's prediction of precipitation for the regional model, at least as measured by correlation. For the global model ensembles, the relationship is less distinct. These results also show that the regional ensembles more frequently have higher skill than the global model, with four members having yield correlations higher than 0.45. However, while the set of ensemble members, as well as the ensemble mean of the regional model, have higher skill, the correlation of the global and regional ensemble mean for Day 50–85 precipitation is very similar, 0.64 versus 0.62, when 2005 is excluded. Thus, we have what appears to be a paradox, in that while the precipitation totals for Days 50–85 appear to be well-predicted by both the global and regional models, the regional model has good skill at predicting yield (2005 notwithstanding), whereas the global model did not. Further analysis indicates this paradox can be explained by exaggerated variability of precipitation of the global model. Table 1 shows the standard deviation across ensemble members (STDEV1) and across years (STDEV2) and counts of very extreme events of <150 mm and >500 mm accumulated precipitation in the first 100 days after planting. It can be seen that the variability of the regional model is much closer to observed, and there are fewer extreme events. By “extreme event,” we mean events that are outside the range that has been observed over the time period 1987–2004. While some extreme events outside the observed range are to be expected in a large number of ensembles, the number produced by the global model appears to be too high, leading to an excessive number of incorrect crop yields.

Figure 6.

Scatter diagram of correlations of precipitation versus correlations of yields for 10 ensemble members. The correlations are computed by model precipitation against observed precipitation (days 50–85) and by model yields versus observed yields.

Table 1. Variance of Precipitation — First 100 Days After Planting
 STDEV1STDEV2<150>500Total Cases
  1. STDEV1: standard deviation across years, averaged over ensemble members.

  2. STDEV2: standard deviation across ensemble members, averaged over 19 years.

  3. <150: occurrences of less than 150 mm accumulation during the 100 days.

  4. >500: occurrences of greater than 500 mm accumulation during the 100 days.

  5. a

    =0 if 2005 is excluded for >500; STDEV1 is 102.52 if 2005 is included.

Observation86.44 01a19
Regional model82.3171.2617190
Global model114.44106.372112190

5 Conclusion

[16] This study re-evaluated the performance of a season-long regional downscaling model compared to that of a global climate model using nonconventional skill metrics. The conventional model evaluation methods, such as temporal correlation of seasonal average rainfall, often cannot reveal the value of dynamically downscaled data. The added value of using the regional model for downscaling was better uncovered by using a crop model as a forecast evaluation tool because the crop yield data include the high-frequency variability of seasonal climate (e.g., dry/wet spell sequences). Compared to the global model, the regional model better simulated intraseasonal variability than the anomalies of the seasonal mean. In addition, the value of the regional model was also exposed by high-frequency statistics, e.g., the time series of accumulated rainfall and Lawn-and-Garden Moisture Index, which are derived from rainfall data only.

[17] This study clearly shows why a dynamical downscaling system could be useful for application models. The resolution of the state-of-the art global models (e.g., those used in the Coupled Model Intercomparison Project 5 [Taylor et al., 2012]) still might be too coarse to capture the signals of regional climate variability. Hence, dynamical downscaling might remain an essential step to use global climate data in application models. It would be interesting to compare statistical downscaling methods with dynamical downscaling methods using the framework presented here.

[18] It is worthwhile to mention that evaluation of downscaling systems in the context of applications is an important concept, not only for assessing the value added of downscaling, but also because the applications reflect the impact of climate variability and change on society. In essence, it is not the crop model application that matters, but instead identification of the climatological factors that influence crop development and assessment of the model (or downscaled model) in terms of those specific factors. In this sense, these results demonstrate that existing metrics do not tell the whole story in terms of value added or impact assessment.


[19] This research was supported via a USDA/NIFA-funded collaborative EaSM project (FLAW-2011-00828).