Evaluation of the DEMETER performance for seasonal hindcasts of the Indian summer monsoon rainfall

Authors

  • Shujie Ma,

    Corresponding author
    1. Catalan Institute of Climate Sciences (IC3), Barcelona, Spain
    • Catalan Institute of Climate Sciences, Carrer del Doctor Trueta 203, 08005 Barcelona, Spain.
    Search for more papers by this author
  • Xavier Rodó,

    1. Catalan Institute of Climate Sciences (IC3), Barcelona, Spain
    2. Catalan Institution for Research and Advanced Studies (ICREA), Barcelona, Spain
    Search for more papers by this author
  • Francisco J. Doblas-Reyes

    1. Catalan Institute of Climate Sciences (IC3), Barcelona, Spain
    2. Catalan Institution for Research and Advanced Studies (ICREA), Barcelona, Spain
    Search for more papers by this author

Abstract

On the basis of the observed all-India rainfall (AIR), Climate Prediction Center (CPC) merged analysis of precipitation (CMAP), and the multimodel rainfall simulated in the Development of a European multimodel ensemble system for seasonal to interannual prediction (DEMETER) project, the model performance in terms of forecast quality of the seasonal mean Indian summer monsoon rainfall (ISMR) was evaluated. The approach used included the individual model (single-model ensemble) comparison as well as its comparison with the multimodel ensemble (MME). The results obtained reveal that systematic biases are the main source of the low skill in predicting the ISMR, while improved reproduction of interannual variability may increase the overall forecast quality for the ISMR. The MME shows superior performance in reproducing AIR features than the single-model ensembles. This improved forecast quality achieved by the MME with respect to the ISMR variations primarily depends on model performance and secondly on ensemble size. The MME has improved capability of simulating the ISMR variations during 1972–1990 decades, whereas it exhibits a lower performance in the 1990s. The reason of this decadal varying forecast quality clearly deserves more investigation. In summary, these results suggest that the MME's increase in skill absolutely depends on enhancing the individual model quality. Copyright © 2011 Royal Meteorological Society

1. Introduction

It is well known that the Indian summer monsoon rainfall (ISMR) from June to September critically affects the Indian agriculture and economy. For example, the impact of the deficit of 19% in the ISMR in 2002 was estimated to be a loss of billions of dollars in India (Gadgil et al., 2004), while the flooding events over India and Bangladesh in 1998 and 2004 summers led to serious damage as well. There is a great need for accurate long-range prediction of the ISMR. However, the current seasonal forecasts of the ISMR are not very accurate and are particularly difficult in the prediction of extreme drought and flooding events by either empirical or numerical models (Gadgil et al., 2005).

The limited dynamical predictability of the ISMR lies largely on the fact that most models still have major systematic biases in simulating the seasonal mean ISMR and large uncertainty in reproducing its interannual variability (Goswami and Xavier, 2005). The Indian summer monsoon and its precipitation are strongly tied to atmospheric dynamics and seem to be quite sensitive to initial conditions (Webster et al., 1998; Goswami and Xavier, 2005; Xavier et al., 2008). After a few days of integration, synoptic-scale systems will exhibit noticeable differences from observations, with planetary-scale anomalies diverging afterwards (Sivillo et al., 1997). Atmospheric instability with abundant moisture transport in the tropical monsoon region and active synoptic-scale convections produce significant ‘climate noise’ for the seasonal mean ISMR in numerical simulations. The systematic biases are inevitable and endemic (Palmer and Williams, 2008). It was estimated that about 50% of the interannual variability of the seasonal mean summer monsoon climate is governed by atmospheric internal processes (Goswami and Xavier, 2005). This means that if the models simulate the ISMR with large systematic biases in the first place, they are not well placed to exhibit easily and correctly the temporal and spatial variations of the ISMR. Thus, the model performance tends to influence largely the forecasts of the seasonal mean ISMR.

How to improve dynamical forecast quality of monsoon rainfall is always one of the important goals in global seasonal forecasting. It has been shown that the ensemble forecasting produced by a single model or by multi models could be a useful technique to achieve some of the improvement in seasonal forecasting of monsoon rainfall due to three major factors. First, an ensemble emphasizes similar features among members of the ensemble and minimizes their differences (Buizza, 1997). These differences may arise from both random error growth due to model deficiencies and the growth of errors in initial conditions (Reynolds et al., 1994). This ability of an ensemble is very important for ISMR predictions because of the sensitivity of monsoon rainfall to initial conditions and the occurrence of the ‘climate noise’. Second, although the forecast skill of ensemble mean is dependent on ensemble size (Kharin and Zwiers, 2002), the forecast skill of ensemble mean on average is higher than the skill of any member of the ensemble (Leith, 1974). This advantage of an ensemble has been clearly seen in the global and regional seasonal forecasts of precipitation by the multimodel ensemble (MME) obtained from the Development of a European MME system for seasonal to interannual prediction (DEMETER) project (Palmer et al., 2004). The MME even shows better results in seasonal forecasting in the tropics than a single-model ensemble with the same ensemble size due to a mixture of additional information from the different models (Hagedorn et al., 2005). Finally, techniques for constructing optimal MME forecasting have been developed (Krishnamurti et al., 2000; Doblas-Reyes et al., 2005). These approaches of the MME have shown some prospect in improving the seasonal forecast quality of precipitation beyond the individual models (Yun et al., 2005).

The assessment of prediction skill in monsoon rainfall from the probabilistic viewpoint has been largely carried out since a landmark paper of Palmer et al. (2000). In this paper, they put great emphasis on the chaotic nature of forecasts and the resulting spread of the ensembles and found almost no skill in the prediction of seasonal rainfall anomalies over the monsoon region as well. In fact, dynamical predictability of monsoon rainfall tends to be influenced greatly by atmospheric intraseasonal motions and external boundary forcings (Webster et al., 1998). The limited predictability is caused partly due to the poor reproduction of observed large-scale atmospheric teleconnection patterns linked to atmospheric intraseasonal variations (Sperber et al., 2001). The large-scale impact ranging from intraseasonal to interdecadal time scales on monsoon rainfall should not be neglected for an accurate seasonal forecasting of the ISMR (Goswami, 2004; Goswami and Xavier, 2005). Therefore, the assessment of ISMR forecast quality has been performed in this study from the deterministic viewpoint.

Systematic biases, linear association of interannual variability between forecasts and observations, the accuracy of the forecasts, and the forecast skill are some basic aspects that may contribute to the quality of a seasonal forecast from the deterministic viewpoint (Murphy, 1993). Therefore, the purpose of this paper was to assess the performance of individual models and the MME in terms of forecast quality of the seasonal mean ISMR on the basis of the attributes as mentioned above and the datasets derived from the DEMETER project. By doing this, the relative importance of model performance and ensemble size for the MME is also highlighted.

The paper is arranged as follows. A brief introduction of the datasets is given in Section 2. The contribution of individual model performance in terms of systematic biases and interannual variability to the forecast skill of the seasonal mean ISMR is described in Section 3. The quality in predicting the interannual variability of the ISMR in the MME is addressed in Section 4. The forecast quality of the spatial variations of the ISMR is evaluated in Section 5. The last section summarizes and discusses the major points.

2. Data

Seven state-of-the-art coupled models in the DEMETER prediction system are described in Table I (Palmer et al., 2004). All models included in this study are named according to the modelling groups. They are combined with four atmospheric general circulation models (AGCMs) and three oceanic general circulation models (OGCMs) in different versions. Thus, European Centre for Medium-Range Weather Forecast, international organization (ECMWF) and Laboratoire d'Océanographie Dynamique et de Climatologie, France (LODYC) share the same AGCM and so do Centre National de Recherches Meteorologiques, Météo-France, France (CNRM) and European Centre for Research and Advanced Training in Scientific Computation, France (CERFACS). The horizontal and vertical resolutions are different among these AGCMs, and the convection schemes closely related to the monsoon precipitation are not exactly the same. Therefore, the evaluated performance for the ISMR is the integrated effect of the model hindcasts.

Table I. Description of the features of the coupled models and initial conditions in the DEMETER prediction system, together with the convection scheme and the integration period used
 CERFACSECMWFINGVLODYCCNRMUKMOMPI
  1. All state-of-art models included in this study are named according to the modelling groups, namely, CERFACS, ECMWF, INGV, LODYC, CNRM, UKMO, and MPI.

AGCMARPEGEIFS/Cycle 23r4ECHAM4IFS/Cycle 23r4ARPEGEHadAM3ECHAM5
ResolutionT63/L31T95/L40T42/L19T95/L40T63/L312.5° × 3.75°/L19T42/L19
Convection schemeBougeault (1985)Tiedtke (1989)Tiedtke (1989)/Nordeng (1994)Tiedtke (1989)Bougeault (1985)Gregory and Rowntree (1990)Tiedtke (1989)/Nordeng (1994)
Atmosphere initial conditionsERA-40ERA-40Coupled AMIP-type experimentERA-40ERA-40ERA-40Coupled run relaxed to observed sea surface temperatures (SSTs)
OGCMOPA8.2HOPE-EOPA8.1OPA8.2OPA8.0GloSeaMPI-OM1 (new HOPE)
Resolution2.0° × 2.0°/L311.4° × (0.3°–1.4°)/L292.0° × (0.5°–1.5°)/L312.0° × 2.0°/L31182 grid points (GP) × 152 GP/L311.25° × (0.3°–1.25°)/L402.5° × (0.5°–2.5°)/L23
Ocean initial conditionsOcean analyses forced by ERA-40Ocean analyses forced by ERA-40Ocean analyses forced by ERA-40Ocean analyses forced by ERA-40Ocean analyses forced by ERA-40Ocean analyses forced by ERA-40Coupled run relaxed to observed SSTs
Integration period1980–20011958–20011973–20011974–20011958–20011959–20011969–2001

To carry out the basic assessment of summertime ISMR forecasting in an MME approach, the total precipitation from the start of May DEMETER hindcasts integrated for 6 months is used in this study. Each retrospective forecast comprises an ensemble of nine members with slightly different initial conditions. These initial conditions were created using ECMWF reanalysis dataset (ERA-40; Uppala et al., 2005) in most models, except for Max-Planck Institut für Meteorogogie, Germany (MPI), which produced both atmospheric and oceanic initial conditions by using a coupling initialization method, as well as Istituto Nazionale de Geofisica e Vulcanologia, Italy (INGV) created the atmospheric initial conditions using an Atmospheric Model Intercomparison Project experiment. Single-model ensemble means, representing individual model performance, are calculated by averaging among all nine members with equal weights. The MME is calculated in a similar fashion, but the average is calculated across the seven individual models as well. All models have been run until 2001 but with different starting times. The longest period begins in 1958, while the shortest starts in 1980.

The all-India rainfall (AIR) has been widely used to assess the variability of the ISMR from interannual to interdecadal time scales (ftp://www.tropmet.res.in/pub/data/rain/iitm-regionrf.txt; Parthasarathy et al., 1992). It consists of an area-weighted average of 306 rain gauges distributed across continental India, where about 90% of the total area of the country is considered except for the hilly regions, accumulating from June to September (JJAS) each year for the period of 1871–2001. The monthly mean Climate Prediction Center (CPC) merged analysis of precipitation (CMAP; ftp://ftp.ncep.noaa.gov/pub/precip/cmap; Xie and Arkin, 1997) for the period of 1979–2001 is also used to study the temporal and spatial variations in monsoon precipitation. The CMAP is constructed by merging rain gauge observations, microwave and infrared satellite images, and forecasted precipitation from National Center for Environmental prediction/National Center for Atmospheric Research (NCEP/NCAR) reanalysis.

3. Systematic biases, interannual variability, and their relations with the forecast skill of the ISMR

In this section, a quantitative assessment of the model performance in terms of forecast quality of the seasonal mean ISMR is performed by using the skill score (SS) for the accuracy of the forecasts described in the study by Murphy (1988). The accuracy of the forecasts is represented by the mean square error (MSE) between forecasts and observations. This means that systematic biases through the routine comparison between simulated and observed climatology may be included in this assessment. So the evaluation here is performed over the common integration period of 1980–2001 for the sake of unity.

The SS is used to indicate the improvement in the accuracy of the forecasts of interest over the reference forecasts:

equation image(1)

where f and x denote the forecasts of interest and the relevant observations, c represents the reference forecasts. Observed long-term mean climatology calculated in cross-validation is used as the reference forecasts for the target seasonal hindcasts in this study (Saha et al., 2006). The cross-validation is used to ensure that the particular year is not included in the calculation of climatology. The negative (positive) SS indicates that the accuracy of the forecasts is less (greater) than the accuracy of the climatological forecasts. A positive SS is usually considered to represent a minimal level of acceptable performance for the forecasts. So the more negative the climatological SS is, the worse the forecast skill is.

Under the condition of the equivalence between the long-term means of reference forecasts and observations (see Section 3 in the work of Murphy (1988)), the SS is decomposed by adding and subtracting both the long-term means of forecasts and observations in the calculation of the MSE in Equation (1):

equation image(2)

where r denotes the correlation coefficient between forecasts (f or c) and observations (x), s and the overbar represent the standard deviation and the long-term mean of forecasts or observations, respectively.

The decomposition of the SS contains four terms in the numerator of Equation (2). The first term, the square of linear correlation (r2fx), is a measure of potential forecast skill that generally indicates an upper limit on the SS (Murphy and Epstein, 1989). The second term represents a measure of linear association of interannual variability between the forecasts of interest and the relevant observations ([rfx − (sf/sx)]2, including the term of amplitude errors through the ratio of the forecast to observed standard deviations). The third term is a clear measure of systematic biases in the forecasts ([()/sx]2). The last term relates to the degree of association between the reference forecasts and the observations. The contribution of this term to the actual skill of the SS is constant and quite small compared with the terms associated with systematic biases and amplitude errors, which can be ignored in this study.

Therefore, the skill of the forecasts tends to increase as the correlations between forecasts and observations increase and also as systematic biases decrease. The forecast skill can be improved to some degree when the interannual variability of the forecasts shows positive correlations with observations greater than the amplitude errors. This means that good model performance in reproducing the ISMR interannual variability makes it possible to improve the forecast skill. The assessment of the SS may provide an insight into basic characteristics of forecasting performance, especially for the forecasts with limited skill.

The forecast skill of the seasonal mean ISMR is generally poor over the investigated regions in all models, with the exception of positive forecast skill over small areas to the south of the equator in the Met Office, United Kingdom (UKMO), INGV, CERFACS and CNRM due to positive SS (Figure 1). The area to the south of the equator between 70°E and 100°E corresponds to the climatological intertropical convergence zone (ITCZ) in JJAS (Figure 2(a)). For the two areas of maximum of the observed climatological ISMR, the forecast skill is lower over the Bay of Bengal (BoB) than that over the west coastal areas of the Western Ghats. The worst forecast skill takes place over the large topographical regions from the southern slope of the Himalayas to the Tibetan Plateau. In the former region of the BoB, synoptic-scale systems easily form and atmospheric intraseasonal oscillations are typical (Hoyos and Webster, 2007). In the latter region of the Himalayas, orographically driven convection is evident. It has been noted that the atmospheric intraseasonal oscillations related to the ISMR variations are one of the severe deficiencies in numerical modelling (Sperber and Annamalai, 2008; Xavier et al., 2008).

Figure 1.

Skill scores obtained in the forecasts of JJAS-mean ISMR during 1980–2001 in (a) UKMO, (b) INGV, (c) MPI, (d) ECMWF, (e) LODYC, (f) CERFACS, and (g) CNRM model, respectively. The intervals for shading areas are − 50, − 30, − 15, − 5, − 1, and 0

Figure 2.

JJAS-averaged rainfall for the interval 1980–2001 in CMAP (a) and the corresponding systematic biases displayed by (b) UKMO, (c) INGV, (d) MPI, (e) ECMWF, (f) LODYC, (g) CERFACS, and (h) CNRM model, respectively. Units are given in millimeters per day and the colour interval is 2 mm d−1

It is noteworthy that the areas of lower negative SS in each single-model ensemble, especially for the evident individual differences of the SS among the climate models, correspond well to the regions with larger systematic biases in the ISMR (Figures 1 and 2(b)–(h)). The component of the forecast skill contributed by systematic biases shows a similar magnitude and distribution to the SS (figures not shown). Therefore, it is the model systematic biases that contribute mainly to the low forecast skill of the seasonal mean ISMR, especially over the BoB and the Tibetan Plateau.

The forecast skill of the ISMR related to the contribution of interannual variability is low except over the regions where the interannual variability of the ISMR exhibits positive correlations between forecasts and observations greater than the amplitude errors (Figure 3). The positive contribution to the forecast skill by statistically significant interannual variability of the ISMR occurs over southwest Burma in ECMWF and LODYC, over the ITCZ in CERFACS and CRNM, over the northern Arabian Sea in LODYC, and over the Indian subcontinent south of the Himalayas in MPI. This implies conversely that these particular models have the potential ability to predict the interannual variability of the ISMR over these specific regions, respectively.

Figure 3.

Difference between the correlations of the seasonal mean ISMR between forecasts and observations during 1980–2001 and the amplitude errors represented by the ratio of the forecast to observed standard deviations, that is, rfx − (sf/sx), in (a) UKMO, (b) INGV, (c)MPI, (d) ECMWF, (e) LODYC, (f) CERFACS, and (g) CNRM model, respectively. The areas with significant correlations at the 90% confidence level are shaded. Positive differences are shown by the thick solid line and negative ones by the thin dashed one

The above analysis reveals that both systematic biases and interannual variability influence the actual skill of the ISMR forecasts, while the former decreases the forecast skill more largely than the latter. However, the forecast skill could be increased when the individual model has good quality in predicting the ISMR interannual variability with significantly positive correlations and low amplitude errors between forecasts and observations. Therefore, in the following section, we focus on the detailed evaluation of the quality of individual models and the MME in predicting the interannual variability of the ISMR.

4. Seasonal forecast quality of AIR interannual variability in the MME

Because the AIR obtained by spatial averaging of the ISMR can increase the correlation relative to the ISMR relation at the grid points (Saha et al., 2006), the forecast quality of AIR interannual variability has been assessed in this section, on the basis of the basic measurement of verification metrics (Yun et al, 2005): anomaly correlation coefficient (ACC) and root mean square error (RMSE). To find out the association of the quality in reproducing AIR interannual variability with the background AIR variability on longer time scales, the evaluation has been performed over the whole integration period in both single-model ensembles and the MME.

4.1. The forecast quality of AIR interannual variability

The model AIR (MAIR) in each single-model ensemble is constructed through accumulating JJAS total rainfall averaged over the land points of the region (70°–90°E, 5°–30°N) with elevations less than 1000 m. The MAIR index is defined as the standardized rainfall anomalies by removing the climatology calculated in cross-validation and so is the observed AIR index. The MME of MAIR indices before 1980 are obtained by computing the arithmetic mean of individual MAIR indices with longer hindcast periods.

The forecast quality of AIR interannual variability in each single-model ensemble and the MME at the different integration periods is shown in Table II. The interannual variability of the AIR is well reproduced by MPI for the period of 1969–2001, with a significant ACC (0.45) at a 99% confidence level (p < 0.01) and a low RMSE (1.03), and by CNRM for the period of 1958–2001 [ACC = 0.35 (p < 0.05); RMSE = 1.13]. It is interesting to see that the MME exceeds CNRM in the forecast quality of AIR interannual variability with a significant correlation at a higher confidence level and a lower RMSE [ACC = 0.43 (p < 0.005); RMSE = 0.94]. The MME shows slightly better forecast quality than MPI over the same period of 1969–2001 with a comparable correlation and a smaller RMSE [ACC = 0.46 (p < 0.01); RMSE = 0.9]. In other words, the MME is superior in terms of forecast quality of AIR interannual variability to each single-model ensemble.

Table II. ACCs and RMSEs between observed AIR index and simulated MAIR indices for each single-model ensemble and the MME at the different integration periods
ACC (RMSE)1958–20011959–20011969–20011973–20011974–20011980–2001
  1. The significant ACCs at the 95% (99% and 99.5%) confidence level(s) are represented in italics (bold italics and vertical bold).

Model(s)ECMWF:CNRM:UKMO:MPI:INGV:LODYC:CERFACS:
 0.250.350.260.450.340.340.16
 (1.21)(1.13)(1.20)(1.03)(1.13)(1.13)(1.27)
MME0.430.430.460.400.370.31
 (0.94)(0.95)(0.90)(0.91)(0.93)(0.93)

The comparatively high forecast quality of AIR interannual variability in MPI is in agreement with the above result as for the positive contribution of interannual variability to the forecast skill of the ISMR over continental India by MPI (Figure 3(c)). The high ability of MPI to reproduce the interannual variability of the ISMR may be associated with the fact that the ECHAM family of models, atmospheric component of ECHAM5 in MPI, shows the best agreement with observations in terms of the magnitude and propagation of the intraseasonal ISMR anomalies among the climate models in the Coupled Model Intercomparison Project-3 (Sperber and Annamalai, 2008). Therefore, the potential impact of atmospheric intraseasonal oscillations on the interannual variability of the ISMR deserves further exploration.

4.2. The background quasi-decadal variability of the AIR

It has been found that 6–7-year lower frequency oscillations dominate the AIR variability during the period 1948–1998 (Figure 1 in Li and Zhang, 2002). The observed variance contribution of 7-year quasi-decadal AIR variability to its interannual variability in the period 1958–2001 is small (6%), but the modulation effect of lower frequency variability on the predictability of the seasonal mean monsoon cannot be neglected (Goswami, 2006). This is because the variability of the ISMR on longer time scales may unfold the background interaction between the atmospheric mean states and slowly varying external forcings. It is well known that the relationship between the El Niño-Southern Oscillation (ENSO) and the ISMR has been weakening during the past 2 decades of 20th century (Krishna Kumar et al., 1999). The change in the ENSO–monsoon relationship on interannual time scale can be modulated by the large-scale circulation changes (the monsoon Hadley and Walker circulations) associated with the monsoon variability on longer time scales (Goswami, 2006). Hence, it is important to investigate the ability of the models and the MME to reproduce the AIR variability on the 7-year time scale.

The quasi-decadal variability of observed and simulated AIR is obtained by the 7-year running mean of the AIR index and the MAIR indices in the individual models and the MME shown in Figure 4(a). The quasi-decadal variability of the observed AIR for the period of 1958–2001 is seen in Figure 4(b). The observed above-average rainfall occurs in the periods of 1961–1964, 1972–1978, and 1991–1997, and below average precipitation happens in the periods of 1965–1971 and 1981–1990. The MME simulates quite well the AIR variations during 1972–1990 decades and the transition from higher than normal to lower than normal AIR variations in the late 1970s, which is seen through the comparatively large positive sliding correlations in a 7-year window (Figure 4(c)). However, the MME tends to exhibit longer quasi-decadal variability of the AIR than the observation (Figure 4(b)). The poor simulations of AIR quasi-decadal variations occur over the periods of 1965–1971 and 1991–1997 for each individual model and the MME, being worse in the latter period than in the former one.

Figure 4.

Observed AIR index (red line with circle) and simulated MAIR indices in single-model ensembles (colour lines) and the MME (thick black line) (a), running mean of observed AIR and simulated MAIR indices in a 7-year window (b), and sliding correlations between the MAIR indices and the AIR index in the 7-year window (c) for the whole investigated periods. Horizontal line with percentage in (c) denotes significant correlations at the 90% confidence level. The colour fonts at the bottom of (c) denote models used in (b) and (c) This figure is available in colour online at wileyonlinelibrary.com/journal/joc

Because it is a common problem for all models and the MME to present poor forecast quality of AIR quasi-decadal variations in the 1990s, the better performance of MPI in reproducing the ISMR interannual variability over continental India in the period of 1980–2001 (Figure 3(c)) may be contributed in part by its good quality in simulating AIR quasi-decadal variability during 1983–1988 (Figure 4(b) and (c)). This indicates a close association of the quality in reproducing the interannual variability of the ISMR with the background variability of the ISMR on longer time scales. The poor quality of the ISMR predictions in the 1990s will be discussed in the last section.

Besides, the ability to reproduce the AIR variations in the 1980s is comparable between CNRM and CERFACS (Figure 4(b) and (c)), which share the same AGCM. This ability is different between ECMWF and LODYC despite sharing the same AGCM. As mentioned above, CNRM and CERFACS have the ability to reproduce the ITCZ interannual variability, and ECMWF and LODYC lack such ability (Figure 3(f) and (g)). It has been found that the good representation of the large-scale organization of the ITCZ play an important role in improving seasonal forecasting of the ISMR (Xavier et al., 2008). The activity of the ITCZ relates closely to the heating state of the equatorial Indian Ocean. Therefore, the mechanism responsible for decadal varying forecast quality of the ISMR also deserves further exploration on the basis of the interaction between the monsoon Hadley and Walker circulations and slowly varying external forcings, particularly from tropical Indian Ocean and Pacific.

4.3. The relative importance of model performance and ensemble size for the MME

Based on the analysis in Section 4.1, one wonders why the MME shows better quality in predicting AIR interannual variability than each single-model ensemble. Does it relate mainly to increased ensemble size in the MME? Or does it relate more to the performance of individual models than to increased ensemble size? To answer these questions, a detailed assessment of the forecast quality of AIR interannual variability has been made using random combinations of the single-model ensembles.

In the random combinations, the same number of individual models involved means that the ensemble size is equal among these multimodel combinations. Hence, the ensemble size increases with more individual models involved. Each multimodel-combined MAIR index is obtained using the same way to calculate the MME MAIR index in Section 4.1.

First, it is found that the multimodel-combined MAIR indices calculated by the models with shorter integration periods than MPI are not statistically significant due to their smaller degrees of freedom for a statistical significance test (Figure 5(a)). More importantly, it is found that the percentage of multimodel combinations with significant ACCs (p < 0.01) increases with increased ensemble size. All multimodel combinations with random six models can reproduce AIR interannual variability quite well with statistically significant ACCs (p < 0.01) and comparatively low RMSEs (p < 0.98). Therefore, increased ensemble size in the MME gives a better chance to improve the forecast quality of AIR interannual variability. However, the forecast quality in the MME created by all individual models is beaten by the combinations [significant ACCs (p < 0.002) and lower RMSEs (p < 0.91)] with three, four, five, and six models that contain CNRM, UKMO, and MPI together and the other models in shorter integration periods than MPI. Interestingly, in all the significant cases (p < 0.01), either MPI or CNRM or both are always included. These results indicate that the forecast quality in the MME depends largely on the ability of individual models to reproduce correctly the interannual variability of the ISMR. The performance of individual models plays a more important role in the MME forecasting than the ensemble size.

Figure 5.

ACCs between observed AIR index and simulated MAIR indices for each single-model ensemble and all possible multimodel combinations, for the whole integration periods (a) and for the period 1980–2001 (b). Dashed and dotted lines with percentage represent significant ACCs at the different significance levels for two initial integration years of 1969 and 1958, respectively. Asterisks denote single-model ensemble of MPI, as well as combinations of MPI with other models in shorter integration periods than MPI. Circles have the same meaning as asterisks, but for INGV and/or LODYC

Even for the common integration period of 1980–2001 in the models (Figure 5(b)), an overall low forecast quality of AIR interannual variability occurs in the MME and most combinations except for the single-model ensemble of MPI and the combinations of (CNRM, MPI), (MPI, INGV), and (CNRM, MPI, and INGV). The low forecast quality in this period is not improved by increasing the ensemble size. In fact, it may be associated with poor model performance for the reproduction of AIR quasi-decadal variability in the 1990s (Figure 4(b) and (c)). This implies again that the forecast quality of the ISMR interannual variability in the MME is primarily determined by model performance and secondly by ensemble size.

5. Seasonal forecast quality of spatial ISMR anomalies

Based on the verification metrics of ACC and RMSE calculated over the grid points for the construction of the MAIR, the forecast quality of the spatial ISMR anomalies for the period of 1980–2001 has been evaluated by using CMAP precipitation (Figure 6). In general, the forecasts of the spatial ISMR variations spread largely among the models due to the inconsistent signs of the ACCs in all years except 1988, 1991, 1997, and 1999 (Figure 6(a)). Even in these four years, the MME and most models successfully capture the spatial ISMR variations in 1988 and 1991 with significantly positive ACCs and low RMSEs, while failing in predicting rainfall anomalies in 1997 and 1999 (Figure 6(a) and (b)). The obvious positive ACCs in 1987, 1994, and 1998 correspond to higher RMSEs than average. The long-term mean ACCs in each individual model and the MME are very small and not statistically significant. Therefore, the forecast quality of the spatial ISMR variations is poor.

Figure 6.

Spatial ACCs and RMSEs of JJAS mean ISMR anomalies over land areas (70°E–90°E, 5°N–30°N) with elevations lower than 1000 m. Plots are for single-model ensembles (colour lines) and the MME (thick black line) (a, b) and in the three significant combinations obtained (c, d). Dashed lines with percentages denote significance levels for the total number of grid points considered in (a) and (c). Long-term mean RMSE in (b) is 0.89, while in (d) is 0.84 This figure is available in colour online at wileyonlinelibrary.com/journal/joc

It is interesting to see that the three combinations obtained as shown in Figure 5(b) tend to show a similar behaviour in the sign of the forecast anomalies (Figure 6(c)), and the long-term mean forecast errors decrease in magnitude as well (Figure 6(d)), if compared with that shown in Figure 6(b). High forecast quality of the spatial ISMR anomalies takes place in 1986, 1996, 1998, and 2001 besides 1988 and 1991 as mentioned above. However, the poor forecast quality in 1997 and 1999 does not change. The forecast quality in 1984 becomes worse. This indicates limited MME improvement with respect to the quality of spatial ISMR forecasting under the current model performance.

However, it is found that positive (negative) spatial ACCs in the MME (Figure 6(a)) tend to correspond to the occurrence of consistent (opposite) variations in sign between the observed and MME AIR indices in the period of 1980–2001 except in 1984 and 1993 (Figure 4(a)). The relation of land area ISMR variations between the MME and the observations is achieved almost similarly in the two observed datasets. This indicates that the MME MAIR index may provide useful information and quality to some degree on seasonal forecasts of the ISMR anomalies.

6. Summary and discussion

A detailed evaluation of the seven state-of-the-art model performances and the MME has been carried out in terms of their forecast quality of the seasonal mean ISMR from the deterministic viewpoint. We used comparisons among single-model ensembles and the MME. Systematic biases of the ISMR can be seen to contribute largely to the current low forecast skill. Instead, adding the interannual variability of the ISMR reproduced by the models can greatly increase the forecast skill provided individual models are capable of predicting the ISMR interannual variability. The MME surpasses each single-model ensemble in terms of forecast quality of AIR interannual variability. This improvement in forecast quality achieved by the MME primarily depends on model performance and secondly on ensemble size.

The ISMR forecasts would benefit from reducing systematic biases. This is in agreement with the result in the study by Berner et al. (2008), who pointed out that the probabilistic skill of seasonal forecasts can be significantly improved in terms of the reduction of systematic biases. At present, the complexity of forecasting the ISMR largely relies on the model ability to simulate the rainfall close to the areas with large topographic gradient, as well as to reproduce synoptic-scale convective rainfall and associated variability. The lack of such ability in current climate models can be changed as model resolution increases to enhance the relevance of scale interactions to model systematic biases (Slingo et al., 2003). Another framework for further improvement is the implementation of ‘stochastic physics’ into climate modelling (Berner et al., 2008).

In addition to reducing systematic biases, increasing the model ability to reproduce ISMR interannual variability stands as a key process for improving MME forecasting. The interannual variability of the ISMR links closely to the intraseasonal oscillations, one of atmospheric internal processes (Goswami and Xavier, 2005; Sperber and Annamalai, 2008), and slowly-varying external forcings, such as the ENSO (Goswami, 2006) and the Indian Ocean Dipole (IOD; Ashok et al., 2001) event. The better performance of the MPI model and the MME in predicting the interannual variability of the ISMR may be associated with the fact that they both display an ability to simulate ISMR variations during the 1970s and 1980s, as well as the transition from higher than normal to lower than normal ISMR variations in the late 1970s. Instead, the lowest forecast quality of the ISMR in the 1990s may be associated with a weakening trend in the relationship between the ISMR and the ENSO during the past 2 decades of 20th century (Krishna Kumar et al., 1999). Of interest also, the IOD events and their role in the modulation of ENSO impact on the ISMR ought to be explained (Ashok et al., 2004). The poor forecast quality in 1997 is a typical example that results from the out-of-phase response of the Indian summer monsoon to the El Niño events, when compared with the expected response (Slingo and Annamalai, 2000). The relative importance of the intraseasonal oscillations and the air–sea interaction for the ISMR at interannual time scale should not be neglected either.

The spatial poor predictability of the ISMR currently limits the practical value of the MME forecasting capacity. Conversely, it is encouraging that the reproduced AIR by the MME provides useful information and some quality for the seasonal forecasting of spatial ISMR anomalies over continental India. New research studies arisen from this study include a better understanding of the dynamics at intraseasonal time scales and the modulation exerted by the air–sea interaction on the variability of the ISMR.

Acknowledgements

The authors appreciate helpful comments from one anonymous reviewer that much improved prior manuscripts. We are grateful to the European Center for Medium-Range Weather Forecast (ECMWF) that freely provided the database produced by the DEMETER project. S. Ma was in receipt of a Torres-Quevedo grant by the MICINN (Spain), through the Climate Research Laboratory at the Barcelona Science Park. X. Rodó was in receipt of support from a NOAA/NSF (cholera) grant (process) and by the PANDORA project (PRJ.CGL63053) of the Spanish Ministry of Science.

Ancillary

Advertisement