Verification of daily precipitation amount forecasts in Armenia by ERA-Interim model

Authors

  • Artur Gevorgyan

    Corresponding author
    • Department of Development and Validation of Hydrometeorological Models, Armenian State Hydrometeorological and Monitoring Service, Yerevan, Armenia
    Search for more papers by this author

Correspondence to: A. Gevorgyan, Department of Development and Validation of Hydrometeorological Models, Armenian State Hydrometeorological and Monitoring Service, Baghyan str. 1, 3750056 Yerevan, Armenia. E-mail: agm86@yandex.ru

ABSTRACT

Verification of 12-h daily precipitation amount forecasts is considered in Armenia making use of ERA-Interim model output data. The performance of ERA-Interim model is evaluated against observations of 30 meteorological stations of Armenia over the period 1996–2010. Daily precipitation amounts possess high spatio-temporal variability in Armenia associated with complex orography. The verification results have shown that the mentioned variability of precipitation amounts is not reproduced appropriately by forecasted data of ERA-Interim model.

1. Introduction

The precipitation regime in Armenia possesses a highly irregular behaviour in both the spatial and temporal dimensions (Gevorgyan, 2010). The spatial distribution of precipitation as well as its seasonal variability may be explained in terms of the broad characteristics of the global circulation and regional climate factors (e.g. latitude, orography, oceanic and continental influences).

Great temporal and spatial variability of distribution of precipitation amounts make it difficult to forecast precipitation amounts in Armenia. However it should be noted that there have been considerable advances in numerical weather prediction (NWP) models in recent decades associated with improvements of model dynamics and physics and availability of observed meteorological information all over the world. There are several studies considering the ability of the European Centre for Medium-Range Weather Forecasts Tropical Ocean and Global Atmosphere (ECMWF-TOGA) analyses, RegCM2 and MM5 regional models to simulate climate and precipitation in the Middle East and Caspian Sea region including Armenia (Evans et al., 2004; Oskouian et al., 2008; Zaitchik et al., 2007). It was shown that accurate simulation of precipitation in the Middle East region requires the correct simulation of storm tracks, topographic interactions and atmospheric stability. The higher resolution of RegCM2, compared with the ECMWF analyses, allows it to capture the spatial variability of temperature and precipitation better despite model biases being present. RegCM2 is better able to simulate the interannual variability averaged over the entire domain compared with the ECMWF analyses; however, they both have difficulty in reproducing the interannual variability in particular sub-regions (Evans et al., 2004).

The ERA-Interim project was initiated in 2006 to provide a bridge between ECMWF's previous reanalysis, ERA-40 (1957–2002), and the next-generation extended reanalysis envisaged at ECMWF Berrisford et al. (2009). Dee et al. (2011) have considered the configuration and performance of the data assimilation system of the ERA-Interim model used in this paper. It was stated that further progress is possible in several key areas of precipitation modelling. For example, unphysical changes in global mean precipitation, while still present in ERA-Interim, are now well-understood and will likely be reduced in future reanalyses. A great deal has been learned about assimilation of cloud- and rain-affected satellite radiances for NWP since ERA-Interim began. Representation of moist physical processes in forecast models has improved, and the ability to usefully assimilate in situ observations of accumulated rainfall is now well within reach. The results of comparison of monthly averaged precipitation rates and anomalies from ERA-Interim with Global Preciptiation Climatology Centre (GPCC), at the 1° × 1° resolution of the GPCC gridded data product, at four different grid points located on the British Isles have shown that month-to-month variability of ERA-Interim rainfall compares quite well with GPCC estimates, as do the anomalies. ERA-Interim produces lower values for rainfall maxima in mountainous regions Dee et al. (2011).

Precipitation amount forecasts are of great importance for hydrological forecasts, agriculture and transportation. The main method of precipitation forecasts is still synoptic method in Armenian State Hydrometeorological and Monitoring Service. These forecasts are probabilistic precipitation forecasts. This study aims to examine the ability of ERA-Interim model to forecast daily precipitation amount in Armenia and to evaluate the possibility of using this model for Armenia.

In Section 'Data and method' the description of data sets used to evaluate the ERA-Interim model is presented, and the statistics calculated to perform the evaluation are described. The model's performance is compared with the observational datasets in Section 'Results'. The conclusions and discussions are presented in Section 'Discussion and conclusions'.

2. Data and method

Forecasted values of daily precipitation amount of ERA-Interim model with lag of 12 h were used in this paper (http://data-portal.ecmwf.int/data/d/interim_daily/; Berrisford et al., 2009). The ERA-Interim atmospheric model and reanalysis system uses cycle 31r2 of ECMWF's Integrated Forecast System (IFS), which was introduced operationally in September 2006. Forecast data on pressure levels and for the surface and single level parameters are archived at the 28 ranges, or steps from daily forecasts at 00:00 and 12:00 UTC. On the ECMWF Data Server forecasts are only available for surface and single level fields and only up to a range of 12 h.

Predicted values of daily precipitation amounts were obtained by summing forecasted total precipitation amounts at 00:00 and 12:00 UTC in each grid point (on a 1.5° × 1.5° grid) located in Caucasus Region and Armenia (Figure 1). It can be seen from Figure 1 that only three grid points are located in the territory of Armenia and the other six grid points are located outside of Armenia.

Figure 1.

ERA-Interim grid with the nine points used for verification of daily precipitation amount forecasts in Armenia.

The physical map of Armenia is presented in Figure 2. Predicted daily precipitation amounts in the selected grid points were re-interpolated into 30 meteorological stations presented in Figure 3 making use of bilinear interpolation method (http://ru.wikipedia.org/wiki/bilinear_interpolation). As can be seen from Figures 2 and 3 the selected 30 meteorological stations provide good spatial coverage in Armenia including both mountain and valley stations.

Figure 2.

The physical map of Armenia.

Figure 3.

The distribution of meteorological stations in Armenia.

Verification of daily precipitation amounts was carried out for the period of 1996 – 2010. Only days with precipitation events (wet days) have been included in the verification. According to (Manual of Short-Range Weather Forecasts, 2002) daily precipitation with amount of greater than 0.4 and 0.2 mm is considered as a wet day in the warm (April to October) and cold (November to March) periods correspondingly. The definition of wet days is based on the observation data. Precipitation forecasts are most challenging during the precipitation-producing processes. Thus, wet days have been selected in the verification to better understand the ERA-Interim model's ability to accurately simulate precipitation in Armenia.

There are substantial differences in precipitation-producing processes in the warm and cold periods. In contrast to the cold period much of the precipitation is convective in nature in the warm period associated with air-mass processes and frontal activity in Southern Caucasus and Armenia. For these reasons, during the warm period precipitation processes are expected to be sensitive to a land–atmosphere forcing. Therefore, the verification results are examined for the cold and warm periods individually.

Total number of wet days considered in the 1996–2010 period is varying from 452 to 1294 in the selected 30 meteorological stations of Armenia (on average 900 d) in the warm period. In the cold period, total number of wet days is ranging from 313 to 845 (on average 500 d). Large number of wet days considered suggests that the verification results are expected to be statistically significant for all stations both in the warm and cold periods.

There are different statistical measures to test the performance of models quantitatively (Evans et al., 2004; Wilks, 2006). In this paper, daily precipitation amount forecasts of the ERA-Interim model are evaluated against observations using the statistics presented below.

It is interesting to estimate the spatial variability of daily precipitation amounts in Armenia according to observed and forecasted data. For this purpose the coefficient of variation (CV) of mean daily precipitation amounts in Armenia was calculated making use of Equation (1).

display math(1)

where CV is the coefficient of variation of daily precipitation amounts in Armenia, math formula the mean daily precipitation amount in Armenia, σ the standard deviation of daily precipitation amounts in Armenia, p(i) the daily precipitation amount in ith station and n the total number of used meteorological stations equal to 30.

In order to evaluate the spatial agreement between the model and observations quantitatively, the pattern correlation between observed and simulated fields of daily precipitation amounts was calculated by Equation (2) Evans et al. (2004).

display math(2)

where ρ is the pattern correlation of daily precipitation amounts in Armenia, Ō the mean observed daily precipitation amount in Armenia derived from 30 stations data, O(i) the observed daily precipitation amount in ith station, math formula the mean modelled daily precipitation amount in Armenia calculated from interpolated values of forecasted daily precipitation amounts in 30 stations, M(i) the interpolated value of modelled daily precipitation amount in ith station and n the total number of used meteorological stations equal to 30.

The relative error of daily precipitation amount forecasts (RE) is the next estimate considered in the verification. RE expressed in percentage was calculated for each of the selected 30 meteorological stations of Armenia making use of Equation (3).

display math(3)

where RE is the relative error of daily precipitation amount forecasts expressed in percentage, pf,i the forecasted daily precipitation amount in a station in ith day of the verification period, po,i the observed daily precipitation amount in a station in ith day of the verification period and n the total number of days included in the verification period.

Finally it is interesting to test the ability of ERA-Interim model to capture the temporal variability of observed daily precipitation amounts in Armenia. For that purpose the correlation coefficient (R) between values of the forecasted and observed daily precipitation amounts was calculated for each of the 30 selected stations making use of Equation (4).

display math(4)

where R is the correlation coefficient between values of the forecasted and observed daily precipitation amounts, pf(i) the forecasted daily precipitation amount in a station in ith day of the verification period, po(i) the observed daily precipitation amount in a station in ith day of the verification period, math formula the forecasted mean daily precipitation amount, math formula the observed mean daily precipitation amount and n the total number of days included in the verification period.

3. Results

The distribution of observed and forecasted mean daily precipitation amounts in Armenia in the warm period from 1996 to 2010 is presented in Figure 4(a) and (b) respectively. It can be seen from Figure 4(a) that there is high spatial variability in the distribution of observed precipitation amounts associated with influence of complex orography. Mean daily precipitation amounts in the selected 30 stations vary from 4.0 to 9.5 mm d−1. Observed precipitation amounts are relatively low in the southwestern regions including Yerevan (less than 5 mm d−1), in the Basin of Sevan Lake and in the southeastern regions (from 5 to 6 mm d−1). Higher precipitation amounts (greater than 6 mm d−1) are observed in the northern and southeastern regions of Armenia (Figure 4(a)).

Figure 4.

The map of mean daily precipitation amount in Armenia in the warm period mm d−1. (a) Observed data and (b) forecasted data.

Figure 4(b) shows that spatial variability of forecasted daily precipitation amounts is strongly underestimated relative to that of observed data. Forecasted mean daily precipitation amounts vary from 4.4 to 6.1 mm d−1 in Armenia. Moreover, the ERA-Interim model fails to capture the distribution pattern of observed daily precipitation amounts. In contrast to observed data higher forecasted precipitation amounts (greater than 5.5 mm d−1) can be seen in the southwestern regions of Armenia and in the eastern part of the Basin of Sevan Lake (Figure 4(b)). As a result, forecasted mean daily precipitation amounts are slightly overestimated (from 0.5 to 1.5 mm d−1) in the regions with lower precipitation amounts and they are strongly underestimated (from −3.5 to −1.5 mm d−1) in the regions with higher precipitation amounts.

The main problems of forecasting of daily precipitation amounts peculiar to the warm period can be seen in the cold period (Figure 5(a) and (b)). Spatial variability of forecasted daily precipitation amounts is strongly underestimated relative to that of the observed data. Mean daily precipitation amounts vary from 2.9 to 6.5 mm d−1 according to the observed data, while forecasted mean daily precipitation amounts vary from 2.8 to 4.7 mm d−1. Forecasted precipitation amounts vary from 3.5 to 4.0 mm d−1 in the regions with lower observed precipitation amounts (less than 4.0 mm d−1). In the mountainous stations with higher observed precipitation amounts (greater than 5 mm d−1) values of forecasted mean daily precipitation amounts mainly vary from 3.0 to 3.5 mm d−1. As in the warm period, forecasted mean daily precipitation amounts are slightly overestimated (from 0.5 to 1.0 mm d−1) in the regions with lower precipitation amounts and they are strongly underestimated (from −2.5 to −1.5 mm d−1) in the regions with higher precipitation amounts.

Figure 5.

The map of mean daily precipitation amount in Armenia in the cold period mm d−1. (a) Observed data and (b) forecasted data.

The calculated values of CV and ρ (Equations (1) and (2)) of mean daily precipitation amount fields in Armenia based on observed and forecasted data (Figures 44(a), 44(b), 55(a) and 5(b)) are presented in Table 1. The spatial variability of mean daily precipitation amounts in Armenia is greater in the cold period both by observed and forecasted data. The ratio between CV calculated by the forecasted data and by the observed data (CVfo/CVob, column 4 of Table 1) shows that about half of spatial variability of mean daily precipitation amounts is reproduced by the forecasted data. It should be noted that the magnitude of the ratio of CVfo/CVob is greater in the cold period (0.50) relative to the warm period (0.42). The latter indicates that the spatial variability of mean daily precipitation amounts is better captured by ERA-Interim model in the cold period.

Table 1. The calculated values of CV and ρ of mean daily precipitation amounts in Armenia
PeriodCVCVfo/CVobρ
ObservedForecasted
12345
Warm0.190.080.42−0.55
Cold0.220.110.50−0.30

Negative values of pattern correlation between observed and simulated fields of mean daily precipitation amounts (ρ) both in the warm period and in the cold period indicate that ERA-Interim model fails to capture the distribution pattern of observed daily precipitation amounts. It is worth noting that the pattern correlation is greater in the cold period (−0.30) than in the warm period (−0.55).

Calculated values of RE (Equation (3)) for the warm and cold periods are presented in Figure 6(a) and (b). Values of RE are greater than 100% in most stations of Armenia both in the warm and cold periods. The later means that the absolute value of difference between the observed and forecasted daily precipitation amounts exceeds the observed daily precipitation amount on average in most stations of Armenia. After comparison of observed mean daily precipitation maps (Figures 4(a) and 5(a)) with RE maps (Figure 6(a) and (b)) it can be seen that the values of RE are higher in the regions with lower daily precipitation amounts and vice versa. The latter can be explained by the fact that the lower precipitation amount in a station the greater the contribution of error of forecasted precipitation amount and vice versa.

Figure 6.

The distribution of relative error of daily precipitation amount forecasts (RE) % in Armenia. (a) Warm period and (b) cold period.

In the warm period values of RE exceed 220% in the southwestern regions with precipitation amounts less than 4.5 mm d−1. Relatively high values of RE (greater than 160%) are obtained for the Basin of Sevan Lake and for the far southeastern regions (Figure 6(a)). The values of RE are less than 120% in the northern and southeastern regions where precipitation amounts exceed 6.5 mm d−1.

In the cold period, values of RE are greater than 160% in the western, southwestern, southeastern regions and in the Basin of Sevan Lake with precipitation amounts less than 4 mm d−1 (Figure 6(b)). Lower values of RE (less than 120%) are obtained for mountainous stations located in the northwestern, central and southeastern regions of Armenia where precipitation amounts exceed 5.0 mm d−1.

Figure 7(a) and (b) presents the distribution of values of R (Equation (4)) in Armenia in the warm and cold periods respectively. Values of R are higher than 0.35 in the northern, northeastern and southeastern regions of Armenia in the warm period (Figure 7(a)). Values of R are less than 0.3 in the western, southwestern regions of Armenia and in the Basin of Sevan Lake.

Figure 7.

The distribution of correlation coefficient between values of the forecasted and observed daily precipitation amounts (R) in Armenia. (a) Warm period and (b) cold period.

Figure 7(b) shows that relatively high values of R (greater than 0.5) are obtained for the northern and northeastern regions of Armenia in the cold period. Values of R vary mostly from 0.35 to 0.5 in the other regions. It should be noted that the values of R are about 1.5 times higher in the cold period than in the warm period in most stations of Armenia. The latter indicates that the ERA-Interim model simulates temporal variability of observed daily precipitation amounts more successfully in the cold period.

The average values of estimates of observed and forecasted daily precipitation amounts over the warm and cold study periods respectively have been considered above. The average values of the estimates over the whole period could mask a sort of annual/monthly/daily variability due to a different precipitation amount in the different years, for example. Therefore, to further check the performance of ERA-Interim model in Armenia mean daily precipitation trend from 1996 to 2010 for the warm and cold periods is examined.

Figure 8(a) and (b) shows time series of mean daily precipitation amounts for the entire Armenia from observed and ERA-Interim data for 1996–2010 in the warm and cold periods respectively. It can be seen from Figure 8(a) and (b) that ERA-Interim model systematically underestimates daily precipitation amount by 0.6 mm d−1 on average both in the warm and cold periods. However it is worth noting that the ERA-Interim forecasts in the cold period (Figure 8(b)) are superior to those in the warm period (Figure 8(a)) which is consistent with the results obtained above. The correlation coefficient between the observed and forecasted mean daily precipitation amounts is higher by approximately a factor of two in the cold period (0.92) relative to the warm period (0.43). Overall, 15-year means show the increase in daily precipitation amounts both in the warm and cold period. The total increase in daily precipitation amount consists 0.6 (0.5) and 0.5 (0.4) mm d−1 in the warm and cold periods correspondingly according to the observed (ERA-Interim) data. However, such a result obtained from relatively short period should not be taken as indicative of an actual climate change in Armenia.

Figure 8.

Interannual variability of mean daily precipitation amounts (mm d−1) for the entire Armenia from the observed and ERA-Interim data for 1996–2010. (a) Warm period and (b) cold period.

4. Discussion and conclusions

Verification of performance of NWP models is very important issue. It is still difficult to achieve good or satisfactory results in precipitation amount forecasting (Wetterhall et al., 2009; Wilby and Wigley, 2000) confirmed by the above presented verification results. Forecasted daily precipitation amounts of ERA-Interim model have relatively coarse resolution for the considered region with significant topography. Therefore, errors related to re-interpolation of forecasted daily precipitation amounts to meteorological stations may significantly impact the verification results. However, the station-based verification of 12-h daily precipitation amount forecasts in Armenia according to ERA-Interim model has shown several interesting and important patterns mentioned below

High values of relative errors of daily precipitation amount forecasts exceeding 100% in most stations of Armenia and the persistently low values of correlation coefficients between values of forecasted and observed daily precipitation amounts make it difficult the use of ERA-Interim model for forecasting daily precipitation amounts in meteorological stations of Armenia.

Forecasted mean daily precipitation amounts are slightly overestimated in the regions with lower precipitation amounts and they are strongly underestimated in the regions with higher precipitation amounts.

Verification results have shown that both temporal and spatial variability of daily precipitation amounts in Armenia are better reproduced by forecasted data in the cold period than in the warm period. This may be the result of the substantial differences in precipitation-producing processes in the warm and cold periods. As was mentioned above there is significant contribution of convective activity in the formation of precipitation in the warm period. The formation and intensification of convective systems are very sophisticated in Armenia as a result of complex topography.

The paper is the first attempt that presents an analysis of performance of NWP model output data in Armenia. The performance of ERA-Interim model and other models in Armenia should be studied further. It is interesting to consider daily precipitation amounts of ERA-Interim model as dichotomous (yes/no) forecasts and to examine such verification measures as Heidke skill score, false alarm ratio (Oskouian et al., 2008) generally applied in verification of non-probabilistic forecasts for discrete predictands (Wilks, 2006).

Even with continuing advances in computing power, general circulation models (GCMs) remain limited to horizontal resolutions to simulate small-scale circulation features, which are important for the accurate simulation of regional climate and precipitation-producing factors. The use of limited-area models nested within GCMs or different statistical downscaling methods (Wetterhall et al., 2009; Wilby and Wigley, 2000) provides the ability to resolve detailed features of orography and the land surface that affect regional climate. The mentioned issues are the subject of future research.

Ancillary