An automated nowcasting system of mesoscale convective systems for the Mediterranean basin using Meteosat imagery. Part II: Verification statistics

Authors


S. Kolios, Department of Geography, University of the Aegean, University Hill, Mytilene 81100, Greece. E-mail: stavroskolios@yahoo.gr

ABSTRACT

A nowcasting system of Mesoscale Convective Systems (MCSs) based on Meteosat Second Generation (MSG) imagery (described in part I) is applied on 109 MCSs identified over the Mediterranean region for a 16 day warm season period in order to statistically asses its performance. The dependence of the accuracy on forecast lead time as well as on the forecast cycle was also investigated. This first statistical verification of the system performance showed encouraging forecast accuracy although a gradual accuracy degradation is observed with forecast lead time. In general, forecasts can be acceptable up to a 105 min duration. Adequately accurate forecasts of the MCS parameters are obtained for a 45 min forecast range while a very good spatial agreement between forecasted and observed MCSs was found for a 60 min forecast range. A tendency for a slight improvement in the forecast skill is observed proceeding to the next forecast cycles. A presented case study demonstrated the value of the forecasting system in predicting both the movement and intensity of an MCS when applied in an appropriate situation. According to the verification results, the forecast scheme presented has proven to be a valuable tool in predicting the evolution of warm season isolated MCSs over the Mediterranean basin, even at the lower boundaries of the mesoscale.

1. Introduction

In general, a Mesoscale Convective System (MCS) nowcasting algorithm based on satellite data often relies on linear extrapolation or advection of satellite data to produce very short-range forecasts of the MCS movement and evolution (Collier, 1989; Brown et al., 1994). The linear extrapolation of recent trends, however, cannot handle the complex MCS lifecycles. To meet the need for forecasting the behaviour and the development stages of convective systems, conceptual life cycle models were established based on local climatology.

Several automatic forecasting systems based on geostationary satellite data combined linear extrapolation methods with MCS conceptual life cycle models to provide very short-range forecasts (up to 2 h) of the movement and evolution of an MCS (Zipser, 1982; Hand and Conway, 1995; Riosalido et al., 1998; Puca et al., 2005; Vila et al., 2008). Such a system has proved to be a valuable and unique tool for monitoring and forecasting mesoscale convection developed over remote land and sea areas such as those in the Mediterranean basin (Riosalido et al., 1998; Morel and Senesi, 2002; Puca et al., 2005).

Statistical verification of forecasting systems is an essential component of their development and use. It helps in evaluating the quality or accuracy of forecasts by providing feedback to developers throughout the development process, as deficiencies in the systems are unveiled, and giving information to forecasters and end users to consider in their decision-making processes (Davis et al., 2006). Moreover, forecast verification can be used to compare forecasts made by different forecasting systems (Mahoney et al., 2002). Most of the MCS forecasting systems developed in previous studies have been evaluated using continuous (mean error, root mean square error, correlation co-efficient etc) and categorical verification statistics (i.e. probability of detection and false alarm ratios) based on the entries of a contingency table.

The objective of this study is to test and evaluate the forecasting accuracy of the automated nowcasting system of MCSs described in the first part of the two-part study (Kolios and Feidas, 2012). The nowcasting system is an algorithm that detects and tracks MCSs in Meteosat Second Generation (MSG) images and then forecasts the movement and the evolution of the physical properties of a selected MCS through its entire lifecycle, at 15 min intervals, over the entire Mediterranean basin and its surroundings. A complete forecast of the MCS evolution (forecast cycle) is provided every 15 min, in accordance to the acquisition of a new satellite image. The first forecast cycle starts 15 min after the first detection of the MCS. Herein, the nowcasting system is applied over the Mediterranean region for a 16 day warm season period to statistically assess its performance. In addition, a case study is presented to demonstrate the value of the forecasting system in predicting both the movement and intensity of an MCS.

2. Verification area and data

The verification analysis was performed over the fixed application domain of the nowcasting system, covering the Mediterranean basin and its surroundings (from 28°N to 50°N, and 5°W to 35°E) (Figure 1). The nowcasting system was applied for the period of 21 May to 5 June 2008 using data in the thermal infrared (10.8 µm) and water vapour (6.2 µm) spectral region obtained with the Spinning Enhanced Visible and Infrared Imager (SEVIRI) instrument on board MSG satellites. They have a 15 min temporal sampling rate and about 3 km × 3 km spatial resolution at the sub-satellite point, reaching 5 × 6 km2 at the easternmost limit of the area of study. One hundred and nine MCSs identified in the study area during the 16 day period were included in the verification.

Figure 1.

The verification domain and orography

3. The verification methodology

Two different methodologies were used to verify the forecast accuracy of the system. The first verification process uses continuous verification statistics based on the comparison between the forecast and observed parameters of an MCS during its entire lifetime. To this end, five parameters were used to measure the closeness of the forecasts to the observed values: the correlation co-efficient (r), the mean error (ME) or bias, the root mean square error (RMSE) after the bias is removed, the mean absolute error (MAE) and the efficiency score (eff) which is defined as follows:

equation image(1)

where Fi are the forecast values and Oi are the observed values. Both MAE and ME express the systematic error whereas RMSE ascertains the random error in a forecast. The eff represents the model's skill relative to the observed data ranging from − ∞ (poor skill) to 1 (perfect skill) with 0 indicating that model forecasts are as accurate as the mean of the observed data (Nash and Sutcliffe, 1970; Legates and McCabe, 1999).

The second verification procedure uses a categorical yes/no dichotomous statement to compute a contingency table, where yes/no denotes a pixel belonging/not belonging to the MCS. The contingency table is a matrix representing the frequencies of forecast and observed MCS pixel occurrence and non-occurrence. This table is built by overlapping the two ellipses that represent the observed and the forecast MCS.

In this study, the following categorical verification statistics have been computed from the entries in the contingency table.

  • The probability of detection (POD), representing the fraction of all MCS pixels that were correctly identified. It is a measure of the ability of the forecast procedure to identify correctly the occurrence of a MCS. The perfect score for POD is 1.

  • The false alarm ratio (FAR), defined as the fraction of forecasted MCS pixels, which were found to be, in fact, non-MCS pixels. A perfect forecast has a FAR score equal to 0. POD and FAR must always be considered together as neither of them alone is sufficient.

  • The critical success index (CSI), or threat score (TS), which measures the fraction of observed and/or forecasted MCS pixels that were correctly forecasted (Donaldson et al., 1975). CSI values vary from 0 to 1 (perfect forecast).

4. Results

4.1. Statistical verification

In order to evaluate the forecast accuracy of the developed algorithm described in the first part of the study (Kolios and Feidas, 2012), the observed and forecast parameters for all the MCSs detected by the algorithm for the period of 21 May to 5 June 2008 were compared. This verification process was applied only in MCSs which did not display any merging or splitting during their lifetime. A total number of 109 MCS trajectories were identified in the area of interest during the verification period. In order to examine how the accuracy varies with forecast lead time, and as a function of the forecast cycle, the verification analysis was performed for all the 109 MCSs and for each forecast cycle and lead time separately using the automated verification module of the forecasting algorithm.

Verification is carried out using two types of processes. The first process was calculation of continuous verification statistics for the main MCS parameters. The second process is based on contingency tables and categorical statistics resulted by the comparison of the ellipses representing observed and forecast MCS.

4.1.1. Continuous statistics

The first verification analysis was carried out by considering the main MCS parameters estimated by the forecasting algorithm (see part I). The verification procedure uses the measures of the closeness of the forecasts to the observed values presented in the description of the verification algorithm (r, ME, RMSE, MAE and eff) and the scatter plots of the forecasts against observations. Comparison statistics were obtained for eight forecast cycles by taking into account the forecasted evolution of all the MCSs detected by the algorithm during the period of verification.

Figure 2 presents the verification scores of five main MCS parameters (areal extent, mean temperature, the 10% lowest temperature at the infrared channel, convective potential and major semi-axis length of the ellipse) for eight forecast cycles. The plot also gives the mean value and the associated standard deviation of the observations. It is noted that the extension of the standard deviation bars to negative values for areal extent is an indication of a skewed distribution and has no physical meaning. The standard deviation of the parameters indicates a considerable variability, since for each forecast cycle the evolution of all the MCSs detected during the period of verification is considered.

Figure 2.

Verification scores for five main MCS parameters (a) areal extent (km2), (b) major semi-axis length of the ellipse (km), (c) mean brightness temperature at the 10.8 µm infrared channel (K), (d) the 10% lowest temperature at the at the 10.8 µm infrared channel (K) and (e) convective potential (%), for eight forecast cycles when considering the forecasted evolution of all the MCSs detected during the period from 21 May to 5 June 2008. equation image: The mean observed values; equation image: the associated standard deviation of the observations; equation image: MAE; equation image: ME; equation image: RMSE

Inspection of Figure 2 reveals a slight overestimation of the MCS areal extent in the first two forecast cycles and an underestimation after the fifth forecast cycle, as indicated by the sign of ME. The exactly opposite pattern is observed in the other morphological parameter, which is the major semi-axis of the best-fitted ellipse. This means that ellipses forecasted in the first two forecast cycles are slightly larger and more circular than the observed MCSs and become smaller and more elongated after the fifth forecast cycle. The pattern of the ME of the mean temperature and the 10% lowest temperature is coincident with that of the areal extent. A systematic underestimation of the convective potential by the forecasting algorithm is present in all the forecast cycles.

When considering the random errors (MAE and RMSE), a tendency for a slight improvement in the forecast skill is observed proceeding from the first to the eighth forecast cycle.

The MAE ranges from 426 to 897 km2, (from 26.5 to 50.1% of the mean observed value, respectively) for areal extent, from 12.3 to 24% for convective potential and from 12.1 to 18.2 km (from 31 to 36% of the mean observed value, respectively) for the major semi-axis length. The MAE for the 10% lowest temperatures is very low lying between 1.18 and 1.97° while for the mean temperature the MAE is even lower spanning from 0.75 to 1.26°. A similar behaviour is evident for RMSE score. Finally, a slight but clear decreasing tendency for both errors (MAE and RMSE) is evident proceeding to the next forecast cycles. This improved forecast skill is attributed to the increased observational input information being taken into account by the model in the subsequent forecast cycles.

Scatter plots of the observed and forecast parameters for three forecast cycles (first, fourth and eighth cycles) are shown in Figure 3. Correlation co-efficient, efficiency score and sample size are presented in Table 1. The best agreement between the observed and forecast values is observed for the areal extent (r from 0.8 to 0.9 and eff from 0.64 to 0.9) whereas the forecasts of the convective potential show the poorest correspondence with observations (r from 0.57 to 0.74 and eff from 0.2 to 0.9), especially for the first forecast cycle.

Figure 3.

Scatter plots of the forecasted and observed parameters: areal extent (a–c), major semi-axis length of the ellipse (d–f), mean brightness temperature at the 10.8 µm infrared channel (g–i), the 10% lowest temperature at the 10.8 µm infrared channel and (j–l), convective potential (m–o), for three forecast cycles: first cycle (left column), fourth cycle (middle column) and eighth cycle (right column)

Table 1. Basic statistics referring to the scatter plots of the observed and forecasted areal extent (Figure 3): correlation co-efficient (r), efficiency score (eff) and total number of cases (N)
MCSStatisticalForecast cycle
parametersparameters 
  FirstFourthEighth
Areal extent (km2)r0.80.910.96
 eff0.640.80.9
 N995614144
Major semi-axis lengthr0.750.760.89
of the ellipse (km)eff0.50.530.67
 N995614144
Mean brightnessr0.830.780.86
temperature in theeff0.670.60.74
channel of 10.8 µm (K)N995614144
10% lowest brightnessr0.850.860.85
temperature in theeff0.70.710.72
channel of 10.8 µm (K)N995614144
Convective potential (%)r0.570.60.74
 eff0.120.810.9
 N995614144

As expected, a gradual improvement in the forecast skill is observed proceeding from the first to the eighth forecast cycle. This improvement is notable on the scatter plots of the areal extent and convective potential, showing a dispersion which is gradually reduced from the first to the eighth forecast cycle. For the length of the major ellipse semi-axis, a substantial amelioration is evident only in the eighth cycle. These patterns are also reflected on the increase of r, which for the areal extent starts from 0.8 at the first forecast cycle and ends up to 0.91 and 0.96 at the fourth and eighth cycle, respectively. The respective increase for eff values are from 0.64 to 0.8 and 0.9. The same behaviour is also observed for convective potential (r from 0.57 to 0.61 and 0.74). On the contrary, the forecast skill improvement for the brightness temperature parameters is not pronounced.

The validation of the forecast MCS location was performed in terms of the distance between the forecast and observed centres of mass of the system. This distance determines as well the displacement and velocity accuracy of the forecasted MCS. The root mean squared error (RMSE) is widely used to determine the residuals in a geometric correction procedure and it was chosen as a measure of closeness between the observed and the forecast MCS location.

Figure 4 exhibits the RMSE between the forecasted and the observed centres of mass of the MCSs for the eight forecast cycles of the verification period. As expected the largest RMSE (7.7 km) is produced in the first forecast cycle, probably due to the short MCS life history available for assimilation in the forecasting system. The RMSE falls to 5 km in the second forecast cycle being gradually reduced to less than 5 km, which is approximately the MSG pixel size in the area of study. This significant improvement in the forecast accuracy of the MCS location can be ascribed to the continuous increase of the MCS life history observations ingested into the model.

Figure 4.

Total RMSE between the forecasted and the observed centre of mass of the MCSs for eight forecast cycles when considering the forecasted evolution of all the MCSs detected during the period from 21 May to 5 June 2008

Another validation test was performed to investigate how the model accuracy varies with forecast lead time in each forecast cycle. In general, lead time refers to the period between the issue time of the forecast and the occurrence of the predicted phenomenon (AMS, 2000). In the present case, the forecast lead time begins at the time of first detection of the MCS being expressed as number of 15 min time steps.

The evolution of the verification scores of two main MCS parameters (areal extent and mean brightness temperature at the 10.8 µm channel) with forecast lead time is presented in Figures 5 and 6, for two indicative forecast cycles (first and third cycles). The plot also gives the mean value and the associated standard deviation of the observations. Note that the x-axes were adjusted to correspond to the same time of the MCS lifecycle.

Figure 5.

Evolution of verification scores (MAE, ME, RMSE) with lead time for areal extent (km2), for the (a) first and (b) third forecast cycle when considering the forecasted evolution of all the MCSs detected during the period from 21 May to 5 June 2008. The mean and the associated standard deviation of the observations are also plotted in the graph. The x-axes were adjusted to correspond to the same time of the MCS life cycle. equation image: Mean observed values; equation image The associated standard deviation of the observations; equation image: MAE; equation image: ME; equation image: RMSE

Figure 6.

Same as Figure 5 but for the mean brightness temperature (K) at the 10.8 µm channel. equation image: Mean observed values; equation image: the associated standard deviation of the observations; equation image: MAE; equation image: ME; equation image: RMSE

As expected, both random errors (MAE and RMSE) deteriorate with forecast lead time, following the same pattern in both forecast cycles. The third forecast cycle, however, provides significantly lower errors than those of the first cycle. For example, for the early first forecast cycle of the areal extent (Figure 5(a)), MAE lies below the 57.4% of the mean for the 90 min forecast range (sixth lead time), whereas for the third forecast cycle (Figure 5(b)) the error for the corresponding times of the MCS lifecycle (up to the forth lead time) is reduced below 38%. The ME points to an overestimation of the observed values for all the lead times of the first forecast cycle. This overestimation is significantly reduced in the third forecast cycle, when more MCS life history observations are ingested into the model.

The MAE of the mean brightness temperature in the first forecast cycle (Figure 6(a)) rises with forecast lead time from 0.4 to 1.5 K, whereas a constant overestimation is evident when considering the ME. Both errors are reduced in the third forecast cycle (Figure 6(b)). These values are considered very small given that the absolute mean brightness temperatures range from 220 to 227 K, spanning a range of 7 K.

In general, the first forecast cycle is capable of providing accurate enough results, although overestimated, for a 45 min forecast range (the first three lead times) with an MAE less than 38% of the mean value for the areal extent and absolute MAE less than 1 K for mean temperature. Forecasts, however, can be acceptable up to the 105 min forecast (seventh lead time), with an MAE up to 60% of the mean value for the areal extent and absolute MAE up to 1.5 K for mean temperature. Similar results were found for other parameters and forecast cycles.

Figures 7 and 8 show the scatter diagrams of the observed and forecasted areal extent and mean brightness temperature for three lead times (second, fourth and sixth) and two forecast cycles (first and third cycles). Correlation co-efficient, efficiency score and sample size are shown in Tables 2 and 3 and respectively. As expected, a gradual degradation in the forecast skill is observed with forecast lead time. This degradation is evident in the dispersion of the points and expressed as well in both the correlation co-efficient (from 0.97 to 0.7 for areal extent and from 0.93 to 0.75 for mean temperature) and efficiency score (from 0.94 to 0.37 for areal extent and from 0.85 to 0.55 for mean temperature). More precisely, for the first forecast cycle, the correlation co-efficient declines from 0.97 to 0.7 for areal extent (Table 2 and Figure 7(a)) and from 0.93 to 0.75 for mean temperature (Table 3 and Figure 8(a)), whereas for efficiency score the decrease is from 0.94 to 0.37 and from 0.85 to 0.55, respectively. A good agreement between the observed and forecast values is observed up to the fourth lead time (60 min). Despite the significant decrease in the forecast skill in the sixth lead time (90 min forecast), levels of statistical scores are still high. The forecast skill of the model is improved, though not considerably, in the third forecast cycle (see scatter plots for corresponding times of the MCS life cycle linked with an arrow in Figures 7 and 8).

Figure 7.

Scatter plots of the observed and forecasted areal extent (km2) of MCSs during the validation period for the second lead time (a and d), fourth lead time (b and e) and sixth lead time (c and f) and two forecast cycles: first forecast cycle (a–c) and third forecast cycle (d–f). Scatter plots for corresponding times of the MCS life cycle are linked with an arrow

Figure 8.

Same as Figure 7 but for the mean brightness temperature (K) at the 10.8 µm channel

Table 2. Basic statistics referring to the scatter plots of the observed and forecasted areal extent (Figure 7): correlation co-efficient (r), efficiency score (eff) and total number of cases (N)
Forecast cycleStatistical parametersSecond lead timeFourth lead timeSixth lead time
Firstr0.970.820.7
 eff0.940.620.37
 N109109109
Thirdr0.840.750.6
 eff0.660.32− 2.8
 N10910976
Table 3. Basic statistics referring to the scatter plots of the observed and forecast mean brightness temperature (Figure 8): correlation co-efficient (r), efficiency score (eff) and total number of cases (N)
Forecast cycleStatistical parametersSecond lead timeFourth lead timeSixth lead time
Firstr0.930.860.75
 eff0.850.710.55
 N109109109
Thirdr0.930.720.48
 eff0.850.520.2
 N10910976

4.1.2. Categorical statistics

The second verification test is a feature-based approach that evaluates how well the forecast MCSs spatially match the observed MCSs based on the overlap of the ellipses representing the cloud system cells. The categorical statistical indices of POD, FAR and CSI were computed from the resulted contingency table to evaluate the forecast performance.

Figure 9 shows the categorical statistics for the first eight forecast cycles when considering the forecasted evolution of all the MCSs detected during the period of verification. POD score shows a slight but insignificant increase during the first five forecast cycles. At the same time, the FAR score is generally decreasing, while the CSI score remains almost constant. More precisely, the POD scores vary from 52 to 55% and the FAR score from 53 to 62% in the first five forecast cycles. These values can be deemed as satisfactory when considering that the spatial match of the ellipses depend on the additive uncertainty arising from the errors in the estimation of the four parameters used to calculate the forecasted ellipse (co-ordinates of the centre, major axis length, areal extent and orientation). For the next three forecast cycles, the previous trends reverse with the POD and CSI scores decreasing and the FAR score increasing. This deterioration in the spatial match between ellipses is attributed to the double penalty effect which becomes important in the last forecast cycles due to the gradual decrease of the observed MCS size during the dissipation phase. The double penalty effect arises when the observed small-scale feature is misplaced by the forecast. The forecast model is then penalized twice; once for not getting the feature at the correct location (miss) and again for forecasting it where it is not (false alarm) (Anthes, 1983; Mass et al., 2002).

Figure 9.

Categorical statistics (POD, FAR and CSI) for the first eight forecast cycles resulted by the comparison of the ellipses that represent the observed and the forecasted MCS during the period from 21 May to 5 June 2008. equation image: POD; equation image: FAR; equation image: CSI

Figure 10 presents the same verification scores (POD, FAD and CSI) for each lead time of the first and third forecast cycle. Again, the x-axes were adjusted to correspond to the same time of the MCS lifecycle. Examination of the time evolution of these categorical statistics shows that all scores deteriorate with forecast time. For example, for the first forecast cycle POD decreases from 74.5% in the 15 min forecast (first lead time) to 36% in the 120 min forecast (eighth lead time), and FAR increases from 45 to 67.5%, respectively (Figure 10(a)). Obviously, the third forecast cycle exhibits a better skill, with significantly lower FAR and slightly better POD (Figure 10(a)). As a result higher CSI values are obtained.

Figure 10.

Evolution of categorical statistics (POD, FAR and CSI) with lead time for the (a) first and (b) third forecast cycle, resulted by the comparison of the ellipses that represent the observed and the forecasted MCS during the period from 21 May to 5 June 2008. The x-axes were adjusted to correspond to the same time of the MCS life cycle. equation image: POD; equation image: FAR; equation image: CSI

According to Figure 10(a), the first forecast cycle of the model provides adequately accurate MCS forecasts for a 60 min forecast range (the first four lead times) with a POD exceeding 66% and FAR being less than 59%.

4.2. A case study

Besides the statistical evaluation of the forecast algorithm for a large number of MCSs, it is also helpful to gain a visual impression of the forecasts and assess the forecast accuracy for individual tracks. For the sake of brevity, a single case study is presented in this section, aiming to show the method operating and to give the reader a sense of the forecasts produced by the system.

At 1200 UTC, 30 May 2008, a cold front (Figure 11(a)) associated with a trough oriented in a west–east direction in the lower and middle troposphere (Figure 11(b) and (c)) was crossing the Balkan Peninsula. The scattered clouds located north of Greece (Figure 12) were the result of the low-level forcing for ascent along the southern edge of the cold front. As a result, several MCSs were formed over the mountainous area of the western Balkans. An isolated convective cloud system with a size of 300 km2 was first detected by the forecasting system in Serbia at 1215 UTC (Figure 12). Three snapshots of the MCS life cycle are presented in Figure 13. The cloud system propagated to the north and dissipated at 1530 UTC, 195 min after the time of its first detection.

Figure 11.

(a) Sea-level pressure analysis map with fronts provided by the UK Meteorological Office, (b) 500 hPa geopotential height (lines with 30 m interval) and temperature analysis (dashed lines with 1 °C interval) (b) 850 hPa geopotential height (solid line with 15 m interval) and temperature analysis (dashed lines with 1 °C interval), for 30 May 2008 at 1200 UTC

Figure 12.

Infrared MSG images at the 10.8 µm channel for 30 May 2008 at 1215 UTC. The inner square with the cross marks the isolated MCS whose temporal evolution is forecasted by the algorithm

Figure 13.

Three snapshots of the MCS life cycle developed in Serbia on 30 May 2008. (a) Infrared MSG images with the MCS denoted by a square. (b) The cloud cell identified by the detection algorithm inside the frame

Figure 14 presents the plot of the MCS track in terms of best fitted ellipses, showing the recent MCS history, the current position, and the forecast for two forecast cycles (first and fourth cycle). The forecast MCS lifespan is 30 min shorter that the observed life cycle duration. There is, however, a quite satisfactory spatial match between forecast and observed ellipses of the studied MCS and a general agreement in the direction of movement. The MCS started to move north-northwestward but after 1 h it turned slightly right into a north-northeast direction. This small change in the trajectory caused by factors such as topography and upper level winds cannot be predicted by a forecasting system using the MCS history to forecast the MCS displacement. In fact, the first forecast headed steadily the system north-northwestward driven by the MCS history during the first two time steps of its life cycle. Moreover, the model moved the MCS less far to the north than the observations. The forecasted trajectory is amended and corrected during the next forecast cycles as more MCS life history observations are ingested into the model (see for example the fourth forecast cycle).

Figure 14.

Observed (gray colour) and forecasted (black colour) ellipses for two forecast cycles representing the lifecycle of the MCS identified by the system in Serbia on 30 May 2008, at 1215 UTC. The solid gray ellipses represent the observations corresponding to the forecasts issued at 1245 UTC and 1330 UTC for the first and fourth forecast cycle, respectively. The dotted gray ellipses depict the MCS history prior the first lead time while the dashed gray ellipses represent the MCS life cycle not predicted by the model. The centres of the ellipses are also indicated in the figure

This is also reflected to the evolution of the distance between the forecasted and observed centre of the ellipses with lead time shown in Figure 15. The increasing trend of the distance observed after the fourth lead time of the first forecast cycle (Figure 15(a)) and the corresponding first lead time of the fourth forecast cycle (Figure 15(b)) is the result of a small change in the direction of movement of the MCS. This error is ameliorated in subsequent forecast cycles (Figure 15(c)). In general, the error in the position of the forecasted MCS is overall small, being of the order of one MSG pixel (less than 5 km), at least up to the sixth lead time of the first forecast cycle (Figure 15(a)).

Figure 15.

Evolution of the distance between the forecasted and observed centre of mass of the examined MCS (30 May 2008) with lead time, for three forecast cycles: (a) first, (b) fourth, and (c) eighth cycles. The x-axes were adjusted to correspond to the same time of the MCS life cycle

Figure 16 exhibits the comparison between the forecasted and observed MCS life cycle expressed as the time evolution of three characteristic parameters (areal extent, temperature of the 10% coldest pixels in the 10.8 µm channel and convective potential) for three forecast cycles (first, fourth, and eighth). The forecasted lifetime for the first and fourth forecast cycles ends at 1500 UTC, 30 min earlier than the time of the MCS last detection. Given that the time of maximum areal extent depends on the MCS lifetime, this 30 min difference between the forecasted and observed lifetime resulted in a 15 min time lag between the estimated and measured maximum areal extent (Figure 16(a)). However, there is a good correspondence between the time evolution of forecast and observed areal extent in the development phase. The main discrepancy is discerned in the dissipation phase, during which the MCS dissipated more slowly than forecast. The correspondence between forecasts and observations is improved only after the sixth forecast cycle (not shown in Figure 16) when observations from the dissipation phase became available to the forecasting system. This is evident in the forecasts of the eighth forecast cycle shown in Figure 16(a).

Figure 16.

Comparison between the forecasted and observed life cycle of the examined MCS (30 May 2008) expressed as the time evolution of three characteristic parameters: (a) areal extent (km2), (b) temperature of the 10% coldest pixels in the 10.8 µm channel (K), and (c) convective potential (%). Forecasts are presented for three forecast cycles (first, fourth, and eighth). equation image: observed values; equation image: first forecast cycle; equation image: fourth forecast cycle; equation image: eighth forecast cycle

The time evolution pattern of the temperature of the 10% coldest pixels (Figure 16(b)) is well predicted in the first forecast cycle. The time lag in the minimum temperature occurrence and the slower rate of the temperature rise during the dissipation phase are the two main differences between observation and forecasts. The discrepancy in the fourth forecast cycle is increased by the inclusion of the minimum observed at 1300 UTC in the least squared equations.

The system provides the best forecast when considering the MCS convective potential, with a notable agreement between forecasts and observations both in terms of life cycle phases and absolute values (Figure 16(c)). This agreement is very encouraging since the convective potential is a crucial parameter in forecasting the strength of the MCS.

5. Conclusions

This study presents an evaluation of the forecasting accuracy of the automated nowcasting system of MCSs described in the first part of this two-part study (Kolios and Feidas, 2012). The nowcasting system, which is able to forecast the movement and the evolution of the physical properties of a MCS through its entire lifecycle using MSG SEVIRI satellite data, is applied over the Mediterranean region for a 16 day warm season period in order to statistically assess its performance.

The forecast accuracy of the developed forecasting algorithm was first evaluated by comparing the observed and forecasted parameters for the 109 MCSs detected by the algorithm in the verification period (21 May to 5 June 2008). Categorical verification statistics were then used to evaluate how well the forecast MCSs spatially match the observed MCSs based on the overlap of the ellipses representing the cloud system cells. Both verification processes investigated how the accuracy varies with forecast lead time, and as a function of the forecast cycle. In addition, the capability to forecast the MCS properties has been demonstrated for an indicative case study. The main results of the verification are summarized as follows.

  • The accuracy of the forecasts is encouraging although a gradual degradation in the model forecast skill is observed with forecast lead time.

  • In general, the first forecast cycle is capable of providing adequately accurate forecasts of the MCS parameters for a 45 min forecast range (the first three lead times). When the categorical statistical indices are taking into account, a very good spatial agreement between forecasted and observed MCSs is found for a 60 min forecast range. A forecast skill decline is distinctive for longer lead times. Forecasts, however, can be acceptable up to the 105 min forecast (seventh lead time).

  • A tendency for a slight improvement in the forecast skill is observed proceeding to the next forecast cycles. This improved forecast skill can be ascribed to the positive impact brought by the continuous increase of the MCS life history observations ingested into the model procedures.

  • The presented case study demonstrated the value of the forecasting system in predicting both the movement and intensity of a MCS when applied in an appropriate situation.

The presented results show that the performance of the developed forecasting algorithm is encouraging. The forecast scheme presented has proven to be a valuable tool in predicting the MCS time evolution for short forecast times and efficient enough for early and accurate warning. It can be used with radar-based nowcasting techniques in a synergetic way to analyse and forecast convection at different spatial scales. A further development of this work will be to the direction of enhancing the forecasting scheme to handle more complex MCS lifecycles (e.g. mergers and splits), using a better object representation of the MCS and improving the forecasts accuracy by incorporating numerical weather prediction models' fields such as wind fields (Corfidi et al., 1996) at different atmospheric levels. Further work will comprise different and more sophisticated criteria for selecting cloud cells, such as the additional use of temperature thresholds in different channels and a less simplistic object representation of the MCS. Finally, the adaptation of the algorithm to the Rapid Scan High rate SEVIRI with the 5 min time sampling could improve the accuracy of forecasts especially for rapidly developing convective cells.

Acknowledgements

This study is a part of a research project co-financed by E.U.-European Social Fund (75%) and the Greek Ministry of Development-GSRT (25%) with the title: ‘Automated system of Mesoscale Convective Systems short range forecast with the use of Meteosat imagery’. The authors would like also to express their appreciation to the anonymous reviewers for their constructive comments that helped to improve the completeness and clarity of the paper.

Ancillary