Verification of objective sensitivity climatologies of Mediterranean intense cyclones: test against human judgement

Authors


Abstract

A variety of sensitivity climatologies of Mediterranean intense cyclones have been recently built owing to the growing international interest in contributing to the basic understanding and the short-range forecasting of high-impact weather events. The verification of these climatologies is essential to ensure the reliability of the sensitivity products and ultimately provide robust guidance to policy-makers on plans to redefine routine observational strategies. This work tackles the arduous task of verifying the available (an adjoint-based and two different ensemble-based) sensitivity climatologies of Mediterranean intense cyclones. We perform Observing System Simulation Experiments (OSSE) with the WRF ARW model for 25 of the most intense Mediterranean cyclones detected in the ERA-40 database to test the ability of each method in identifying areas where perturbations in the initial conditions derived from the sensitivity fields lead to a greater impact on the forecast of intense cyclones. For the sake of a sensible reference, the performance of the available sensitivity climatologies is tested against the judgement of an experienced severe weather meteorologist. In addition, a control measure of the background random response is also carried out. The impact on the prediction of intense Mediterranean cyclones of prescribed perturbations to the initial conditions is evaluated comparing each perturbed experiment with a control simulation. Furthermore, a quantitative study of the linearity of the evolution of the perturbations is performed using twin perturbations. Results confirm a statistically significant superior skill of the human and adjoint sensitivity fields against both ensemble sensitivity climatologies. Climatological ensemble sensitivities only show a noticeable improvement upon non-sensitivity experiments when an ad hoc classification of cyclones is used. This reveals one fundamental limitation of the ensemble sensitivity technique in climatological mode when it is applied to rare events insufficiently sampled in the available datasets. Copyright © 2011 Royal Meteorological Society

1. Introduction

The value of current numerical weather predictions is severely limited by the persistent presence of large errors in socially sensible forecast products. These errors originate from two main sources: imperfections in the models, including those from the parameterization schemes, and errors in the analysis fields. Similarly, the quality of the analyses depends on two main factors: the quality of the data assimilation system and the available observational dataset. In current forecasting systems, atmospheric observations are aimed at providing innovative information to the analysis, relaxing the previous cycle forecast to the actual atmospheric state, generally leading to more accurate initial conditions (IC) and eventually issuing improved predictions. Indeed, national weather services allocate substantial resources to improving upon the World Weather Meteorological Organization Global Observing System (WMO-GOS) or its climate equivalent (WMO-GCOS). The Network of European Meteorological Services Composite Observing System project (EUMETNET-EUCOS, http://www.eumetnet.eu/conteucos.html) is a relevant example of the European commitment to optimize the integrated observing system network at a European scale to improve short-range forecasts over Europe. The ever-growing pressure from the public and authorities to improve forecast skill without increasing significantly the overall cost makes unfeasible the inefficient solution of a homogeneous increase in the number, quality and type of in situ observations.

The sensitivity analysis to precursing states and processes involved in the evolution of a system is a fundamental methodology used across many scientific disciplines as it reveals causal information about certain aspects of interest of the system. Regarding weather prediction, sensitivity analysis techniques highlight atmospheric features at earlier times that have a relevant effect on a particular forecast aspect of interest. This allows estimating areas in the initial conditions where errors will grow fast into the forecast, and thus opens the door for establishing optimized observational plans. Information derived from sensitivity analyses should be the basis for decision makers regarding the design of both efficient routine observing networks and special targeted observation strategies.

Over Europe and for all weather regimes, climatological sensitivities of forecast errors are located mainly upstream of the westerlies, over the northeastern Atlantic (Marseille and Bouttier, 2000). However, for the sake of efficiency, the design of the permanent component of observational networks should take into special account high-impact weather (HIW) episodes because of the huge associated economic and human losses. Therefore, sensitivity information about high-impact events is very valuable owing to the larger potential benefits of associated forecast improvements. International programs such as the Mediterranean experiment oncyclones that produce high impact weather in the Mediterranean (MEDEX, http://medex.aemet.uib.es) or the Hydrological cycle in the MediterraneanExperiment (HyMeX, http://www.hymex.org/) show a growing interest in defining an observational network that improves the accuracy of HIW forecasts. The Mediterranean area is frequently affected by events of extreme adverse weather such as strong winds and heavy rain. Not all high-impact events in the Mediterranean region are related to cyclones and most of the cyclones do not produce extreme weather; however, most of Mediterranean HIW events are linked to cyclones (Jansà et al., 2001). In accordance with the continuing interest in contributing to the basic understanding and short-range forecasting of HIW events related to Mediterranean cyclones, Homar et al. (2006, 2007) and Jansà and Homar (2006) discuss the generation of a climatology of short-range sensitive areas for intense cyclone events in the Mediterranean based on adjoint model calculations.

Traditional sensitivity studies analyze the effects of one factor by comparing a control experiment with one in which the factor is altered, allowing one to easily track all effects of that cause along the evolution of the system. However, with the adoption of adjoint models by the atmospheric numerical community, the inverse approach is also possible: to calculate the set of causes that are responsible for one effect. This effect is typically a characterizing scalar measure of a particular aspect of interest in the forecast and is known as the response function, J. Tangent linear adjoint models follow a phase-space trajectory that is tangent linear to the basic nonlinear state evolution and trace back in time the gradients of the response function with respect to the model state (Errico, 1997). Therefore, adjoint models compute a linear estimate of the sensitivities of a forecast aspect to initial and boundary condition fields. The tangent-linear character of the operator limits the validity of the sensitivity fields to time-spans and perturbation sizes within the linear regime (Gilmour et al., 2001). This interval may extend up to 48–72 h for smooth integrated response functions but is shorter than 12–18 h when diabatic processes affect J substantially (Homar and Stensrud, 2004).

The climatology of sensitivities of Mediterranean intense cyclones built by Jansà and Homar (2006) using adjoint sensitivity calculations indicates that North Africa, the Mediterranean Sea and the eastern North Atlantic, which are poorly covered by in situ observing networks, are relevant for the short-range forecast of Mediterranean intense cyclones. These results are confirmed and complemented by the sensitivity climatology of Mediterranean intense cyclones obtained by Garcies and Homar (2009), which used a new approach of sensitivity analysis not linked to a particular forecasting model. Garcies and Homar (2009) show that the average evolution of these high-impact systems 24 h prior to its maturity depends largely on structures located over western Europe, the northern African lands and parts of the eastern North Atlantic. The theoretical framework of the ensemble sensitivity technique used by Garcies and Homar (2009) was first described by Hakim and Torn (2008) and was explored in detail and compared to adjoint sensitivity by Ancell and Hakim (2007a). They defined ensemble sensitivity as the linear regression of a forecast response function onto the initial conditions and showed that an ensemble sensitivity field is proportional to the projection of the analysis-error covariance matrix onto the adjoint sensitivity field. Garcies and Homar (2009) applied the ensemble sensitivity approach not to ensembles of simulations but to classes of Mediterranean intense cyclones from the climatology produced by Jansà and Homar (2006). By linearly correlating atmospheric features, Garcies and Homar (2009) produced sensitivity fields without dependence on any forecasting system. Once the ensemble sensitivity method was proved to produce sensible results from a climatological perspective, Garcies and Homar (2010) built a new classification of Mediterranean intense cyclones oriented to the application of this statistical sensitivity technique. In general terms, the new climatological sensitivity products are in accordance with those obtained by Garcies and Homar (2009). However, the accuracy of the final products is expected to improve when an ad hoc classification of Mediterranean intense cyclones is used.

Torn and Hakim (2008) suggested the ensemble sensitivity analysis as an alternative to adjoint sensitivity analysis and one that may also prove useful for observation targeting as this technique provides, prior to knowing the observation value, the impact of a hypothetical observation on the forecast metric variance. In addition, Ancell and Hakim (2007b) examined the potential of both ensemble and adjoint methods in reducing forecast variance in a practical, real-time forecasting environment. However, the comparison of the ability of ensemble and adjoint sensitivity analysis in identifying, from a climatological perspective, locations where observations would have a significant positive impact on a future numerical weather forecast has not yet been performed. Garcies and Homar (2009) carried out a qualitative comparison between ensemble and adjoint sensitivity products for Mediterranean intense cyclones but these results were not objectively verified. On the other hand, Garcies and Homar (2010) carried out illustrative numerical experiments to test the accuracy of the new climatological statistical sensitivities against the analogous previous results for only a single type of cyclone. Here, we aim to assess the accuracy of all these sensitivity climatologies by means of an objective, statistically significant, verification testbed.

The most application-oriented verification approach one can envision is the installation of permanent observational means over the most sensitive areas given by the different climatologies and collect a long record of Mediterranean cyclone episodes. However, the costs of this hypothetical verification strategy would be similar to those of the final implementation stage, becoming a practically useless planning test. In fact, due to the difficulties of experimental observation campaigns and the lack of robust climatological findings, the use of simulation techniques to evaluate the potential contribution of proposed observing systems to forecasts is widespread (Arnold and Dey, 1986). We propose the use of observation system simulation experiments (OSSEs) to simplify the analysis and considerably reduce costs. Indeed, OSSEs simulate possible observing systems within weather prediction models and test their impact on a certain aspect of interest in the prediction. Here, we design a verification testbed based on the most sensitive areas for each cyclone class, as errors over these regions are expected to grow fastest within the linear regime and have a major impact on the mature cyclone prediction. As a matter of fact, we do not quantify the value of a particular observing system but evaluate impacts on the prediction of intense Mediterranean cyclones of prescribed perturbations to the initial conditions. This is a first step to further research work accounting for the data assimilation cycle. Therefore, in a strict sense, we do not explicitly infer details about particular observational means because we are not using any kind of data assimilation system. Here, we assess the value of various sensitivity fields of the forecasting stage as a necessary step towards an eventual analysis of sensitivities to the entire forecasting system, including the data assimilation cycle.

This paper is organized as follows. Section 2 describes the main characteristics of the Mediterranean intense cyclones sensitivity climatologies under evaluation. The methodological details of the verification are discussed in section 3. Section 4 presents the results of one representative case study and the summarized results for the 25 most intense cyclones common to all climatologies. Conclusions and final remarks are given in section 5.

2. Mediterranean intense cyclones sensitivity climatologies

The database of Mediterranean cyclones used in the sensitivity climatologies which will be tested in this study is based on the reanalysis fields from the European Center for Medium-Range Weather Forecast (ERA-40; Uppala et al., 2005). Those Mediterranean cyclones detected over the 45-year period covered by the ERA-40 (September 1957 to August 2002) with a maximum circulation exceeding 7 107m2s−1 and a lifetime of at least 24 h (Campins et al., 2010) constitute the cyclone database. In the classification obtained by Jansà and Homar (2006) (hereafter, JHC06), 1202 days with Mediterranean intense cyclones were classified in 25 clusters grouping events with both similar location of the cyclone at the time of maximum intensity, hereafter t, and similar precursory low and mid-level synoptic fields of temperature and geopotential height. In this classification, classes such as the Adriatic and Ionian seas, the Gulf of Genoa and Cyprus present a large population of intense cyclones confirming the well-known persistent occurrence of intense cyclonic systems over these areas.

For each class, Jansà and Homar (2006) used the MM5 adjoint modeling system (Zou et al., 1997) to compute the sensitivities of the pressure perturbation over each cluster's mean sea-level pressure (MSLP) field with respect to the initial conditions 48 h before time t. Then, they computed a synthetic sensitivity field for each cluster over all fields and levels, obtaining a standardized index without physical units ([pressure units] / [‘mixed IC units’]) in order to provide an indication of the most sensitive regions for each cyclone class. For the sake of brevity, the mean sensitivity field at t − 48 h computed over all clusters and weighted by the climatological frequency within the database is shown here. This summarizing field reveals a sensitivity maximum centered over northern Italy and eastern France and expanded along the Italian peninsula towards the Aegean Sea. This structure extends also towards the North Atlantic and North African lands, where a second significant sensitivity maximum emerges (Figure 1).

Figure 1.

Mean adjoint sensitivity field to central cyclone pressure at t − 48 h computed over all cyclone classes (darker colors indicate higher sensitivities) (adapted from Jansà and Homar, 2006). This figure is available in colour online at wileyonlinelibrary.com/journal/qj

Garcies and Homar (2009) built an analogous climatology of sensitivities of Mediterranean intense cyclones using an alternative methodology not linked to a particular forecasting model or numerical set-up. For each Mediterranean cyclone class of JHC06, the sensitivity fields were derived by linearly correlating the precursor conditions and the response function, which was defined as the average of the MSLP at the time of maximum intensity of the cyclone over an area centered over the cyclone center of the centroid's MSLP field. As a result, the final standard sensitivity product expresses variations in the response function associated with typical IC perturbation amplitudes and have the same units as the response function (i.e. mb). With the aim of providing a general outline of which regions are sensitive for the central MSLP values at the time of maximum intensity for Mediterranean intense cyclones, the mean sensitivity field for each cyclone class and the summarized sensitivity map for all clusters were computed. For t − 48 h, the correlation coefficient correction applied by Garcies and Homar (2009) produces a significant degradation of the sensitivity field due to the lack of sufficient homogeneity in the clusters. The remnant sensitivity patterns highlight regions upstream of the westerlies and over northwest Africa (Figure 2).

Figure 2.

Mean ensemble sensitivity field for all cyclone types computed over all considered precursor conditions and levels (mb, shaded) for t − 48 h (updated from Garcies and Homar, 2009). This figure is available in colour online at wileyonlinelibrary.com/journal/qj

In order to improve the climatological ensemble sensitivity results, Garcies and Homar (2010) built a new ensemble-sensitivity-oriented classification using the aforementioned Mediterranean intense cyclones database (hereafter, GHC10). In addition to performing a regional classification and a subsequent division according to the preceding conditions that lead to cyclone formation, a further classification step was carried out. With the aim of optimizing the homogeneity of the new classes, these were pruned at the expense of sample size. Thus 406 episodes were classified into 23 cyclone types covering 13 regions across the Mediterranean basin. As a result, more homogeneous clusters and more significant sensitivity products were obtained. Furthermore, Garcies and Homar (2010) added some improvements to the sensitivity computations such as an increased temporal resolution of the products, stricter criteria to smooth out non-significant sensitivity signals and higher precision in the definition of the response function by centering it over each individual cyclone's center. Once the sensitivity fields for each of the 23 clusters were computed following the methodology proposed by Garcies and Homar (2009), the effects of the reduced sample size became evident with the emergence of spurious correlations for classes with fewer than 14 members, which were discarded from further analysis. The remaining classes were used to compute a summary sensitivity map for Mediterranean intense cyclones. At t − 48 h, this averaged sensitivity map highlights areas over eastern North African lands, parts of the central Mediterranean, central Europe and the European Atlantic coasts (Figure 3).

Figure 3.

Mean ensemble sensitivity field for sufficiently populated clusters from the classification of Garcies and Homar (2010) computed over all considered precursor conditions and levels (mb, shaded) for t − 48 h (adapted from Garcies and Homar, 2010). This figure is available in colour online at wileyonlinelibrary.com/journal/qj

Keeping in mind the characteristics of each sensitivity climatology considered, some important differences are evident between them even in the rather generic measure of averaged sensitivity obtained for each climatology for t − 48 h. In addition, differences for each individual cyclone type are also remarkable. Therefore, a rigorous analysis of the accuracy of each sensitivity climatology is necessary before robust guidance and firm recommendations about design strategies for permanent vigilance networks for Mediterranean intense cyclones are issued.

3. Verification method

The verification of climatological sensitivity results is an ever-challenging task. The ultimate goal of these studies is to unequivocally guide policy-makers about plans to redefine routine observational strategies. Therefore, verification is essential to quantify the reliability of the sensitivity products and reduce the risk of inefficient budget allocation. This study aims at testing the reliability of the sensitivity regions identified by the three available climatologies of intense Mediterranean cyclones presented in the previous section. We focus on the regions pointed out by the summarized sensitivity field of each cyclone type since these fields indicate where, on average, errors in the initial conditions which evolve linearly would most rapidly grow and have a major effect on the mature cyclone depth. Thus, this study tests the self-contained information given by the mean sensitivity fields of individual cyclone classes.

The 25 most intense Mediterranean cyclones (25MIC) common to both classifications have been chosen for this test (Table I). These systems are among the 65 most intense Mediterranean cyclones detected in the ERA-40 fields. The highest density of these extreme events occur over the Tyrrhenian and Adriatic seas. Most of the cyclones that reach a mature stage over these areas are likely of alpine origin. In fact, Petterssen (1956) highlighted the southern alpine flank as the most cyclogenetic area in the Mediterranean. Nevertheless, the 25MIC are distributed throughout the Mediterranean region, from the Balearic Sea to Cyprus, including other well-known cyclogenetic areas such as the Aegean and Black seas, which become two of the most active areas in the Mediterranean winter (Trigo et al., 2002).

Table I. Date and time of maximum intensity of the cyclone (formatted as YYYYMMDDHH), circulation (107m2s−1) and region of the 25 most intense Mediterranean cyclones of the database selected for verification.
RegionDateCirculation
Balearic Sea197111091810.9
North Tyrrhenian Sea198701111214.7
 199312261813.4
 197801210611.6
 199212090011.3
South Tyrrhenian Sea199711230611.9
 198701180011.3
 195903130611.1
Adriatic Sea196912060613.1
 196912191812.3
 197111200611.9
 197912311811.1
 196701061210.8
 195901161810.7
Ionian Sea198312020613.1
 197812051811.4
 196612071811.2
 199303270610.9
 197802051210.8
Aegean Sea196212210611.5
 198101170011.2
Black Sea197103031812.7
Turkey196411200611.4
 198012101810.8
Cyprus199302090610.8

For the sake of a confronting reference, the performance of the sensitivity climatologies is tested against the judgement of a human meteorologist with experience in Mediterranean climate. For each case study, the meteorologist was asked to highlight the region where a perturbation in the initial conditions would lead to the largest impact in the forecast cyclone. In the search of a fair comparison, the meteorologist made the selection of the subjective sensitive location over the mean initial conditions of the corresponding cluster instead of the actual precursing fields of the particular cyclonic event. Bear in mind that the products we test were built from a cluster perspective and no specific information about the particular member used here for the verification was used in the generation of the sensitivity fields. After all, a density map of the regions pointed out by the human meteorologist highlights the European Atlantic coasts, specifically the Bay of Biscay, the eastern part of Europe and the Mediterranean Sea as persistently preferred areas (Figure 4).

Figure 4.

Density map of influential areas pointed out by the human meteorologist for the 25 verification cyclone cases. This figure is available in colour online at wileyonlinelibrary.com/journal/qj

Besides the four sensitive locations determined by the human, adjoint and both ensemble climatologies, a fifth location –not highlighted by any of the four methods –is also included in the test as a control measure of the unintended model responses to the specific definition of the initial condition perturbations.

3.1. Definition of the perturbation

For each verification experiment we perturb each prognostic model variable and level at t − 48 h with a Gaussian field, defined as

equation image(1)

where equation image is the Euclidian distance, in kilometers, between each grid point (i,j) and the Gaussian's center, (i,j′). The variance of the Gaussian distribution, s2, is set to 150 000 km2. The amplitude of the perturbation, Aφ, is the mean of the zonal standard deviation, σ, of each prognostic model variable φ at a certain pressure level. The coefficient α, which is set to 0.1, is an amplification parameter that controls the perturbation size and is set to keep the evolution of the perturbation in the linear regime. By guaranteeing the linear evolution of the perturbations we minimize the impact of spurious balancing effects in the verification runs.

Regarding the location of the center of the perturbation, the human and non-sensitivity experiments provide a direct indication of it. For the case of the adjoint and ensemble sensitivities we search for the location that is expected to maximize the impact on the response function, ΔJ. To do so, we compute a pseudo equation image at each grid point (i,j′) as the inner product between the corresponding mean sensitivity field derived for each cluster and a Gaussian standardized perturbation:

equation image(2)

where Sij is the mean sensitivity product for the corresponding cluster derived in each climatology (e.g. Figure 5 for the Ionian Sea class). The measure equation image takes into account the non-uniform spatial distribution of the grid points by considering the area of each grid box (ΔxijΔyij). Finally, the center of the perturbation, (a,b), is set at the grid point where equation image is a maximum:

equation image(3)

and Pijφ;ab following Eq. (1) is used.

Figure 5.

(a) Vertically averaged adjoint-computed sensitivity field (darker shaded colors show larger sensitivities) (adapted from Jansà and Homar, 2006) and (b and c respectively) mean ensemble-1 and ensemble-2 sensitivity field (mb, shaded) for Ionian Sea cluster at t − 48 h. Geopotential height field at 500 hPa (gpm, solid lines) and temperature field at 850 hPa (°C, dashed lines) averaged over all the cluster's members. The center of the perturbation that maximizes the impact on the response function (Eq. (3)) is labeled by *. This figure is available in colour online at wileyonlinelibrary.com/journal/qj

3.2. Evaluation of the linear regime

Doubtless, evaluation of the linear evolution of the adjoint and ensemble sensitivity perturbations is essential to ensure the verification experiments lie within the sensitivity validity range, as both methods are built upon linear assumptions. The duration of the linear regime depends on the size and orientation of the perturbation as well as the particular dynamical situation of the day. We investigate the linearity of the evolution of perturbations over the 500 hPa geopotential height field because it is a main driver of the mid and low tropospheric dynamics and it is widely used in linearity studies, being considered the norm in which the evolution will be most linear. In this study, the relative impact of the nonlinear terms is assessed by monitoring the evolution of twin perturbations (i.e. perturbations of equal amplitude and opposite orientation) under the full nonlinear model. Thus, while the model dynamics are approximately linear, twin perturbations will remain roughly equal and opposite. Buizza (1995) computed the correlation between twin perturbations to explore the breakdown of the linear regime in an operational model from ECMWF (European Centre for Medium-Range Weather Forecasts). However, the correlation measure only takes into account the orientation of the evolved perturbations and does not care about their relative magnitudes. Gilmour et al. (2001) defined the relative linearity of the evolution, θ, as a measure to quantify the degree of linearity which depends on variations in both magnitude and orientation of a perturbation. Therefore, in order to discuss the duration of the linear regime, we use the spatial correlation r and the relative nonlinearity θ, which are both necessary conditions for linear evolution. These indicators are defined as

equation image(4)
equation image(5)

with δ+ (δ) denoting positive (negative) initially anticorrelated perturbations, and cov (var) their covariance (variance). The norm | · | is defined by the inner product (·,·). In a perfectly linear evolution, relative nonlinearity and correlation measures would remain at θ = 0 and r = −1, respectively. For random realizations, they amount to r = 0.5 and equation image (Hohenegger and Schär, 2007b) and r = −0.25 and equation image are typically considered as thresholds for the breakdown of the linear regime (Hohenegger and Schär, 2007a).

3.3. Numerical set-up

The numerical experiments are run with the WRF ARW limited area model. It is a fully compressible, non-hydrostatic model widely used in research and operations (Skamarock et al., 2008). Our simulations use 35 vertical σ levels and 240 × 150 grid points, with 30 km grid spacing. The domain is centered in the Mediterranean region and stretches across northern Africa, Europe and parts of the eastern North Atlantic and the Arabian peninsula (e.g. Figure 6). Initial and boundary conditions are provided by the ERA-40 reanalysis fields. Regarding the physical parameterizations, the WRF Single-Moment 6-class scheme (Hong et al., 2004), including ice sedimentation and other ice-phase parameterizations, is used for sub-grid microphysics calculations. Moist convection is parameterized using an improved version of the Kain and Fritsch (1990) and Kain and Fritsch (1993) schemes, based on testing within the Eta model (Kain, 2004). A modified MRF PBL (Medium Range Forecast Model Planetary Boundary Layer; Hong and Pan, 1996), Yonsei University scheme, accounts for planetary boundary layer processes, whereas the Rapid Radiative Transfer Model (Mlawer et al., 1997) is used to parameterize radiation effects. All simulations for each of the 25MIC span over the last 48 h before the time of maximum intensity of the cyclone (i.e. start at t − 48 h) and use the same numerical set-up.

Figure 6.

Adjoint (a), human (b), ensemble-1 (c), ensemble-2 (d) and non-sensitivity (e) perturbation (gpm, shaded) over geopotential height 48 h prior to the mature state of the cyclone of 2 December 1983 (gpm, solid lines). This figure is available in colour online at wileyonlinelibrary.com/journal/qj

4. Results

For each of the 25 cases listed in Table I, five OSSEs are performed based on: adjoint climatology; ensemble-1 climatology (by Garcies and Homar, 2009); ensemble-2 climatology (by Garcies and Homar, 2010); human meteorologist; and non-sensitivity. In order to illustrate the method and discuss particularities of the verification measures, the first part of this section describes one cyclonic case in detail. The main results of this study are then presented in terms of statistical measures of the performance of each sensitivity estimate for all 25MIC cases.

4.1. Example case: Ionian Sea cyclone

An Ionian Sea cyclone which occurred on 2 December 1983 has been selected to illustrate the verification method. For JHC06, the cluster it belongs to is made up of 52 members and the average precursing synoptic pattern at t − 48 h is characterized by an Atlantic ridge and a wide European trough (Figure 5(a) and (b)). For GHC10, this case belongs to a smaller cluster with 14 members and the centroid geopotential height field at t − 48 h reveals a similar pattern to JHC06, although with a shorter and deeper wave (Figure 5(c)). Specifically, an acute ridge is extended along the European Atlantic coast and the deep trough with positively tilted axis is located over central Europe.

Regarding the t − 48 h sensitivity fields, on the one hand, the adjoint model produces a single sensitivity structure on the northern side of the northwesterly jet accompanying the trough, with special emphasis in the central region of the trough (Figure 5(a)). On the other hand, ensemble sensitivities seem less tightened to dynamically active features. Ensemble-1 sensitivity field shows significant signals pointing towards the westerly flow associated with the Atlantic high-pressure system and also highlights northwestern African lands and parts of the central region of the trough (Figure 5(b)). Ensemble-2 sensitivity climatology produces the two more consistent sensitivity structures centered over the western part of the Iberian peninsula and over the central Mediterranean basin and northern African lands (Figure 5(c)). Furthermore, signals of sensitivity also appear over north-central Europe. Admittedly, some small-scale sensitivity structures may be caused by spurious correlations due to the reduced size of the cluster. Also, bear in mind that for each sensitivity map in Figure 5 the center of the Gaussian perturbation, (a,b)method, is obtained maximizing the expected change in the cyclone's depth produced by a perturbation to the initial conditions (Eq. (3)).

Regarding the human meteorologist decision, mainly based on a quasi-geostrophic thinking and basic advection reasoning, the evolution of this cyclone depends largely on structures located upstream of the main trough, over the northwesterly jet flow. This area is presumed to be decisive in the evolution of the southern half of the main trough and eventually greatly impact the Ionian low-level cyclogenetic process. Therefore, a region over the British Isles was selected as the most sensitive region (i.e. (a,b)human) at t − 48 h (indicated by ‘H’ in Figure 5(b)). Finally, the non-sensitivity perturbation must be located over an area which does not match the sensitivity regions already highlighted by the methods. For this case study, the region between the Canaries and Azores was considered as a reasonable non-sensitivity location (indicated by ‘NS’ in Figure 5(b)) that would not have a direct dynamical connection with the Mediterranean cyclogenesis but could affect the response function by means of spurious perturbation balancing effects due to the particular perturbation definition adopted here.

The main purpose of a climatological sensitivity field is to identify key regions where the recurrent prediction of a particular climatological feature (e.g. a cyclone type) would mostly benefit from observational improvements over that region. The prediction of individual cyclones of a certain type will benefit more from the improved observations as they resemble more the fields used to generate the sensitivity products. As far as the cluster is sufficiently homogeneous, the perturbations added to the IC, for the ensemble methods, lie in a dynamical region coherent with the cluster fields used to compute the climatological sensitivities. For example, note the better accordance between the case study and the GHC10 cluster mean fields due to the increased homogeneity of the GHC10 cyclone classes compared to those of JHC06 (Figure 6, ensemble-1 vs. ensemble-2). A remarkable mismatch is obtained for the ensemble-1 sensitivity perturbation when it is placed over the actual case study IC fields. The main structure of sensitivity (and thus the (a,b)ensemble-1) is shifted to the western part of the Atlantic ridge when placed over the actual IC of the case study. This key challenge of climatological sensitivity products reflects the fundamental complexity of building long-term plans for network designs focusing on specific types of events.

Once the perturbations are added to the initial conditions, following Eq. (1), 10 perturbed experiments, taking into account the twin (positive–negative) perturbations, and the control run are rendered. As discussed in section 3, the linear evolution of the adjoint and ensemble perturbations must be tested. For completeness, the linearity study is also performed over the human and non-sensitivity perturbations. Therefore, the spatial correlation (Eq. (4)) and the relative linearity (Eq. (5)) are computed for initially anticorrelated perturbations added to the 500 hPa geopotential height field. The temporal evolution of these linearity indicators reveals that all perturbations remain in the linear regime throughout the 48 h simulation (Figure 7). Then, the adjoint and ensemble perturbations evolve within the linear regime as hypothesized in their respective theoretical formulations. Therefore, we take advantage of this and only the positive perturbations are considered for further computations.

Figure 7.

Time series of the (a) correlation coefficient and (b) relative linearity between twin positive–negative perturbation pairs for 2 December 1983. This figure is available in colour online at wileyonlinelibrary.com/journal/qj

One way to address the information derived from the simulations is to quantify the impact of each perturbation on the forecast aspect of interest. Despite all sensitivity fields being computed using a measure of the cyclone's depth as the response function, its definition is not common between climatologies. For the sake of fair comparison, it is necessary to use a common measure focused on this particular aspect of the predicted cyclones to evaluate the perturbation's impact. Here, this measure is defined as the root mean square difference (RMSD) between the MSLP field of each perturbed simulation and the control one over a region centered on the cyclone's center of the control run. Bearing in mind that over 65% of the Mediterranean cyclones have a maximum radius less than 550 km (Trigo et al., 1999), we compute the RMSD over a circle of 350 km of radius centered on the control cyclone center in order to obtain an accurate measure of the variation of the MSLP near to the cyclone's center in concordance with the definition of the response function.

Despite sensitivity estimates in the tested climatologies being, stricto sensu, informative about changes in the cyclone at time t (final simulation time), the RMSD is computed every 6 h throughout the 48 h simulation period (Figure 8). Thus, the RMSD computation region follows the track of the cyclone in the control run. This gives an estimate of the evolution of the differences throughout the cyclogenesis between the perturbed simulations and the control one. The perturbations defined in Eq. (1) introduce initial imbalance and rapid model adjustment exciting acoustic and gravity waves that last for about 6–12 h in the model integrations. This is reflected in the rapid/sudden growth of the RMSD at 6 h and the subsequent decay. Eventually, the dynamically adjusted part of the perturbations remains and evolves according to the dynamics of the day. As expected, poor RMSD values from the non-sensitivity simulation are obtained for the whole simulation period, indicating that the perturbed area does not have a relevant influence on the cyclone deepening, and the spurious balancing effects are also negligible. However, the ensemble-1 perturbation does not produce much better results. On the other hand, the other sensitivity perturbations produce higher values of RMSD and show an overall increasing trend. Equivalent results are obtained (not shown) when the WRF Data Assimilation system is used to assimilate soundings with similar characteristics to the described perturbations, revealing the consistency of the results despite the particular definition of the perturbation adopted in this study.

Figure 8.

Time series of the RMSD between perturbed and control simulations for the MSLP field over a region centered on the cyclone's center of the case of 2 December 1983. This figure is available in colour online at wileyonlinelibrary.com/journal/qj

Since the verification deals with 48 h sensitivity fields, we mainly focus on the impact of the perturbation on the MSLP at the final time step (Figure 9). At the cyclone's mature stage, the human experiment produces the highest variation on the cyclone's central MSLP field, although the perturbation is also significant downstream of it. The human-driven perturbation consists basically of an extensive dipole around the mature cyclone region, which indicates a northeastward shift of the cyclone structure in the perturbed simulation (Figure 9). The adjoint sensitivity perturbation produces a deeper cyclone slightly shifted to the north and the change is comparable to the one obtained in the human experiment (Figure 9). In this case, the perturbation also has effects far to the northeast of the control cyclone, producing, in contrast, a higher-pressure area. Regarding the ensemble-2 sensitivity experiment, a weak impact is found around the control cyclone center and signals of a lower pressure area are also obtained near the Atlantic European coasts (Figure 9). Finally, the lowest RMSD obtained by the non-sensitivity and ensemble-1 perturbations confirm the expected insignificant difference between perturbed and control simulations over the mature cyclone center (Figure 9). Note that the poorest impacts over the mature cyclone center are obtained when IC perturbations are not dynamically linked with the main trough. These results indicate that the adjoint and human sensitivity estimates indeed have skill in identifying influential areas for the formation of the intense cyclone well above the impact of identical perturbations located over other areas. Therefore, for the most important question to be addressed here, the adjoint and human estimates of climatological sensitivity would render larger benefits when used as guiding ruler to redesign observation networks for this particular case of 2 December 1983 than the other methods analyzed.

Figure 9.

Difference between MSLP of the control simulation (mb, solid lines) and the adjoint (a), human (b), ensemble-1 (c), ensemble-2 (d) and non-sensitivity (e) simulations (mb, shaded) at the time of maximum intensity of the Ionian Sea cyclone of 2 December 1983. The circle indicates the region used for RMSD computations.

4.2. General results

By definition, climatological sensitivities ought to be verified over a long-term climatology of events. Although a significant improvement for all intense Mediterranean cyclones would be ideal, climatological sensitivities are strictly not expected to help improve the prediction of each individual member of a cluster but to render significant average improvements across the cluster (i.e. Mediterranean intense cyclone type). In order to have a general outline of the performance of the adjoint and ensemble sensitivity climatologies, as well as the human estimates, the procedure described in section 4.1 is reproduced for the 25 most intense Mediterranean cyclones which belong to all climatologies (i.e. 25MIC).

For each case study, the duration of the linear regime is evaluated as in section 4.1. The correlation and relative linearity between twin positive–negative perturbation pairs on the 500 hPa geopotential height reveal that, in spite of a few perturbations drifting away from the linear regime, adjoint and ensemble sensitivity perturbations evolve, in general, in the linear regime and so do the human and non-sensitivity perturbations (Figure 10 and 11). Therefore, the adjoint and ensemble perturbations used here fit the linear assumptions of their respective theoretical foundations.

Figure 10.

Time series of the relative linearity between twin positive–negative perturbation pairs for all 25 verification cyclone cases for the adjoint (a), human (b), ensemble-1 (c), ensemble-2 (d) and non-sensitivity (e) experiments. The bold solid line indicates the mean relative linearity over the 25 cyclone cases. This figure is available in colour online at wileyonlinelibrary.com/journal/qj

Figure 11.

Time series of the correlation coefficient between twin positive–negative perturbation pairs for all 25 verification cyclone cases for the adjoint (a), human (b), ensemble-1 (c), ensemble-2 (d) and non-sensitivity (e) experiments. The bold solid line indicates the mean correlation coefficient over the 25 cyclone cases. This figure is available in colour online at wileyonlinelibrary.com/journal/qj

In order to summarize all information derived from the 25MIC, a mean normalized RMSD index over the 25 verification cyclone cases is computed. Note that the dynamics involved in each cyclone typology are different and perturbations are naturally going to grow larger for certain cyclone types than for others. Thus a simple averaging of RMSD would skew the final results towards the methods which have larger skill in cases with large perturbation growth. In order to take into account the natural dynamical variability among cyclone classes and remove this effect, the RMSD of the MSLP of each case is normalized using the standard deviation value among sensitivity estimates. The average RMSD value for each method and time step is then computed over the 25 verification cyclone cases. The standard error of this mean is also computed and represented together with the 25MIC mean normalized RMSD value given an estimate of the confidence interval of the results (Figure 12).

Figure 12.

Time series of mean normalized RMSD over the 25 verification cyclone cases. This figure is available in colour online at wileyonlinelibrary.com/journal/qj

As expected, the non-sensitivity experiments produce the poorest average changes on the Mediterranean cyclones. This is less than half the impact of the adjoint and human experiments, evidencing the ability of these methods in identifying reliable sensitivity areas. In fact, adjoint and human methods show similar skill, as revealed by the insignificant differences between them on the mean normalized RMSD results at the time of cyclone maturity. However, some disappointment has followed the results for the ensemble sensitivity experiments that clearly make explicit the insurmountable limitations of this sensitivity calculation method in identifying the climatological sensitivity regions. While the ensemble-1 experiments produce slightly better results than those of non-sensitivity experiments, the ensemble-2 climatology shows a noticeable improvement upon these results without reaching the superior skill of adjoint and human methods. This outstanding improvement is probably derived from the ensemble-sensitivity-oriented classification used to compute the ensemble-2 sensitivity fields (Table II).

Table II. Main characteristics of the considered sensitivity climatologies of Mediterranean intense cyclones. T indicates temperature, U and V the components of the wind, UV wind speed, PP pressure perturbation, Q specific humidity, H geopotential height and R relative humidity.
Sensitivity methodMediterranean intense cyclone classificationSensitivity products
Adjoint1202 cyclones classified into 25 clustersTimes: t − 48 h.
 Clustering steps: regional classification,IC considered: T, U, V, PP, Q
 subclassification considering precursor conditions.for the centroid.
 Jansà and Homar (2006)J: centroid cyclone's depth.
  Jansà and Homar (2006)
Ensemble-11202 cyclones classified into 25 clustersTimes: t − 48 h and t − 24 h.
 Clustering steps: regional classification,IC considered: H, T, UV.
 subclassification considering precursor conditions.J: individual MSLP
 Jansà and Homar (2006)at centroid's cyclone center.
  Garcies and Homar (2009)
Ensemble-2406 cyclones classified into 23 clustersTimes: from t − 48 h to t − 6 h
 Clustering steps: regional classification,every 6 h.
 subclassification considering precursor conditions,IC considered: H, T, UV, R.
 cluster pruning.J: individual cyclone's depth.
 Garcies and Homar (2010)Garcies and Homar (2010)

4.2.1. Significance tests

In order to make sure that the reliability of these global verification results is not at stake by the spurious balancing effects of the particular perturbation definition adopted here, we perform a statistical study of the significance of the verification results for the 25MIC. On the one hand, we test whether there is no difference in means between the distributions of the normalized RMSD of the five perturbation methods at time t (Figure 13). In other words, the null hypothesis states that the five perturbation methods produce the same average change in the mature cyclone's depth, i.e. a mean value of 0.2 for the normalized RMSD distribution. This is the expected result if random noise is assumed to mark the underlying perturbation signal. The results of the significance test over all 25MIC reveal that the adjoint and human methods show higher normalized RMSD mean than the one established by the null hypothesis, while the ensemble-1 and non-sensitivity means of the normalized RMSD are lower than 0.2 with 99% confidence. Lastly, the ensemble-2 normalized RMSD mean is not significantly distinguishable of an equal distributed response. On the other hand, we also test whether or not the observed frequency distribution of succeeds is significantly different from an equally likely distribution, defining a success as the ability of a sensitivity method to produce the highest change over the mature cyclone's depth for a verification case. As shown in Figure 13, adjoint and human methods produce the highest normalized RMSD in 17 cases, while the other three methods show the best skill, identifying sensitivity areas only in eight verification cases. Given the demonstrated significant skill of the human and adjoint methods in producing higher average changes over the cyclone's depth, we test the null hypothesis stating that the frequency distribution of successes of these two methods is consistent with a fair distribution of frequency of successes with the other three methods. By means of Pearson's chi-square test, the null hypothesis of an equal distribution of successes is rejected with 99% confidence. Therefore, the human and the adjoint sensitivity methods reveal a statistically significant skill in identifying reliable sensitivity regions for Mediterranean intense cyclones.

Figure 13.

Bars represent the normalized RMSD for each verification case at time t. The height of each segment is proportional to the RMSD of the perturbation method it represents. Dots indicate which method produce the highest normalized RMSD in each case.

4.2.2. Verified adjoint sensitivities

Once the adjoint sensitivity products are identified as the most reliable of all objective climatological sensitivities considered here, the global-averaged adjoint sensitivity map shown in Figure 1 is thus the best available linear estimate of the climatological short-range sensitive areas of hazardous intense cyclones across the Mediterranean. Admittedly, the verification procedure considers a limited sample of 25 case studies, and many untested events contribute to the synthetic climatological product shown in Figure 1, leaving a small margin for doubts about the representativity of the RMSD scores for the complete intense Mediterranean cyclone catalogue. Despite 25 OSSEs providing a robust representation of the catalogue, and thus Figure 12 being safely considered an informative measure of the overall adjoint climatology results, these 25MIC have a relevant role per se when planning new observational means as representatives of the most intense cyclones in the database. Consequently, we compute the sensitivity field averaged over the 25MIC as an illustrative product of strictly verified climatologically sensitive areas of extremely intense (among the 0.2% most intense cyclones detected by Campins et al., 2010) Mediterranean cyclones (Figure 14). This product highlights the western and central Mediterranean as important regions with extensions towards the eastern Atlantic and North African coastlands, although the maximum values of sensitivity for the 25MIC lie along the Italian peninsula and surrounding seas.

Figure 14.

Mean adjoint sensitivity field computed over the 25 verification cyclone cases (darker colors indicate higher sensitivities). This figure is available in colour online at wileyonlinelibrary.com/journal/qj

5. Conclusions

In recent years, the growing interest in improving the permanent observing networks to support weather prediction offices in issuing more valuable and timely forecasts has led to the emergence of objective sensitivity calculations. In favor of efficiency, special attention is placed on the sensitivities of high-impact weather. As a matter of fact, Jansà and Homar (2006) and Garcies and Homar (2009, 2010) built sensitivity climatologies of Mediterranean intense cyclones with a certain degree of disagreement among them. None of these key results for future plans of observational means redesign was extensively tested and verified.

Doubtless, the verification of sensitivity results is an ever-challenging task and it is essential to provide a reliable guidance for decision makers to focus on areas where an increased observational effort would significantly improve the quality and value of short-range numerical weather predictions. Here we make use of OSSEs to evaluate the quality of these sensitivity climatologies derived using adjoint and ensemble techniques. The impact of synthetic perturbations applied to realistic forecasting models for cases of Mediterranean intense cyclones is adopted as a verification testbed to contrast the skill of the climatologies.

The 25 most intense Mediterranean cyclones common to all sensitivity climatologies are chosen to test the ability of adjoint and ensemble sensitivity methods in identifying areas where perturbations in the initial conditions produce the largest impact on the forecast intense cyclone. For the sake of a relative calibration measure for the verification results, a reference sensitivity proxy is used. It consists of the indications of an experienced severe weather meteorologist who was asked to point at the region where a perturbation in the initial conditions would have the largest impact on the forecast cyclone's depth. In addition, a control experiment is also performed to test the effects of our particular definition of the perturbation shape and amplitude. In this sense, a non-sensitivity area is perturbed to obtain a measure of the unintended random responses due to imbalances induced by the perturbations in the initial conditions. Needless to say that this verification framework does not evaluate the impact of a particular observing system but is aimed at quantifying the value of climatological sensitivity products in improving the prediction of Mediterranean intense cyclones. A similar procedure is followed by Torn and Hakim (2009), who introduced a perturbation amplitude over each grid point and evaluated the impacts on the forecast metric. In contrast, Torn (2010) quantitatively validated ensemble sensitivity values assimilating a hypothetical observation. Note that both studies show that the predicted response in the forecast metric matches the predicted change given by the ensemble sensitivity technique in its genuine version (i.e. using an ensemble of simulations).

The perturbations are defined in such a way that they evolve linearly at all time-spans, which is a fundamental hypothesis of the adjoint and ensemble sensitivity calculation methods. We evaluate the evolution of twin perturbations to determine the breakdown of the linear regime and the safe time-span for the sensitivity products. Specifically, we quantify the linearity of the evolution of the perturbations, computing the relative linearity and correlation of twin perturbations which take into account both amplitude and orientation of the perturbations. The results show that a great majority of perturbations evolve linearly, and so the nonlinear effects on their evolution are very unlikely altering and masking the verification results.

Regarding the performance of the considered sensitivity methods, the adjoint sensitivity fields and human judgement show the best skill in identifying actual sensitive areas for the forecasting of the 25 Mediterranean intense cyclones under study. Given the absence of a human-based sensitivity map, but only a density distribution of targeted regions, the adjoint sensitivity fields are shown to be the most reliable source for objective climatological sensitivities for Mediterranean intense cyclones. It is noteworthy that Jansà and Homar (2006) derived adjoint sensitivity fields from a climatological perspective; i.e. for each of the cyclone classes the set of initial and boundary conditions for the MM5 adjoint simulations were averaged fields over all the individual members of the clusters to obtain a representative simulation of the cyclone class. On account of this, better climatological results may be expected if the adjoint model is applied on each member of a cluster and mean sensitivity fields are produced. However, the extremely high cost of running the adjoint model for such a large number of events becomes a severe limitation to accomplish this. On the other hand, Garcies and Homar (2009) proposed an alternative cheap approach to climatological sensitivity analysis that produced results allegedly competitive with adjoint sensitivity fields. However, the verification results of these ensemble sensitivity fields only exceed those of non-sensitivity experiments, showing an insufficient skill in identifying influential areas. Garcies and Homar (2010) attempted to improve the quality of climatological ensemble sensitivity results, building an ad hoc classification of Mediterranean intense cyclones. Indeed, this notable improvement is confirmed through the verification test for the 25MIC, which shows significantly higher responses on the forecast cyclone for the latter climatology. Nevertheless, some cyclone classes contain too few members for the statistical results and are subject to sampling errors. The application of this statistical sensitivity method to rare events such as some classes of intense cyclones, together with the use of databases covering relatively short time spans, makes explicit one fundamental limitation of this technique in climatological mode for Mediterranean intense cyclones. Hopefully, longer databases available in the future will undoubtedly overcome this handicap.

The findings in this study pave the way for the construction of a climatology of sensitivities of high-impact weather on a Mediterranean, European or even global scale. The infrequent task of verifying sensitivity products is tackled here on perhaps the most complex product, that of climatological sensitivities. The results confirm the statistically significant superior ability of the adjoint dynamical methods against statistical approximations given the currently available datasets. The generation of sensitivities within a general nonlinear context is a latent research line as operational forecasting systems operate at higher spatial resolutions and longer time-spans. This complete paradigm change is left to future research.

Acknowledgements

The ECMWF is acknowledged for providing the ERA-40 reanalysis. The authors also thank the Mediterranean Studies Section of the Centro MeteorológicoTerritorial en Illes Balears of the Spanish Weather Service (AEMET) for building the cyclone database and the two anonymous reviewers for their helpful comments, which have helped to improve this text substantially. This research has been supported by MEDICANES project (CGL2008-01271/CLI). L. Garcies also acknowledges support from the Spanish MEC through FPU grant (AP2007-01367).

Ancillary