Do subseasonal forecasts take advantage of Madden–Julian oscillation windows of opportunity?

This study proposes an objective methodology to highlight windows of opportunity in a numerical subseasonal forecasting system. The methodology is based on a contingency table and is applied to the prediction of heavy tropical precipitation by the European Centre for Medium‐range Weather Forecasts (ECMWF) subseasonal‐to‐seasonal (S2S) reforecasts in the November‐to‐April season, in relation with the Madden–Julian oscillation (MJO). As a slowly propagating signal of enhanced convection, the MJO may indicate favorable conditions for heavy precipitation a few weeks ahead in some tropical areas. The combined knowledge of these climatological impacts and the current phase of the MJO at initialization defines observation‐based “climatological windows of opportunity.” We then investigate whether the ECMWF S2S forecasts are indeed more performant when there is increased likelihood of heavy rainfall, that is, whether the model converts “climatological windows of opportunity” into “model windows of opportunity.” Our results show that, by Week 2, this is only verified for a limited number of tropical areas, mostly located in the western Pacific and Africa. Meanwhile, failures to seize the opportunities lie in misplaced MJO impacts, signal loss, or too many false alarms.


| INTRODUCTION
Despite the growing interest for numerical forecasting in the subseasonal-to-seasonal range (S2S, from 2 weeks to 2 months approximately), the average quality of these forecasts is often modest after the first week (e.g., Hudson et al., 2011;Malloy & Kirtman, 2020). It is increasingly recommended that applications of S2S and longer climate predictions should rely on specific conditions, called "windows of opportunity," when the predictable signal extends beyond weather timescales (Mariotti et al., 2020). The concept of "window of opportunity" acknowledges that skill mostly comes from periods of higher signal-to-noise ratio, to be identified, for example, through statistical models (Albers & Newman, 2019;Mayer & Barnes, 2021) or preexisting information about certain phenomena.
The Madden-Julian oscillation (MJO) is the most prominent mode of subseasonal climate variability at the global scale. It consists in the eastward propagation of zonal wind anomalies and a deep convection center (the convective envelope) around the globe in the equatorial band at the average speed of 5 mÁs À1 , for a total duration of 30-90 days (Zhang, 2005). Because the MJO is a slowly propagating signal, the location of the convective envelope may indicate heavy precipitation in other areas a few weeks ahead (Lefort & Peyrillé, 2020). Thanks to these lagged relationships, periods of active MJO are expected to provide windows of opportunity for S2S prediction, since MJO phases are typically precursors of rainfall east of the positive convection anomalies.
Consequently, a wide range of studies have examined if subseasonal forecasting models performed better at predicting precipitation under certain MJO initial conditions. The general framework of these studies is to make comparisons between the prediction scores in several sub-samples from a set of reforecasts. A first approach is to distinguish between forecasts initialized in active versus inactive MJO conditions (e.g., Marshall et al., 2011;. A second approach is to stratify according to the initial MJO phase and consider the phases coinciding with the most skillful predictions in the area of interest (e.g., Jones et al., 2011;Vigaud et al., 2018;Vigaud, Robertson, Tippett, & Acharya, 2017). Meanwhile, these approaches have also been applied to assess forecasts of other variables such as atmospheric rivers (Baggett et al., 2017), geopotential (Tseng et al., 2018), and the north atlantic oscillation (Feng et al., 2021). The conclusions of these studies may substantially differ depending on the target variable, the region of interest, and the numerical model. Moreover, the windows of opportunity identified from a simple comparison of conditional scores are sometimes difficult to interpret in light of the propagating MJO signal.
In this study, we propose a two-step objective method to identify windows of opportunity in a numerical subseasonal forecasting system. First, we assess whether an increase in "Base Rate" (i.e., frequency of occurrence) of heavy rainfall events, 2 or 3 weeks after a favorable precursor MJO phase, leads to more "hits" (i.e., correctly detected events) in the S2S forecasts compared to a situation with no MJO signal. Then, we analyze if periods with more "hits" result in windows of higher forecast quality, when "false alarms" are also taken into account.
After a preliminary description of the data and verification methodology (Section 2), Section 3.1 describes the lagged relationships between the MJO and heavy rainfall at Weeks 2 and 3 across the Tropics. The two-step method is illustrated in Section 3.2, while Section 3.3 refines the results from Section 3.2 at the regional scale for a range of selected areas. Finally, a summary of the main findings can be found in Section 4.

| Forecast and observation data
This study uses daily precipitation data in the ensemble reforecasts provided to the S2S project database (Vitart et al., 2017) by the European Centre for Medium-range Weather Forecasts (ECMWF). We use the reforecasts computed on the fly in relation with the real-time forecasts between November 2, 2020 and April 29, 2021. In doing so, we cover 20 November-April seasons from 2000-2001 to 2019-2020 with model configuration CY47R1 1 . The November-April season was chosen as an extended austral summer season with a peak in MJO activity (Madden, 1986) and impacts (Zhang, 2013). There are 52 start dates per season (separated by 3 or 4 days) and a total sample of 52 Â 20 = 1040 start dates. The ensemble size is 11 members. Precipitation is extracted on the 1.5 archiving grid of the S2S database over the whole tropical band (30 S-30 N). ECMWF precipitation reforecasts are verified against daily precipitation data from the Global Precipitation Project Database (GPCP) version 1.3 (Huffman et al., 2001), which goes from October 1, 1996 onward. GPCP data were bilinearly interpolated from their original 1 grid to the 1.5 S2S grid in order to match the reforecast data.

| Heavy precipitation events
A heavy precipitation event is defined to occur at day d and grid point g if the average precipitation over the weekly period [d, d + 6] exceeds the upper quintile (80th percentile) of the local climatology for the corresponding calendar week. GPCP reference data and ECMWF ensemble members are averaged on 7-day moving windows, such that forecasts at d days lead correspond to the days [d, d + 6] average, while forecasts at d + 1 days correspond to the days [d + 1, d + 7] average. The focus is particularly laid on Day 12 (average of Days 12-18) and Day 19 (average of Days 19-25), for which the forecasts match the usual Week 2 and Week 3 from a common S2S convention (Coelho et al., 2018;. Seven-day averaged observations and ensemble member values are then converted to binary data indicating whether they exceed the upper quintile threshold computed from their own climatology, and a probabilistic forecast is defined with the fraction of members in which the event occurs. Forecast and observed climatology are both determined using a leave-one-year-out cross-validation (i.e., 20 À 1 = 19 years) following the same approach as . The forecast climatology sample size at a given lead time is therefore 19 years Â 11 members = 209, while the observed climatology sample size is 19 years Â 3 weeks = 57 (obtained by using 1 week before and 1 week after, in addition to the corresponding calendar week). Grid points for which the average observed upper quintile over the full seasonal cycle does not exceed 7 mm in a week (1 mmÁday À1 ) are removed from the analysis.

| Contingency table and verification
The methodology to highlight windows of opportunity relies on several indicators from a contingency table (Hogan & Mason, 2012). A schematic example of contingency table is reproduced in Table 1.
• the Base Rate corresponds to the observed frequency of the event in the sample, and is independent of the forecast: • the Forecast Rate corresponds to the forecast frequency of the event in the sample, and is independent of the observations: • the Hit Rate is the percentage of correct forecasts knowing that the event of interest occurred in observations: • the False Alarm Rate is the percentage of forecasts predicting the event although it did not occur in observations: • the Peirce Skill Score (PSS) is the difference between Hit Rate and False Alarm Rate. It summarizes the overall ability of the forecasting system to predict the event correctly by representing the trade-off between hits and false alarms: In this study, to compare a probabilistic ensemble forecast to a single binary observation, the counts in the four categories a, b, c, and d are computed using the probabilistic contingency table approach introduced by Gold et al. (2020). Moreover, in order to remove spatial noise when using map representations (Figures 1-3), the indicators are computed from a contingency table obtained by pooling 3 Â 3 grid points in a neighborhood around the central grid point.

| Conditioning on initial MJO phase
The aim is to compute the indicators for sub-samples of the reforecasts according to specific initial MJO conditions. These initial conditions can be inactive MJO, or active MJO in phases 8-1, 2-3, 4-5, or 6-7, following the eight-phase classification from Wheeler and Hendon (2004). The active MJO phases are grouped by two in order to increase sample size, in a way which is consistent with well-defined geographical locations of the convective envelope: 2-3 for the Indian Ocean, 4-5 for the Maritime Continent, 6-7 for the Western Pacific, and 8-1 for the Western Hemisphere and Africa. Inactive MJO periods correspond to a real-time multivariate MJO amplitude (see Wheeler & Hendon, 2004) less than 1. For each reforecast start date, the initial MJO phase is the MJO phase at Day 1 provided by the Australian Bureau of Meteorology at http://www.bom.gov.au/climate/mjo/ graphics/rmm.74toRealtime.txt.
Once the ensemble forecasts are stratified according to their initial MJO condition, the metrics from the contingency tables (Section 2.3) are computed for each subsample. The aim is to compare the metrics between initially active MJO in any of the four phases (8-1, 2-3, 4-5, 6-7) and initially inactive MJO. The sub-sample sizes are given in Table S1. To assess the significance of differences, we use the 90% confidence interval of the metrics. If the value for the inactive MJO sample does not lie within the confidence interval of the phase-specific MJO sample, the difference is deemed significant.
The 90% confidence intervals are estimated with a bootstrap procedure by resampling the corresponding start dates 1000 times with replacement. Taking the example of the Hit Rate of forecasts initialized in MJO phases 8-1 (sample size 132), 1000 resamplings of the 132 forecasts are verified against their corresponding observations: the confidence interval lies between the 5th and 95th percentiles of the bootstrapped Hit Rate values.

| Climatological windows of opportunity
It is widely documented that the MJO increases tropical rainfall where the convective envelope is located (e.g., Zhang, 2013). This is verified for mean precipitation, but also for the upper tail of the distribution with an increase in the frequency of extremes. The left column of Figure 1 illustrates the locations where a significant increase in the Base Rate of the upper quintile of weekly precipitation can be observed in GPCP data, simultaneous to the MJO phases indicated on the left and compared to inactive MJO. The propagating signal is clearly visible, and is in good agreement with the MJO-induced precipitation anomalies such as those described by Zhang (2013) (Figure 1). The middle and right column show a similar analysis when considering the weekly GPCP observations, respectively, 12 and 19 days after the ECMWF reforecast start dates. The first key point is the gradual decrease in the extent of the areas experiencing a higher frequency of heavy precipitation. This highlights the natural limits of the MJO as a precursor for heavy rainfall: the MJO is not a perfectly regular wave and it often returns to an inactive state before completing its circumglobal propagation. The second key point is the eastward shift of the areas where the Base Rate increases, compared to the leftmost figures. This is in agreement with the expected displacement of the convective envelope.
The lagged relationships between the MJO and heavy rainfall-illustrated in the middle and right columns of Figure 1-indicate "climatological windows of opportunity": by relying only on the past observational record, a forecaster is able to know that heavy rainfall will be more likely at day d + 2 or 3 weeks considering the observed state of the MJO at day d.

| Model windows of opportunity
The question that arises when climatological windows of opportunity are identified is whether the S2S prediction system benefits from them by issuing more performant forecasts, thus defining "model windows of opportunity." The first element to investigate is the improved detection of the events that occur in agreement with an increased climatological probability (e.g., a favorable precursor MJO phase), compared to those occurring in the absence of signal (e.g., an inactive MJO period). This is examined with the first two columns of Figure 2, by comparing the increase in Base Rate and Hit Rate for Week-2 lead time (Week 3 is shown in Figure S1). Most of the time, the areas with a higher Hit Rate are included within areas where the Base Rate is also higher, such as East Africa after phases 8-1 and 6-7, the central Pacific Ocean after phases 2-3, and the southwest tropical Pacific after phases 4-5. Nonetheless, we note a slight reduction in the extent of significant areas between Base Rate and Hit Rate, for instance over the Indian Ocean in phases 8-1 and over the Maritime Continent in phases 2-3. In these locations, the F I G U R E 1 Base rate anomalies of the upper quintile of weekly precipitation for the four active MJO phases, relative to inactive MJO. Only significantly positive anomalies (based on bootstrapping) are shown, while non-significant or negative anomalies are left blank. Left column: Base rate simultaneous to the MJO phase. Middle and right column: Base rate, respectively, 12 days (lead time Week 2) and 19 days (lead time Week 3) after the MJO phase. Grid points in dry areas removed from the study (upper quintile <1 mmÁday À1 ) are in gray expected improvement of heavy precipitation detection does not appear anymore in the ECMWF S2S forecasts as early as Week 2.
The simultaneous increase in Base Rate and Hit Rate can be considered as a "Type 1 model window of opportunity," showing that the forecasting system is able to tap into predictable subseasonal signals. Yet, it might not be enough to identify fully actionable forecasts if more hits are associated with too many false alarms. This is why higher overall forecast quality is also explored   (Figure 2, third column). It clearly appears that a lot of grid points with a significant increase in Hit Rate do not show a significant increase in PSS. This is particularly true for Africa and the Indian Ocean, in phases 8-1 or 6-7. It can be assumed that in such cases, the forecasts take into account the MJO signal but tend to overpredict heavy precipitation when conditions are favorable. As a whole, the results for PSS are rather disappointing: by Week 2 there are very few large and consistent areas where PSS is enhanced, and most grid points with significant PSS improvement are located in the ocean with no inhabited land.
In order to summarize Figures 2 and S1, Figure 3 classifies the grid points according to the relationships between climatological and model windows of opportunity for each MJO phase, at Day 12 and Day 19. Three categories of grid points are identified: • Grid points with a Type 1 model window of opportunity (in orange): a significant increase in Base Rate leads to a significant increase in Hit Rate but no significant increase in PSS on account of an excessive False Alarm Rate.
• Grid points with a Type 2 model window of opportunity (in brown): a significant increase in Base Rate leads to a significant increase in both Hit Rate and PSS. • Grid points with a significant increase in both Hit Rate and PSS that cannot be explained by a significant increase in Base Rate (in yellow).
A general result from Figure 3 is that the largest areas of significant Hit Rate increase from Figures 2 and S1 are compounds of core regions experiencing Type 2 opportunities in brown, surrounded or connected by grid points experiencing Type 1 opportunities (with too many false alarms) in orange. "Yellow" areas are either isolated or located at the edge of the Type 1 and Type 2 regions. They often reveal the misplacement or shifting of the MJO impacts in the forecasts.

| Regional scale analysis
The aim of this section is to illustrate how the analysis of Figures 2 and 3 can be refined at the local scale, which is F I G U R E 4 Day-by-day evolution of the five contingency table indicators (Base Rate, Forecast Rate, Hit Rate, False Alarm Rate and PSS) of the upper quintile of weekly precipitation in the 7 selected regions indicated by rectangles on Figure 3. The evolution is shown for forecasts initialized with a specific MJO phase (colored curve, see title for phase number), for forecasts initialized in inactive MJO conditions (black curve) and for all forecasts (gray curve). Significant differences between the active phase (color) and inactive MJO (black) are shown in full bold line. The gray dotted vertical lines indicate Week 2 and Week 3 lead times essential for potential applications. From Figure 3, we select seven inhabited areas across the tropical band (black rectangles) so as to sample different categories of positions along the MJO track. They are all encompassed in 12-gridpoint boxes so that the contingency tables are computed on the same sample size for all areas. From west to east, we name these areas according to the main islands, countries, or territories they encompass: Uganda (UG), Comoros (COM), Java (JAVA, Indonesia), Darwin (DWN, Australia), New Caledonia (NC), Samoa (SAM), and Northeast Brazil (NEB). Their coordinates are given in Table S2. Figure 4 shows how the five metrics from Section 2.3 evolve with lead time in each of these zones, for a subsample of forecasts initialized with a specific MJO phase. It also shows the comparison with forecasts initialized when the MJO is inactive, and with the whole set of reforecasts. For the sake of brevity, only one MJO phase was chosen for each zone on the basis of Figure 3. If the metric for the specific MJO phase (in color) is significantly different from inactive MJO (in black), the curve appears as a full bold line. Note that because the weekly windows for two consecutive daily lead times overlap by 6 days, we consider the windows of opportunity are robust when they last for at least 7 consecutive days.
Compared to the "snapshots" provided by Figures 2  and 3, Figure 4 illustrates the whole extent of the windows of opportunity on the full range of forecast times. Furthermore, it provides finer information about: • the representation of the MJO impacts by the model ( In Figure 4, we observe a similar behavior for Uganda following phases 6-7 (Figure 4a), Comoros following phases 8-1 (Figure 4b), and Darwin following phases 2-3 ( Figure 4d). There is a peak in the occurrence (Base Rate) of weekly precipitation in the upper quintile between 2 and 3 weeks after the given phase. Such an increase is reproduced, although weakly, by the ECMWF reforecasts (Forecast Rate). As a result, the increase in Base Rate translates into a model window of opportunity at Weeks 2 and 3 for the Hit Rate. However, the False Alarm Rate tends to evolve in close relationship with the Forecast Rate and this compensates the increase in Hit Rate, such that there is no significant improvement of the PSS, apart from a few days around Week 2. Consequently, the opportunity in Hit Rate fails to convert into an opportunity in skill: it is only a Type 1 opportunity as illustrated by the orange points in Figure 3. On the contrary, Samoa (Figure 4f), which shows the same type of Base Rate peak at Weeks 2 and 3 after MJO phases 4-5, undoubtedly exhibits a Type 2 opportunity with a robust peak in Hit Rate and PSS around Week 2, thanks to a False Alarm Rate that is no higher than average. A similar conclusion can be drawn in the Northeast Brazil region (Figure 4g) between Day 5 and Day 15 after MJO phases 8-1.
Finally, Java ( Figure 4c) and New Caledonia (Figure 4e) are "yellow" areas where the increase in Hit Rate and/or PSS is not related to an increase in Base Rate (around 3 weeks after phases 2-3 for Java, between 2 and 3 weeks after phases 4-5 for New Caledonia). This is particularly striking for New Caledonia, for which the improved performances actually coincide with a significant decrease in the probability of heavy precipitation. It means that some rarer events are more easily detected than average by the ECMWF forecasts. The interpretation is that the forecasts tend to unduly predict more intense precipitation than observed, as suggested by the comparison between the Base Rate and the Forecast Rate curves.
It should be stressed that Figure 4 highlights significant model windows of opportunity thanks to the large start date sub-samples. However, it must also be acknowledged that the maximum improvement of Hit Rate or PSS is small (around 0.1), and that their values remain modest in the subseasonal range regardless of the MJO initial conditions.

| SUMMARY AND CONCLUSION
This study proposes a very simple approach to identify windows of opportunity in a subseasonal forecasting system, when specific precursor conditions suggest that an event is more likely to occur according to the observational record. The approach relies on a contingency table analysis (Hogan & Mason, 2012). A first step looks for the coincidence between a lagged increase in the event frequency (Base Rate) and an improvement of the Hit Rate. A second step uses the decomposition of the PSS between Hit Rate and False Alarm Rate, to see if the improvement in Hit Rate converts into an overall improvement of forecast skill when false alarms are also taken into account. These two steps determine if a climatological window of opportunity translates into a model window of opportunity of Type 1 or Type 2. Although Type 2 (improved PSS) is preferable, it is also worth highlighting Type 1 (improved Hit Rate only), as it may already prove valuable to users that are more sensitive to hits than false alarms (e.g., when the forecast event is very rare).
In this article, the method is applied to the state of the MJO at forecast initialization and relies on one-toone comparisons between an active phase and the inactive phase. It is implemented on 20 years of ECMWF reforecasts from the S2S database in the November-April season. It shows that, by Week 2, favorable precursor MJO conditions do reflect in the Hit Rate over the western Pacific and Africa, while they do not over the Indian ocean and the Maritime Continent. This illustrates the ability of the ECMWF system to anticipate heavy tropical rainfall in the former regions. Unfortunately, the improvements in Hit Rate often coincide with an increase in False Alarm Rate, showing the tendency of the system to overforecast heavy precipitation related to lagged MJO impacts. Finally, the approach also highlights enhanced model performances that do not match an increase in Base Rate (Figure 3, yellow grid points): they are most likely related to "lucky hits" in areas where the model predicts an increased frequency of the event while there is none.
The same contingency table approach can be used to diagnose the ability of S2S systems to benefit from any climatological window of opportunity. For instance, another illustration is shown in Appendix S1 with the forecast of weekly precipitation below the lower quintile, in relation with the MJO suppressed convection ( Figures  S2-S4). Beyond the MJO, the method could also be applied to other phenomena that precede extreme events (e.g., Sudden Stratospheric Warmings that can lead to mid-latitude cold waves, Domeisen & Butler, 2020), as long as the forecast sub-sample with the corresponding precursor is large enough.