Large Uncertainties When Diagnosing the “Eddy Feedback Parameter” and Its Role in the Signal‐To‐Noise Paradox

A too‐weak eddy feedback in models has been proposed to explain the signal‐to‐noise paradox in seasonal‐to‐decadal forecasts of the winter Northern Hemisphere. We show that the “eddy feedback parameter” (EFP) used in previous studies is sensitive to sampling and multidecadal variability. When these uncertainties are accounted for, the EFP diagnosed from CMIP6 historical simulations generally falls within the reanalysis uncertainty. We find the EFP is not independent of the sampled North Atlantic Oscillation (NAO). Within the same dataset, a sample containing larger NAO variability will show a larger EFP, suggesting that the link between eddy feedbacks and the signal‐to‐noise paradox could be due to sampling effects with the EFP. An alternative measure of eddy feedback, the barotropic energy generation rate, is less sensitive to sampling errors and delineates CMIP6 models that have weak, strong, or unbiased eddy feedbacks, but shows little relation to NAO variability.


Introduction
The winter North Atlantic Oscillation (NAO) has been shown to be predictable on seasonal (Scaife et al., 2014) and decadal (Smith et al., 2019) timescales.However, the predictable NAO signal in models (variability of the ensemble mean) is weaker than expected given the skill, meaning forecasts are underconfident (Scaife & Smith, 2018).This underconfidence occurs despite models having a relatively good representation of total NAO variability and has been coined the signal-to-noise paradox (Scaife et al., 2014;Scaife & Smith, 2018).This underconfidence could be a manifestation of a too-large component of forecast noise or a too-weak predictable signal (Eade et al., 2014;Scaife & Smith, 2018).
Several studies have investigated whether predictable NAO signals are poorly captured in models, including the representation of teleconnections from the tropics to the North Atlantic (O'Reilly et al., 2019;Williams et al., 2023), the response to Arctic sea ice anomalies (Smith et al., 2022), the response to North Atlantic sea surface temperature (SST) anomalies (Simpson et al., 2018), the response to solar cycle variability (Gray et al., 2013;Scaife et al., 2014) and the response to predictable tropical stratospheric variability (Andrews et al., 2019).
There are currently two main hypotheses to explain the NAO signal-to-noise problem.
1. Weak air-sea coupling in the North Atlantic.This has been shown to contribute to an underestimation of winter North Atlantic eddy-driven jet variability on multidecadal timescales (Bracegirdle et al., 2018;Simpson et al., 2018) and summer NAO variability on decadal timescales (Ossó et al., 2020).
• The "eddy feedback parameter" is a highly non-stationary quantity, making reanalysis and model comparisons problematic on short time periods • Sampling uncertainty in the eddy feedback parameter from reanalysis data is comparable to the intermodel spread • Barotropic energy generation rate is a more stable quantity, but does not explain model spread in North Atlantic climate variability

Supporting Information:
Supporting Information may be found in the online version of this article.
2. Weak eddy feedbacks in midlatitudes.Eddy momentum fluxes can act to reinforce the zonal-mean flow and increase the persistence of jets (Lorenz & Hartmann, 2001, 2003) and the NAO is known to be driven by momentum forcing from synoptic and stationary eddies (Luo et al., 2007).Smith et al. (2022) calculated an "eddy feedback parameter" (EFP) to quantify the relationship between eddy forcing and the midlatitude jet (see Section 2.2.1).Smith et al. (2022) showed the EFP in present day climate correlated with the amplitude of the midlatitude zonal wind response to projected Arctic sea ice loss across a set of climate models.They showed that models underestimated the EFP compared to reanalyzes and used an emergent constraint approach to derive a constrained spread of the modeled jet shift.Hardiman et al. (2022) found that models with a weaker EFP (further from reanalysis) generally have less skill and worse signal-to-noise errors for predicting the Arctic Oscillation (AO).
Eddy momentum fluxes are both a driver of, and a response to, mean flow variability (e.g., Trenberth (1984); Karoly (1990)).Quantifying the eddy feedback on the mean flow requires a separation of these components, for example, using lagged approaches or time filtering (e.g., Lorenz & Hartmann, 2001;Lorenz & Hartmann, 2003;Simpson et al., 2013).However, the EFP as calculated in (Smith et al., 2022) does not separate the timescales of eddies and the mean flow, so it is not clear that it does diagnose eddy feedbacks.
The EFP is based on zonal-mean data and therefore blends together variability from separate regions which may make it more difficult to link to the NAO which is focused on a specific region.Therefore, we also analyze a spatially-resolved diagnostic of eddy forcing, the barotropic energy generation rate (Mak & Cai, 1989), which allows us to investigate the relationship between North-Atlantic eddies and NAO variability using time-filtered data.Although the barotropic energy generation rate includes time filtering, it also does not formally separate the eddy feedback from the response to the mean flow, so we refer to it as an eddy forcing rather than a feedback.
Most of the work on the NAO signal-to-noise problem has focused on seasonal-to-decadal timescales; it remains an open question as to whether similar issues manifest in multidecadal projections of the NAO including externally forced trends (McKenna & Maycock, 2021).The initial motivation of this work was to test the eddy feedback hypothesis in climate simulations by examining whether the EFP is related to multidecadal NAO variability.However, we found that our results were strongly affected by sampling issues with the EFP not accounted for in past studies.In this study, we address the sampling uncertainty in the EFP within reanalysis and climate model datasets, as well as the inherent relationship between the EFP and NAO characteristics within a sample.
This study is laid out as follows: Section 2 describes the datasets used in the study and methods for quantifying eddy-mean flow feedback, Section 3 presents the results and Section 4 presents a summary of the key findings.

Data Sets
Climate model data is taken from Phase 6 of the Coupled Model Intercomparison Project (CMIP6) (Eyring et al., 2016).We use the historical experiment (1850-2014) from 12 CMIP6 models that provide the required variables (monthly-mean mean-sea-level pressure, and daily-mean zonal (u) and meridional (v) wind on pressure levels) for at least 10 ensemble members (see Table S6 in Supporting Information S1).We select models that provide large ensembles in order to quantify sampling effects and the role of internal variability in calculating the EFP and its relationship with the NAO.Diagnostics are calculated from data regridded to the coarsest resolution climate model (CanESM5, roughly 2.8°).All diagnostics are for Northern hemisphere winter (DJF) with the year labeled by the JF (e.g., 2009/10 is labeled 2010).
We use the ERA5 (Hersbach et al., 2020) and ERA20 C (Poli et al., 2016) reanalysis datasets.The back extension of ERA5 covers the period 1940 to 1978 and the standard ERA5 covers 1979 to present.ERA20 C covers 1900-2010 and only assimilates surface pressure and surface marine wind observations.For ERA5 and ERA20 C winds, we aggregate 6 hourly data (00,06,12,18) to daily means to provide an equivalent comparison to the CMIP6 data.We also use monthly-mean mean-sea-level pressure data from 20CRv3 (Slivinski et al., 2019), a reanalysis with a longer record (1836-2015) that only assimilates surface pressure, and HadSLP (Allan & Ansell, 2006), a gridded dataset produced from surface pressure observations, to calculate NAO timeseries in the supplement.

Eddy Feedback Parameter
where ρ is density, ϕ is latitude, a is Earth's radius.Overbars represent a zonal mean, and primes represent local deviations from the zonal mean.Here, we calculate the EFP following Hardiman et al. (2022).The differences in methodology for calculating the EFP can give a different absolute value, but give similar results for the uncertainty (see Fig. S1 in Supporting Information S1).

Barotropic Energy Generation Rate
The barotropic energy generation rate (G) diagnoses the exchange of energy between eddies and the large-scale flow based on an energy equation for the ageostrophic perturbation flow in quasi-geostrophic dynamics (Mak & Cai, 1989).If (U, V) describes the large-scale geostrophic wind and (u′, v′) the eddies, then the barotropic energy generation rate is given by where is the E-vector, which describes the elongation of the eddy, and is the deformation of the large-scale flow (Mak & Cai, 1989), where λ is latitude.Note that we use the spherical coordinate version of these equations from Fukotomi and Yasunari (2002).We diagnose G using daily-mean winds at 250 hPa that are separated into a high frequency (2-6 days) eddy component and a slowly varying (>10 days) large-scale component using Lanczos filters with a window of 61 days.We then average the daily values of G to a seasonal mean.
In comparison to the EFP, G is spatially-resolved, giving a measure of the local energy exchange.To provide a comparison with the EFP and relate G to NAO variability, we average seasonal-mean values of G over a box in the North Atlantic (60°-25°W, 30°-45°N) giving G NA .This region is where the models and reanalysis show climatological negative values (see Fig. S2 in Supporting Information S1), indicating exchange of energy from the eddies to the large scale flow.
From the NAO timeseries we calculate variance.Multidecadal NAO variance is also calculated by first applying a 20-year running mean.

Geophysical Research Letters
10.1029/2024GL108861 The NAO has not been detrended, which could lead to an overestimation of NAO variance in the CMIP6 models compared to ERA5 because we are retaining longer-timescale variability.However, multidecadal variability is only a small part of the total NAO variance (see Section 3.3), so the difference in NAO variance due to including these longer timescales is small.

Statistics
To estimate sampling uncertainty, we recalculate the EFP in ERA5 by resampling winters with replacement (bootstrapping) using the same sample size as the input dataset (e.g., for 1940-2022, each sample is 82 years), repeating 1,000 times.We also recalculate the EFP, NAO variance, and G NA in ERA5 in the same way, but with a sample size matching the historical simulation length (164 years) to compare with the CMIP6 simulations.Each diagnostic is calculated using the same sample years, allowing us to assess relationships between these diagnostics due to sampling.Relationships between variables are estimated using linear least squares regression.

Uncertainties in Reanalysis Derived Eddy Feedback Parameter
In this section, we show how the EFP is affected by sampling uncertainty and multidecadal variability.Figure 1 shows the calculation of the EFP in ERA5 broken into constituent steps.Figures 1a and 1b show the DJF-mean input variables as a function of latitude and year: ū and the acceleration of ū diagnosed from the quasigeostropic component of the horizontal EP-flux divergence.The EFP is calculated by calculating the correlation coefficient (r) between these two variables at each latitude and then averaging r 2 across latitudes.r is defined as the covariance of two variables normalized by their standard deviations.To understand how different years and latitudes contribute to the EFP, Figure 1c shows the anomalies of the input variables, relative to the time mean at each latitude, multiplied together, so the time mean is the covariance as a function of latitude.Figure 1d shows the same, but normalized by the standard deviations of the input variables at each latitude, so the time mean is r as a function of latitude.1. Calculating r at each latitude and then taking a spatial average overemphasizes latitudes with weaker variability.This can be seen by comparing Figures 1c and 1d: anomalies are weaker closer to the equator for the covariance but have a larger contribution to r because the standard deviation at those latitudes is smaller.2. A single outlier season can make a large contribution to the EFP (e.g., 2009/2010 in Figure 1d).This undermines comparisons of the EFP in reanalysis data and climate models when they do not span a common period and do not sample the same internal variability.For example, if a model with inherently weak eddy feedback happens to simulate a season like 2009/2010, it may appear to have a larger EFP than a model with a strong eddy feedback that by chance does not simulate a season like 2009/2010.
Building on point 2, to quantify the sampling uncertainty we recalculate the EFP by sampling years from ERA5 with replacement (see Section 2.3). Figure 2a shows results with the resampling period varied to show the dependence of "observed" EFP on time period: the full ERA5 period ; the pre-satellite backward extension period only ; and the satellite period only .In all cases, the sampling uncertainty in the EFP (≈0.2-0.3) is comparable to the median value based on the 95% confidence interval.This sampling effect represents a substantial uncertainty that has not been acknowledged in previous studies (e. Figure 2a also shows the EFP is dependent on time period: the satellite period has a larger EFP than the presatellite back extension period, with no overlap of the 95% intervals.To better understand the dependence of EFP on time period, we calculate the EFP using a rolling 23-year window (consistent with the 1993-2016 period used in Hardiman et al. (2022)).ERA5 shows a systematic increasing trend in the 23-year EFP (Figure 2b)).A long-term trend in the EFP could be spurious if the reanalysis is poorly constrained by observations and behaves more like the underlying atmospheric model further back in time.Figure 2b also shows the EFP from ERA20 C, which extends back to 1900.Longer-term reanalyzes that only assimilate a limited set of surface observations, such as ERA20 C, have been shown to produce unrealistic trends as the density of the observation network evolves with time (Befort et al., 2016;Bloomfield et al., 2018;Krueger et al., 2013;Oliver, 2016).However, ERA20 C actually shows a larger EFP in the 1930s/1940s when there is less observation data and reproduces the increase in EFP over the late twentieth century.This shows the apparent EFP trend is unlikely to be due to an intrinsic bias of weak eddy feedback in the model that produces ERA5 and instead is related to multidecadal variability in the input parameters.Interestingly, the increase in EFP over the late twentieth century closely mirrors the positive trend in NAO variance over this period, though this common temporal behavior is weaker in the earlier period covered only by ERA20 C. It makes sense that the NAO and EFP are related.Eddy-driven jet latitude is related to the NAO (Woollings et al., 2010) and NAO predictability (Parker et al., 2019;Strommen, 2020), and zonal-mean zonal wind is one of the inputs to the EFP calculation.The EFP calculation also emphasizes large seasonal deviations in jet latitude.For example, winter 2009/10 had a strongly southward shifted jet and negative NAO (Santos et al., 2013).Figure 1d showed how the shift in jet in 2009/10 is emphasized in the correlation calculation and Figure 2b shows a step increase of almost 0.1 when 2009/10 is included in the rolling window.
The time period used by Hardiman et al. (2022Hardiman et al. ( ) (1993Hardiman et al. ( -2016) ) is very close to the maximum EFP over the entire twentieth century due to the inclusion of 2009/10 and being close to the peak of multidecadal variability in the EFP.The results in this section show that previous studies have likely overestimated the long-term mean EFP in reanalysis data.

Comparison of Climate Models and Reanalysis Eddy Feedback Parameter
We next address the comparison of EFP in climate models with reanalysis data in the context of the sampling uncertainties described in the previous section.Figure 3 shows the range of EFP calculated from the CMIP6 ensembles (a, c) and from repeatedly sampling 164 years from ERA5 with replacement (b), as well as the relationship with NAO variance (discussed in the following section).In contrast to previous results, we do not find that the EFP is weaker in models than in reanalysis.The EFP diagnosed from CMIP6 models is generally within the uncertainty from ERA5, with some models potentially having too large EFP (CanESM5, CESM2, CMCC-CM2-SR5).If we only considered the EFP and its associated uncertainty from the satellite period of ERA5 (Figure 2a), then we would conclude that some CMIP6 models underestimate the EFP.This highlights the importance of considering longer-timescale variability, as well as sampling uncertainty, when quantifying the EFP and the limitation of using the EFP as a diagnostic for model performance.

Relationship Between the Eddy Feedback Parameter and the North Atlantic Oscillation
Section 3.1 highlighted a relationship between long-term variations in the EFP and the NAO.We next show how this relationship can lead to correlations that should be interpreted as a sample with larger NAO variability giving a larger EFP, rather than stronger eddy feedbacks leading to stronger NAO variability.Most CMIP6 models capture NAO variability well (Figure 3a) compared to ERA5 (Figure 3b).Only MIROC-ES2L is systematically too weak.Some models are potentially too weak (MIROC6, INM-CM5-0) or too strong (IPSL-CM6A-LR, CESM2), but still overlap with the uncertainty from ERA5.
The lines in Figure 3 show linear regressions calculated from the data in each panel in different ways.
1.For "ERA5" (gray line in Figure 3b) the regression is across the bootstrap samples.Because EFP and NAO variance are calculated using the same sets of sample years, this tells us how the EFP relates to NAO variability purely due to sampling.2. "Mean" is the regression across the ensemble mean points of all models.This relates to model biases and is what would typically be used for emergent constraints (e.g., Smith et al. ( 2022)) 3. "Weighted" is a weighted average regression across all models.For each model, a regression is calculated across ensemble members.The average slope and intercept are then calculated from these individual model regressions, weighted by the number of ensemble members for each model.This indicates whether a sampling relationship between EFP and NAO variability is present, on average, in individual models.4. "All" is the regression across all ensemble members of all models with each sample treated independently.
This gives a mix between "Mean" and "Weighted".
The full set of results from the linear regressions are given in the supplement (Tables S1-S5 in Supporting Information S1).Note that many of the individual model regressions in 3) are not significant due to low sample sizes and the p-value test is less meaningful for the "ERA5" and "All" regressions because the points are not independent.However, the analysis is intended to show how sampling issues with the EFP can produce spurious relationships with the NAO rather than identifying significant relationships.
All three regressions in Figure 3a show a similar relationship between EFP and NAO variance and are well reproduced by sampling ERA5 (r = 0.34-0.55).This means that the across model relationship between the EFP and NAO variance ("Mean"), which could have been interpreted as physically related model biases, is most likely an extension of the sampling relationship found in ERA5: a model with stronger NAO variability is diagnosed with a larger EFP.
Although total NAO variability is relatively well represented for models exhibiting the signal-to-noise paradox (Scaife & Smith, 2018), weak multidecadal NAO variability Bracegirdle (2022); Bonnet et al. (2024) could be evidence of signal-to-noise issues in climate models.However, similar relationships are found when multidecadal variability is isolated (Figure 3c), suggesting this is still only identifying sampling relationships.We haven't estimated the reanalysis relationship between the EFP and multidecadal NAO variance because ERA5 is too short for sampling and longer-timescale reanalyzes give less consistent values of NAO further back in time (see Figs. S3 and S4 in Supporting Information S1).

Alternative Measure of Eddy Forcing
We next show that an alternative measure of eddy forcing targeted at the North Atlantic (G NA , see Section 2.2.2) suffers much less from the sampling issues identified for the EFP. Figure 4 shows G NA for ERA5 and the CMIP6 ensembles and its relationship to NAO variance and the EFP.G NA is better able to identify models that are weak (CanESM5, CESM2, IPSL-CM6A-LR, CMCC-CM2-SR5, INM-CM5-0, MIROC6, MIROC-ES2L), strong (MPI-ESM1-2-LR/HR), or unbiased (UKESM1-0-LL, CNRM-CM6-1, CNRM-ESM2-1) compared to ERA5 due to having much smaller sampling uncertainty.
The sampling relationship between G NA and NAO variability in ERA5 is much weaker (r = 0.07) in contrast to that of the EFP and NAO variability (r = 0.55).Furthermore, the relationship differs from the (nonsignificant) Geophysical Research Letters 10.1029/2024GL108861 across model relationship.Similar results are found for multidecadal NAO variability.Interestingly G NA shows no sampling relationship to the EFP and very little relationship across the models used here (Figures 4d and 4e).This suggests that either the EFP is capturing different aspects of eddies, due to G NA being more localized, or that the EFP is a poor measure of eddies due to the sampling issues shown in earlier.
G shows similar uncertainty when averaged over the same region as used for the EFP (see Fig. S6 in Supporting Information S1).This suggests that the uncertainties in the EFP are due to the sensitivities in how the EFP is diagnosed (identified in Section 3.1) rather than variability in the underlying behavior of the eddies.

Conclusions
Previous studies have suggested that seasonal prediction systems and free running climate models systematically underestimate Northern hemisphere midlatitude eddy feedbacks (Screen et al., 2022;Smith et al., 2022), and that this bias may explain the signal-to-noise paradox (Hardiman et al., 2022;Scaife et al., 2019).However, we find that the eddy feedback parameter  We have shown that the EFP is sensitive to individual outlier seasons and also exhibits strong multidecadal variability.This makes the EFP problematic to interpret as an intrinsic property of a model or the real world because very large sample sizes are needed to produce an estimate with sufficiently small uncertainties.Previous published estimates of the EFP in modern reanalysis data are close to the maximum value derived within the 1940-2022 period because of the pronounced effect of an outlier season (2009/2010) and the phasing of multidecadal variability in the EFP.When sampling uncertainty is taken into account, the EFP in CMIP6 historical simulations is largely consistent with ERA5.Previous results using the EFP as an emergent constraint (Screen et al., 2022;Smith et al., 2022) should have much larger error bars to account for these sampling uncertainties.These uncertainties may be the reason that Screen et al. (2022) found that the reanalysis EFP in the Southern Hemisphere is roughly in the middle of the model values, while the Northern Hemisphere EFP appeared too weak in models.
We have also shown that the sample EFP correlates with sample NAO variability and this can lead to spurious across-model correlations between the EFP and NAO variability.The across model correlation could have been interpreted as a stronger model eddy feedback causing stronger NAO variability, but is actually due to a sample with stronger NAO variability being diagnosed with a stronger EFP because the EFP and NAO are not independent.The relation between the EFP and NAO makes sense because both variables have an underlying relationship with jet latitude.For example, winter 2009/10 had an anomalously southward shifted jet and negative NAO (Santos et al., 2013) and makes the largest single contribution to the EFP in ERA5.It could be argued that models with stronger eddy feedbacks would produce more years like 2009/10; however, it is clear that we need a much larger sample of data than is available for reanalyzes to determine if this is the case.
We have investigated another measure of eddy forcing, the barotropic energy generation rate G, which more cleanly separates eddy forcing and mean flow terms and can be calculated locally for the North Atlantic region (G NA ).G NA shows much smaller sampling uncertainty than the EFP and a much weaker sampling relationship with NAO variability, suggesting that it is better at describing intrinsic properties of the models and reanalysis.We find no systematic bias in G NA , but G NA does better distinguish which models are too weak, too strong or unbiased.
In summary, our results raise questions about previous interpretations that weak eddy feedbacks can explain the signal-to-noise paradox.Firstly, we find that models do not systematically underestimate the EFP when accounting for sampling uncertainty or using an alternative, better constrained, diagnostic (G NA ).Secondly, the diagnosed EFP from a sample is dependent on the sample NAO variability, which makes it difficult to interpret differences associated with the EFP as being caused by eddy feedbacks rather than some confounding variable.Therefore previous results should be re-examined with a diagnostic of eddy forcing that is more robust to climate variability and where clearer causality can be determined, such as the barotropic energy generation rate or a more in depth lead-lag approach that formally isolates the feedback of eddies on the mean flow (e.g., Lorenz and Hartmann (2001)).
Smith et al. (2022) defined the eddy feedback parameter (EFP) as the squared correlation coefficient (r 2 ) between the DJF-mean zonal-mean zonal wind ( ū) and the DJF-mean of the horizontal component of the Eliassen-Palm flux (EP-flux) divergence, calculated as a function of latitude and pressure, and then averaged over 25-72°N, and 200-600 hPa.Hardiman et al. (2022) used a similar formulation, but calculated the EFP at a single level (500 hPa) and only included the quasi-geostrophic component of EP-flux divergence, expressed as a zonal acceleration: Equation 1 fromHardiman et al. (2022),

Figure 1 .
Figure 1.Calculation of the EFP using ERA5.Variables used to calculate the EFP as a DJF mean, (a) zonal-mean zonal wind and (b) the acceleration of the zonal-mean zonal wind diagnosed from the quasi-geostrophic component of the horizontal EPflux divergence.(c) The product of anomalies of (a) and (b), where the anomalies are calculated against the time mean (mean across rows) by latitude.(d) shows the same as (c), but normalized by the standard deviations, at each latitude, of (a) and (b).The time mean of (c) and (d) give the covariance and correlation as a function of latitude, respectively.

Figure 1
Figure 1 reveals two potential issues with the EFP.

Figure 2 .
Figure 2. Uncertainties in the EFP identified from reanalysis data.(a) The EFP calculated by resampling ERA5 over different periods.Orange lines show the median, crosses show the EFP from the original set of years, boxes show the 25%-75% range, whiskers show the 2.5%-97.5% range, and circles show points outside this range.(b) The EFP and NAO variance calculated over 23-year rolling windows for ERA5 and ERA20C data.The x-axis shows the middle year in each sample.The vertical line is for 1993-2016, the years used in Hardiman et al. (2022).

Figure 3 .
Figure 3.The relationship between the EFP and NAO variance for (a), (c) CMIP6 historical simulations (1850-2014) and (b) ERA5 (full period, 1940-2022).(a) The EFP and NAO variance for CMIP6 ensemble members and mean for each model ensemble (outlined symbols).(b) EFP and NAO variance calculated using 164 years sampled from ERA5 with replacement (repeated 1000 times).The outlined dot shows the EFP and NAO variance for the full ERA5 data.(c) The same as (a), but for NAO variance calculated after applying a 20-year running-mean filter.The lines on each subfigure show linear regressions calculated from each set of data in the subfigures (see text for details).The lines from (a) are duplicated in (b) for comparison.

Figure 4 .
Figure 4.The same as Figure 3, but with North-Atlantic DJF-mean barotropic energy generation rate (G NA ) on the x-axis instead of EFP and extra panels (d) and (e) with EFP on the y-axis for CMIP6 models and ERA5, respectively.