### Abstract

- Top of page
- Abstract
- 1. Introduction
- 2.
*In situ* SST observations from 1850–2008 - 3. Large-scale reconstruction using the reduced-space Kalman smoother
- 4. Framework for the reconstruction of the mid-scales
- 5. Specification of the prior covariance
- 6. Verification
- 7. Ensemble reconstructions of the mid-scale SST
- 8. Discussion
- Acknowledgements
- A. Matrix form for the moments of the joint posterior
- B. Parameter determination for mid-scale covariance matrix
- References

Existing historical records of sea-surface temperature extending back to the mid-1800s are a valuable source of information about climate variability on interannual and decadal time-scales. However, the temporal and spatial irregularity of these data make them difficult to use in climate research, where gridded and complete data fields are expected for both statistical analysis and forcing numerical models.

Infilling methods based on constraining the solution to the linear space spanned by the leading eigenvectors of the global-scale covariance, otherwise known as reduced-space methods, have proven very successful in creating gridded estimates of sea-surface temperature. These methods are especially useful for infilling the vast regions of unobserved ocean typical of the earliest segments of the data record. Regional variability, on the other hand, is not well represented by these methods, especially in data-poor regions. Here we present a method for augmenting the established large-scale reconstruction methods with a statistical model of the mid-scale variability. Using high quality sea-surface temperature data from the last 30 years including satellite-derived records, we specify a spatially non-stationary, anisotropic covariance model for the mid-scale sea-surface temperature variability. With the parameters of the covariance model estimated from the modern record, historical observations are used for conditioning the posterior distribution. Specifically, we form the expected value and correlated uncertainty of the mid-scales as well as generating samples from the posterior.

While this work focuses on a limited domain in the midlatitude North Atlantic Ocean, the method employed here can be extended to global reconstructions. Copyright © 2011 Royal Meteorological Society

### 1. Introduction

- Top of page
- Abstract
- 1. Introduction
- 2.
*In situ* SST observations from 1850–2008 - 3. Large-scale reconstruction using the reduced-space Kalman smoother
- 4. Framework for the reconstruction of the mid-scales
- 5. Specification of the prior covariance
- 6. Verification
- 7. Ensemble reconstructions of the mid-scale SST
- 8. Discussion
- Acknowledgements
- A. Matrix form for the moments of the joint posterior
- B. Parameter determination for mid-scale covariance matrix
- References

Prior to the current era of satellite data acquisition, the main source of information on sea-surface temperatures (SST) came from the logs of ships of opportunity. These records stretch back to the mid 19th century, making them a tantalizing source of information about climate variability on interannual and decadal time-scales. However, the temporal and spatial inhomogeneity of these data make them difficult to use in standard statistical analysis procedures. Gridded fields of SST are also needed for initialization and verification of ocean models and as time-dependent boundary conditions in atmospheric models. As a result, interpolation schemes for infilling these sparse data are tremendously important in climate research.

One popular approach to the interpolation of historical datasets is reduced-space estimation (Shriver and O’Brien, 1995; Smith *et al.*, 1996; Kaplan *et al.*, 1998, 2000; Rayner *et al.*, 2003). One of its advantages over more conventional methods such as simple kriging (which uses a stationary, localized covariance function) is its emphasis on the reconstruction of the largest and most energetic spatial scales over the entire domain of interest. This is a natural advantage for modelling climate variables such as SST, because the dynamics of the climate system often result in global-scale coherency.

It is worth considering why reduced-space estimation has been so useful in climate applications. For climate variables that possess a large spatial dimension, the relative temporal ‘shortness' of the reliable observational data record that can be used for computing a sample covariance matrix often leads to rank-deficiency. Assuming a lower dimensionality of the system via truncation of the less energetic eigenvectors of the covariance matrix can circumvent this problem. Another advantage of reduced-space techniques becomes evident when the data used for reconstruction are clustered in limited areas, leaving large regions completely unobserved. Under these circumstances, inference in the interiors of the unsampled regions would be imprudent using methods that rely solely on local spatial estimation methods.

The disadvantage of using a reduced-space technique for interpolation is that there is no guarantee that the patterns of covariability that dominate within smaller subregions of the global domain will be well represented. The truncation of trailing eigenvectors necessarily excludes some structures that are better suited to local estimation techniques. Ideally, a reconstruction methodology would draw from the strengths of both types of interpolation, with the aim of representing behaviour over a range of spatial scales.

We restrict our focus to the statistical modelling and reconstruction of SST anomalies in the northern hemisphere Atlantic Ocean. We present a method to augment an existing historical SST reconstruction that uses a reduced-space Kalman smoother (Kaplan *et al.*, 1998) to capture what we will term the ‘global-scale’ or ‘large-scale’ modes of variability. The contribution of this work is to model and reconstruct what we will term ‘mid-scale’ variability. For the remainder of this article, we will use the terms global and mid-scale to distinguish between variability captured by the reduced-space technique and the more locally dominant variability on which we are focused.

The separation into global and mid-scales is not based on physical processes. No objective criteria for parsing between these covariance models is used, nor do we mean to imply that a given length-scale of covariability will be uniquely contained in either model. In the context of this study, mid-scales can be interpreted as the most dominant local variability not captured by the globally-based reduced space reconstruction.

Section 2 describes the historical temperature data, extending back to 1850, that are used in this reconstruction. Section 3 gives a brief description of the reduced-space Kalman smoother that was used in the published reconstruction of the large-scale SST anomalies. As we discuss, there are subjective choices that go into reduced-space techniques and our definition of mid-scales is implicitly impacted by these. Given this caveat, it is still instructive to note that the mid-scales tend to have geographic coherency of the order of 500–1300 km.

There are two main areas of emphasis in this work. They are (1) the statistical modelling of our prior knowledge of the mid-scale variability not present in the established reduced space reconstruction and (2) description of the mid-scale reconstruction in terms of the mean, covariance and samples from the posterior distribution. Section 4 outlines the statistical procedure that we use to form the posterior distribution for our mid-scale reconstruction. In section 5 we present our model for the covariance of the mid-scale variability. We employ a novel covariance parametrization developed by Paciorek and Schervish (2006) that allows for non-stationarity in the length-scales and anisotropy of the spatial correlation functions. This parametrization gives our model the flexibility to capture geographic variation in the underlying covariability of SST anomalies while still ensuring a positive-definite covariance matrix defined over the entire domain. This is a useful feature for analyses of SST in the northern Atlantic Ocean basin, where the dominant physical processes vary over the domain.

We verify the statistical model in section 6 and section 7 presents a selection of the resultant reconstructions. Because the quantification and representation of uncertainty has become an area of increased interest within the climate research community (Rayner *et al.*, 2009), we pay special attention to the uncertainty estimates implied by the posterior distribution. Specifically, we note the temporal evolution of the uncertainty due to changes in data availability through time and the spatial correlations inherent in the posterior distributions. We conclude in section 8 with a discussion of some of the broader issues relevant to this work, some of its limitations and prospects for its extension.

### 2. *In situ* SST observations from 1850–2008

- Top of page
- Abstract
- 1. Introduction
- 2.
*In situ* SST observations from 1850–2008 - 3. Large-scale reconstruction using the reduced-space Kalman smoother
- 4. Framework for the reconstruction of the mid-scales
- 5. Specification of the prior covariance
- 6. Verification
- 7. Ensemble reconstructions of the mid-scale SST
- 8. Discussion
- Acknowledgements
- A. Matrix form for the moments of the joint posterior
- B. Parameter determination for mid-scale covariance matrix
- References

Our reconstruction is based on the Hadley Centre Sea-Surface Temperature, version 2(HadSST2) dataset of monthly *in situ* SST anomalies from 1850–2008. HadSST2 is based on the International Comprehensive Ocean–Atmosphere Data Set (ICOADS) archive of surface marine observations collected from ships and buoys (Worley *et al.*, 2005). In HadSST2, the ICOADS SST data are subjected to quality checks, corrected for systematic bias, converted to climatological anomalies and averaged on to a 1° × 1° grid.* Note that HadSST2 is not an interpolated product, so grid boxes where no data are present in the ICOADS database remain empty. Rayner *et al.* (2006) document the extensive work that was done to create the HadSST2 dataset, including a detailed description of the bias-correction methods.

Because data in the early part of the record were primarily collected on a volunteer basis by merchant ships, they tend to be concentrated along trade routes, leaving large portions of the ocean unobserved. While the number of records generally increases in time, sociopolitical events such as the two World Wars and the Great Depression are marked by temporary decreases in the availability of data. However, the *in situ* coverage becomes dense in the second half of the twentieth century and by the mid 1960s observations are routinely available over nearly 90% of the North Atlantic Ocean basin. The reader is referred to Worley *et**al.*(2005) and Rayner *et al.* (2006) for details on the time-evolving *in situ* data coverage.

### 3. Large-scale reconstruction using the reduced-space Kalman smoother

- Top of page
- Abstract
- 1. Introduction
- 2.
*In situ* SST observations from 1850–2008 - 3. Large-scale reconstruction using the reduced-space Kalman smoother
- 4. Framework for the reconstruction of the mid-scales
- 5. Specification of the prior covariance
- 6. Verification
- 7. Ensemble reconstructions of the mid-scale SST
- 8. Discussion
- Acknowledgements
- A. Matrix form for the moments of the joint posterior
- B. Parameter determination for mid-scale covariance matrix
- References

Central to all reduced-space estimation techniques is the idea that the global covariability can be adequately expressed in terms of a limited number of spatial basis functions. If these basis functions are found as the leading eigenvectors of a sample covariance matrix, these basis functions are called empirical orthogonal functions (EOFs). Typically, the number of EOFs retained in a reconstruction is far fewer than the spatial dimension of the state variable. The exact number of basis functions retained is a subjective compromise between the desire to capture a large portion of the variability in the system and the desire to drastically reduce the size of the problem. At the very least, the truncation must be severe enough to eliminate any rank-deficiency in the full-space sample covariance matrix.

In this framework, the reduced-space expression of a state vector of global SST anomalies can be written as

- (2)

Here ℰ is an orthogonal matrix, the columns of which are the EOFs, and *α* is a vector of weighting coefficients. In reduced-space estimation it is generally assumed that the EOFs are fixed and *α* is the probabilistic variable of interest. Well-known methods of data assimilation (Kalman filtering, smoothing, variational methods, etc. ) can be used to construct a posterior distribution of *α*.

The historical SST reconstruction that serves as the large-scale base for our mid-scale features is the reduced-space Kalman smoother described in Kaplan *et al.* (1997, 1998). Hereafter we will refer to this reconstruction as the KaplanSST. Since the details of the KaplanSST are described elsewhere in the literature and are not the focus of this work, we provide only a brief description below.

The KaplanSST is a near-global SST reconstruction wherein the EOFs are computed from the global sample covariance and the reconstructions make use of observations from all ocean basins. The version of this reconstruction used here is based on the 1° × 1° HadSST2 dataset of *in situ* SST observations described in the previous section. The analysis is done for SST anomalies, i.e. deviations of full SST values from their monthly climatological values. The climatology is that of Smith and Reynolds (1998) for the 1961–1990 period. SST anomalies from the relatively data-rich period 1951–2007 were used for calculating the sample covariance matrix from which the orthogonal basis functions were computed and the data from 1850–2008 were used as the observations on which posterior distributions are conditioned. Consistent with Kaplan *et al.* (1998), only 80 global EOFs are retained. The 1° × 1° HadSST2 is area averaged onto a 5° × 5° grid prior to computing EOFs and generating the reconstruction. The resulting reconstruction is then bilinearly interpolated on to a 1° × 1° grid. Unlike Kaplan *et al.* (1997, 1998), where only the expected value of the reduced-space analysis was computed, here the posterior covariance of the reduced space is also estimated as described in Kaplan *et al.* (2000). From the KaplanSST, then, we have a sequence of global SST anomalies (along with their corresponding error covariance) at each month from 1850–2008. By construction, the KaplanSST estimates only large-scale, globally relevant modes of variability.

As pointed out by Dommenget (2007), describing the covariability of a system in terms of a set of EOFs is purely a statistical convenience. It does not imply that the underlying covariance could not also be described using another statistical or physical model. Reduced-space techniques are useful not because they are unique models of the covariance but because they are a parsimonious way of describing (and ranking in terms of importance) the types of variability that are typically attractive to the climate science community. However, there can be coherent scales not captured by the reduced-space reconstruction that have regional importance. These scales can be modelled using localized covariance models, as described in the following sections.

### 4. Framework for the reconstruction of the mid-scales

- Top of page
- Abstract
- 1. Introduction
- 2.
*In situ* SST observations from 1850–2008 - 3. Large-scale reconstruction using the reduced-space Kalman smoother
- 4. Framework for the reconstruction of the mid-scales
- 5. Specification of the prior covariance
- 6. Verification
- 7. Ensemble reconstructions of the mid-scale SST
- 8. Discussion
- Acknowledgements
- A. Matrix form for the moments of the joint posterior
- B. Parameter determination for mid-scale covariance matrix
- References

Because the SST anomalies are the sum of the variability on global scales and mid-scales, the most complete reconstruction procedure considers the joint distribution of these processes. At each month, we can define a joint state as the large-scale SST anomalies in reduced space (*α*) appended with mid-scale SST anomalies (*z*′) in full grid space:

- (3)

The matrices *λ* and *C* are the specified prior covariances on large and mid-scales (respectively). The observations can be written as

- (4)

where *y* is a vector of the HadSST2 observations at a given month and is the corresponding vector of measurement error (both described in section 2). The columns of the matrix ℰ correspond to the basis functions of the KaplanSST that have been bilinearly interpolated on to a 1° × 1° grid. ℋ is a submatrix of the identity matrix that maps from the 1° × 1° geographical state space of *z* to the observational space of *y*.

Although the large and mid-scales are assumed independent in the prior (3), once *Z* is conditioned on *y* they are no longer independent. We do not show this here, but it can easily be seen by forming the joint posterior *p*(*Z*|*y*). This links the problem of reconstructing the mid-scales with the problem of large-scale reconstruction. This combination increases the effective dimension of the problem from , which represents the number of EOFs retained in the KaplanSST analysis, to , the number of grid points in the geographical state space. The assumed time autocorrelation in *α* further exacerbates this issue because smoother solutions are naturally more computationally demanding than sequential filters.

An alternative to forming the joint distribution is to recognize that the variable of interest is actually the sum of the large and mid-scales (*z* = ℰ*α* + *z*′), which can be written

- (5)

The marginal posterior probability *p*(*α*|*y*) can be approximated by the reduced-space KaplanSST solution, which is multivariate normal (in time and space) with mean *μ*_{α} and covariance *P*_{α}. In forming the marginal distribution of *α*, the KaplanSST assumed that all variability unresolved by the leading EOFs was uncorrelated. We show in the next section that in fact there are significant correlation structures, but because the structures in *z*′ are considerably smaller in scale than the dominant global patterns in this is a tolerable approximation.

Using the KaplanSST distribution as a close approximation for the marginal posterior, the solution becomes

- (6)

This hierarchical form allows us to focus on forming only the first factor in the integral (with the understanding that samples can be drawn from the second). Applying Bayes' theorem, we can write

- (7)

where the first term on the right-hand side is a Gaussian likelihood and the second term is a Gaussian prior distribution with mean *α* and covariance of the mid-scales (*C*). We can then recognize *p*(*z*|*α,y*) as also being normally distributed, with expected value

- (8)

and covariance

- (9)

The full solution given in (6) is then simply the integral over the product of two Gaussians:

- (10)

Samples from this distribution can be formed following a simple Monte Carlo approach: given a sample from *N*(*α*|*μ*_{α}*,P*_{α}), we can draw from *N*(*z*|*μ*_{z|α}*,P*_{z|α}). This is a particularly attractive tactic, because only the expected value *μ*_{z|α} depends on *α* and the covariance *P*_{z|α} need only be computed once regardless of the number of samples needed.

It is also possible to write the full solution by performing the integral over *α*. We show in Appendix A that there is a compact matrix form for this integral. However, while it is easy to write, it is not a matrix that one would like to form explicitly because of its full rank in time and state space. We draw on the result that the expected value of the full solution is

- (11)

Defining a data residual term,

- (12)

we can write the mid-scale portion of the expected value as

- (13)

To demonstrate the type of correlated uncertainty associated with the mid-scales, we focus in the remainder of this article on only the conditional covariance *P*_{z|α} given by (9). It is worthwhile to keep in mind, however, that the full uncertainty, as shown in Appendix A, contains the KaplanSST posterior uncertainty (*P*_{α}) as well as terms involving the interaction between large and mid-scales.

In the following section we describe our model for the prior covariance matrix *C*. Once this mid-scale covariance matrix has been formed, it is straightforward (albeit computationally expensive) to generate samples from the posterior distribution defined by (9) and (13).

### 6. Verification

- Top of page
- Abstract
- 1. Introduction
- 2.
*In situ* SST observations from 1850–2008 - 3. Large-scale reconstruction using the reduced-space Kalman smoother
- 4. Framework for the reconstruction of the mid-scales
- 5. Specification of the prior covariance
- 6. Verification
- 7. Ensemble reconstructions of the mid-scale SST
- 8. Discussion
- Acknowledgements
- A. Matrix form for the moments of the joint posterior
- B. Parameter determination for mid-scale covariance matrix
- References

To verify the reconstruction, we withheld observations for comparison. Every month from 1950 to 1974 we withheld 10 randomly chosen observations from the HadSST2 dataset in the Northern Hemisphere (NH) Atlantic (3000 in total). We chose this part of the record because it is the most well observed segment not overlapping with the NCEP OI period. We then reconstructed the mid-scale SST using our non-stationary covariance model and compared results to the withheld observations. As the simplest baseline for comparison, we use the large-scale solution without any mid-scale reconstruction (i.e. *z*′ = 0).

Empirical distributions of the absolute difference between the 3000 observation values and the predicted expected value are shown Figure 3. The average correction imparted by the mid-scale reconstruction is about 0.11°C and 25% of the locations used in the verification show an improvement of over 0.25°C. Furthermore, we expect that the mid-scale reconstruction is most important in eddying regions like the Gulf Stream (30°N–50°N, 75°W–45°W). Corrections to withheld observations in the Gulf Stream region tend to be larger, averaging 0.2°C with 40% of the instances showing improvement of over 0.25°C.

We also consider how well this non-stationary covariance model performs relative to a more traditional stationary statistical model for the mid-scale variability. For this test, we repeated the reconstruction with the same withheld observations but using spatially stationary zonal and meridional length-scales *L*_{x} = 1000 km and *L*_{y} = 500 km, while retaining the non-stationary prior variance.‡ Compared with this more sophisticated baseline solution, the results are more subtle. The stationary reconstruction is shown by the black dashed lines in Figures 3 and 4. Over the entire domain, we see only a small reduction in absolute difference from using the non-stationary covariance model. In the Gulf Stream region, however, the improvement is more evident, albeit with a subtle mean improvement of 0.07°C and about 5% of the locations showing improvements greater than 0.25°C.

It is important to characterize the types of errors we expect to result from using a stationary covariance model when the underlying stochastic process is, in fact, non-stationary. Let us focus again on the Gulf Stream, because it is a high-variance region. There the stationary covariance model overestimates the length-scales. This type of mis-specification effectively reduces the resulting degrees of freedom in the system and erroneously dampens both the pointwise uncertainty in the posterior and the spatio-temporal variance of the expected value. In our experiment, for example, we had a 56% reduction in the average pointwise uncertainty variance in the Gulf Stream region when the stationary model was used and a 30% reduction in the spatio-temporal variance of the expected value. Naturally, there was also a reduction in the spatial gradients of samples drawn from the posterior. These kinds of errors are important in applications in which SST analyses are used to give boundary conditions for atmospheric models, because in frontal zones the atmosphere is responsive to the Laplacian of the SST (Minobe *et al.*, 2008).

### 7. Ensemble reconstructions of the mid-scale SST

- Top of page
- Abstract
- 1. Introduction
- 2.
*In situ* SST observations from 1850–2008 - 3. Large-scale reconstruction using the reduced-space Kalman smoother
- 4. Framework for the reconstruction of the mid-scales
- 5. Specification of the prior covariance
- 6. Verification
- 7. Ensemble reconstructions of the mid-scale SST
- 8. Discussion
- Acknowledgements
- A. Matrix form for the moments of the joint posterior
- B. Parameter determination for mid-scale covariance matrix
- References

Here we present a selection of the resultant reconstructions of the non-stationary mid-scale SST. Figure 5 qualitatively illustrates the relative importance of including the mid-scales in the SST analysis. For the month of January in 1850, 1942 and 1980, the first column shows the SST anomaly from the KaplanSST reduced-space Kalman smoother. The second column shows the expected value of the mid-scale SST modelled in this work, and their sum is shown in the third panel. While the overall signal is dominated by the large-scale KaplanSST, the mid-scale reconstruction provides a significant higher resolution correction to the analysis.

Over the entire time period of the reconstruction, the spatio-temporal variability (in the expected value of SST anomaly) in the NH Altantic is increased by ∼60% due to the explicit modelling of the mid-scales. The variance spectrum of the SST anomaly with and without the inclusion of mid-scales helps us to quantify the relative importance of modelling these scales (Figure 6). This spectrum is calculated from the eigenvalues of the long-term covariance of NH Atlantic SST with and without mid-scales (circles versus dots, respectively). We computed the eigenvalues of this spectrum only for the NH Atlantic; they are not the eigenvalues of the global KaplanSST analysis. In general, the leading eigenvalues are associated with larger scale structures and the trailing eigenvalues are structurally smaller. We see that including the mid-scales in the analysis adds power to the trailing modes of the spectrum. This flattening of the spectrum is due to the reintroduction of spatial scales that are not captured by reduced-space reconstruction methods.

The mid-scale reconstruction adds variance across the entire spectrum. In particular, there is a very modest, but detectible, contribution to the leading few modes. Figure 7 shows a similar spectrum, but with the reconstructions projected on to the orthogonal structures of the KaplanSST. Here we see unequivocally that the mid-scale reconstruction has a projection on to the global solution. This is not unexpected. Independence in the prior does not guarantee independence in the posterior. It simply reinforces the idea that a joint modelling of the large and mid-scales is an important next step.

Figure 8 illustrates the long-term mean in large and mid-scale solution components. The mid-scale reconstruction introduces a cooling in the temperatures along the northern edge of the Gulf Stream region, along with a smaller warming along the southern edge. This is a course resolution of the Gulf Stream pathway. Figure 9 shows a time series of the reconstruction at 50°W and 43°N, a locally cool pivot point where the Gulf Stream turns northward to form the North Atlantic Drift. We see that this feature is temporally consistent, emerging in the late 1800s and persisting through most of the record. The grey shadow in Figure 9 is the posterior uncertainty bound (95 confidence intervals). Since the prior covariance is temporally stationary, the fluctuations in the posterior uncertainty are driven by the availability (or absence) of *in situ* HADSST2 data near this location.

A close examination of this SST anomaly time series in the context of its uncertainty reveals a slightly different story. There is little evidence, in fact, that this cool feature was ever absent. The lack of observations leads to a large uncertainty during the preceding three-quarters of a century.

Of course, pointwise uncertainties are only part of the story. The posterior uncertainties in the mid-scales are correlated in space. We illustrate these covariance structures in Figures 10–12. For January in 1850, 1942 and 1980, we have contoured the expected value and pointwise uncertainty of the mid-scale SST anomaly (top panels). In the remaining four panels we present a sample drawn from the full posterior uncertainty distribution. The black dots on the maps are points where *in situ* observations were available. As we expect, the realizations tend to be most similar (to each other and to the expected value) when the ocean is densely observed. In the early record, as well as during times of changing shipping routes (such as the early 1940s), we can see that the ensemble members cluster in agreement at the observation locations. We can also see the underlying covariance structures that were specified in the prior covariance are emerging in the posterior.

To demonstrate the utility of representing the reconstructions via an ensemble of realizations, we can compute the long-term covariance eigenspectrum multiple times from samples taken from the posterior distribution. The line marked with ‘+’ in Figure 6 shows the average eigenspectrum for 20 realizations. This increase (∼10%) reflects the additional variability in the SST due to posterior uncertainty in the mid-scales. This additional variability will make up an even greater percentage in the early part of the record, when observations are scarce. Variability estimated from many individual realizations approximates the ‘true’ variability, whereas variability found from the expected value will always be biased low.

### 8. Discussion

- Top of page
- Abstract
- 1. Introduction
- 2.
*In situ* SST observations from 1850–2008 - 3. Large-scale reconstruction using the reduced-space Kalman smoother
- 4. Framework for the reconstruction of the mid-scales
- 5. Specification of the prior covariance
- 6. Verification
- 7. Ensemble reconstructions of the mid-scale SST
- 8. Discussion
- Acknowledgements
- A. Matrix form for the moments of the joint posterior
- B. Parameter determination for mid-scale covariance matrix
- References

This work presents the statistical modelling and reconstruction of mid-scale SST anomalies in the NH Atlantic. We use a recently developed statistical parametrization by Paciorek and Schervish (2006) for the prior covariance that allows for non-stationarity in the anisotropic correlation length-scales. The benefit of this approach is that it allows us to form a positive-definite posterior uncertainty covariance from which samples can be easily drawn. The correlated error structures in the prior naturally emerge in realizations drawn from the posterior.

Because this work is focused on the modelling and reconstruction of the mid-scale variability, we have not combined their estimates with the posterior distribution of the KaplanSST. A complete modelling of the SST anomaly would treat the global and mid-scales as a joint distribution because, while these scales can be assumed independent in the prior, they are correlated in the posterior. However, one might want to avoid joint modelling for a number of reasons. Primary among them is that the large-scale part of the solution is global in size and autoregressive in time. Dimensional reduction is thus computationally necessary for a tractable solution. In subdomains where we also want to compute the mid-scale part, such as the NH Atlantic basin considered here, it is helpful to draw on the results of the large-scale reconstruction while avoiding a global multiscale reconstruction. The hierarchical form of the full solution shown in section 4 suggests a simple Monte Carlo approach to drawing samples from the full large and mid-scale solution. This is a fruitful direction for future work.

Lorenc (1986) notes that geophysical fluids exhibit energy over a wide range of spatial scales. It is not altogether clear how these scales should be separated. We have implicitly assumed in this study that global scales of covariability can be captured via a reduced-space representation and the remaining variability can be modelled with a locally supported covariance. However, even within this framework it is clear that there are multiple scales present at this mid-level too. We can interpret our non-stationary parametrization as an attempt to select for correlation structures that are locally dominant. Even with this simplified description, mid-scale reconstructions contribute significantly to both the long-term mean (Figures 8 and 9) and the long-term covariance spectrum (Figure 7).

For problems of the size often encountered in the climate sciences, computational constraints can limit the direct evaluation of (8) and (9). Approximations and iterative methods used to deal with this practical limitation tend to focus on the computation of the expected value (Lorenc, 1986). Lorenc (1986) and Pedder (1993) point out that methods focused on computation of the expected value tend to be best suited for problems with reasonably reliable and densely distributed observations. When data are sparse and noisy, as happens in the early part of the record presented here, a full specification of the prior covariance becomes preferable.

A description of the covariance matrix and its evolution through time is expensive to store and disseminate. A multivariate Gaussian distribution with *n* spatial points and *m* time points requires *mn*(*mn* + 1)/2 numbers to quantify the uncertainty. Users of climate datasets may not have the computational resources or mathematical expertise to make use of a full covariance matrix of uncertainty information. As an alternative, samples from the posterior distribution can be distributed. To the extent that the data model is adequate, the samples form an ensemble of possible realizations of the true SST. Users can perform standard climate data analyses on the multiple realizations, building up uncertainty information in a Monte Carlo fashion.

In the specific case of generating SST datasets, another important application is the boundary forcing of atmospheric models. It is not uncommon for analyzed SST datasets to be viewed as non-probabalistic, with the uncertainty in the system assumed to stem from the internal variability of the atmosphere only. Instead, atmospheric modellers could use ensembles as a set of readily accessible and statistically rigorous possibilities with which to force their models.

The presentation of an ensemble of possible realizations of SST is especially important in data-poor regions of the ocean. It is a natural consequence of Bayesian inference that the expected value of the reconstruction in unobserved areas will relax towards the mean of the prior distribution. When considered outside the context of the full covariance information, data users can falsely interpret these locations in the data record as less energetic. A proper interpretation, in contrast, would be that there is little constraint on the possible states of the system. The sparse and irregular nature of historical data thus makes ensemble presentation an important contribution to the research community.

Finally, we suggest that with some ingenuity this method could be extended to global reconstructions. While the computational expense of covariance generation and sampling of mid-scales on a global domain is a significant obstacle, it may be possible to exploit the inherent sparsity of the correlation matrix for computational efficiency. One could also make use of the idea that mid-scale features are a relatively minor part of the total reconstruction in many regions of the ocean. It may be the case that mid-scale reconstruction in isolated domains is sufficient.

### B. Parameter determination for mid-scale covariance matrix

- Top of page
- Abstract
- 1. Introduction
- 2.
*In situ* SST observations from 1850–2008 - 3. Large-scale reconstruction using the reduced-space Kalman smoother
- 4. Framework for the reconstruction of the mid-scales
- 5. Specification of the prior covariance
- 6. Verification
- 7. Ensemble reconstructions of the mid-scale SST
- 8. Discussion
- Acknowledgements
- A. Matrix form for the moments of the joint posterior
- B. Parameter determination for mid-scale covariance matrix
- References

Section 5.3 presents the parameters of the mid-scale covariance matrix that we estimate from the NCEP OI mid-scale data. Here are the details of the maximum pseudo-likelihood method used for the estimation.

To isolate the spatial correlation from the variances, we begin by standardizing the NCEP OI mid-scale data to have zero mean and variance of unity over the period 1981–2008 (). Based on exploratory analysis of the NCEP OI mid-scale data, a priori we prescribe the Matérn shape parameter *ν* = 3. Using this value of the smoothness parameter, we proceed to estimate the kernel matrix (16) centred at each grid point *i*. We can express the bivariate distribution of two points as

- (B1)

where *C*(*i,j*) = *f*(Σ*,ν*) is the anisotropic Matern covariance from (14)–(16) with *σ*_{i} = *σ*_{j} = 1. We define a likelihood function for the data centred at grid point *i* as the product of these bivariate normals over all points *j* within 20° of the location of *i*. We can express the maximum-likelihood estimate of the parameter Σ_{i} as

- (B2)

where Θ is the set of all 2 × 2 real, symmetric, positive-definite matrices. Since the multivariate normal likelihood function in (B2) is nonlinear in all three variables that comprise Σ, its maximization is performed numerically using a standard Nelder–Mead search algorithm.

This reduces the problem to finding the maximum pseudo-likelihood estimate within a series of smaller (20° × 20°) subregions centred on the point *i*. is constrained at its upper bound by the pointwise sample variances of the NCEP OI mid-scale data. Although the minimization is done over the entire subregion, only the solution at grid point *i* is retained. As the pseudo-likelihood maximization procedure cycles through all grid points *i*, the solution is also constrained by any variances within the subsets that have previously been computed. This enforces continuity between neighbouring grid points. This process is iterated over the domain until convergence is achieved.