Air–Sea fluxes from ICOADS: the construction of a new gridded dataset with uncertainty estimates



The methods used to calculate a new in situ global dataset of air–sea exchanges, called the NOCS Flux Dataset v2.0, are described. The fluxes have been derived from in situ weather reports from Voluntary Observing Ships (VOS) covering the period 1973–2006. The reports have been adjusted for known biases and residual uncertainties estimated. The dataset is constructed using Optimal Interpolation (OI) using new estimates of random uncertainty in the observations. Daily fields have been calculated on a 1° latitude by 1° longitude grid, each grid box and time step have an associated uncertainty estimate. Monthly fields have been calculated from simple averages of the daily fields and monthly uncertainty estimates from the daily uncertainties, using estimates of the autocorrelation between the daily uncertainty estimates. The uncertainties due to the choice of flux parameterisation have not been accounted for. Bias adjustments applied to the data are shown to reduce trends in the data and to improve the consistency of estimates of air temperature, sea surface temperature (SST) and specific humidity. The bias adjustments also improve the agreement of NOCS v2.0 with independent data from research moorings. Cross-validation of the dataset suggests that the uncertainty estimates are realistic, but that the uncertainties are probably underestimated in high variability regions and overestimated in regions with lower variability. Copyright © 2009 Royal Meteorological Society

1. Background

Reliable estimates of air–sea fluxes of heat, freshwater and momentum are needed to improve our understanding of the coupled ocean–atmosphere system. The main physical interactions at the sea surface are the turbulent heat, moisture and momentum fluxes between the ocean and the atmosphere, the radiative warming of the ocean surface by solar radiation and the net cooling of the ocean surface by thermal radiation. Direct measurements of the turbulent fluxes are typically only made on air–sea interaction research cruises and dedicated moored buoys and platforms. This limits the availability of direct flux measurements in space and time and their usefulness for global studies. As a result, global marine flux datasets are typically constructed using bulk estimates of the mean meteorological parameters and bulk parameterisations (or bulk formulae) developed using coincident measurements of the direct and radiative fluxes and mean meteorological variables from research cruises.

The meteorological parameters required by the bulk formulae are: air temperature and humidity; sea surface temperature (SST); wind speed; sea level pressure; and cloud cover. These parameters are available from different sources including ship and buoy observations, satellite retrievals and model output. In this paper we use only in situ measurements made by Voluntary Observing Ships (VOS). The VOS observations are known to contain significant biases, to be of variable quality and to have uneven sampling with the observations clustered over the major shipping lanes. However, the observations have been characterised well in terms of random errors (Kent and Berry, 2005), bias (Cardone et al., 1990; Berry et al., 2004; Kent and Kaplan, 2006) and metadata on observing practices (Kent et al., 2007). Additionally, the VOS provide all the parameters required to estimate the fluxes on multi decadal time scales.

There is a long history of estimating regional and global air–sea fluxes from the VOS data. Early examples include the atlases of Bunker (1976), Hsiung (1986) and Oberhuber (1988). More recently, estimates have been made by da Silva et al. (1994) and Josey et al. (1999) who calculate the fluxes for individual VOS reports and average the estimates to give monthly mean and climatological values. Successive correction (da Silva et al., 1994) is used in both datasets to smooth and fill gaps in the fields produced. One problem currently suffered by all in situ flux datasets is an imbalance in the global surface net heat budget: typically the oceans gain around 30 W m−2 (Josey et al., 1999). This has led to an inverse analysis of the heat fluxes in the UWM/COADS (da Silva et al., 1994) and NOCS v1.1 climatologies to bring the net heat flux into agreement with hydrographical estimates of ocean heat transport (NOCS v1.1a, Grist and Josey, 2003).

In addition to the in situ based estimates, flux datasets have been generated using satellite retrievals and atmospheric reanalysis model output (Jost et al., 2002; Kubota et al., 2002; Bentamy et al., 2003; Chou et al., 2003; Yu and Weller, 2007). Estimates are directly available from reanalysis models (Kalnay et al., 1996; Uppala et al., 2005; Onogi et al., 2007) on sub-daily timescales.

This paper describes the development of the new NOCS v2.0 flux dataset fields and uncertainty estimates. A separate paper, Berry and Kent (2009), describes the fluxes and basic meteorological variables together with a comparison to independent buoy observations. The new dataset forms a substantial update to the previous NOCS v1.1 climatology (Josey et al., 1999), expanding the period covered, using significantly different methods to grid the data and calculate the fluxes and including uncertainty estimates (Table I).

Table I. Comparison of NOCS v1.1a and NOCS v2.0
 NOCS v1.1NOCS v2.0
Source dataData from COADS Release 1a (1980–1993), later extended to 2005ICOADS v2.4
Data types usedAll data excluding those from Coastal-Marine Automated Network (C-MAN, Platform Type 13)Ship data (Platform types 0–5 only)
PeriodClimatology for 1980–19931973–2006, periodic updates planned
 Monthly files for 1980–2005 
VariablesLatent heat flux, sensible heat flux, net shortwave flux, net longwave flux, directional wind stress, precipitation, 10-m wind speed, 10-m air temperature, 10-m specific humidity, SST, total cloud amount, sea level pressureLatent heat flux, sensible heat flux, net shortwave flux, net longwave flux, 10-m wind speed, 10-m air temperature, 10-m specific humidity, SST, total cloud amount, sea level pressure
  Extension planned to include precipitation & directional wind stress
Grid1° monthly1° monthly, based on 1° daily fields
Quality controlICOADS 4.5-sigma trimmingICOADS 4.5-sigma trimming, data outside range ± 10 data units ( °C, m s−1, g kg−1, mb) of daily background field also excluded
Bias adjustmentsSST (Josey et al., 1999)SST—none
 Air temperature (Kent et al., 1993)Air Temperature (Berry et al., 2004)
 Visual wind speed (Lindau, 1995)Visual wind speed (Lindau, 1995), additional adjustment for time varying difference between visual and anemometer wind speeds (see text)
 Humidity (Josey et al., 1999)Humidity (see text)
Height adjustment to 10 mStability dependent adjustment (Smith, 1980, 1988) for wind speed, air temperature and humidityas v1.1
Heat flux parameterisationSensible and Latent Heat: Smith (1980, 1988)as v1.1
 Wind Stress (Smith, 1980) 
 Net Shortwave (Reed, 1977; Payne, 1972), Net Longwave (Clark et al., 1974) 
Ice maskClimatological monthlyWeekly ice concentration from Reynolds et al2002 interpolated to daily values. Prior to 1982 climatological values are used (Berry and Kent, 2009)
Uncertainty estimatesNoRandom (measurement plus sampling) and bias uncertainty estimates for each grid box
Global heat flux balanceVersion 1.1a available for climatological fields only (Grist and Josey, 2003)No
Dataset constructionSuccessive correction applied to 1° monthly averages of individual flux estimates and meteorological variablesFluxes calculated from 1° daily gridded fields of basic variables calculated usingOI
Random (residual bias)Not used (not specified)Air temperature: 1.07 (0.2) °C
uncertainties for input data Specific Humidity: 1.25 (0.2) g kg−1
  SST: 1.24 [max(0.15, 0.1 · |TseaTair|)] °C
  Sea level pressure: 2.1 (0.0) mb
  Wind speed: 2.15 (0.2) ms−1
  Cloud cover: 1.8 (0.0) octas

Optimal interpolation (OI) is used in NOCS v2.0 to grid the individual VOS observations and to take into account the random errors in the observations and the irregular sampling by the VOS. The natural variability of the fields being constructed and the nonlinear nature of the flux calculation formulae are accounted for by the OI and in the flux estimation. Section 2 describes the input data, the bias adjustments applied to the data and the random uncertainty estimates required by the OI. Section 3 describes the flux calculation and Section 4 the OI procedure adopted, including the calculation of monthly means and their uncertainties. Section 5 assesses the resulting OI fields and considers the accuracy of the accompanying uncertainty estimates. Section 6 contains a discussion.

2. Data sources

2.1. VOS observations, metadata and sampling characteristics

Observations from VOS, contained in the International Comprehensive Ocean-Atmosphere Data Set (ICOADS) v2.4 (Worley et al., 2005) and made between 1973 and 2006, have been used together with observational metadata from World Meteorological Organization (WMO) Publication Number 47 (Kent et al., 2007). Measurement methods and heights have been assigned to the observations by associating the metadata with the observations based on the callsigns available in both ICOADS and the metadata (Josey et al., 1999). The period 1973–2006 has been chosen for the new dataset due to the availability of the metadata on the observing practices and heights required for the various bias and height corrections (data from 1970 to 1972 have also been used in the spin up phase of the OI, see Section 4.1.2). Over this period all the variables required to calculate the fluxes are available from the VOS observations. Quality Assurance (QA) has been applied using the ICOADS ‘trimming flags’ (Wolter, 1997), excluding data outside 4.5 standard deviations of the climatological monthly mean. A track checking algorithm has been applied to the observations following Kent and Challenor (2006). Observations that cannot be tracked, for example because of missing identifiers, are included in the dataset.

Over the period chosen, the VOS observations are clustered over the major shipping lanes with VOS typically making weather reports containing the parameters required to calculate the fluxes every 3 or 6 h. In the coastal regions of Europe, North America and West Africa up to 100 observations per month can be made in a single 1° grid cell. This decreases to between 10 and 20 observations per month in the mid-ocean shipping lanes and to less than 5 observations per month away from the shipping lanes. This irregular and often poor sampling is one of the largest contributions to the uncertainty in gridded fields and flux estimates based on the VOS observations (Gulev et al., 2007) and is accounted for in the OI scheme used to grid the observations in NOCS v2.0. Although we have attempted to account for irregular and sometimes sparse sampling in the analysis, it is not possible to account for any systematic lack of observations under particular conditions, usually known as ‘fair-weather bias’. Fair-weather bias is hard to quantify within a sampling pattern that is inherently irregular; Kent and Taylor (1995) failed to find evidence for a significant effect of individual high wind events on VOS sampling in North Atlantic high latitudes. However, VOS sampling is typically lower in winter months, especially in high latitudes, and extreme events such as hurricanes are likely to be entirely unsampled.

2.2. Uncertainties, biases and error characteristics

The OI scheme used to grid the observations (Section 4.1) requires estimates of the uncertainty in the individual VOS observations. The random errors, or uncorrelated component of the uncertainty, have been estimated using a semi-variogram analysis following Kent and Berry (2005) but with several changes. Coincident observations made within 24 h of each other have been used to estimate the sample semi-variograms rather than the 1 h used in Kent and Berry (2005). This wider time separation limit has been used to include the representativeness errors in using a single value to represent a daily mean. A number of different semi-variogram models are also used to extrapolate to zero separation distance, with the Gaussian model (Isaaks and Srivastava, 1990) found to give the more consistent results for the different variables excluding wind speed (Berry, 2009). The exponential model is found to give more consistent results for wind speed. Table I gives the values of the random errors estimated for each variable used in the OI. Although spatially and temporally varying estimates of the uncertainties can be made (Kent and Berry, 2005), the values used in NOCS v2.0 are constant.

The correlated component of the uncertainty, or bias uncertainty, is by its nature difficult to estimate, especially where we have too little information to estimate and remove any bias in the observations. As a result, this quantity has been set to zero in the OI. However, we wish to estimate the magnitude of the bias uncertainty in the flux estimates. Tentative estimates of the bias uncertainty for each input variable to the flux formulae are made in the following sections based on either residual differences between methods (humidity and wind speed), residual biases (air temperature) or the range of differences observed between methods (SST). These estimates are included in the dataset, both as separate quantities and added in quadrature to the uncertainty estimates from the OI to give the total uncertainty. The residual bias uncertainty in each flux component is calculated using the estimated bias uncertainty for the input variables (Section 4.2).

2.2.1. Air temperature

Air temperature (Tair) observations are adjusted for bias due to the solar heating of the instruments' and ships' superstructure following Berry et al. (2004). This adjustment models the heat budget of the ships' structure and sensor between sunrise and the time of measurement, balancing the heat storage, convective and conductive cooling with solar heating. To apply the correction, appropriate empirical coefficients must be determined for each application. Berry et al. (2004) used differences between well and poorly exposed sensors to estimate the bias whilst Berry and Kent (2005) used differences between observations and model output. This information is generally unavailable for the VOS and, as a result, the bias has been estimated as the difference between the individual observations and a night-time only version of the optimally interpolated air temperatures. Annual estimates of the coefficients have been fitted to a subset of these bias estimates following Berry et al. (2004) to account for any change in ship characteristics over time. The bias uncertainty in the corrected air temperature observations has been estimated at 0.2 °C (Berry et al., 2004). Figure 1 shows the air temperature adjustment as a function of local time for a fixed set of environmental conditions. The variations in the daily mean adjustment with the model parameters, cloud cover, relative wind speed, location and time are illustrated in Figure 2.

Figure 1.

Mean diurnal cycle of the air temperature bias adjustment using the parameters of the best fit from 1973 to 2006 (solid line) and fixed environmental conditions of: relative wind speed = 7 ms−1, cloud cover = 3 octas; latitude = 40°N. Shading indicates maximum and minimum values of annual adjustment over the period for the same fixed environmental conditions

Figure 2.

Contour plot of daily mean air temp correction at 40°N (left) and 0°N (right) as a function of cloud cover and relative wind speed. Mean values for January (solid) and July (dashed) are shown. Note that solar radiation values for the Octa model (Dobson and Smith, 1988) peak at 1 octa rather than for clear sky

In addition to biases due to solar heat, spurious trends exist in the surface marine air temperature record due to the changing size of the VOS and the height above the sea surface at which the observations are made. The observing height has increased from around 16 m in 1973 to over 24 m by 2006 (Kent et al., 2007) which would introduce a spurious downward trend of ∼0.1 °C over the past 30 years (Rayner et al., 2003). Air temperature observations were adjusted to a standard reference height of 10 m using the bulk flux formulae and the parameterisation of Smith (1980) for the drag coefficient and Smith (1988) for the heat and moisture transfer coefficients after Josey et al. (1999).

2.2.2. Humidity

Humidity observations have been made routinely by the VOS since the early 1950s and are usually made using wet and dry bulb thermometers housed in marine screens or sling psychrometers (Kent et al., 2007). Biases have been reported in the humidity observations made using marine screens due to the inadequate ventilation of the wet bulb thermometers (Kent et al., 1993). A modification of the adjustment proposed by Kent et al. (1993) was used in the NOCS v1.1 (Figure 3).

Figure 3.

Bias corrections for screen-derived humidities. Kent et al. (1993) (solid: 1.032*Tdew-0.839), Josey et al. (1999) (dashed: 1.029* Tdew-1.080) and the present study (dotted: 0.966*qair). Tdew is the dewpoint temperature and qair is the specific humidity

Figure 4 shows a comparison of the humidity estimates from each method before and after adjustment. The uncorrected screen observations are 0.4 g kg−1 too high compared to psychrometers, with larger biases in the tropics than higher latitudes. The correction of Kent et al. (1993) is too small, that of Josey et al. (1999) under corrects at lower latitudes and over corrects at higher latitudes. An adjustment that reduces screen-derived specific humidities by 3.4% [incorrectly quoted as 4% in Berry and Kent (2009)] gives the best agreement between screens and slings, although geographical and temporal biases remain. After application of this adjustment the residual bias uncertainty is estimated at 0.2 g kg−1 based on the residual differences between observations from the two different methods. Humidity observations are adjusted to 10 m using Smith (1980, 1988) as for air temperature.

Figure 4.

Differences between screen and psychrometer specific humidity observations averaged globally (left) and zonally (right). Differences before correction (grey), after correction following Kent et al. (1993) (dashed) and after correction following Josey et al. (1999) (dotted) are shown. The differences after a reduction of 3.4% in the screen specific humidity observations are also shown (black). The inset shows the regions with at least three observations from both methods and at least 5 years of data over which the comparisons are made

2.2.3. Sea surface temperature

The VOS SST measurements are typically made by measuring the temperature of water samples either collected using a bucket or from the engine room intake (ERI). A small, but increasing, number of measurements are made using hull contact sensors (Kent and Taylor, 2006; Kent et al., 2007). The proportion of observations made using buckets has declined since 1973 as more ships have switched over to using the ERI or hull contact sensors. Buckets can be biased cold when there is heat loss from the water sample between collection and measurement (Kent and Kaplan, 2006; Kent and Taylor, 2006). ERIs can be biased warm due to the proximity of the intake to the engine (Blanc, 1986; Kent et al., 1993; Kent and Kaplan, 2006; Kent and Taylor, 2006). Corrections have been proposed for the different measurement methods (Kent and Kaplan, 2006), however, there is still uncertainty in both the magnitude and sign of the bias. SST observations are therefore left unadjusted and the bias uncertainty is estimated based on the results of Kent and Kaplan (2006) as the greater of 0.15 °C and 0.1 · |TseaTair | °C.

2.2.4. Wind speed

Observations of the wind speed and direction over the oceans by the VOS are either visual estimates or from anemometers (Thomas et al., 2005, 2008; Kent et al., 2007). The visual estimates are based on the sea state and reported using the WMO1100 Beaufort Equivalent Scale (Thomas et al., 2008). This scale is known to be biased and a polynomial adjustment has been applied to the reported visual wind speed estimates to convert them to the Beaufort Equivalent Scale of Lindau (Lindau, 1995) following Thomas et al. (2005). A further adjustment has been made to account for the increasing influence of anemometers on visual observations (Thomas et al., 2008). A factor of 1 prior to the end of 1985 and then a factor that linearly decreases to 0.95 by 2000 applied to visually observed wind speeds only brings the trends seen in the visual winds into agreement with those seen in height corrected anemometer winds (Figure 5).

Figure 5.

Global mean height and Beaufort-Scale adjusted wind speed from OI datasets constructed from anemometer winds only (solid line); visual winds only (dashed line); all data (dash-dot) and all data with bias adjustment to visual winds, see text for details (dotted line)

As for air temperature and humidity, the anemometer observations have been adjusted to a standard reference height of 10 m using the bulk formulae and parameterisations of Smith (1980, 1988). The VOS anemometer measurements also contain biases due to flow distortion by the ships' superstructure (Blanc, 1986; Moat et al., 2006a, 2006b) that can be of either sign and depend on the location of the anemometers and the relative wind direction. Currently, there is not enough information on the location of the anemometers on board the VOS to estimate the effects of flow distortion routinely. Errors have also been reported in the anemometer wind speed measurements due to the conversion from relative wind speed and direction to the true wind speed and direction (Gulev, 1999). These errors will increase the random uncertainty.

The bias uncertainty in the wind speed observations have been estimated at 0.2 ms−1 based on the residual differences between the data from the different measurement methods.

2.2.5. Sea level pressure

The VOS usually use aneroid barometers to measure the pressure, but with an increasing number of digital aneroid barometer measurements (Kent et al., 2007). A small number in the early 1970s also use mercury barometers (Kent et al., 2007). The bias uncertainty in the pressure observations is assumed to be zero.

2.2.6. Cloud cover

The VOS make visual reports of total cloud cover, cloud types for high, medium and low cloud, cloud base height and amount of low cloud cover. In NOCS v2.0 only the total cloud cover estimates are used in the parameterisations of the shortwave and longwave radiative fluxes (Section 3). Whilst there is diurnal variability in the cloud cover and small biases reported in nighttime estimates under low-illumination conditions (Hahn et al., 1995) the uncertainty that this introduces will be included in the random error estimates. Bias uncertainties have been set to zero.

3. Surface flux calculation

There are several conflicting requirements to consider when deciding how best to calculate flux estimates from VOS data. Fluxes can either be calculated for each observation (called the sampling approach) or the meteorological observations can be gridded and the fluxes calculated from the gridded values (the classical approach). Most early flux datasets used the classical approach to calculate monthly mean fluxes from monthly mean variables (Hsiung, 1986; Oberhuber, 1988) as it is computationally efficient, requiring the iterative flux calculation to be performed only once for each month and grid cell. However, biases are introduced in the flux fields due to the loss of the synoptic correlation between the basic variables in the nonlinear flux formulae (Ledvina et al., 1993; Josey et al., 1995; Gulev, 1997). This led to the adoption of the sampling approach (da Silva et al., 1994; Josey et al., 1999). However, if observations do not contain all of the parameters required to calculate the fluxes they cannot be used in the sampling method, leading to loss of data and therefore increased uncertainty (Gulev et al., 2007). A further consideration is the effect of random uncertainties propagating through the nonlinear flux formulae and producing biased flux estimates.

The goal is therefore to maximise the amount of data used, minimise random uncertainties and to retain synoptic scale correlations between the variables. In the new dataset, the classical method of calculating fluxes from gridded meteorological fields has been used. Daily fields on a 1° spatial grid have been constructed using OI from observations adjusted to a 10-m reference height; the true resolution of the resulting fields will however be coarser. Estimating the mean fields on a daily timescale forms a compromise between the competing requirements for reducing the random errors whilst maintaining the synoptic scale correlation between the variables. Correlation between variables on shorter time scales will be lost however and may result in errors of order 5% (Gulev, 1994). In data-rich regions the approach taken will reduce random errors through combining multiple observations whilst maintaining the synoptic scale correlations. In data-sparse regions the random errors will not be reduced, but the results will be comparable to the sampling method. The OI accounts for spatial sampling and data uncertainty when interpolating the observations to a regular grid. Additionally, estimates of the minimized error variance are produced as part of the output from the OI allowing uncertainty estimates to be made for the individual fields. The fluxes and flux uncertainty can then be calculated from the daily meteorological fields. As in NOCS v1.1, we use the bulk formulae of Smith (1980, 1988) for the turbulent fluxes, Reed (1977) for net shortwave flux using the albedo values of Payne (1972) and Clark et al. (1974) for the net longwave flux (Table I).

4. Dataset construction methodology

4.1. Optimal interpolation

4.1.1. Introduction

The OI used is based on the scheme developed by Lorenc (1981) and implemented following Reynolds and Smith (Reynolds and Smith, 1994). The OI performs the interpolation on the difference of individual observations relative to a first guess field (denoted data deviations) in a dimensionless form. The data deviations and error terms are normalised using the uncertainty in the first guess field (Lorenc, 1981), with the analysis value from the OI given by

equation image(1)

where Ak is the daily analysis value at point k and Pk the first guess value at point k. equation image is the uncertainty in the first guess field and required to make the analysis dimensional. rk is the sum of weighted normalised data deviations, i.e.

equation image(2)

where qi is the deviation of observation i from the first guess normalised by first guess error and wik is the least squares weight. Following Reynolds and Smith (1994), subscripts i and j are used to indicate the individual data points and k the analysis location. The weights are determined by a least square minimization of the expected error variance in the analysis values (see Lorenc (1981) or Reynolds and Smith (1994) for details). The minimization gives a set of N linear equations (Reynolds and Smith, 1994) that can be expressed in matrix notation as

equation image(3)


equation image(4)

The < πiπj> and < πkπi> terms are the ensemble average correlations between the first guess errors and < βiβj> the ensemble average correlation between the observation errors. Correlation terms between the observation errors and first guess errors are neglected. The ε terms are the normalised expected errors in the observations. Once the error terms (i.e. uncertainties) are known for the observations and first guess field together with the correlations between the different errors, the minimization can be performed and the analysis increment and error variance in the solution calculated. The analysis increment is calculated using Equation (2) and the error variance by

equation image(5)

4.1.2. Application to VOS observations

Application of the OI scheme to the VOS observations requires estimates of the first guess field and its uncertainty, the observation errors and the correlation terms in Equation (4). Observation errors have been estimated using a semi-variogram analysis (Lindau, 1995; Kent and Berry, 2005) and are summarised in Table I. Following Reynolds and Smith (1994) the previous day's analysis is used as the basis for the first guess field and the analysis error as the basis for the uncertainty in the first guess field. To ensure maintenance of the annual cycle in data-sparse regions the previous day's analysis is incremented by

equation image(6)

where ΔCt is the daily increment for day t; Cm the climatological monthly mean for month m; Dm the day of year for the mid-point of month m and Dm−1 < t < Dm. The first guess is then given by

equation image(7)

where Pk, t is the first guess for point k at time t, Ak, t−1 the analysis from the previous day and ΔCt the increment to allow for the annual cycle. The climatological monthly mean, Cm, has been estimated by averaging those VOS observations which have passed the QA tests and have been adjusted for bias and to 10 metres height were appropriate, onto a 1° monthly grid between 1973 and 2002. The monthly values were averaged to give a monthly climatology and any gaps filled by linear interpolation from adjacent grid cells. Any remaining gaps were filled using the zonal mean value and a 1-2-1 smoother applied.

The uncertainty in the first guess has been set to the previous day's analysis uncertainty plus an exponential decay model that allows the uncertainty in the first guess to rise to a preset value after a period without any data. This preset value is set to the uncertainty that would be introduced by using the climatological mean value as the estimate of the field rather than the analysis value. This has been estimated as the mean standard deviation of the observations in each grid box plus the standard deviation of the grid boxes in time. For grid boxes filled using the zonal mean, the uncertainty has been set to the mean standard deviation of the observations across the grid boxes plus the standard deviation of the grid boxes used to calculate the zonal mean. A 1-2-1 smoother was then applied. The uncertainty in the first guess is given by

equation image(8)

where equation image is the first guess for the current time step; equation image the analysis error from the previous time step; equation image the analysis error from the last time step with data; equation image the preset value the first guess error decays to for the current month; n the number of days since the last data for the current location and f(n) an exponential decay model, i.e.

equation image(9)

where λt is an e-folding timescale and n the number of days since the last observation. equation image can be calculated from the previous day's analysis error as:

equation image(10)

The initial first guess and first guess error used in the OI have been generated by running the OI for the period 1970–1972. Climatological values have been used as the starting point for 1st January 1970 and the fields from 31st December 1972 used as the first guess and first guess error for the new dataset. This allows the first guess errors to stabilise and the start point of the new dataset is coincident with the reliable matching of call signs between ICOADS and Pub. 47 (Kent et al., 2007).

In order to calculate the weights for the OI, the correlation terms in Equation (4) are required. The observations are assumed to be from different sources with uncorrelated errors and the correlation terms, < βiβj>, set to the Kronecker delta, δijij = 1 for i = j and δij = 0 when ij). The correlation between the first guess errors has been estimated using a Gaussian function:

equation image(11)

where (xixj) is the scalar distance between observations i and j, and λ the e-folding space scale for the field. The e-folding scales for both the spatial (Equation (11)) and temporal (Equation (9)) correlations have been set to 300 km and 3 days respectively. These have been chosen for pragmatic reasons, balancing the capture of the synoptic variability within the constraints of a limited number of observations. These choices set the expected minimum spatial and temporal resolution of the dataset; in poorly sampled regions the resolution will be coarser.

4.2. Flux uncertainty estimates

There are uncertainties in the flux estimates due to the errors in the input data and deficiencies in the bulk formulae. Uncertainties in the bulk formulae remain poorly known (WGASF, 2000) and have not been included in the uncertainty estimates presented with this dataset. The uncertainties due to errors have been estimated following Gleckler and Weare (1997) and using propagation of errors (Taylor, 1997). The errors in the input data have been assumed independent as a first approximation, with the uncertainties given by Equation (12).

equation image(12a)
equation image(12b)
equation image(12c)
equation image(12d)

The sigma terms are the uncertainty in the respective basic meteorological variables given by the subscripts.

The uncertainty due to the different sources of error (i.e. sampling, random and systematic errors) can be evaluated separately and summed in quadrature to give the total uncertainty in the fluxes (Gleckler and Weare, 1997). In this study the contribution of bias uncertainty to the total uncertainty has been evaluated separately from that due to random and sampling errors. The uncertainty due to random and sampling errors has been estimated using the output from the OI and Equation (12) whilst the bias uncertainty has been estimated using the values given in Section 2 for the different variables. The total uncertainty in the daily flux estimates is then given by summing these two uncertainty estimates in quadrature. Following averaging onto longer time scales (Section 4.3) the random and sampling errors will reduce, but the uncertainty due to the bias errors will not reduce.

4.3. Calculating monthly means and uncertainties

4.3.1. Monthly mean

A number of methods could be used to estimate the monthly mean fluxes and meteorological variables. The individual days could be given equal weighting, weighted by their uncertainty or only days with an analysis value used. However, applying a non-uniform weighting to the individual days will lead to the uncertainty being biased towards the well-sampled days and hence underestimated. To ensure that the uncertainty estimates correctly represent the measurement and sampling uncertainties in the monthly mean, each day must be given the same weight in both the calculation of the monthly mean and the monthly mean uncertainty. This has been done in the new dataset.

4.3.2. Monthly mean uncertainty

To produce accurate estimates of the uncertainty in the monthly mean values the correlation between the errors in the different daily analyses needs to be taken into account. This will be a function of the amount of data available for each analysis and the weight given to the preceding day's analysis. Ignoring the small effect of the increment to account for the annual cycle, the correlation between errors on two adjacent days for a grid cell can be estimated by

equation image(13)

where ρt, t−1 is the correlation between the errors on day t and the preceding day (t − 1), and wt, i the sum of the weights applied to the N observations for day t from the OI. The correlation between errors for fields multiple days apart will then be the product of correlations for adjacent days, i.e.

equation image(14)

The uncertainty in the monthly mean (or other time mean) can then be estimated using propagation of errors, i.e.

equation image(15)

where σrandom is the uncertainty in the mean of the analyses due to random and sampling errors, σr, i the uncertainty in the analysis for day i and ρk, k−1 the correlation between analyses for days k and k − 1. The uncertainty in the monthly mean fluxes has been estimated in a similar manner, replacing the daily sum of weights with the minimum daily sum of weights from the basic input variables used to calculate the fluxes that day. This will act as an upper limit for the uncertainty in the fluxes.

In addition to sampling and random errors in the OI fields there is also uncertainty due to systematic biases, either due to uncertainties in the bias corrections applied or where we know potential biases exist in the observations but are not able to apply a bias correction. These uncertainties are assumed to be constant in time and correlated across the daily analysis fields. As a result, these will not be reduced by averaging across days. The total uncertainty in our monthly fields is then given by

equation image(16)

where σmonthly is the uncertainty in the monthly mean field, σbias the estimate of the bias uncertainty and σrandom the uncertainty due to random and sampling errors.

5. Results

5.1. Validation of OI procedure

The first test of the OI dataset is whether the output fields are unbiased when compared to the input data. This was tested by calculating statistics for the differences of the ICOADS input data and the resulting OI fields. For all variables the mean global differences over the period were less than 0.01 and the range of monthly mean differences was less than ± 0.2, all values quoted in units appropriate to the variable ( °C, g kg−1, m s−1 or hPa).

5.2. Effect of data adjustments

Annual average time series and maps of the basic variables and fluxes before and after adjustment are shown in Figures 6 and 7 respectively. The region analysed is where 90% of the monthly means have a standard deviation of the daily values within the month greater than the uncertainty estimate which is a good indicator of data quality.

Figure 6.

Left panel: Annual means of the basic meteorological variables from the OI variables before (dotted line) and after adjustment (solid line) for well-sampled regions (shown by maps). Top: air temperature also showing SST as dashed line; 2nd row: specific humidity; 3rd row: wind speed. Right panel: average of difference (adjusted—unadjusted) for period 1973–2006. This figure is available in colour online at

Figure 7.

As Figure 6 but for the turbulent fluxes. Top: latent heat flux; bottom: sensible heat flux. Right panel: average of difference (adjusted—unadjusted) for period 1973–2006. This figure is available in colour online at

The adjusted air temperature is on average 0.4 °C lower than the unadjusted, increasing the mean air–sea temperature difference by over 30%. The adjustment depends strongly on the incoming solar radiation so is largest in the tropics and lowest at the Poles, modulated by cloud cover and by local variations in the typical relative wind speed over the ship. The humidity is reduced by 0.1 g kg−1 on average, increasing the air–sea specific humidity difference by 3%. The spatial variation of the humidity adjustment depends on the specific humidity itself and on the proportion of measurements made using screens, which is higher in the Pacific due to the national preferences of VOS operators. The adjustments increase the correlation between the air temperature, SST and humidity fields in the new dataset, suggesting the consistency is improved by the adjustments. The mean wind speed is increased in the early period due to the Beaufort Scale adjustment and decreased after the 1980s due to the height and bias adjustments. Over the full period the adjustments decrease the mean wind speed by 0.1 m s−1. Wind speed is reduced by a greater amount in the Pacific than in the Atlantic as anemometers are more commonly used in the Pacific and tend to be located at greater heights (Kent and Taylor, 1997). The mean sensible and latent heat fluxes are increased in magnitude by the adjustments by 3 W m−2 and 4 W m−2 respectively. The mean shortwave flux is unaffected by the adjustments and the longwave flux reduced by 0.4 W m−2.

The impact of the adjustments on the air temperature, humidity and radiative flux trends is minimal with the impact of the height adjustment on the temperature and humidity masked by the natural variability. In contrast, the trends in wind speed, latent and sensible heat fluxes are all reduced by more than half. This reduction in trend can be detected above the background natural variability and is significant at the 95% confidence level. The minimum two sample t-statistic, with 405 degree of freedom, is 3.7 (for the sensible heat flux). The Cochrane–Orcutt procedure (Thejll and Schmith, 2005) has been used to account for AR1 errors in the linear regressions used to estimate the trends. The reduced trends in the turbulent heat fluxes and wind speed are believed to be more realistic (Thomas et al., 2008; Berry, 2009).

The reduction of trends and an improvement in the consistency of temperature related variables gives us confidence that the adjustments applied to the data are improving the accuracy of the output fields. Table II shows the effects of the data adjustments on comparisons of the daily OI fields with data from a series of buoy deployments since 1981 by the Woods Hole Oceanographic Institution (WHOI) Upper Ocean Processes Group (UOP), see Berry and Kent (2009) for information on the deployments used. Comparison of gridded fields representative of a spatial average to point measurements is complicated by errors of representativeness, the characteristics of which will vary with location and with time. However, these comparisons are important as independent validation. Mean air temperatures are closer to buoy values after adjustment for 10 of the 12 deployments, the overall difference (NOCS v2.0—buoy) reducing from 0.34 °C to − 0.13 °C. There are two possible reasons why a negative difference might be expected. Firstly the buoy observations are also subject to heating errors (Anderson and Baumgartner, 1998) so might be expected to overestimate the air temperature. Secondly OI daytime air temperatures may have been slightly over corrected by referencing to night-time temperatures and might therefore underestimate the air temperature. Humidity estimates also agree better for 10 of the 12 deployments following adjustment of the OI fields, but a high bias of 0.3 g kg−1 in the OI humidities remains. This suggests that the VOS humidity observations, including sling measurements, may require further adjustment (Kent et al., 1993). Wind speed agreement is slightly improved by adjustment and adjusted OI wind speeds are about 0.4 m s−1 greater than buoy wind speeds. Thomas et al. (2005) found a similar difference between ship and buoy wind speeds, although the buoys in that study were of a different type to the WHOI UOP moorings. Latent heat flux agreement is improved following adjustment for 9 of the 12 moorings; mean differences reduce from 10 Wm−2 to 3 Wm−2. Similar results are seen for sensible heat flux, although mean differences are close to zero.

Table II. Differences between OI fields and WHOI UOP moorings (Arabian Sea Mixed Layer Dynamics Experiment; Acoustic Surface Reverberation Experiment, 1991; Coastal Mixing and Optics Moored Array, Central Mooring; Coupled Ocean-Atmosphere Response Experiment; Marine Light-Mixed Layers Experiment, 1991; Severe Environment Surface Mooring; Shelf Mixed Layer Experiment (SMILE) and five buoys from the Subduction Experiment)
FieldMean difference unadjustedMean difference adjustedBias uncertainty estimateNo. of comparisons improved by adjustment
  1. Differences are shown before and after adjustment of the OI fields. Values in brackets are differences if the coastally located SMILE mooring results are excluded. Bias uncertainties are mean values of long term buoy accuracy from (Moyer and Weller, 1997) and the OI bias uncertainty added in quadrature. Random uncertainties are negligible for these comparisons. Also given is the number of deployments where agreement with the buoy was improved by adjustment of the OI fields.

Air temperature ( °C)0.34 (0.59)− 0.13 (0.11)0.310 of 12
Specific humidity (g kg−1)0.49 (0.59)0.31 (0.41)0.310 of 12
SST ( °C)0.02 (0.18)0.2
Wind speed (m s−1)0.45 (0.38)0.38 (0.32)0.49 of 12
Latent heat flux (W m−2)9.7 (11.3)2.8 (4.1)179 of 12
Sensible heat flux (W m−2)2.3 (3.5)− 1.9 (−0.9)38 of 12

5.3. Accuracy of uncertainty estimates

In order to test the accuracy of the uncertainty estimates a cross-validation approach is taken, sub-sampling the data and re-running the OI to produce an ensemble of fields and uncertainty estimates. For the 5-year period 1970–1974 an ensemble of 10 datasets has been constructed, each member using a random selection of half the available data. The standard deviation of the different realisations is used as an alternative estimate of the uncertainty in the daily fields, recognising that this is likely to be underestimated due to the common data between the realisations. A further ensemble was constructed using only 30% of the data for each realisation to try to estimate the size of this effect. In regions with poor sampling the daily analysis fields from the OI will be strongly influenced by the first guess field and the daily analysis values will be strongly correlated with each other across the ensemble of runs. Poorly sampled grid cells are therefore excluded from this analysis following Berry (2009).

Figure 8 shows the mean ratio of the ensemble standard deviation (i.e. the alternative uncertainty estimate from the cross-validation) to the ensemble mean daily uncertainty for air temperature for the 5-year period. Regions where the ratio is greater than 1 indicate that the alternative uncertainty estimate is higher than the values from the OI, suggesting that the OI is underestimating the uncertainty. In regions where the ratio is significantly less than 1 the opposite is true, with an overestimation by the OI. In regions where the ratio is close to but less than 1 the results are less conclusive where we may be underestimating the uncertainty using the alternative approach.

Figure 8.

Ratio of standard deviation of cross-validation ensemble to OI uncertainty estimate for regions where the standard deviation of the daily values within the month is greater than the uncertainty estimate. Contour intervals are 0.25 and unity is shown by a thick line. This figure is available in colour online at

Whilst the alternative approach may be underestimating the uncertainty the results are still useful and indicate where we may be overestimating or underestimating the uncertainty in the new dataset. The results shown in Figure 8 suggest that the uncertainty is underestimated in the new dataset in high variability regions, such as over the Kuroshio and Gulf Stream currents, and overestimated in regions of lower variability. Similar results (not shown) are seen for the SST, wind speed and pressure fields but with the region of underestimation expanded for pressure and wind to cover the storm tracks. This is due to the greater variability of the wind and pressure fields. In contrast, due to the lower variability of humidity, the ratios are generally less than 1 over the majority of the oceans for humidity and close to 1 in the regions of higher variability (not shown).

5.4. Comparison of uncertainty estimates

For comparison the uncertainty estimates for the net heat flux in the new dataset have been compared to those estimated by Gulev et al. (2007). Table III lists regional average ‘interpolation uncertainty’ from Gulev et al. (2007) together with the average random and sampling uncertainty estimates for the same regions from the new dataset. The interpolation uncertainty gives an estimate of the uncertainty in the grid cell monthly mean value due to both sampling errors and the uncertainty due to interpolation in data-sparse regions. The contribution from measurement errors is not included. The random and sampling uncertainty estimates in the new dataset similarly include not only the uncertainty due to sampling errors and interpolation in data-sparse regions but also the contribution from measurement errors. It should be noted that the uncertainty estimates cover different time periods.

Table III. Comparison of regional uncertainty estimates in the monthly mean net heat flux from Gulev et al. (2007) and NOCS v2.0
RegionInterpolation uncertainty (Gulev et al., 2007)Random and sampling uncertainty from NOCS v2.0
Subpolar North Atlantic, 50°–70°N47.611.333.512.4
Subpolar North Pacific, 40°–70°N42.99.728.115.6
Midlatitudinal and subtropical North Atlantic, 30°–50°N10.
Midlatitudinal and subtropical North Pacific, 20°–40°N12.14.317.420.6
Tropical oceans, 20°S–20°N5.24.844.544.7
Southern Ocean, south of 30°S23.227.958.250.1

In the subpolar northern oceans both sets of uncertainty estimates show large uncertainties in the winter and smaller uncertainties in the summer. In the winter months, when the variability is greatest, the uncertainty estimates from Gulev et al. (2007) are larger than those in NOCS v2.0. This is consistent with the underestimation of the uncertainty in the new dataset in high variability regions identified in Section 5.3. In the summer, when the variability is lower and the underestimation of the uncertainty in NOCS v2.0 reduced or negligible, the interpolation uncertainties from Gulev et al. (2007) are lower than those in NOCS v2.0. The larger uncertainties in NOCS v2.0 are expected due to the inclusion of the measurement uncertainties. In regions of moderate variability and good sampling, such as the northern mid-latitudes, the two sets of uncertainty are comparable. In the poorly sampled tropical and southern oceans the uncertainty in NOCS v2.0 is much larger than the interpolation uncertainty estimates, again probably due to the measurement uncertainties included in NOCS v2.0 but not in the estimates of Gulev et al. (2007).

6. Summary and discussion

The methods used to calculate a new in situ global dataset of air–sea exchanges, called the NOCS Flux Dataset v2.0, have been described. The development of this dataset has been guided by the potential uses of the new dataset together with a focus on minimizing and accurately estimating the different sources of uncertainty. A discussion of the requirements for different applications can be found in WGASF (2000) and Fairall et al. (in press). The requirements for forcing oceans models have not been considered during the development of NOCS v2.0 as it is unlikely the accuracy and resolution requirements can be met on a global scale by in situ observations alone. Instead, the development has focussed on the requirements for validation of other flux datasets—such as model output or forcing fields (reanalysis, NWP, climate) and satellite flux estimates—and on climate variability studies. Essential to these applications are the realistic uncertainty estimates, enabling differences between datasets to be understood and regions where the flux estimates are poor to be excluded.

The new dataset forms a major update to the Josey et al.'s (1999) NOCS v1.1 flux dataset and is based on VOS reports from the period 1973 to 2006. The major improvements over the previous versions of the NOCS dataset are: improved bias corrections; better treatment of inhomogeneous sampling by the VOS; and the estimation of the uncertainty in the new dataset routinely as part of the dataset construction method. Prior to gridding, bias and height adjustments are applied to the air temperature, humidity and wind speed observations. The observations have then been gridded on a daily timescale using OI to take into account the uneven sampling by the VOS observations and also to produce uncertainty estimates for the gridded fields. These are then used to calculate daily estimates of the fluxes and the flux uncertainty. Monthly mean values of the different meteorological parameters and fluxes have been estimated as the average of the daily OI values. The uncertainty estimates for the monthly mean values are derived from the daily uncertainty estimates with the autocorrelation between the errors for the successive analyses taken into account.

The impact of the bias and height adjustments on the gridded fields has been shown, with the adjustments shown to reduce spurious trends in the gridded data and to improve the internal consistency of the different variables. The adjustments also improve the agreement of the NOCS v2.0 dataset with independent data from research moorings. The magnitude of the uncertainty estimates produced by the OI and estimated using propagation of errors for the fluxes are thought to be reasonable. However, cross-validation of the uncertainties suggest they are likely to be overestimated in low variability regions and underestimated in high variability regions. Further work is required to quantify this overestimation and underestimation for the uncertainty estimates; however, the uncertainty estimates can still be used as an indicator of the quality of the gridded values and flux estimates.

In addition to improving the consistency of the different variables, the adjustments reduce the global imbalance found in the net heat fluxes in NOCS v2.0 (and common to other observation based datasets). However, a significant imbalance remains in NOCS v2.0 (Berry and Kent, 2009) and the source of this imbalance is unclear. Similar imbalances are seen in satellite flux datasets, for example OAFlux (Yu and Weller, 2007) has an imbalance of 30 W m−2 over the period 1984–2004 with the oceans gaining heat. For comparison, over the same period NOCS v2.0 has an imbalance of 22 W m−2 (35 W m−2 prior to bias adjustment). The availability of uncertainty estimates will allow improved constraints to be used in any inverse analysis applied to bring the fluxes into global balance and into agreement with hydrographic estimates (da Silva et al., 1994).

Whilst NOCS v2.0 has been released development of the dataset is ongoing. Priorities for future releases are the addition of the wind stress and wind component fields, and the extension of the dataset back in time to give a 50-year record. In the longer term, improvements to the spatial and temporal length scales to improve the uncertainty estimates and also to include uncertainties due to the parameterisations themselves will be a priority.

NOCS v2.0 monthly mean meteorological data and surface fluxes can be downloaded via the project website at: Daily fields for well-sampled regions can be made available on request.


This work was funded by the UK Natural Environment Research Council (NERC) Oceans2025 programme with additional funding from the NERC and Ministry of Defence Joint Grants Scheme and the Met Office through the National Centre for Ocean Forecasting. ICOADS individual reports were obtained from the Research Data Archive managed by the Data Support Section of the Computational and Information Systems Laboratory (CISL) at the National Center for Atmospheric Research in Boulder, Colorado. We are also grateful to CISL for archiving and serving the NOCS v2.0 fields through their Research Data Archive. The WHOI mooring data was obtained from the WHOI UOP website at NOAA OI SST V2 ice data were obtained from the NCEP Environmental Modelling Centre ( The FORTRAN code for OI was adapted from code provided by Richard Reynolds and Diane Stokes of NOAA. The Ferret program, a product of NOAA's Pacific Marine Environmental Laboratory, was used for some analysis and graphics in this paper ( We thank the reviewers for their help in improving this paper.