By continuing to browse this site you agree to us using cookies as described in About Cookies
Notice: Wiley Online Library will be unavailable on Saturday 7th Oct from 03.00 EDT / 08:00 BST / 12:30 IST / 15.00 SGT to 08.00 EDT / 13.00 BST / 17:30 IST / 20.00 SGT and Sunday 8th Oct from 03.00 EDT / 08:00 BST / 12:30 IST / 15.00 SGT to 06.00 EDT / 11.00 BST / 15:30 IST / 18.00 SGT for essential maintenance. Apologies for the inconvenience.
Denoising of passive and active microwave satellite soil moisture time series
Authors
ChunHsu Su,
Corresponding author
Department of Infrastructure Engineering, University of Melbourne, Parkville, Victoria, Australia
Corresponding author: C.H. Su, Department of Infrastructure Engineering, University of Melbourne, Engineering Block D, Parkville, Vic 3010, Australia. (csu@unimelb.edu.au)
[1] Satellite microwave retrievals and in situ measurements of surface soil moisture are usually compared in the time domain. This paper examines their differences in the conjugate frequency domain to develop a spectral description of the satellite data, suggesting the presence of stochastic random and systematic periodic errors. Based on a semiempirical model of the observed power spectral density, we describe systematic designs of causal and noncausal filters to remove these erroneous signals. The filters are applied to the retrievals from active and passive satellite sensors and evaluated against field data from the Murrumbidgee Basin, southeast Australia, to show substantive increase in linear correlations.
[2] Soil moisture (SM) plays a key role in hydrological and meteorological processes. Spaceborne microwave remotesensing technology provides the most practical means to achieve realtime global mapping of surface SM over a range of land surface and meteorological conditions. Microwave technologies have underpinned specialized satellite missions—the operational Soil Moisture and Ocean Salinity (SMOS) mission by the European Space Agency and the Soil Moisture Active/Passive mission planned for 2014 by National Aeronautics and Space Administration (NASA). Significant efforts have been taken to evaluate SM retrieval accuracies in field campaigns [e.g., Albergel et al., 2012], which generally show high rootmeansquare differences caused by a rather high bias but moderate to fair correlations give cause for optimism. While various renormalization methods for matching statistical moments against reference data may circumvent systematic errors in bias and variance, Su et al. [2013] found evidence that they are limited in improving correlations due to significant short time scale fluctuations.
[3] Most SM retrieval algorithms are based on instantaneous observations and have not fully exploited historical observations. This has motivated Du [2012] to improve correlations by combining highfrequency spectral components from direct sensor observations with the retrieved SM using Fourier filters. The rationale is that short time scale SM signals are better preserved in direct observations. Here we follow an alternative line of attack to improve the retrieval accuracy of current products by addressing the question: How can erroneous signals be removed from these products by using information in past satellite observations? Our starting point is to develop a semiempirical model of satellite data in the conjugate frequency domain and combine it with methods in digital signal processing to develop nearoptimal denoising filter designs. The utility of the filters is demonstrated by evaluating denoised SM against in situ measurements from southeastern Australia.
2 Data Sets
[4] We consider SM retrievals from AMSRE (Advanced Microwave Scanning RadiometerEarth Observing System), ASCAT (advanced scatterometer), and SMOS for their distinctive characteristics. AMSRE used C and Xband radiance observations to retrieve nearsurface SM. The host satellite Aqua provided daily scans of Australia during the midday ascending (1330 h local time) and nighttime descending (0130 h) orbits. We use the version 5 C/Xband 0.25° resolution product described by Owe et al. [2008] for the period July 2002 to October 2011. In contrast, SMOS provides Lband observations at 0600 h and at 1800 h. We use the most recently reprocessed (RE01, January 2010 to March 2012) daily global SM product provided by Centre Aval de Traitement des Données [Jacquette et al., 2010]. Finally, the ASCAT sensor onboard the MetOpA is a Cband scatterometer that provides observations over Australia about twice a day at about 2130 h and 0830 h. Robust retrievals are achieved using a changedetection algorithm where the moisture content is retrieved as relative wetness. We convert the version 5.4 data set (January 2007 to August 2011) produced by Vienna University of Technology to volumetric measures using soil water retention data reported by McKenzie et al. [2000]. Finally, field measurements of SM at 0–5/8 cm depth are obtained from the 82,000 km^{2} Murrumbidgee Basin at southeast Australia. Smith et al. [2012] provide details of the regional climatology and monitoring network OzNet, and we follow the data selection and resampling methods described in Su et al. [2013] for quality assurance. For comparison with the AMSRE and SMOS SM, arithmetic averages of the in situ measurements colocated in their satellite footprint are used. For ASCAT SM, measurements from each station are compared with the nearest retrievals. The ground data are subsampled to match the different observation times of the respective satellites.
3 Spectral Analysis
[5]Katul et al. [2007] presented a spectral description of in situ soil moisture dynamics. The nearsurface SM time series can be regarded as a mixture of longterm trend consisting of seasonally varying cycles (S_{S}) originating from the seasonal fluctuations in rainfall and solar radiation, and irregular fluctuations (S_{E}) from individual rainfall wetting and drying events. The longerterm trends can also result from instrument drift and longerterm climate cycles and trends, but in this study, these are ignored since only decadelong observations were considered and instrument drift was not observed. The dynamics {S_{S}, S_{E}} can be described by a onedimensional (1D) water balance equation:
dθtdt=pt−ltμ,(1)
where θ is the topsoil water content, p is the throughfall precipitation rate, l is the total loss rate from the surface layer, and μ is the layer thickness. In Fourier space, the combined power spectrum of S_{S} and S_{E} is Θ(ω)^{2} = ω^{− 2}P(ω) − L(ω)^{2}, where Θ(ω), P(ω), and L(ω) are the Fourier transforms (FTs) of θ(t), p(t)/μ, and l(t)/μ, respectively, and ω = 2π/T is the angular frequency of a spectral component with period T. This spectrum describes the distribution of total power over the different frequency components of the stochastic soil moisture process. Since typical storms last for hours while losses have a relatively slow response, the spectral amplitude of P and L decreases with increasing ω, so that the small time scale SM dynamics mimic Brownian motion and have a brownlike spectrum, Θ(ω)^{2} ∝ ω^{− 2 − κ} [Katul et al., 2007].
[6] The discrete Fourier transform (DFT) of the averaged 30 min ground data from six monitoring stations in a 25 × 25 km^{2} area in Kyeamba Creek provides experimental support to this model. In order to perform the spectral analysis, we apply a 1D gapfilling algorithm based on the discrete cosine transform (DCT) [Wang et al., 2012] to infill missing values in the data. As the FT of an individual realization of a random process does not yield an ordinary, wellbehaved function of frequency, its power distribution can be better represented by power spectral density (PSD, in the units of m^{6} m^{−6} h) calculated using Welch's method of averaging the periodograms of the time series segmented with a moving Hamming window. However, a tradeoff exists: The use of longer windows enables identification of longerperiod components and better resolutions, while shorter windows allow more samples for averaging to reduce the variance of the estimation. The results are shown in Figure 1a, depicting an overall power scaling of ω^{− 2.25}, and spectral peaks (black arrows) corresponding to the diurnal and halfdaily cycles due in part to daily temperature variations, and seasonal cycles due to largescale temporal variations of vegetation and climate.
[7] Fourier analysis of the satellite product can also provide insights into the observed dynamics and more importantly, retrieval error characteristics. While most evaluations of satellite SM [e.g., Su et al., 2013, and references therein] have focused on comparing satellite SM against in situ measurements in the time domain, our analysis represents a complementary framework in the Fourier domain. To apply conventional DFT, we assume that periodic temporal sampling by satellite sensors at approximately 12 h intervals. For ASCAT, where there are multiple overpasses in one morning or night, their averages are used as single measurements. The DCTbased infilling is then applied to provide estimates for the missing data points (34–55%) of the satellite time series, where the lengths of intermittent gaps are mainly 0.5–1 day long.
[8] Figures 1b–1d plot the PSDs of AMSRE, SMOS, and ASCAT SM time series over Kyeamba. They are representative of the spectra obtained at other locations in the catchment. Their spectra are significantly different from the in situ spectra, especially at high frequencies, suggesting that two forms of retrieval error exist in the satellite data. First, the appearance of relatively flat power distributions for T < 10^{3}h suggests a stochastic noise component. A wide range of shortterm stochastic processes such as background contributions to sensor field of view, coarse spatial sampling, and sensor uncertainties can combine to introduce such retrieval errors. Second, the spectra contain additional resonant peaks, which manifest as extraneous cycles in the time domain, suggesting a form of systematic retrieval error. The AMSRE spectrum contains 10 false resonances (red arrows) with periods (T) in the range of 1–8 days, and SMOS has about 12 false resonances. For ASCAT, the false resonances are spectrally broad or absent, and there is a noticeable gap in the T = 1.5–2 day spectral window. All three products also contain energetic components with T = 10–40 days, although the repeat cycles of satellite orbits in the range of 19–26 days may be partly responsible for this. This form of systematic error is distinct from the consistent differences in means and spread, and among other reasons, might be attributed to temporally varying biased sampling of a land surface with highspatial variability. From synthetic experiments with the ground data (not shown), we found that these features do not arise from the irregular temporal sampling by the satellites sensors.
4 Denoising Methods
[9] In this section, we build upon the water balance model in equation ((1)) to devise two denoising strategies, namely a lowpass Wiener filter and a bandstop filter to remove the identifiable stochastic and systematic errors in satellite data in the highfrequency regime. The details of their mathematical derivations and Matlab pseudocode are provided in the supporting information.
4.1 Wiener LowPass Filter
[10] To the zeroth order, the stochastic noise error can be assumed to be additive white noise so that the transform of an erroneous satellite retrieval is FT[θ_{sat}] ≡ Θ_{sat}(ω) = χΘ(ω) + E, where Θ(ω) is the solution of equation ((1)), χ is some constant scaling factor, and E is the constant amplitude of the noise. Given the linearity of FT, this statement assumes the Brownian process characteristics of pointscale SM and the water balance equation extends to the satellite footprint scale. Some evidence supporting this includes linear associations between the FTs of in situ SM at six sites colocated within the Kyeamba area (not shown) and temporal stability between pointscale dynamics [e.g., Grayson and Western, 1998].
[11] Focusing on the short time scales, we invoke firstorder approximations by linearizing equation ((1)) with l(t) = ημθ(t), where η is a constant, and assume a Poisson model of precipitation arrival to obtain the model spectrum of noisy satellite SM time series:
Θsatω2=A+ηE2+ω2E2η2+ω2(2)
for ω ≠ 0, where P(ω) ≈ P is the spectral amplitudes of the rainfall forcing and A ≡ χP is introduced. Since the noise is typically weaker than the signal, three regimes can be identified, namely, Θ_{sat}(ω)^{2} ≈ A^{2}/η^{2} in the lowfrequency regime for time scales over a month, where ω < < η; Θ_{sat}(ω)^{2} ≈ A^{2}/ω^{2} in the brown spectrum regime, where the soil acts as an integrator of random rainfall forcing; and Θ ′ (ω)^{2} ≈ E^{2}, where E^{2} > AE/ω > (A/ω)^{2} in the noise regime where the stochastic noise dominates. This model produces similar spectra to the in situ and satellite spectra, as shown by the model fits in Figure 1.
[12] We apply the standard treatment of signal processing to design a filter with a transfer function H_{L}(ω) to eliminate the stochastic error that is optimal, conditional on the validity of the signal and noise models. Optimality in the least squares sense [Wiener, 1949] requires a filter to produce a filtered signal θ_{filt}(t) or Θ_{filt}(ω) = H_{L}(ω)Θ_{sat}(ω) that minimizes the L^{2}norm of Θ_{filt}(ω) − Θ(ω). Consequently, the transfer function for such a noncausal discretetime (DT) filter is
HL,ncDTz=A22γE211−e−γz−1+11−e−γz,(3)
where z ≡ exp(iω) and γ^{2} = (A^{2} + E^{2}η^{2})/E^{2}. For a given DT input signal {θ_{sat}[n]; n = 0, 1, 2 …} sampled at times t = nΔt, the filtering can be implemented in the time domain with the following inputoutput equation
θfiltn=A22γE2∑m=0∞e−γmθsatn−m+e−γmθsatn+m.(4)
[13] This noncausal filter design that requires prerecorded data can be extended to a causal filter that is suitable for realtime operations,
HL,cDTz=γ−η1−e−γz−1,(5)
and in the time domain,
θfiltn=e−γθfiltn−1+γ−ηθsatn.(6)
[14] The gain response H_{L}(ω) ∝ (cosh γ − cos ω)^{− 2} ≈ (cosh γ − 1 + ω^{2}/2)^{− 2} describes the level of attenuation to be applied to the spectral component at frequency ω. The Wiener filter constitutes a lowpass filter that leaves the lowfrequency regime T < 10^{3}h of the spectra unchanged with a unit gain but suppresses the components at high frequencies with decreasing gain H_{L}(ω) ~ O(1/ω), resulting in a smoothed time series in the time domain and a brown spectrum in the Fourier domain. However, as the highfrequency components are also responsible for describing rapid SM changes following intense rainfall events, a tradeoff exists between removing noise and retaining this information. It is also important to note that a filtering problem is equivalent to a signal estimation problem, where the Wiener filter provides the best estimation of SM based solely on imperfect observations. While designed primarily for its gain response, the phaseshift response of a causal filter can induce undesirable signal distortions [Hamming, 1998], i.e., different spectral components within the filter bandwidth undergo temporal delays or advances of τ(ω) = ∠ H_{L,nc}(ω)/ω. These delays are most significant for components with short periodicities, where τ/T can be up to 0.21 for the T = 1 day component. To overcome this, the data can be processed twice with the same filter in forward and backward directions to negate the phase shifts, but this procedure can be done only on prerecorded data.
[15] The Wiener filter design is mathematically similar to the existing exponential filter [Wagner et al., 1999], θ_{filt}[n] = (1 − α)θ_{filt}[n − 1] + αθ_{sat}[n] for estimating profile soil water from topsoil retrievals. As the firstorder approximation and η < < γ < 1 (see later), equation ((6)) becomes θ_{filt}[n] ≈ (1 − γ)θ_{filt}[n − 1] + γθ_{sat}[n], but with notable contextual difference, where α is related to the characteristic time of the percolation process while γ is related to the signaltonoise ratio.
[16] To utilize these filters, we require estimates of the model/filter parameters {A,E,η}. As shown in Figure 1, one approach is to fit the model (equation ((2))) to the power spectrum of the historical SM time series via minimization of the L^{1}norm of Θ_{data}^{2} − Θ_{model}^{2}. This objective function is chosen over least squares so that the routine is less sensitive to spectral peaks, and fitting to Welch's PSD is preferred over a power spectrum as the filter parameters are independent of the data length and Hamming window size. Finally, we note in the supporting information that solving for the amplitude E provides a standalone method to estimate the variance of the stochastic random error in the satellite data.
4.2 Bandstop Filter
[17] Our empirical studies of satellite data have revealed the presence of extraneous resonant peaks. Owing to the linearity of the FT, we can extend the model with another additive term, ∑ _{k}δ(ω − ω_{k}), where ω_{k} are frequencies of these modes. A series of bandstop filters centered at {ω_{k}} can be used to systematically remove these resonant modes, and the simplest filter design is
where HBDTz=0 when ω = ω_{k} produces the desired gain response. The constant H_{0} is determined by setting a boundary value HBDT1=1, and the parameter r (r < 1) controls the bandwidth of the filter that increases with 1 − r, which can be optimized based on the width of the resonant mode. Finally, we implement the causal filter in the time domain with
[18] The proposed method is applied to the three satellite products to improve their agreement with the ground data. Its greatest impact is on reducing the mismatch in timing and shape of the satellite and in situ time series. By focusing on attenuating the highfrequency components, the denoising leaves the temporal mean unchanged and variance largely unaffected. Therefore, Pearson's correlation R is used as the performance metric in the evaluations and its 95% confidence interval (CI) is calculated using Fisher's transform. Correlations for SM anomalies, where the seasonal trend is removed from the satellite and ground data using a 5 week moving window [Albergel et al., 2012], are also considered.
[19] For a given site, four steps are taken to apply denoising. (1) Estimate the PSD of the historical SM time series with Welch's method and identify the frequencies ω_{k} of the false resonances with a peakfinding algorithm. (2) Apply the bandstop filters to the target data using equation (8). (3) Calibrate the Wiener filter parameters {A,E,η} by fitting the model (equation (2)) to the historical data. (4) Apply the calibrated filter to the prerecorded or realtime data using equation (4) or (6) for reanalysis or realtime applications. In effect, a lowpass filter and multiple bandstop filters are combined in a cascade to yield an output spectrum, Θ_{filt}(ω) = H_{L}(A,E,η)[∏_{k }H_{B }(ω_{k})]Θ_{sat} (ω). No ancillary or in situ data is required in the process.
[20] The denoising of the satellite data over the Kyeamba site will first be discussed. In Figure 2, red curves show reduction of these resonances of T < 200h with bandstop filters (Step 2). In this case, the bandstop filter was not applied to ASCAT due to the absence of pronounced false resonances. Step 3 yields the calibrated Wiener filter parameters listed in Table 1. With a long historical record of >4 years, AMSRE and ASCAT show good agreement with sitespecific parameters A and η of the in situ data. By contrast, the fitted A value for SMOS (~16 months) is overestimated, highlighting the limitation of using shorter records to estimate the lowfrequency end of the PSD. In other words, while the estimates of parameters E and η converge with relatively short historical record (<2 years), an accurate estimation of A requires longer time series. To overcome this, it might be reasonable to use calibrated parameters from another data set (e.g., AMSRE) for filtering SMOS data, given their similar satellite footprints; however, further evaluation is needed given differences in penetration depths at L and C/Xbands.
Table 1. Fitted Parameters of Equation ((2)) to the Estimated PSD of In Situ and Satellite SM in Figure 1a
Data Set
W
η
A
E
γ
^{a}The calculation is performed on the average of six Kyeamba stations (K*) and on single station K1. The Hamming window sizes W (in units of days) used in PSD estimation are chosen as the minimum of 3 years or the total data length. The parameters η and γ are in units of rad/h, and A and E are in units of m^{3}m^{−3} h^{1/2}.
In situ (K*)
1095
0.0010
0.0028
0

In situ (K1)
1095
0.0009
0.0025
0

AMSRE
1095
0.0009
0.0028
0.1101
0.0254
SMOS
502
0.0012
0.0033
0.0950
0.0348
ASCAT
1095
0.0010
0.0029
0.0874
0.0332
[21] Finally, in Step 4, the lowpass filters reduce the amplitude of the highfrequency components and smooth the time series (insets). We note that the mismatch between the fitted model and the ground data (Figure 1a) highlights a tendency for oversmoothing such that the filtered spectra fall below the in situ spectrum. This could potentially be improved with a better rainfall forcing model that accounts for nonzero κ in the ω^{− 2 − κ} scaling of typical SM spectra. Nevertheless, we observe statistically significant improvements (at 95% confidence level) in correlations between AMSRE and in situ data, from R_{0} = 0.727[12] (i.e., ±0.012) to R_{filt} = 0.776[10] when bandstop filters are applied and to R_{filt} = 0.808[9] following the lowpass filtering. Similar levels of improvements are found with the SMOS data where correlations increased from 0.783[25] to 0.825[21] and 0.863[17] after Steps 2 and 4, respectively. By comparison, the benefit of lowpass filtering the ASCAT data is more modest with R improving from 0.654[21] to 0.691[19].
[22] Extending the evaluation to other monitoring stations within the catchment, Figures 3a–3c show that on average across all the evaluation sites, the bandstop filters yield modest but consistent improvements of 〈R_{filt} − R_{0}〉 = 0.021 and 0.031 for AMSRE and SMOS, respectively. Further improvement is achieved through the causal lowpass filtering, notably by a further 0.054 (AMSRE), 0.099 (SMOS), and 0.050 (ASCAT) on average. Small temporal delay of SM dynamics induced by the causal filter does not result in worse performance than the noncausal counterpart. Finally, we eliminate seasonal influences on reported R scores by repeating the analyses with SM anomalies in Figures 3d–3f. With preprocessed R_{0} = 0.45–0.51 on average, the bandstop filtering improves AMSRE and SMOS data by 0.040–0.045 in correlations and combined with the lowpass filtering to yield an overall increase by 0.154–0.175. In general, improvements in correlations are greatest for low preprocessed correlation and were modest where that correlation was strong.
[23] The methodology and these results should however be taken with care. Since our analyses and filter designs are based on the physical water balance model (equation ((1))), any data periods during which SM dynamics deviate from the model (e.g., frozen soils) should be removed prior to Step 1. The formalism also treats extraneous signals from vegetation and background emission, and perturbations to soil emission due to vegetation masking as noise; thus, applying the denoising schemes renders the filtered time series more representative of an in situ SM and less representative of a remotely sensed surface layer. Two interpretations of the results are therefore possible. First, the filters remove erroneous signals in the data sets and nonSM information of the surface has been treated as random noise. Alternatively, one can interpret that information has been added and the filters create a new SM product with the Brownianprocess property expected of water balance model.
6 Conclusions
[24] Spectral analysis is a promising method for studying the error characteristics of long global SM data sets. This study provides evidence of the presence of false resonances and stochastic noise that degrade the correlations between the SM retrievals and in situ observations. Based on a semiempirical water balance model for erroneous satellite observations in the Fourier domain, designs of nearoptimal (Wiener) filters and bandstop filters are proposed and the variance of the stochastic error can also be estimated. While the characterization of the model relies on longterm records, data sets from other sensors can offer alternatives to aid in defining model parameters pertaining to onsite input forcing and moisture loss. By removing extraneous noise and errors, the proposed denoising filters are capable of improving the correlations between three distinct satellite products and in situ data. While the model/filters might be improved with more realistic spectral descriptions of precipitation and loss processes, our results give cause for optimism that the current scheme is relevant to a broader range of SM retrieval records dating back to 1978 and to future observations in operational environments.
Acknowledgments
[25] We are grateful to two anonymous reviewers and Gabrielle De Lannoy for their valuable comments on the manuscript. This research was conducted with financial support from the Australian Research Council (Linkage project LP110200520).
[26] The Editor thanks two anonymous reviewers for their assistance in evaluating this paper.