Characterizing the retrieval of cloud properties from optical remote sensing

Authors


Abstract

[1] This paper presents a new approach to the formal characterization of the optical retrieval of cloud optical thickness and effective droplet radius based on a nonlinear methodology that is derived from a general stochastic inverse problem formulation similar to standard Bayesian estimation theory. The methodology includes efficient use of the precomputed radiative transfer model simulations which are already available in standard retrieval algorithms. Another important property of the methodology is that it does not require performing the retrieval with actual measurements in order to characterize the retrieval results. One utility of this analysis is the quantification of information content in the standard retrieval problem, and the increase of information through adding channels (radiances at different wavelengths) to the inversion. This was demonstrated for the five-wavelength retrieval using airborne hyperspectral shortwave irradiance measurements. The ability of the method to evaluate the impact of observation and radiative transfer model uncertainties on the retrieved cloud properties is also demonstrated. Further benefits from this study will be in its application to the cloud retrieval algorithms to be developed for future space- and airborne instruments. The present study puts forth the framework necessary to quantify that increase in information and to optimize new retrieval algorithms that efficiently accommodate the enhanced measurement space.

1. Introduction

[2] Knowledge of cloud properties, including their spatial and temporal variability, is needed for understanding and quantifying the role of clouds in climate variability and for modeling clouds and their effects in climate and weather models. Many techniques have been developed for cloud property retrievals from airborne and ground-based remote sensing measurements of reflected irradiance/radiance in the shortwave region. Most of the current retrieval techniques are based on using precomputed radiative transfer model simulations (commonly referred to as the look-up table) and a procedure for evaluating the minimum distance (the best fit) between the modeled and measured cloud reflectance or transmittance in the visible and near-infrared regions of the spectrum (Twomey and Cocks [1989], Nakajima and King [1990], and Foot [1988], among many others). The techniques differ in the selection of the radiative transfer model used to compute the simulated measurements for clouds of varying optical depth and effective radius, distance function (e.g., the choice of weights used for spectral regions), measurement type (e.g., irradiance and radiance) and the spectral bands used in the retrieval. In general, the accuracy of the retrieved cloud properties depends on the accuracy of the radiative transfer model, including knowledge of the uncertainties in independent nonretrieved inputs, modeling assumptions and numerical approach (e.g., the plane parallel assumption and the computation of scattering), and on the accuracy of the measurements and their sensitivity to the cloud properties that are retrieved. The importance of evaluating cloud properties requires that the accuracy and uncertainty of the cloud property retrievals are well characterized to ensure robust estimates of uncertainties in the applications. Additionally, characterization of sources of uncertainty in the retrieval algorithms and the effect of them on the accuracy of the retrieved cloud properties is needed to help with determining observing strategies and approaches for improving the algorithms.

[3] The retrievals are often validated against in situ flight measurements [King et al., 1997, section 3.3.2] to estimate accuracy and errors. This approach may provide a direct measure of the accuracy, but it is limited by the availability and characteristics of the in situ measurements. For example, the in situ measurements are only approximately representative of the cloud properties on a spatial scale of the remote sensing measurement that is used in the retrieval for the same site, unless the clouds are approximately homogenous. In addition, the in situ measurements are not representative of the wide variety of clouds on global scales to which the retrieval algorithms are typically applied. Besides direct comparisons to in situ measurements, uncertainties in the retrieved cloud properties have been evaluated by the method of error propagation [Coddington et al., 2010]. By this method the retrieval algorithm is employed repeatedly with slightly different inputs for the same observed scene. The different inputs are assumed to be representative of the uncertainties in the quantities that are not retrieved such as for example, atmospheric profiles of temperature and humidity, surface optical properties, and aerosol effects. This method requires large numbers of simulations with the retrieval algorithm to obtain a robust estimate of the uncertainties in the retrieved cloud properties. Because of this property the error propagation method is not typically used in standard operational cloud retrievals such as MODIS (Moderate Resolution Imaging Spectrometer) retrievals [King et al., 1997].

[4] The number of computations of error estimates needed for explicit error propagation could be reduced by employing the error estimation theory, where the error of the retrieved cloud properties is modeled by an assumed transformation of the input statistics of the uncertainties into the equivalent output statistics [Rodgers, 2000]. Typically, in order to make the modeling of the error statistics tractable it is assumed that both the input and output uncertainties are well represented by normal or Gaussian statistics. This assumption implies that the transformation between the input and output of the retrieval algorithm is linear [Tarantola, 2005]. The condition of linear transformation is not satisfied for cloud property retrievals because the radiative transfer model is not linear. Examples of nonlinear effects in the radiative transfer model calculations on the retrieval uncertainties in ice cloud properties retrieved from MODIS [King et al., 1997] measurements are shown by Posselt et al. [2008]. In this study a Monte Carlo type technique is used for the retrieval. The retrieval results are provided in terms of the estimates of the probability density function (pdf) of the retrieved quantities, specifically the optical depth and Ice Water Path (IWP). The results by Posselt et al. [2008] show that the pdfs of IWP and effective radius are not Gaussian, implying that the associated retrieval error statistics are also not Gaussian. The Monte Carlo type retrievals by design produce the desired uncertainty estimates and include the full nonlinear relationship between the retrieved and input quantities, but are computationally as demanding if not more so than the explicit error propagation method.

[5] In the current study an alternative, computationally efficient, nonlinear method is presented for characterizing the cloud property retrievals. The method is based on the general stochastic inverse problem theory as set forth by Mosegaard and Tarantola [2002]. The theoretical basis for the method is presented by Vukicevic and Posselt [2008] in the context of diagnostic analysis of inverse problem solutions in atmospheric and oceanic data assimilation problems. A detailed presentation of the mathematical theory is included by Tarantola [2005]. The theory is similar to standard Bayesian statistical estimation theory [Jazwinski, 1970] in that each quantity that contributes to the total quantitative knowledge of modeled and observed states is treated as a stochastic quantity with an associated pdf. A unique feature of the Mosegaard and Tarantola [2002] approach is that contributions from the model (e.g., the radiative transfer model) and the associated modeling errors (e.g., the errors from nonretrieved inputs or model formulation), the observation errors, and the prior information (e.g., the first guess in the retrieval) are each represented by a separate pdf which, when combined together result in a joint posterior pdf. The joint posterior pdf represents the most complete available knowledge of the parameters given the measurements and the (nonlinear) forward model which transforms the parameters into the measurement space. The properties of the joint posterior pdf explicitly determine the accuracy and uncertainties of the inverse solution (e.g., the retrieved cloud properties) and provide the basis for computing the Shannon information content of the measurements that are used in the retrieval [Rodgers, 2000; Posselt et al., 2008].

[6] In the work of Vukicevic and Posselt [2008] the theoretical approach by Mosegaard and Tarantola [2002] is used as the basis for developing a numerical algorithm which is used for diagnosing the impact of the model nonlinearities and model and observation errors on the joint posterior pdf without explicitly computing the propagation of errors through the model. In the current study this diagnostic approach is adapted to the problem of characterizing cloud property retrievals. In this application the nonlinear model in the problem is the radiative transfer model used in the retrieval. The computational efficiency of the new method for characterizing the cloud property retrievals, relative to the explicit error propagation and the Monte Carlo methods, results from using the precomputed radiative transfer model simulations which are already available in the standard retrieval algorithms.

[7] The new method for characterizing cloud property retrievals is presented in section 2. In section 3 this method is applied to retrievals of cloud optical thickness and cloud droplet effective radius from passive remote sensing which were performed by Coddington et al. [2010]. Coddington et al. [2010] examined the impact of aerosols on cloud optical properties retrieved from airborne measurements of reflected irradiance from marine stratus clouds made by the Solar Spectral Flux Radiometer (SSFR) [Pilewskie et al., 2003]. In the current study it is demonstrated that the equivalent effect could be quantified through the impact of the modeling errors on the posterior pdf of the retrieval. An analysis of Shannon information content of the measurements with the associated measurement errors from a selection of five-wavelength bands spanning the visible to near-infrared used in the SSFR retrievals is also presented. A summary and discussion on the potential broad application of this new diagnostic approach for characterizing cloud property retrievals are included in section 4.

2. Method

2.1. Background on Generalized Inverse Problem Theory

[8] Mosegaard and Tarantola [2002] demonstrate (also presented in detail by Tarantola [2005] and Vukicevic and Posselt [2008]) that complete, quantitative information about stochastic parameters, denoted m, which results from numerical information or data provided by stochastic measurements, denoted y, and a modeled relationship between the parameters and measurements is represented by the following equation:

equation image

where, ϕ(m) is the modeled relationship between measured quantities and model parameters and pm(m) is the posterior pdf of the quantitative information about the parameters. The stochastic information from measurements and the model is represented by pd(y) and pt(ϕ(m)∣m), respectively, while the a priori information about the parameters is contained in pp(m). The integral in equation (1) is performed over the measurement space, denoted D, and γ is a normalization constant needed to render the integral of the posterior pdf equal to unity over the parameter space. In general the parameter space is multidimensional and consists of possible discrete values of the parameters in each dimension. Equation (1) presents the generalized inverse problem solution or equivalently the generalized parameter estimation solution. The posterior pdf in the parameter space is the most complete estimate of the parameters because it includes all possible quantitative information about the parameters given the available data by the measurements and model. This information consists of possible discrete parameter values and the associated probabilities of these values.

[9] In equation (1) the model pdf is represented as conditional because the model is a transformation from the parameters into the measurement space, which implies that the model result in the measurement space is conditioned on the choice of parameters. The model pdf is also a function in the joint space of measurements and parameters by the following relationships

equation image

The joint space is at least two-dimensional (2-D) because the minimal input to the inverse problem (e.g., the retrieval) would be one parameter and one measured quantity. Vukicevic and Posselt [2008] show that pt(ϕ(m)∣m) can be obtained by associating a pdf to the model solution in the measurement space for every value of the parameter from the parameter space. This formulation of the model pdf is of critical importance for deriving the new method for characterizing the cloud property retrievals using the precomputed radiative transfer model simulations (described in section 2.2).

[10] The pdfs of the actual physical measurements and prior parameter values are more straightforward to formulate than the model pdf. As expected, pd(y) includes the systematic and random measurement error characteristics, and is interpreted as probabilities of the measurement taking value from discrete intervals (yi, yi + Δy) that span a range of possible values given the measurement uncertainty. Similarly, the prior pdf represents probabilities of the parameter taking the values within the intervals (mk, mk + Δm) from a range of possible values (e.g., physically plausible values). The prior pdf could be uniform if there is no other knowledge other than that the parameters exist within the plausible range. The increments Δy and Δm denote unit intervals, corresponding to a discretization in the corresponding spaces. The unit interval can be interpreted as the minimum error or maximum accuracy in each space.

[11] In the examples shown in section 3 the parameters are the cloud optical thickness and droplet effective radius. Therefore, the prior and posterior pdfs in equation (1) are two-dimensional pdfs over the range of possible values in these parameters. The measurements used in section 3 are spectral albedos at different wavelength, derived from the ratio of upwelling to downwelling irradiances. The model ϕ(m) in this example is the radiative transfer numerical model described by Bergstrom et al. [2003] and Coddington et al. [2008].

2.2. Representing the Cloud Property Retrieval as the Generalized Inverse Problem

[12] Regardless of the technique used, the cloud property retrieval can be expressed in the form of a generalized parameter estimation solution using equation (1) because the retrieval involves measurements and a model with the associated uncertainties. This implies that the measurements and radiative transfer model results are by definition the stochastic quantities and can be fully characterized by the associated pdfs [Posselt et al., 2008]. The prior information statistics may not be explicitly included in the standard retrieval technique, but as discussed in the previous section, the prior pdf could be represented by a constant over the given range of physically plausible values (the uniform pdf). Such a pdf indicates that all values within the range are equally probable before the measurements are used together with the model to perform the retrieval. Using the example of the retrieval of cloud optical thickness and effective droplet radius [Coddington et al., 2010] (also section 3 of this study), the generalized inverse solution is written

equation image

where (τ, r) are the optical thickness and effective radius, respectively.

[13] The measurement values in the actual retrieval are not necessarily the direct physical measurements by an instrument. In the accompanying study by Coddington et al. [2010] and in section 3 of this study the measured quantities included in y are spectral albedo and the ratio of albedo to that at a fixed wavelength where liquid water is nonabsorbing (hereafter referred to as the scaled albedo), using the formulation by Twomey and Cocks [1989]. Because these quantities are derived from actual measurements of upwelling and downwelling spectral irradiance the measurement probability density function pd(y) should be derived starting from the pdfs of actual measurements using the following relationships

equation image
equation image

where a and ar denote spectral albedo and the scaled spectral albedo, respectively, I and I are upwelling and downwelling irradiances and λ and λ* denote the wavelengths that are used in the retrieval, with λ* being the fixed wavelength used to normalize the albedo at all other wavelengths. The measurement pdf in equation (2) should be derived from the pdfs of actual physical measurements because these represent the independent data from which characteristics are known from the measurement technique. Typically it is assumed that the actual measurements are characterized by Gaussian errors. The associated pdf is then expressed

equation image

where equation imageλ denotes the mean value of the measured irradiance and image is the associated error variance. Typically, the actual physical measurement at each wavelength would represent the mean value, unless multiple instruments are used in the experiment. The error variance would be derived from the known or expected error characteristics of the measurement technique. The numerical algorithm for computing the pdf of the albedo and scaled albedo using relationships (3–4) is described in section 3.2.

[14] The formulation of the model pdf in equation (2) requires special attention. Vukicevic and Posselt [2008] demonstrated that the model pdf in the joint space of measurements and parameters could be formed by associating a 1-D pdf of model errors in the simulated measurement space to each point on a grid of discrete values of the parameters in the parameter space. This formulation is based on representing the model in terms of a discretized transformation between the parameters and measurements. The discretized transformation is referred to as the discrete transfer function. The role of the transfer function in generic parameter estimation problems is discussed at length by Tarantola [2005]. Several tutorial examples of the different nonlinear transfer functions and their effect on the posterior pdf in the inverse problem are presented by Vukicevic and Posselt [2008].

[15] In the current study the model pdf is formed by associating a 1-D pdf of model errors in albedo for each simulated albedo on a grid of discrete values of (τ, r). The 1-D pdf for each (τ, r) represents the uncertainties in the simulated albedo that originate from the uncertainties in the model inputs other than (τ, r). For example, the uncertain input could be related to the presence or absence of aerosols in the model atmosphere. This example is presented in section 3.3. In addition, structural model errors such as those that relate to numerical techniques or simplifying assumptions could be included in the 1-D pdf of model errors. Representing the model uncertainties by the pdf in the measurement space is equivalent to statistical representation of the model errors that could be obtained either from verification against the albedo observations or from comparing the model result to solutions of a more sophisticated model (e.g., 3-D radiative transfer if the plane-parallel model is used in the retrieval).

[16] In the retrieval techniques that make use of the precomputed simulations of the radiative transfer model, such as in the study by Coddington et al. [2010], the grid of discrete values of (τ, r) is already contained in the look-up table together with the model results in the measurement space. This condition suggests that the model pdf in equation (2) could be evaluated using the precomputed look-up table. The look-up table is equivalent to the discretized transfer function between the parameters and measurements that is required by the algorithm given by Vukicevic and Posselt [2008]. The availability of the table implies that evaluating the posterior pdf of the retrieval, for the purpose of characterizing the retrieval solution, would require only inexpensive computations of 1-D pdfs of the model and measurement errors in the measurement space. The numerical technique for computing 1-D pdfs in the albedo and scaled albedo space is presented in section 3.2.

2.3. Evaluating the Probability Density Function of the Retrieval

[17] The procedure for evaluating equation (2) for cloud property retrievals would involve the following relatively simple steps

[18] 1. Computing the model pdf: For each triplet (an, τn, rn) in the look-up table a 1-D discrete pdf in the albedo (or the scaled albedo) space would be computed, where n is a location index in the table. This 1-D pdf is denoted pan(a). The computation of the discrete pdf in the albedo space would require defining a grid of discrete values, denoted bins, of the albedo (or the scaled albedo). The binning in the albedo (or the scaled albedo) space is necessary for computing the products of the pdfs in equation (2). This discretization could be as fine as the minimum expected error in the measurements. As discussed in section 2.2, pan(a) would include knowledge of the error statistics of the simulated albedo (i.e., the albedo computed by the radiative transfer model). The total number of these pdfs would be N = Nτ × Nr, where Nτ and Nr are the number of discrete values of the optical thickness and effective radius values that are included in the table, respectively. The resulting set of N, 1-D pdfs would form the discrete representation of the model pdf.

[19] 2. Computing the measurement pdf: The 1-D pdf that represents the uncertainties in the actual measurement-based quantities, denoted pd(a), would be computed using the grid of discretized albedo values (the bins) as in the previous step. The numerical procedure for computing pd(a) is described in section 3.2.

[20] 3. Combining of the model and measurement pdfs: In this step the following integral would be computed for every n, on the grid of values of a

equation image

The final result which includes all n would be the discrete function of the parameters only because the integration eliminates any dependence on variability in a. In terms of the statistical estimation theory this function is the discrete representation of the likelihood function given the measurement and model results [Tarantola, 2005; Rodgers, 2000].

[21] 4. Computing the posterior pdf of the retrieval: The final step would include multiplication of a discretized prior pdf on the grid of (τn, rn) with the result of step 3, for each n by the following equation:

equation image

The posterior discrete pdf in the parameter space would be the 2-D map of these probabilities on the grid of values (τn, rn) from the look-up table. γ is the normalization constant needed to make the integral over this space equal to 1.

[22] Steps 1–4 would apply to each individual measurement used in the retrieval, such as the albedo or scaled albedo at a specific wavelength. Because the retrievals normally include the combined effect of the measurements from different wavelengths, the procedure for computing the posterior pdf should include the effect of all wavelengths that are used. Similarly, when both albedo and scaled albedo are included in the retrieval as by Twomey and Cocks' [1989] approach, the posterior pdf should include the combined effect. The combined effect of multiple measurements on the posterior pdf can be achieved by updating the prior pdf with the result of step 4, after the first measurement. Clearly, starting from the uniform prior, the first measurement would produce a posterior pdf that would characterize the retrieval from this measurement alone. Updating the prior pdf for each consecutive wavelength with the previous posterior pdf would include the information added from each wavelength. The final pdf would be the posterior pdf of the entire retrieval. This pdf should then be used to characterize the final retrieval. The evaluation of the added information from different wavelengths on the posterior pdf is itself, however, a useful result because it enables explicit demonstration of the information contained in the measurements from the different spectral bands. Such a result is presented in the specific example shown in section 3.2.

[23] The important property of the described method for evaluating the posterior pdf of the retrievals is that it does not require performing the actual retrieval with the actual measurements. The procedure described in steps 1–4 requires only data from the look-up table and the computation of 1-D model and measurement error pdfs in the measurement space. The possibility of characterizing the retrieval by the posterior pdf without performing the retrieval suggests the usefulness of the method for evaluating the retrieval technique over a wide range of cases, including different cloud environments and effect of aerosols. The method could also be used to study the impact of new measurements on the retrieval (e.g., the study of expected results of new space missions). The diagnostic results that could be obtained from the knowledge of the retrieval pdf are discussed in the following section. For short, the new method is denoted GENRA (Generalized Nonlinear Retrieval Analysis) method.

2.4. Characterizing the Retrieval Using the Posterior pdf

[24] The retrieval pdf contains information about the possible discrete values of the parameters and their probability. The information about the possible discrete values would depend mainly on practical constraints pertinent to the resolution of the look-up table. Ideally the resolution should correspond to a desired maximum accuracy or minimum error in the retrieval. The information about the probability of the values by the pdf is clearly the most important result, not only because it quantifies the likelihood of the parameter taking the value but also because it relates uniquely, by equation (2), to the accuracy and uncertainty of the measurements and the model and the sensitivity of the measured quantities to the parameters that are being retrieved. The sensitivity is included explicitly in the model pdf by the discretized transformation function that is represented by the look-up table. The explicit relationship between the posterior and the model and measurement pdfs could then be used to test the impact of the accuracy and uncertainties without performing the retrieval with the actual measurements. For example, the impact of systematic model error due to the aerosol effects on retrieved cloud properties is examined in section 3.3.

[25] In addition, several standard diagnostics about the retrieval could be derived from the posterior pdf. These are as follows:

[26] 1. Maximum likelihood value or the most likely value of the parameters, written

equation image

The maximum likelihood value is typically evaluated by the retrieval algorithms which minimize what is commonly referred to as the cost function (the weighted sum of the square differences between the modeled and actual measurements). For example, the maximum likelihood retrieval is computed in the study by Coddington et al. [2010]. Thus, the posterior pdf using the new method and the associated maximum likelihood solution that could be obtained without performing the retrieval would directly relate to the typical actual retrieval result.

[27] 2. Marginal pdfs for each parameter and the associated mean and error variance. The statistical mean values could be chosen to represent the retrieved optical properties instead of the maximum likelihood result. It is well known that the mean and maximum likelihood values would be the same if the posterior pdf is symmetrical such as for the Gaussian distribution. The variance is often used to represent the retrieval uncertainty regardless of the choice of the criterion for the single value retrieval (the maximum or mean). To date in the standard retrieval procedures the error variance would be evaluated either by verification against the in situ measurement or by the error propagation method. By the new method the variance would be evaluated directly from the marginal pdfs.

[28] 3. The information content of the measurements. In the current study the Shannon information content measure is considered [Shannon and Weaver, 1949; Rodgers, 2000; Posselt et al., 2008], by the following equation:

equation image

where N is as before the number of discrete points in the parameter space (the look-up table coordinates), pmn is the posterior pdf in this space (the result of the procedure in section 2.3) and pp0n is the prior pdf. For the purpose of computing the information content added by each measurement in the retrieval (e.g., albedo at each wavelength), the prior pdf in equation (5) should be uniform to emphasize the total information increase provided by that measurement. This approach is used in section 3.2. The information content could also be computed for each measurement relative to the previous, which would then represent the relative information added. In this case the prior pdf in equation (5) would be the posterior pdf from the last measurement used when computing the pdf by the procedure that is described in the previous section (section 2.2).

3. Examples of Characterizing the Cloud Property Retrieval by the New Method

[29] In this section the new method for characterizing the cloud property retrievals (the GENRA method) is demonstrated on the examples of retrievals of cloud optical thickness and droplet effective radius that were studied by Coddington et al. [2010]. A brief summary of the retrieval technique, model and measurements that are used by Coddington et al. [2010] is presented in section 3.1. Applying GENRA to evaluate the impact of albedo and scaled albedo measurements from different spectral bands on the cloud retrieval and the associated Shannon information content is presented in section 3.2. Analyzing the potential impacts of an aerosol layer above the clouds on the cloud retrieval is demonstrated in section 3.3 by including a systematic model error that is equivalent to the explicitly modeled aerosol effect in the Coddington et al. [2010] study.

3.1. SSFR Retrievals: Model, Look-Up Table and Measurements

[30] Coddington et al. [2010] presented results of retrieved cloud optical depth and effective radius from irradiance [W m−2 nm−1] measurements made using the Solar Spectral Flux Radiometer (SSFR) [Pilewskie et al., 2003]. The SSFR, with nadir and zenith light collectors to measure upwelling and downwelling irradiance over the wavelength range 350 to 1700 nm, was flown on the Sky Research Jet stream-31 (J-31) aircraft during the Intercontinental Chemical Transport Experiment/Intercontinental Transport and Chemical Transformation of anthropogenic pollution (INTEX-A/ITCT) [Singh et al., 2006] study based out of Portsmouth, New Hampshire. Also flown on the J-31 was the 14 channel Ames Airborne Tracking Sun photometer (AATS-14) [Russell et al., 1999], which measures spectral aerosol optical thickness over a similar range as the SSFR. The remainder of section 3.1 includes a brief summary of the experiment, SSFR forward modeling calculations, and the SSFR cloud retrieval statistic and results presented by Coddington et al. [2010].

[31] Aircraft provided the unique ability to fly a low-level leg above cloud (and below an overlying aerosol layer as measured by AATS-14) and then an upper-level leg above both cloud and aerosol layers. The upwelling irradiance measured on the upper legs was influenced by the properties of cloud and aerosol particles below. The lower leg, however, allowed for the characterization of the aerosol extinction and account for its impact on the measured cloud albedo. The aerosol (continental pollution outflow from industries along the northeast coast of the United States) was lofted out over the Atlantic Ocean and above stratus clouds by a cold stable marine boundary layer. The aerosol single scattering albedo (from ground-, ship- and aircraft-based measurements or retrievals) spanned a range of values (at 550 nm) encompassed by values of 0.8 and 1.0.

[32] A plane-parallel radiative transfer model [Coddington et al., 2008; Bergstrom et al., 2003] was used to simulate the upwelling and downwelling irradiance for a range of cloud optical thickness (from 0.5 to 100, with variable resolution followed by spline interpolation to unit spacing) and effective radii (from 1 to 30 μm in 1 micron increments). The model covered a wavelength range of 300 to 1700 nm, with 1 nm sampling resolution, and spectral resolution and filter function equivalent to the SSFR (8–12 nm). Details of the Mie scattering theory calculations undertaken to represent the scattering phase function of a gamma distribution of liquid water droplets are presented by Coddington et al. [2010]. Aerosols were included in forward modeling calculations by altitude, optical thickness (as measured by AATS-14), two values of wavelength-independent single scattering albedo (0.8 and 1.0), and wavelength-independent asymmetry parameter of 0.7. Resulting were three spectral libraries of modeled (upwelling and downwelling) irradiance for each of the upper and lower flight legs defined by 450 unique pairs of cloud optical thickness and effective radii. One library represented a baseline of cloud-only irradiance simulations. A second library included the addition of a nonabsorbing (aerosol single scattering albedo equal to 1.0) aerosol layer above the cloud. The final library included the addition of absorbing aerosols (aerosol single scattering albedo equal to 0.8) above the cloud.

[33] Following the method described by Twomey and Cocks [1989], SSFR retrievals of cloud optical thickness and effective radius (defined as the ratio of the third moment of cloud drop size distribution to the second) were calculated by finding the minimum residual in a weighted chi-square statistic composed of differences between measured and modeled albedo (defined by the ratio of upwelling to downwelling irradiance) at five wavelengths (515, 745, 1015, 1240 and 1625 nm). A detailed description of the physical basis for the statistic and choice of wavelengths are provided by Coddington et al. [2010].The retrieval statistic is presented by the following equation:

equation image

where yk and y*k denote the measured and modeled albedo, respectively, at wavelength with arbitrary index k. Due to the behavior of liquid water absorption, droplet absorption increases with particle size and is negligible at visible wavelengths. Droplet extinction is also dependent upon particle size. Therefore, it is not possible to completely separate the cloud optical thickness and effective radius variables in cloud retrievals. However, at visible and very near-infrared wavelengths where cloud absorption is extremely small the reflected signal from cloud top will be mainly influenced by optical thickness. At near-infrared wavelengths, there is a strong dependence on absorption and therefore particle size. In equation (6), the absolute difference between measured and modeled albedo at shorter (less absorbing) wavelengths is accorded more weight than at longer wavelengths because this most directly defines optical thickness. To remove the sensitivity to optical thickness, the relative difference defined by ratios of albedo at a specific wavelength to albedo at the shortest of the five retrieval wavelengths (k1 = 515 nm) are employed, and are weighted more heavily at longer (more absorbing) wavelengths. The latter contribution to the statistic provides particle size information and accounts for the smaller reflectances at longer, absorbing wavelengths. The best fit solution is identified at the minimum of the statistic in equation (6).

[34] Coddington et al. [2010] illustrated an absorbing aerosol overlying the cloud decreases the measured upward irradiance. Neglecting these absorbing aerosols in forward modeling calculations resulted in a best fit with an optically thinner cloud of smaller effective radii. The potential for a retrieval bias was not found with an overlying nonabsorbing aerosol layer.

3.2. Examining the Impact of Different Wavelengths and Scaling of the Albedo

[35] The retrievals by Coddington et al. [2010] include albedo and scaled albedo measurements at five wavelengths. To demonstrate the impact of each wavelength on the retrieval we examine the change of the posterior pdf due to information added by each wavelength. The effect of scaling the albedo is examined by comparing the posterior pdf results for the unscaled and scaled albedo measurements. To test the sensitivity of the results to cloud type, the computations were performed for four different “true” or reference cloud types. These are defined by four pairs of cloud optical thickness and effective radius values, denoted (τ0, r0). The following pairs were used: (10, 10), (20, 20), (60, 10) and (60, 20), representing clouds with different combinations of small and large optical thickness and small and large droplet effective radius. The four-step procedure that is described in section 2.2 is applied in each experiment to the set of five wavelengths used in the Coddington et al. [2010] retrievals. The wavelengths used are: 515, 745, 1015, 1240 and 1625 in units of nm. The experiments in this section assume a cloudy atmosphere without the presence of aerosols.

[36] As discussed in sections 2.22.3, the 1-D pdfs of the model and measurement data in the measurement space should be numerically derived assuming the Gaussian pdfs for the irradiance data. The pdfs of the albedo and scaled albedo were computed in the following way. First, a large Monte Carlo sample (approximately 10,000 values) of the upwelling and downwelling irradiances using the Gaussian statistical distribution was generated. Second, the corresponding samples of the albedo and scaled albedo values were computed using the relationships (3a–3b). Finally, normalized histograms were computed from these samples on the grid of discrete values (the bins) in the albedo and scaled albedo (described in section 2.3). The normalized histograms represent the discretized pdf in the albedo and scaled albedo quantities.

[37] The Gaussian distributions for the model and measurement pdfs in the irradiance space by equation (4) include different mean and variance values. For the measurement pdf the mean value is set to a slightly augmented value of the irradiance from the look-up table at the reference pair of the optical thickness and effective radius (the “true” cloud type). The augmentation is expressed as follows

equation image

where ϕ(τ0, r0) is the radiative transfer model result from the look-up table and δ is a small deviation from that value. This deviation is introduced to represent the expected small systematic error in the actual measurements. The systematic error was set to 3%. The variance of the irradiance measurements errors (σI2 in equation (4)) was set to 0.1.

[38] As discussed in sections 2.22.3, the model 1-D pdf in the irradiance space should be computed for each pair of the optical thickness and droplet effective radius values in the look-up table. Assuming that there is no systematic error in the model relative to the prescribed true cloud type, the mean of each of the model 1-D Gaussian pdfs (pan in section 2.3) would be equal to the corresponding irradiance from the look-up table. In the current experiments the variance of the model pdf in the irradiance space is set to 10 times of the variance of the actual measurements. Using these conditions for the model and measurement 1-D pdfs, the posterior pdfs of the retrieval were computed for each cloud type using the four-step GENRA procedure over the sequence of five wavelengths.

3.2.1. Impact of Albedo Measurements at Different Wavelengths

[39] First we present the 2-D posterior pdfs that correspond to the final retrieval with all five wavelengths of the albedo measurements for the four different cloud types. These are displayed in Figure 1. The results show that the posterior retrieval pdfs are not Gaussian and have different shapes for the different cloud types. The non-Gaussian pdfs demonstrate that the relationship between the albedo and the cloud optical thickness and effective radius is nonlinear and that the nonlinearity is different for the different cloud types. The common feature of the pdfs for the different cloud types is that they are more symmetrical in optical thickness than in effective radius. The asymmetry in the effective radius is toward the larger values, suggesting that the retrievals for the presented cloud types tend to “favor” a larger effective radius. Another common feature in the results shown in Figure 1 is that the maximum likelihood solutions correspond closely to the true specified values of the optical thickness and effective radius. This result is expected because the model was assumed to be free of systematic error while the measurement systematic error was set to a small value.

Figure 1.

Posterior joint pdfs of the retrieval with five wavelengths for four different cloud types: (a) τ = 10, r = 10, (b) τ = 20, r = 20, (c) τ = 60, r = 10, and (d) τ = 60, r = 20. The results apply to reference atmosphere without aerosol layer.

[40] The impact of different wavelengths on the retrievals for the different cloud types is presented by the sequence of marginal pdfs and the associated Shannon information content for the optical thickness and effective radius in Figures 25. As indicated in the introduction to this section, the sequence of pdfs was made by successively adding wavelengths to the computation of the posterior pdf. The solid curves in Figures 25 represent the final retrieval results, after all wavelengths were included, and they correspond to the 2-D pdfs in Figure 2. Overall, the results in Figures 2a and 2b through 5a and 5b demonstrate that the uncertainty of the effective radius retrieval is larger than the uncertainty associated with the optical thickness for all cloud types except for τ = 60 and r = 10 μm pair (Figure 4). The largest uncertainties are associated with the cloud type with the larger effective radius (Figures 3a and 5a).

Figure 2.

Sequences of (a and b) posterior marginal pdfs and (c) Shannon information content for the cloud type with τ = 10 and r = 10 μm. The sequences consist of the retrievals with successively added albedo measurements from five different wavelengths. The wavelengths (in nm) are included in the legend.

Figure 3.

The same as in Figure 2 for cloud type with τ = 20 and r = 20 μm.

Figure 4.

The same as in Figure 2 for cloud type with τ = 60 and r = 10 μm.

Figure 5.

The same as in Figure 2 for cloud type with τ = 60 and r = 20 μm.

[41] The sequence of marginal pdfs for the optical thickness show (Figures 2b5b) that this parameter is, as expected, well constrained by the albedo measurements at short wavelengths but that albedo at long wavelengths does provide additional information for the optically thick clouds (Figures 4b and 5b). The results for the effective radius are more diverse. For the optically thin cloud type (Figures 2a and 3a), each wavelength longer than 515 nm progressively adds more information, with the largest impact resulting from the longest wavelength. In the optically thick cloud regime the result depends on the reference effective radius. For the optically thick cloud type with the small effective radius (Figure 4a), the retrieval of the effective radius is already well constrained by albedo at wavelength of 745 nm. On the other hand, albedo at the longest wavelength is needed to produce a well constrained retrieval of the effective radius for the optically thick cloud with the large effective radius (Figure 5a).

[42] The information added by each wavelength is more formally evaluated by using the Shannon information content results shown in Figures 2c5c. The Shannon information content is equivalent to the signal-to-noise ratio assuming approximately Gaussian distribution [Rodgers, 2000]. Thus, an information content larger than 1 implies that the retrieval is useful because the signal is larger than the noise. In the current experiments the noise was represented by applying a uniform prior pdf over the range of the look-up table coordinates. Regarding the strength of constraint by the different wavelengths, the Shannon information content results lead, as expected, to qualitatively the same conclusions as the analysis of the sequence of marginal pdfs but provide more precise measure of sufficient information. For example it is evident from the information content results for the optical thickness in Figures 2c5c that the short wavelengths alone provide sufficient information to the retrieval, significantly above the noise. The information added by the longer wavelengths then helps to further reduce the error variance. In addition, the information content results show the short wavelength where sufficient information occurs and that the amount of additional information from the longer wavelengths varies with the cloud type. The added information at the longer wavelengths is largest for the optically thick cloud with small effective radius (Figure 4c).

[43] For effective radius, the Shannon information content analysis demonstrates that for the cloud type with the large effective radius (Figures 3c and 5c), the signal is better than the noise only when the longest wavelength (1625 nm) is added, implying that the retrieval became useful then. Also, the results for the optically thin clouds with a small effective droplet radius (Figure 2c) indicate that the retrieval becomes useful only after the albedo measurement at the wavelength of 1240 nm is added although the maximum likelihood solution appears identifiable already at the wavelength of 745 nm (the marginal pdf in Figure 2a), Overall, the results in this section demonstrate that the GENRA method of the retrieval posterior pdfs and the Shannon information content diagnostic enables characterization of the impact of individual spectral information and sensitivity of the impact to the cloud type.

3.2.2. Impact of the Scaled Albedo Measurements

[44] In this section the computation of the marginal posterior pdfs and the Shannon information content for the different cloud types uses scaled albedo at wavelengths longer than 1000 nm (the last three wavelengths in the sequence). The short wavelength used for scaling the albedo is 515 nm as in the Coddington et al. [2010] study. The retrieval simulated in this way by the GENRA method is similar but not exactly equivalent to the approach given by Coddington et al. [2010] because the weights in the cost function are different. In the current study the weights are set by the inverse problem theory and are equal to the inverse of combined error variances of the measurements and model [Tarantola, 2005; Vukicevic and Posselt, 2008], while in the standard retrieval approach the weights are set based on the physical arguments that the shorter wavelengths have more sensitivity to the optical thickness and longer wavelengths to the effective radius. The exact numerical values of the weights are otherwise set arbitrarily by the standard retrieval method which makes use of equation (6).

[45] In Figures 67 we show examples of the sequence of marginal pdfs and the Shannon information content for optically thin and thick cloud types having a large effective radius using the combination of unscaled and scaled albedo measurements. The results indicate that scaling of the albedo at longer wavelengths significantly improves the retrieval of the effective radius. The associated improvement in the optical thickness retrieval is evident only for the pair τ = 60 and r = 20 μm (compare Figure 7b with 5b). The retrieval of the optical thickness for the optically thin cloud type was already perfect without using the scaled albedo (compare Figures 5b and 3b). The results in this section demonstrate that the GENRA method could be effectively used in diagnosing the impact of the change of variable and the associated error variance in the retrievals. Such diagnostic analysis could be applied to evaluate the optimal weights in the retrieval for different spectral bands. This will be explored in future studies.

Figure 6.

The same as in Figure 3 for the experiment that includes combined albedo and scaled albedo measurements and reference cloud type with τ = 20 and r = 20 μm.

Figure 7.

The same as in Figure 6 for the cloud type with τ = 60 and r = 20 μm.

3.3. Potential Impact of Aerosols by Including the Systematic Model Error

[46] In this section, we describe the resulting shifts of mean value and change in uncertainty in the 2-D posterior pdfs that result from adding a spectrally dependent systematic model error in the albedo. This systematic model error simulates the addition of an overlying aerosol layer above the cloud. The maximum likelihood solution for the posterior pdf simulates the retrieved cloud optical thickness and effective radius pair that would result when the radiative effects of the overlying aerosol layer are included in forward modeling calculations (as discussed in section 3.1). The shift between this solution with respect to the true cloud optical thickness and effective radius pair (i.e., for the baseline case where aerosols are absent as described in section 3.2) represents the cloud retrieval bias that would result from neglecting these aerosols in forward modeling calculations.

[47] Coddington et al. [2010] demonstrate that the presence of an overlying absorbing aerosol layer above cloud reduces the albedo. Neglecting this aerosol presence in forward modeling calculations results in a biased cloud retrieval. In the absence of other, independent measurements, this bias would be indistinguishable from an aerosol indirect effect. The presence of an overlying nonabsorbing aerosol was shown to have negligible effect on the albedo, and a potential cloud retrieval bias did not occur. Therefore, we examine two cases of systematic model error in the albedos: “small” and “large.” “Small” systematic model error is defined by the difference between modeled cloud albedo in the absence and presence of an overlying nonabsorbing aerosol layer with single-scattering albedo of 1.0. “Large” systematic model error is defined by the difference to baseline of an overlying absorbing aerosol layer with single-scattering albedo of 0.8. The spectral aerosol optical thickness and wavelength-independent aerosol asymmetry parameter are the same in both systematic model error cases.

[48] Figure 8 shows the spectral albedo and percent difference in albedo (for cloud optical depth of 60 and effective radius of 10 μm) with respect to the baseline, cloud-only case (black) for the small and large cases of systematic model error represented by a nonabsorbing (red) and absorbing (blue) overlying aerosol. The values of modeled cloud albedo and cloud-plus-aerosol albedo used in defining the systematic model error were obtained from forward modeling calculations as discussed by Coddington et al. [2010] An overlying absorbing aerosol decreases the upwelling irradiance, decreasing the albedo. This effect is apparent at wavelengths short of 800 nm, where the aerosol optical thickness, having an approximate inverse wavelength dependence [Bergstrom et al., 2007], has the strongest effect. The five vertical lines represent the wavelengths used in the SSFR cloud retrievals. These wavelengths avoid gas absorption bands (mainly water vapor). The percent difference plot illustrates that percent differences in spectral albedo range from >5% to approximately 1% over the wavelength range 350 and 800 nm when absorbing aerosols are included and are small (fraction of 1%) with the nonabsorbing aerosols. A positive percent difference means the above-cloud albedo exceeds the above-cloud-and-aerosol albedo. While the magnitudes of spectral albedo vary for other cloud optical thickness and effective radii pairs, the percent differences between above-cloud albedo and above-cloud-and-aerosol spectral albedo change very little (not shown).

Figure 8.

(top) Spectral albedo and percent difference of spectral albedo related to the baseline cloud-only cases (black) for cloud optical thickness = 60 and effective radius = 10 μm. The baseline (black) is obscured by spectrum that includes nonabsorbing (purely scattering) aerosol (red). Vertical lines in spectral albedo plot indicate the five wavelengths used in the SSFR retrieval, where the albedo is sequentially applied using the five-step procedure described in section 2.2. (bottom) Changes in above-cloud albedo due to overlying absorbing aerosol are shown in blue.

[49] Figures 9a9d show 2-D maximum likelihood plots of cloud optical thickness and effective radius for the case of “large” systematic model error. Solutions are shown for the same four pairs of cloud optical thickness and effective radius defined in section 3.2. Results for the case of “small” systematic model error are not shown because it resulted in negligible change to the above-cloud albedo (see Figure 8). As seen in Figure 1, the maximum likelihood solutions for the 2-D posterior pdfs are also well defined. The white cross indicates the cloud optical thickness and effective radius pair in the absence of systematic model error (referred to as “truth” in the following discussion). Figures 9a and 9b show results for the cloud optical property pairs with smaller cloud optical thickness (<20). A shift in the maximum likelihood solution with respect to the “truth” is seen toward larger effective radii (from 1 to 2 μm), with no shift in the maximum likelihood solution for cloud optical thickness. Figures 9c and 9d show results for the cloud optical property pairs of larger cloud optical thickness (>60). In this case, a shift in the maximum likelihood solution with respect to the “truth” is seen toward smaller cloud optical thickness (by approximately 10), with no shift in the maximum likelihood solution in effective radius.

Figure 9.

Posterior joint pdfs of the retrieval with five wavelengths for four different cloud types: (a) τ = 10, r = 10, (b) τ = 20, r = 20, (c) τ = 60, r = 10, and (d) τ = 60, r = 20. The results apply to an atmosphere which includes a “large” systematic model error representing a cloud with an overlying absorbing aerosol layer. The white cross indicates the maximum likelihood retrieval result for the atmosphere without an aerosol layer above the cloud (shown in Figure 1) and is referred to as the “truth.”

[50] The behavior shown in Figure 10 (taken from Coddington et al. [2009] for above-cloud top albedo at very near-infrared (850 nm) and near-infrared (1625 nm) wavelengths) is instructive in understanding the differing biases discussed above. The near-orthogonality between cloud optical thickness and effective radius can be seen for clouds of larger optical thickness (>40). This suggests that the variables cloud optical thickness and effective radius are near-independent in this region. The solid near-vertical lines each represent a constant cloud optical thickness, and the dashed near-horizontal lines represent values of constant effective radius. For clouds with optical thickness below 20, the behavior deviates from orthogonality, which suggests that the variables cloud optical thickness and effective radius are no longer near-independent. At zero optical thickness, the albedo is equivalent to the surface albedo.

Figure 10.

Albedo at two wavelengths (850 and 1625 nm). Solid, vertical lines are of constant cloud optical thickness. Dashed, horizontal lines are of constant effective radius. For clouds with optical thickness less than 20, these lines deviate from orthogonality.

[51] An absorbing aerosol overlying cloud reduces the cloud albedo as seen in Figure 9. Consequently, albedo at 850 nm (horizontal axis in Figure 8) is reduced more than albedo at 1625 nm (vertical axis in Figure 8). For clouds of larger optical thickness, the reduction in albedo at 850 nm will result in a smaller retrieved cloud optical thickness and no effect in effective radius due to the near-independent relation between the cloud optical thickness and effective radius at larger cloud optical thickness. However, for clouds of smaller optical thickness (<40), a reduction in the albedo at 850 nm can also translate into a bias in the retrieved effective radius, which is primarily dependent upon the albedo at near-infrared wavelengths.

[52] The results shown in Figure 9 cannot be completely inferred from Figure 10. Figure 10 describes albedo at two wavelengths only, whereas albedo at five wavelengths is used for the results shown in Figure 9. The wavelengths 850 and 1625 nm are chosen because they are near the band centers for MODIS bands 2 and 6 that are used in (nonoperational) cloud retrievals [King et al., 1997; Platnick et al., 2003]. Haywood et al. [2004] have shown theoretical results indicating that this MODIS channel combination is sensitive to an overlying absorbing aerosol layer, and that a potential exists for this bias to be interpreted as an apparent indirect aerosol effect. They show that an overlying absorbing aerosol can cause a retrieval bias to smaller cloud effective radius (a retrieval bias to smaller cloud optical thickness is also shown) which has consequences for remote sensing of the indirect aerosol effect.

4. Summary and Discussion

[53] This paper presents a new method to formally characterize the optical retrieval of cloud optical thickness and effective droplet radius based on a nonlinear methodology, similar to standard Bayesian estimation theory, described by Vukicevic and Posselt [2008]. The new methodology efficiently uses the precomputed radiative transfer model simulations which are already available in standard retrieval algorithms. Several utilities of this analysis are the quantification of information content in standard retrieval problems, the increase of information obtained through adding channels (radiances at different wavelengths) to the inversion, and the dependence of information content on cloud type. This was demonstrated for the five-wavelength retrieval algorithm of Twomey and Cocks [1989], the method adopted by Coddington et al. [2010] in using airborne hyperspectral shortwave irradiance measurements to retrieve cloud optical properties. It was also demonstrated that the new method could be effectively used to quantify the potential impact of systematic errors on the retrieval, such as neglecting the presence of aerosols.

[54] Further benefits from this study will be in its application to the cloud retrieval algorithms to be developed for future space- and airborne instruments. The community is driving toward instruments with more complete and continuous spectral coverage that will make available several hundred (rather than two or three) channels from which clouds or aerosol properties can be derived. Although the information content distributed among the larger set of measurements is largely redundant, it will likely represent an enhancement over the handful of bands available from today's discrete band imagers. The present study puts forth the framework necessary to quantify that increase in information and to optimize new retrieval algorithms that efficiently accommodate the enhanced measurement space. At least two of the proposed Decadal Survey Missions (CLARREO and ACE) will fly hyperspectral imagers measuring reflected shortwave radiance; studies such as this one will assist in meeting the challenges of interpreting the largely expanded data sets acquired in those missions.

Acknowledgments

[55] This study was supported by NSF award ATM 0754998 while the first author was at University of Colorado, Boulder. Contributions by the second and third author were supported by NASA grant NNX08AI83G and NOAA grant NA06OAR4310085.

Ancillary