A new methodology for cloud detection and classification with ASTER data



[1] High spatial resolution sensors such as Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) have the potential to produce gridded, large area datasets of surface parameters such as elevation and emissivity. These datasets are typically derived by combining all clear-sky pixels over a given location for a specified time period necessitating the use of an automated cloud detection and classification algorithm. The current ASTER L1A cloud mask lacks several key features needed to use it for this purpose. We have developed a new cloud detection algorithm which addresses these limitations by using a 2-pass approach similar to Landsat-7 and including further spectral tests for cirrus and cloud shadows from MODIS. The new cloud detection methodology is described together with several case studies that highlight key aspects of the algorithm and comparisons with the MODIS and the current ASTER L1A cloud mask.

1. Introduction

[2] The majority of geophysical parameters derived from Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) data such as elevation, reflectance and emissivity, are generated on a scene by scene basis with no attempt to merge the data into seamless, spatial and temporal datasets. Gridded datasets are essential for many models that are not designed to assimilate scene-based data. For example, a climate model may require the land surface emissivity as an input for modeling surface-atmosphere interactions. Many of these geophysical parameters are derived from clear-sky only pixels and a reliable, accurate cloud detection algorithm is required to mask out cloudy pixels prior to generation of the clear-sky surface parameter product. The current ASTER Cloud Cover Assessment Algorithm (ACCAA) [ASTER Science Team, 1996] product is provided with the L1A data in which the bands are neither radiometrically calibrated nor geometrically registered. It was developed to enable registration of the short-wave infrared bands and lacks several key spectral tests necessary for accurate cloud detection. Other cloud masking algorithms such as the Landsat-7 Automatic Cloud Cover Assessment (ACCA) [Irish, 2000], the Moderate Resolution Imaging Spectroradiometer (MODIS) cloud mask product (MOD35) [Ackerman et al., 1998] and the AVHHR (Advanced Very High Resolution Radiometer) Processing scheme Over cLoud Land and Ocean (APOLLO) [Saunders and Kriebel, 1988] provide more sophisticated and rigorous approaches in discriminating clouds from different types of land surfaces.

[3] Spectral tests for cloud detection are based on the fact that clouds are highly reflective in the visible, near-, and mid-IR bands and are cold in the thermal bands. These characteristics are then used to set reflectance and temperature thresholds for detecting most types of cloud. However, land surfaces such as snow or highly reflective soils in desert regions can produce very similar spectral features as clouds. Also, a cloud detection scheme that may work in one area, may fail in others. We have developed an accurate and reliable cloud detection algorithm for ASTER by blending the techniques used from existing algorithms and adjusting various thresholds to appropriate values for ASTER. The new cloud mask can be used in conjunction with mosaicing algorithms to compute seamless, gridded products autonomously on a global scale. An example of such a product is the ASTER land surface emissivity database of California and Nevada [Hulley et al., 2008].

2. Background

[4] The current ASTER cloud mask (ACCAA) uses ASTER bands 2, 4 and 11 centered on 0.66, 1.65 and 8.65 μm respectively in a 4-filter process with temperature thresholds derived from Earth Observation Satellite Company [1988]. The algorithm differentiates between clouds and snow but does not account for cloud shadows, cirrus, and clouds over desert and highly reflective vegetation. The cloud mask is therefore of limited value, further reduced by the fact that the data is sub-sampled to 20 SWIR (30 m) pixels, giving a coarse spatial resolution of 600 m.

[5] The Landsat-7 ACCA algorithm employs a more rigorous approach for detecting clouds and is based on techniques used by Landsat-4 and 5 and the MODIS cloud mask. The ACCA algorithm uses eight different filters in four bands to distinguish clouds and eliminate problematic land surfaces such as snow and highly reflective desert sands. The first pass through a scene uses a spectral analysis to capture cloudy pixels, while a second pass is used to develop a statistical analysis based on the pass-1 clouds in order to identify any remaining clouds that were classed as ambiguous in the first pass. The only disadvantage of ACCA is it does not account for cloud shadows and thin cirrus.

[6] The MODIS cloud mask uses 19 channels and 13 threshold-based filters to distinguish clouds. There are additionally several ancillary data inputs such as a land-water map, topography, ecosystems and a snow-ice map. The algorithm is used for a wide variety of applications and provides four levels of confidence with regard to whether a pixel is clear or cloudy; confident clear (99%), probably clear (95%), uncertain clear (66%) and cloudy. The MODIS cloud mask is currently the most sophisticated in terms of the number of spectral tests used and the method of cloud classification. The primary drawback is the low resolution of the cloud mask (1 km) - due to the moderate spatial resolution of the input data.

3. Methodology

[7] The strategy for developing a New ASTER Cloud Mask Algorithm (NACMA) involved selecting key tests from the Landsat-7, MODIS, and AVHHR algorithms highlighted above. Using this approach enabled us to build a hybrid cloud detection algorithm that capitalized on the previous approaches. For identifying clear-pixels for surface parameter datasets, the cloud mask needs to be clear-sky conservative (i.e. overestimate rather than underestimate the amount of cloud), and filter threshold values were adjusted to account for this. Additionally, a cloud-filling technique was used to fill in gaps between clouds and shadows. Since Landsat-7 uses four bands that match the VNIR (15 m), SWIR (30 m) and TIR (90 m) ASTER bands, we decided to base our new cloud mask on the Landsat ACCA, and then adjust thresholds to utilize the unique characteristics of ASTER.

[8] The first step involves converting the L1B radiance from ASTER bands 1 through 4 centered on 0.56, 0.66, 0.82 and 1.65 μm respectively to Top Of Atmosphere (TOA) reflectance [Markham and Barker, 1986],

equation image

where, L is the TOA spectral radiance for band i (W/(m2 · sr · μm)), E is the exoatmospheric solar irradiance for each spectral band (W/(m2 · μm)), d is the Earth-Sun distance in Astronomical Units, and θ is the solar zenith angle. The at-satellite temperature for band 13 (10.6 μm) is computed from the TOA spectral radiance,

equation image

where, T is the at-sensor temperature (K), K2 is is the calibration constant, 1282.71 K, K1 is the calibration constant, 666.09 (W/(m2 · sr · μm)) and L13 is the spectral radiance for band-13. In order to isolate clouds from other features on the ground such as snow, soils and vegetation, a set of 8 filters are used to filter out non-clouds and ambiguous pixels (Table 1). All threshold values are identical to the ones used for Landsat [Irish, 2000], except filter-7 which was modified and determined experimentally, filter-6 was modified for ASTER data using sensitivity tests, and filter-4 was modified depending on the percentage of snow in the scene computed from the Normalized Difference Snow Index (NDSI) [Hall et al., 1989]. Pixels with NDSI > 0.4 and band-3 reflectance, r3 > 0.11, qualify as snow and thresholds for filter-4 are adjusted to develop a more conservative cloud estimate (Table 1). Band-3 reflectance is used to distinguish snow from water, as their NDSI values are similar. These adjustments are critical since the presence of snow has the potential to cause significant cloud classification problems.

Table 1. Pass-1 Filters and Threshold Tests Using Reflectance, ri and Temperature, Tsat, Values From Equations (1) and (2)
FilterThreshold TestFunction
1 Brightness Thresholdr2 > 0.08Eliminates low reflectance, dark pixels
2 Snow ThresholdNDSI = (r1r4)/(r1 + r4) < 0.7Eliminates snow
3 Temperature ThresholdTsat < 300Eliminates warm surface features
4 Band 4/5 Composite(1 − r4)Tsat < 240 ⇒ snow presentEliminates cold surfaces - snow, tundra
 (1 − r4)Tsat < 250 ⇒ snow absent 
5 Growing vegetationequation image < 2Eliminates reflective growing vegetation
6 Senescing vegetationequation image < 2.3Eliminates reflective senescing vegetation
7 Rocks and Sandequation image > 0.83Eliminates reflective rocks and sand
8 Warm/Cold Cloud(1 − r4)Tsat > 235 ⇒ warm cloudWarm and cold cloud classificiation
 (1 − r4)Tsat < 235 ⇒ cold cloud 
Cloud Shadowr3 < 0.05 and equation image > 1.1Detects cloud shadows

[9] Pass-2 of the algorithm replicates that of Landsat as detailed by Irish [2000] which uses a further 15 filters to classify any remaining ambiguous pixels. A thermal cloud profile from pass-1 is computed using the cold/warm cloud discriminator from filter 8 and two new thermal thresholds calculated by histogramming the band-13 temperatures and using their 97.5 and 83.5 percentiles. The percentage of snow and desert pixels in a scene along with the mean cloud temperature are used to further classify the ambiguous cloud pixels into upper (warm) or lower (cold) threshold classes using the pass-1 cloud thermal profile. The thresholds are computed using pass-1 cloud statistics that include the mean and maximum cloud temperature, standard deviation and histogram skewness. Once the cloud identification is complete, the cloudy pixels accepted in pass-2 are combined with the pass-1 clouds to give a total cloud coverage estimate.

[10] NACMA also includes two additional cloud tests that were not designed for Landsat-7 ACCA. The first is the detection of cloud shadows using MODIS spectral tests. Cloud shadows need to be masked out since they cause emissivity values to lose spectral contrast. Cloud shadow detection is based on band-3 (0.82 μm) reflectance, and is refined by using a ratio test between band-3 and 1 (0.56 μm) to further distinguish between shadows and water features (Table 1). The thin-cloud/cirrus test utilizes a brightness temperature difference test between ASTER bands-13 (10.6 μm) and 14 (11.3 μm) (BT10.6 − BT11.3) and works well for thin cirrus, the edges of thicker cloud, low reflectance clouds and a variety of other types of cloud. This test was based on the AVHRR approach developed by Saunders and Kriebel [1988]. The brightness temperature differences arise due to the different emissivities of the cloud at the two wavelengths as a result of the non-linear nature of the Planck function. The calculation involved computing ASTER BT differences for a variety of clear-sky scenes over varying terrain features including desert (259 scenes), mountainous terrain (197 scenes) and vegetation (257 scenes) in California. For clear sky conditions, the differences were generally less than 1 K, but could be as large as 6 K for cloudy conditions. The maximum differences were noted and used to set appropriate thresholds (Table 2). For a pixel with given BT10.6, BT differences greater than the threshold values were classified as cloud. However, on occasion it was found that snow at high altitude was classified as cloud, although this only applied to a few limited cases. To overcome this problem, the BT differences were adjusted depending on the percentage of snow in the scene. If the snow percentage was more than 50%, more conservative thresholds were set, and vice versa (see Table 2).

Table 2. Brightness Temperature Thresholds for the Thin Cloud/Cirrus Test
BT10.6 (K)BT10.6–11.3 (K)
Snow > 50%Snow < 50%

4. Cloud Mask Results and Comparisons

[11] Validating any cloud detection scheme is a difficult task as quantitative analysis can only be attained from collocated ground truth measurements such as lidar or from weather stations [Ackerman et al., 1998]. However, this remains a challenge as ground measurements are typically not collocated in time and also observe different cloud properties to passive cloud measurements [Baum et al., 1995]. In this study we evaluated the performance of NACMA by visual image inspection and from comparisons with the MODIS and ACCAA. In the comparisons, ASTER and MODIS-Terra data were used and the MODIS scenes were geolocated with the ASTER scene of interest acquired at the same time. We developed a comprehensive dataset (38 scenes) for testing the cloud mask using statistics generated from the mask to look at a variety of different types of cloud over different land surfaces (snow, water, vegetation, desert). From comparisons with ASTER false-color visible images at 15 m resolution (using bands 3, 2, 1 as red, blue and green respectively), we found excellent agreement with the NACMA cloud mask. Four challenging cloud detection cases are now presented in more detail.

[12] Figure 1 shows the the three cloud masks for a spotted cumulus cloud distribution. This scene was chosen to illustrate the significant improvement of the NACMA mask at high resolution (90 m), when compared to ACCAA (600 m) and MODIS (1 km). The NACMA mask provides excellent visual agreement with the clouds in the ASTER false-color image, in both resolution and cloud position. The MODIS mask flags most of the scene as uncertain (66% confidence), and areas flagged with clouds have low correlation with the visible image cloud positions. The ACCAA mask shows partial agreement, but misses the vast majority of clouds.

Figure 1.

Comparisons between an ASTER false-color image at (a) 15 m resolution on 7 July, 2001, (b) the new ASTER cloud mask, (c) the current ASTER L1A mask, and (d) the MODIS (MOD35) cloud mask. The legends are: for all three cloud masks, black is clear and yellow is cloudy, but for NACMA, blue is shadow, and for MODIS blue is uncertain.

[13] Figure 2 shows the cloud mask results for high, thin cirrus over a snow-covered region near Mono Lake, CA. Visually, the cirrus is hard to see in the false-color image, except for the blurred appearance of cirrus over the lake. The NACMA and MODIS mask however show thin cirrus is present for ∼53% of the scene. The detection of thin cirrus over snow is a challenging task for NACMA, as ASTER does not have MODIS band 26 (1.38 μm) and 27 (6.7 μm) that are particularly useful for detecting thin cirrus. Furthermore, cirrus and snow have almost the same brightness temperature differences at high altitudes, making it hard to distinguish between them. Nonetheless, the NACMA and MODIS masks agree reasonably well, except for a portion of cirrus north of the lake. The ACCAA results do not show any correlation with either NACMA or MODIS mask, and this reflects the lack of any spectral tests for cirrus. A further ten cases were examined under similar conditions and NACMA showed good agreement with MODIS in all except for one case, where snow on mountain tops above 2 km altitude were classified incorrectly as cloud.

Figure 2.

Same as Figure 1, except for a scene on 3 March, 2005 showing cirrus over an area of snow near Mono Lake, CA.

[14] Figure 3 shows the NACMA cloud mask for a scene over a desert landscape, making it a challenging case for cloud detection. We can see excellent correlation between the mask (Figure 3b) and clouds in the visible image (Figure 3a). The mask overestimates the amount of cloud since the filters are tuned for maximum cloud detection. Figure 3c and 3d show the corresponding ASTER emissivity maps for band 10 (8.3 μm), with (Figure 3d), and without (Figure 3c), the cloud mask and other Quality Assurance (QA) parameters applied. Clearly most, if not all, of the spurious emissivity values resulting from cloud contamination (e < ∼0.85) have been removed (white areas in Figure 3d).

Figure 3.

An example of the new cloud mask applied to an ASTER emissivity scene (AST_05) on 8 July, 2000: (a) the ASTER false-color, (b) the new ASTER cloud mask, (c) the original ASTER emissivity image, and (d) the resulting image after the cloud mask and Quality Assurance (QA) parameters are applied. For the cloud mask, blue is clear, yellow is cloudy, and cyan is shadow.

[15] Figure 4 shows a scene with multiple cloud layers consisting of altostratus above a layer of cumulus. Again this is a difficult case for cloud detection because of the large temperature variation of almost 10 K. This could potentially cause cloud classification errors with filter-4 and pass-2 statistics. Figure 4b shows the brightness temperature at 10.6 μm, where the clouds are seen as cooler temperatures (blue-green). The result of the pass-1 mask is shown in Figure 4c. By comparing the pass-1 mask with the true color image, we can see that the bulk of the clouds are detected except for areas of thin cloud and the edges of the thicker cloud noted by areas A, B, C and D (Figure 4c), and the corresponding yellow-green colors in the BT map (Figure 4b). Pass-2 of the algorithm (Figure 4d) and the thin cloud/cirrus test (Figure 4e) then successfully classify these remaining wispy cloud areas, and the final cloud mask is shown in Figure 4f. Once again, the clouds are overestimated as expected and areas between shadow (blue) and cloud (yellow) have been filled by using a cloud-shadow proximity test.

Figure 4.

Cloud mask algorithm applied to a scene on 7 July, 2000: (a) the false-color image, (b) BT10.6, (c) the cloud mask from pass-1, (d) the pass-2 mask, (e) the thin cirrus mask, and (f) the final cloud mask. See text for further discussion.

5. Conclusions

[16] The ability to discriminate between clear and cloudy conditions using a cloud masking algorithm is essential for producing seamless, gridded surface geophysical parameter products from ASTER on a global scale. We have described a New ASTER Cloud Mask Algorithm (NACMA) which provides consistent and accurate cloud assessments for a wide range of conditions from clouds over snow, ice, desert conditions and thin cirrus. The strength of the new ASTER cloud mask lies in the fact that it benefits from a variety of existing cloud spectral tests from the MODIS, Landsat-7, and AVHRR cloud mask algorithms, and also its very high resolution of 90 m. The new cloud mask algorithm will help produce surface geophysical parameter products from ASTER, such as Land Surface Temperature and Emissivity (LST&E), which have a broad range of science applications from climate modeling and cryospheric research, to improving retrieval methodology and numerical weather prediction models. Furthermore, the new cloud mask can be applied and used for a host of other applications that require cloud masking at high resolution or sensors with similar spectral coverage.


[17] The research described in this paper was carried out at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration.