SAR and Passive Microwave Fusion Scheme: A Test Case on Sentinel‐1/AMSR‐2 for Sea Ice Classification

The most common source of information about sea ice conditions is remote sensing data, especially images obtained from synthetic aperture radar (SAR) and passive microwave radiometers (PMR). Here we introduce an adaptive fusion scheme based on Graph Laplacians that allows us to retrieve the most relevant information from satellite images. In a first test case, we explore the potential of sea ice classification employing SAR and PMR separately and simultaneously, in order to evaluate the complementarity of both sensors and to assess the result of a combined use. Our test case illustrates the flexibility and efficiency of the proposed scheme and indicates an advantage of combining AMSR‐2 89 GHz and Sentinel‐1 data for sea ice mapping.

• We propose an adaptive scheme for the fusion of passive microwave radiometers and synthetic aperture radar data for sea ice classification • We demonstrate the flexibility and efficiency of the proposed scheme on a test case by evaluating the performance of various data scenarios • We illustrate the advantages and limitations of applying the data of each sensor simultaneously and separately lower frequencies at the expense of much reduced spatial resolutions at tens of kilometers. The main benefit of PMR is that their wide swaths allow a daily coverage over most of the ice-covered polar regions which is extremely useful for monitoring sea ice at synoptic scales (Heinrichs et al., 2006).
In the past, some attempts were undertaken to combine PMR and SAR data, to improve sea ice concentration estimation that is normally performed with only PMR (Beaven et al., 1996;Karvonen, 2014;Wang et al., 2016). In our study, we focus on exploring the potential of improving SAR-based sea ice classification by additionally using PMR imagery.
We apply a data fusion scheme that includes the graph-based information selection method which relies on information theory metrics in conjunction with a supervised classification approach (Liaw & Wiener, 2001). The flexibility of the proposed scheme is well-suited for efficiently analyzing different data combinations that can be beneficial for sea ice monitoring. In this short letter, we investigate the potential of combining SAR and PMR data for sea ice classification, which we demonstrate for a test case using a combination of data from the Sentinel-1 C-band (5.405 GHz) SAR, and the 89 GHz channel of the Advanced Microwave Scanning Radiometer 2 (AMSR-2).
The rest of this paper is organized as follows. Section 2 describes the data sets used in this study. Section 3 provides details of the proposed data fusion scheme. Section 4 presents the experimental validation of the proposed method. Finally, the discussion and conclusions are presented in Section 5.

Data Set
The following section describes the Sentinel-1/AMSR-2 data set. The Sentinel-1 image was acquired in extrawide swath mode in dual-polarization (HH and HV) at 40 m spatial resolution, which is commonly used for mapping sea ice. From AMSR-2 we use the brightness temperature (BT) of the 89 GHz channel at horizontal (H) and vertical (V) polarization which has a footprint of 3 × 5 km. The data set was acquired over the southern part of Svalbard on 17 March 2021. The Sentinel-1 data were corrected for thermal noise and calibrated to sigma-naught in dB using the ESA Sentinel-1 Toolbox. Both data sets were collocated and the AMSR-2 data were upsampled to the Sentinel-1 pixel-size which means that adjacent pixels can be completely correlated. Figure 1 shows the false-color composites of (a) SAR, and (b) PMR images, as well as (c) a natural-color composite of an optical image acquired by the Ocean and Land Color Instrument on Sentinel-3. The time gap between SAR and PMR scenes was a few minutes, while the optical scene was acquired a few hours later. In particular over the Open Water (OW) area on the right side, indications of cloud liquid water and atmospheric water vapor are visible, which corresponds to the occurrence of clouds over the same area in the optical image.
The Sentinel-3 optical data were used in addition to the SAR and PMR scenes for identifying various sea ice classes, especially the Grey Ice (GI) in the Storfjorden. One main motivation to add PMR data for sea ice mapping is based on occasional difficulties to separate OW and ice in SAR imagery. Despite its sensitivity to atmospheric parameters, we selected the 89 GHz channel because of its higher spatial resolution compared to the lower-frequency PMR bands. Using the three different data sources together was extremely beneficial for visual inspection and manual identification of various sea ice and water classes. Supported by the input of an expert from the Norwegian Meteorological Institute (MET Norway) we could distinguish the following sea ice types: brash ice (BI), GI, and thin first-year ice (TFYI), as well as OW areas. The spatial distribution of the regions of interest (ROIs) that were used for the classification is illustrated in Figure 1d, while the classes, number of training samples, and ROIs used are shown in Table 1. We note that the ROIs are placed with large enough distances between them to avoid mapping upsampled PMR pixels corresponding to a single original PMR pixel into different ROIs. Nevertheless, the degree of correlation between pixels is considerably higher for the PMR data than for the SAR data.

Methods
In the following subsections, we briefly describe the main steps of the proposed fusion scheme, namely pre-processing including the collocation and upsampling of the original data sets, as well as extraction of textural features for SAR data, the selection of relevant attributes, and finally the parallel classification. A more detailed technical description of the information retrieval method can be found in Khachatrian et al. (2021).

Pre-Processing
The Sentinel-1 and AMSR-2 images differ in their areal coverage, spatial resolution, and the coordinate system used for presenting the data. Therefore as a first step, we make the data comparable by means of collocating, upsampling the PMR data to the SAR resolution, and extracting the overlapping area. In addition to the original radar intensities, we extract 10 texture features available in the ESA's Sentinel Application Platform (SNAP) for each SAR polarization using the Gray-Level Co-Occurrence Matrix (GLCM) (Haralick et al., 1973;Kandaswamy et al., 2005) for 0°, 45°, 90°, and 135°. The results are averaged which is common practice to account for the possible rotation of different sea ice or ocean surface structures.

Attribute Selection
In this section, we briefly describe the information selection method (referred to as GKMI, i.e., Gaussian kernel and mutual information) that is part of the proposed scheme which we employ to select relevant attributes (Khachatrian et al., 2021). This approach consists of three main steps: segmentation, graph building, and graph clustering. It should be noted that even though the whole processing scheme proposed in this study is supervised due to the selected classification method, the information selection step is unsupervised.

Segmentation
In order to preserve the particularity of distinct areas in the observed Sentinel-1 and AMSR-2 scenes and to optimize the algorithm in terms of execution time (ET), we implemented the attribute selection on superpixels, that is, groups of neighboring pixels showing homogeneous characteristics throughout the considered multivariate data set. It should be noted that there are other patch-wise approaches, for example, using regularly spaced windows. However, fixed windows often still cover variations of characteristics, which is avoided in the superpixel approach. Moreover, superpixels allow employing the algorithm on a local scale, which is crucial since some image parts might require different types of attributes to effectively represent different ice types and characteristics. In the SAR images, the decrease of the local incidence from near-to far-range has to be considered as well. An image can be split into superpixels using different segmentation methods, such as Watershed (Beucher, 1992) or Felzenszwalb (Felzenszwalb & Huttenlocher, 2004). In this work, we determine the superpixels using the Simple Linear Iterative Clustering segmentation method (Neubert & Protzel, 2014). Furthermore, the number of superpixels, as well as their size, are parameters that can be changed within the algorithm depending on user preferences and applications. The superpixel segmentation is only used as a part of the information selection step.

Graph Building
To find the attributes which are best suited for sea ice classification, we apply a selection method, that relies on information theory metrics and on a representation based on graph Laplacians. Unlike existing graph-based Note. The number of training samples refers to the number of pixels in the ROIs specifically used for training.
clustering methods that are only using kernels as similarity measures (representing the mutual relations between the data points), we are also considering the information content of the original data. Therefore, the similarity is quantified using two metrics simultaneously, which allows us to capture relevant information at different scales which improves the precision of the selection. The mutual information (MI) is performed globally and provides a better estimation of the attributes-shared information (Vergara & Estévez, 2014), while the Gaussian kernel (GK) is applied locally and preserves the structure of the original data (Luxburg, 2007).

Graph Clustering
Once the graph is defined according to the operations that have been previously introduced, we perform the partition of the graph using a procedure inspired by the spectral clustering approach (Luxburg, 2007) in order to identify and select the most relevant attributes in the data set. The GKMI method forms groups of similar attributes, according to MI and GK metrics, and selects the most relevant from each grouping. Thus we obtain subsets of attributes that preserve the structure and the information content within a particular superpixel. The grouping is performed using k-means, which is a simple and commonly used clustering algorithm (Theodoridis & Koutroumbas, 2008).

Adaptive Selection
As was mentioned above we assume that different homogeneous parts of the image, that is, superpixels, might have different relevant subsets of attributes. Not only the attributes might differ between superpixels, but also their number, depending on the specific ice characteristics. Therefore, we include an adaptive attribute selection based on the kneedle method (Satopaa et al., 2011) in the proposed scheme, which allows us to automatically determine the relevant number of attributes. The kneedle algorithm finds the maxima of the curve to detect a beneficial point, or a "knee." In our case, this means determining the optimal number of attributes after which adding any additional attributes will be redundant.

Classification
The final step of the proposed scheme is the application of the PMR and SAR data in combination for sea ice classification. The classification is carried out using the Random Forest method, which is a widely applied classifier in remote sensing (Liaw & Wiener, 2001). Furthermore, for various superpixels, different numbers and attributes are selected. We consider this in the pixel-wise classification which improves the performance of the algorithm. In our experiments, we use pixels from the predefined ROIs for training and testing the algorithm. Moreover, employing different ROIs avoids overfitting.

Experiments
In this section, we evaluate the performance of the proposed scheme for sea ice mapping and characterization of different sea ice types. In order to quantitatively evaluate the result of sea ice classification, we apply several metrics: the Overall Accuracy (OA) index, Average Accuracy (AA), Cohen's Kappa coefficient (k), and ET. The OA shows the percentage of correctly classified samples, AA quantifies the mean of class-specific accuracies for all classes, while Kappa measures the agreement between the classification and the reference data (Bharatkar & Patel, 2013).

Performance Analysis
In order to properly evaluate the SAR and PMR fusion scheme for sea ice classification, we conducted different experiments by varying the input data combination and assessing the performance of each combination (Table 2). Table 2 shows the OA, AA, ET, and Kappa coefficient (k) obtained for the various cases: single-and multi-sensor scenarios without adaptive attribute selection, and with the proposed scheme using different numbers of selected attributes. The classification accuracies are lower if only a single sensor is used, while execution times are higher in comparison to multi-sensor combinations. It is especially evident in the case of SAR intensity (Table 2: N = 2 for the two polarizations) that the ET is significantly larger than for other data combinations with more attributes (N > 2). Because of limited complementary information, the algorithm struggles to properly assign each point to a particular class which results in lower accuracy and higher ET. Using single-sensor scenarios with a low number of considered attributes, which here are the SAR intensity or the PMR BT at two polarizations, results in a lower classification performance and reveals the lowest accuracy among all the experiments. Due to the lower resolution, PMR misses narrow variations of ice structures and types, which causes lower accuracies if the classification scheme is more detailed. In the case of SAR, low accuracies are most likely due to the influence of speckle and thermal noise. Furthermore, SAR intensity contrasts do not differ strongly between the GI and TFYI in the selected scene, which makes their separation more difficult. The best classification result of the single-sensor cases is found for the combination of intensity and GLCM textural features. Nevertheless, the multi-sensor case outperforms the single-sensor cases in terms of accuracy and computation time. The best performance is achieved for the multi-sensor case with integrated information selection. We tested this on two main scenarios, with a fixed number of attributes and adaptive/automatic selection. For all the cases with selection, except for a fixed (N = 5), the performance scores are higher and the ET is almost halved in comparison to the whole set of attributes without the selection. Moreover, the highest accuracy scores and the lowest ET were achieved with the proposed fusion scheme with an automatic determination of the number of attributes. These results strongly support the relevance of automatic information selection and parallel classification which are the crucial parts of the proposed multi-sensor fusion scheme.
Figure 2 displays the classification results for different data scenarios. Furthermore, the red contours indicate several challenging areas for classification that was improved considerably when using SAR and PMR in combination, together with variations of the proposed scheme as shown in Table 2. The classified maps on the first row of Figures 2a, 2b, and 2c show the single-sensor scenarios. The presented images clearly demonstrate the difference in spatial resolution between PMR and SAR sensors. Because of its coarser resolution, a smoother classification map is obtained from PMR. However, it is not possible to identify sea ice structures and variations of sea ice types that are smaller than the PMR resolution cell. Furthermore, there are areas of misclassification in the OW on the right side of Figure 2a, which are caused by cloud liquid water and water vapor in the atmosphere. The original SAR image is affected by speckle and thermal noise which is transferred into the calculation of texture parameters and the final classification. Furthermore, GI is very often misclassified in the SAR data. Nevertheless, the classified map obtained from SAR considering GLCM textural features illustrates a significant improvement, especially for the GI and OW classes that were misclassified in the intensity-only case.
The classified maps on the second row of Figures 2d, 2e, and 2f illustrate the multi-sensor examples with and without the proposed attribute selection. Each of these maps provides a more accurate classification compared to the single-sensor examples. The combination of SAR and PMR shows, for example, an improvement compared to the single PMR case because the potential influence of cloud liquid water and water vapor in the atmosphere is slightly reduced. On the other hand, the classified maps based on a combination of SAR and PMR preserve the smoothness of the PMR and the ability of the SAR image to identify smaller sea ice and water surface details, such as BI in the marginal ice zone. The classified map obtained with the combined data set without and with the proposed attribute selection depicts more of the GI in Storfjorden between Spitsbergen and Edgeøya in comparison to the PMR and SAR combination without texture. The performance evaluation along with visual inspection of the classified maps shows the advantages of combining various sources in order to obtain unique information regarding the area of interest. Moreover, even though the PMR is mostly used for coarse-resolution wide-coverage products, such as in the case of sea ice concentration, our study demonstrates the usefulness of integrating the PMR for sea ice type classification.

Conclusions
In this test case study, we demonstrate the application of a new, flexible, adaptive, highly accurate, and efficient multi-sensor fusion scheme on specific data combinations obtained from SAR and PMR, the most commonly used sensors for sea ice classification, ice charting, and climatological sea ice monitoring. We presented sea ice classification results separately for each sensor. The results indicate that the addition of PMR can improve the SAR-based classification in certain cases. The suggested fusion scheme reveals a better classification performance and needs less computation time compared to other investigated methods.
Because of the promising results, we plan to extend our study, which here is limited to one test case, to different ice conditions. The test case data were acquired in cold and dry winter conditions, therefore the PMR provided an image that is only marginally affected by cloud liquid water and water vapor in the atmosphere. It is also known that sea ice classification with SAR and PMR is more difficult in the melting season. Thus, future work should additionally assess the seasonal robustness of the approach and its applicability to different times of the year. We are preparing an extended data set with a larger number of Sentinel-1/AMSR-2 data acquisitions over a full season and with more complex ice conditions. In addition, we consider to investigate the usefulness of lower PMR frequency channels and derived ice concentration, considering the sensitivity of the 89 GHz channel to atmospheric parameters.