Canny optimization technique for electron microscope image colourization

Authors


K.S. Sim. Tel: (606) 252 3480; fax: (606) 231 6552; e-mail: kssim@mmu.edu.my

Summary

Images of scanning electron microscope are usually in the monochrome mode. A simple and user-friendly approach is proposed to improve the mechanical contrast of the scanning electron microscope grey images. Also, most colourization techniques involve image segmentation or region tracking, which tend to degrade the image with fuzzy or complex region boundaries. A technique is proposed, which is a hybrid between the Canny edge detection technique and the optimization technique. Compared with existing methods, the new Canny optimization technique gives satisfactory results for scanning electron microscope images.

Introduction

The scanning electron microscope (SEM) is one of the most powerful instruments that can image and analyse bulk specimens (Lee, 1992; Reimer, 1998). Usually, SEM images are in the greyscale level. With intensity different levels, mechanical contrast and artefacts can be detected properly.

Spivak et al. (1975) developed a procedure for the investigation of luminescent solids in a SEM operating in the cathode luminescence regime. Spivak et al. (1983) described a colour-coding system for qualitative and quantitative analysis of information in the SEM images. The method is based on an ‘intensity-colour’ conversion, which matches a certain colour with a certain parameter or property of the object.

In the area of fluorescence electron microscope, Hiroshi et al. (1994) developed laser diodes with the multilayer heterostructure to be observed in cathodoluminescence images using an analytical colour fluorescence electron microscope. Newbury and Bright (1999) developed an algorithm to do multiband colour coding in order to view k-value maps more effectively. This method was used for display and comparison of compositional maps in electron probe X-ray microanalysis.

Because shells and pearls often show iridescence colour, Liu et al. (1999) used a shell of the mollusk Pinctada Margaritifera, which showed strong iridescence colour, to study how the colour is produced in the layers of nacre in shell. From SEM observations, the shell showed fine scale diffraction grating structure. However, if the particular SEM images were coloured, the iridescence colour of shell could be exhibited clearly.

SEM images lack colour and are often incapable of imaging bone modifications because of magnification and chamber size limitations (Gilbert & Richards, 2000). A digital imaging method for producing extremely high depth of field enlargements of three-dimensional, sub-millimetre scale objects circumvented these problems.

The principles of image formation in natural colour scanning electron microscopy (NC-SEM) were discussed by Oho and Watanabe (2001). The method is based on the frequency characteristics of the human visual system. It was shown that the Mach effect and the masking effect are important in the characteristics. The former, which can enhance structural details, is visually similar to the edge effect in secondary electron images, and the latter is required for proper representation of much degraded colour information obtained from a light microscope. When using these effects suitably, an NC-SEM image with the resolution equivalent to that of an SEM image can be acquired, although it is composed of an SEM image and a special video microscopy image with a resolution much lower than the SEM image of identical view. The NC-SEM is more effective than the SEM in observation, interpretation and analysis of various samples with important colour information.

Takenoshita (2002) did a comparative study of SEM and electron acoustic microscopy images using quasi-colour-coded SEM and electron acoustic microscopy images of a test element group of metal-oxide semiconductor large-scale integrated chips. A new type of SEM using the coaxial backscattered electrons was developed by Jiang et al. (2002). They built a third (colour) image that allows users to give finally at the same time, in a single image, both chemical and topographic information. So far, there are no colour SEM images. If SEM can produce suitable colour images, we will be able to diagnose better the mechanical contrast of particular materials, and better analysis can be made. For materials science images, better understanding of the material structure can be realized.

Although there are many researches in image colourization, the techniques are not developed based on the SEM platform. Colourization is a process adding colour to monochrome image, and it was first invented by Markle in 1970 (Anat et al., 2004). Nowadays colourization is used extensively in magnetic resonance imaging and computerized tomography to enhance image visualization (Welsh et al., 2002). In most cases, the pseudo-colouring technique (Any pixels whose grey level is above the intensity-slicing plane will be coded with one colour, whereas any pixel below the plane will be coded with the other.) is used for these images (Gonzalez & Woods, 1992). Although pseudo-colouring technique may be applicable to SEM images, it is not suitable because the bright intensity pixels are coloured with bright colours and dark intensity image pixels with dark colours, based on the limited colour mapping. One of the main limitations of colourization technique is the requirement of colour assignments from users. It would be troublesome and time consuming if a large number of colour marking pixels are needed. Hence, a new method for colourization techniques is needed to simplify the work. In 2004, Anat et al. applied optimization techniques on colourization. Colourization with the aid of adaptive edge detection was introduced by Huang et al. (2005). In 2006, Noda et al. presented a colourization algorithm based on the MAP estimation of a colour image. Liu et al. (2006) discussed the colourization-based animation broadcast system. There are other works on the colourization technique (Horiuchi, 2002, 2003; Noda et al., 2005, 2006a, 2006b; Dongdong et al., 2007).

Previous work

Still colourization method

Figure 1 shows a still colourization image (Ming, 2005). Its colour marking is shown in Fig. 1(a). The algorithm of still colourization fails to differentiate between object and background, which leads to the uncontrollable colour flow that happens in the desired regions as shown in Fig. 1(b). The colourization algorithm starts by determining the intrinsic distance of two adjacent pixels: non-colour pixel s and colour pixel t. Equation 1 is used to define the intrinsic distance.

Figure 1.

Crack images at horizontal full-width = 10 μm. (a) Colour markings image and (b) Colourized image by still colourization method.

image(1)

where Y(.) denotes the grey intensity of a pixel after converting image to YCbCr colour space. Suppose users have specified N different colours and the set of pixels with the same colour are denoted as Ωn, n= 1, …, N, the intrinsic distance of pixel s to Ωn is defined as

image(2)

The final chrominance pixel that transferred to the monochrome pixel (as shown in Eq. 3) is the weighted sum of the top k= 3 chrominance with the minimum intrinsic distances to s.

image(3)

where the weight function w(d)=dr and r= 4. The factor r can control the smoothness of the chrominance transition.

Segmentation and optimization methods

Figure 2 shows a watershed transform to perform image segmentation. Developed using the basic concept of mathematical morphology, watershed segmentation includes all the basic functions such as Sobel edge detection technique, dilation, erosion and so on. The initial practice was to pre-process a greyscale original SEM crack image using the gradient magnitude prior to using the watershed transform for segmentation. The gradient magnitude has high pixel values along object edges, and low everywhere else. Ideally, the watershed transform should result in watershed ridge lines along object edges. However, some results show that colour is randomly assigned. Figure 2(b) illustrates this issue. In fact, Fig. 2(a) shows that the automatic segmentation cannot detect some edges when the region is small or having fuzzy boundaries.

Figure 2.

Crack image at horizontal full-width = 10 μm. (a) Original image and (b) Image after the process of watershed colourization.

Figure 3(a) shows the colour markings in a SEM image with optimization method. It needs more colour markings. In Fig. 3(b), the green colour from the top and bottom parts of the image are leaking out to the nearest block of other crack areas after being colourized by the optimization method. Hence there will be colour conflicts between objects and background when the intensity differences fail to control the colour transfer between pixels in the algorithm.

Figure 3.

Crack images at horizontal full-width = 10 μm. (a) Colour markings image and (b) Colourized image by optimization method.

In order to transfer colour from colour pixels to non-colour pixels in the optimization algorithm, the variance of monochromatic luminance value r) for eight connected neighbouring pixels of a non-colour pixel, pixel r, is calculated. The weighting function rs) is derived as inversely proportional to the intensity difference between pixels r and s, where pixel s is the colour pixel (Anat et al., 2004). The equation is

image(4)

where Y(r) and Y(s) are intensities of the pixels; and k is the proportional constant. The weight is large when the difference between intensities Y(r) and Y(s) is small, and the colour transfer between colour pixel to non-colour pixel is based on the weight as is the following equation,

image(5)

where U(r) and U(s) are the chrominance values of the pixels, and J(U) is the amount of colour transferred to pixel r from pixel s. In Fig. 3(b), the top portion of the crack is greatly influenced by the red colour from the right side of the crack area. This is because of the large weighting function rs) in the area between the top and right part of the crack area where the intensity differences in these areas are small. Therefore, the optimization algorithm has a disadvantage in controlling colour transfer between pixels when the object and background have similar intensities. It works well without the need of segmentation if pixels between objects and background have large intensity differences.

Thus, the Canny optimization method is proposed in this paper for SEM colourization. In this method, Canny edge detection is performed. After that, the edges detected are overlaid on the greyscale image to create large intensity differences between objects and background before using optimization algorithm for colourization.

New methodology

Theory

In this new Canny optimization method, we use Canny edge detection, which has shorter execution time, to produce the desired edges. A standard PC with Microsoft Windows XP operating system, 1GByte RAM, Pentium 4 processor running at 3.2 GHz, is used to perform segmentation on a 256 × 256 pixels image. The theory of Canny optimization is easily illustrated using secondary colour mixing circles shown in Fig. 4. Figure 4(a) is the greyscale image after marking with colours. Figure 4(b) shows the colourized result using optimization algorithm, with colours inside the circles overflowing the boundaries without control, after applying the optimization method. But for the overlapping parts of circles which do not have any colour assigned, optimization algorithm has made a good guess on colour assignment based on the intensity values, because the colours are mixed up very well. Red, green, blue and black are shown clearly in the image.

Figure 4.

(a) Image after marking with colours; (b) Colourized image using optimization; (c) Image with overlaying edge after colours marking; (d) Colourized image by using colour markings image with overlaying edge to perform optimization; (e) Edge modified image for Canny optimization method and (f) Colourized image using Canny optimization method.

In Fig. 4(c), a similar colour marking process on the greyscale image is used to perform optimization algorithm, except that the greyscale image has been overlaid by Canny edges using superposition theorem before doing colour markings. For the Canny edge detection, the greyscale image is first smoothened by a Gaussian filter (Canny, 1986). The equation is

image(6)

where * denotes multiplication; I[i, j] is the greyscale SEM image with i×j pixels; G[i, j, σ] is the Gaussian smoothing filter, where σ is the degree of smoothing and S[i, j] is an array of smoothened image data. Next, the gradient of S[i, j] is used to calculate x and y partial derivatives J[i, j] and K[i, j], respectively, as

image(7)
image(8)

The magnitude and orientation of gradient is computed as

image(9)
image(10)

where M[i, j] is the magnitude of gradient and θ[i, j] is the orientation of gradient. After that, non-maxima suppression is performed by labeling any gradient value that is not local maxima as zero. Thresholding with hysteresis is performed after the non-maxima suppression and the edge data are accumulated using feature synthesis. The equation is

image(11)

where N[i, j] is the edge data, T() is the threshold process and nms() is the non-maxima suppression process. The edge in edge data, N[i, j], with one pixel width is thickened through dilate process and inverted before being added to the greyscale image using superposition theorem. The equation is

image(12)

where I′[i, j] is the greyscale image after adding up with Canny edge. The edge is required to be thickened using the dilate process in order to fully isolate the object and background by creating intensity differences. The colours between object and background still influence each other if an edge with one pixel width is used, because the optimization algorithm will transfer colour between pixels through eight inter-connected neighbouring pixels, and edge with one pixel width is insufficient to perform isolation. Any pixel having negative values after the summing process should be set to zero to avoid errors because intensity values of the image cannot be negative.

In Fig. 4(d), colours of the circles are well controlled inside the boundaries after adding the Canny edge to the images, before performing optimization algorithm. However, the colours in the overlapping parts cannot be assigned properly because of the edge control. To solve this problem, another edge-modifying process is introduced in the Canny optimization method, by adding missing edges or canceling extra edges. During the edge-modifying process, the same greyscale image with an overlaid edge is used to modify the edge. The user can assign white colour onto the edge to cancel the extra edges or black colour to add in the missing edge in the image. In this case, extra edges occurred in the circle's overlapping parts so that the user can assign white to the extra edges as shown in Fig. 4(e) to cancel the edges. The mechanism of edge-modifying process for processing the modified edge image from user is shown as the following:

image

where C[i,  j] is the modified edge image from user, D[i,  j] is the greyscale image overlaid with edges before modified from user and I[i, j] is the original greyscale image. By taking these two input images which are colour marking image in Fig. 4(c) and edge modified image in Fig. 4(e), the final colourization result through the Canny optimization method is shown in Fig. 4(f), where the colour is assigned properly in the overlapping parts and are better controlled inside the circles. The Canny optimization method combines the advantages of transfer colour based on the intensities in optimization algorithm and colour flow control of edge segmentation. It allows assigning more than one colour within a segmented region and provides colour transfer control between objects and the background. The detail of a pseudo-code is shown in Appendix 1.

Figure 5 shows two SEM images. For Fig. 5(a), it is an integrated circuit (IC) compound filler sample image marking with colour whereas Fig. 5(b) is the colour marking image for gold on carbon. In the next section, we discuss the technique on how to assign threshold and sigma values for Canny edge detection technique.

Figure 5.

The SEM colour marking images. (a) The IC compound filler sample and (b) the gold on carbon sample.

Appropriate threshold and sigma values for Canny edge detection

In order to choose the preference sigma and threshold values, we decide to carry the experiments based on 200 sets of data. Without loss of generosity, we choose the IC compound filler and the gold on carbon sample images.

Figure 6 shows the IC compound filler and the gold on carbon sample images with sigma of 1 and at 3 threshold values. Figure 6(a) and (b) are the IC compound filler images with threshold at 0.1 and gold on carbon image with threshold at 0.1, respectively. By changing the threshold value to 0.2, we have IC compound filler image and gold on carbon image shown in Fig. 6(c) and (d), respectively. Lastly, Fig. 6(e) and (f) are images with threshold value at 0.3.

Figure 6.

The IC compound filler and the gold on carbon sample images with same sigma value of 1 but different thresholds.

Figure 7(a–f) are colourized with the same sigma value 2 but with different thresholds. With the increasing threshold values from 0.1 to 0.3, the effectiveness of adjusting the threshold values is shown in Fig. 8(a–f); they are colourized with same sigma value 3 but with different thresholds.

Figure 7.

The IC compound filler and the gold on carbon sample images with same sigma value of 2 but different thresholds.

Figure 8.

The IC compound filler and the gold on carbon sample images with same sigma value of 3 but different thresholds.

To measure the appropriate values for threshold and sigma in colourization SEM images for Canny optimization method, tested images are shown in three groups with different sigma values 1, 2 and 3 as shown in Figs 6–8. In each group, we set the threshold value for each image from 0.1 to 0.3. From the results shown in Figs 6–8, we can clearly observe that images with threshold 0.2 are the best in each sigma group, where the sigma 2 group has the preference colourization result. Figure 6 shows the result after applying the Canny optimization technique with sigma of 1 and threshold values from 0.1 to 0.3. Obviously, Fig. 6(c) and (d) show better colour image although the pictures are not fully coloured. This shows that the possible threshold value is 0.2. In order to confirm the results in Fig. 6, another set of image are carried out with sigma of 2 and threshold values from 0.1 to 0.3. Similarly, we show another set of images in Fig. 8. Comparing Figs 6–8, it is obvious that Fig. 7(c) and (d) have fully filled their own regions without overlapping. This exercise helps to decide the preference values of sigma and threshold within the respective ranges.

In fact, all the images in Figs 6–8 are noisy images. The reason for this exercise is that we want to show that for either noise-free or noisy images, the suitable values of sigma and threshold are able to produce good edge detection result as shown in Figs 6–8. If the chosen sigma and threshold can handle noisy images, then there is no issue in handling noise-free images. Additional images to the wire bonding IC and the IC pad, with different sigma values and thresholds, are shown in Appendix 2.

Flow chart

In the new algorithm, the grey image is first converted into luminance and chrominance colour system, which is the National Television System(s) Committee (NTSC) format, using YIQ colour system; Y is the luminance channel having grey intensity level, whereas I and Q are the chrominance channels containing colour information. After the conversion, a black and white luminance image is produced. Next, the Canny edge detection with appropriate threshold and sigma value is performed onto the luminance image. The resultant edge is overlaid onto the luminance image and this not only can create large intensity differences on boundaries of the objects during colourization process, but also will give better indication to the user for marking the colour. The ambiguity on the need of colour marking in some positions is solved once the edge segmentation regions are known. At least one colour mark is needed within an edge-segmented region. During the colour marking process, if a grey level colour such as black and white colour is marked, it preserves the original greyscale colour in that marked segment region because grey level colours contain zero colour information in the chrominance channels. Thus, no colour is transferred from these pixels. After the colour marking, a process to detect the colour marking pixel positions is performed to cancel the edges that intersect with the colour marking. This provides the alternative for the user to cancel the extra edges instead of performing the edge-modifying process. The user can mark the colour across the edge if the user would like to merge the segmented regions with the same colours by cancelling the edges between these segmented regions. The elimination of the edges is performed on the pixels where colour marking and edges are intersecting. After that, chrominance channels I and Q are extracted from the colour marked image.

The user can still use the edge-modifying process. The edges after performing edge cancellation on the intersections between colour and edge positions previously are overlaid onto the luminance image again to allow the user to modify the edges. The user can add black colour for edge adding and white colour for edge erasing, based on the edge-modifying process mechanism described previously.

Next, the modified edge image is used as Y channel and combined with I and Q channels that are previously extracted from the colour marked image to form a new NTSC image. The optimization algorithm is performed onto this image. Once the image is colourized, the original luminance channel replaces the Y channel again, so that the edge would not be visible. Finally, edge re-colouring process is performed before the final colourized image is produced to minimize erroneous of colour assignment on the edges. This problem occurs when the colourization algorithm considers that the dilated edges are themselves another segmented region. Some edges are still visible after the colourization process although the original luminance channel is restored. The edge pixels would have a different colour from the edge neighbouring pixels because the edges have different intensities from the surrounding pixels. The optimization algorithm would transfer the colour based on the edge intensity values. Thus, a simple edge re-colouring process is performed to minimize the errors by transferring colour from neighbouring non-edge pixels to the edge pixel itself after the optimization colourization process. The following simple algorithm shows the process:

For every edge pixels {

If (neighbouring pixel is not an edge pixel)

Colour of the edge pixel = colour of neighbouring pixel;

Else if (all edge neighbouring pixels are edge pixels)

For every neighbouring pixels {

If (neighbouring pixel is not an edge pixel)

Colour of the edge pixel = colour of neighbouring pixel;}}

In the edge re-colouring process, each of the edge neighbouring pixels is examined. If one of the edge neighbouring pixels is detected as a non-edge pixel, the colour will be transferred from the non-edge pixel to the edge pixel. If there are more than one non-edge pixels around the edge pixel, only the colour from the non-edge pixel which is firstly detected will be transferred to the edge pixel. If all edge neighbouring pixels are edge pixels, each of the neighbouring pixels of the edge neighbouring pixel will be examined. If the neighbouring pixel is not an edge pixel, the colour of the edge pixel is transferred from the neighbouring pixel of edge neighbouring pixel. The details are summarized in the flow chart in Fig. 9.

Figure 9.

Canny optimization flow chart.

Results and discussions

The high-resolution images of different types of sample surfaces are captured using SEM. Originally SEM images are captured as greyscale images. After the Canny optimization colourization, the SEM images have a more attractive three-dimensional appearance. The colours from the human perception of the samples at high detail resolutions in micro- or nanometre precision no longer can be traced. Hence, the colours assigned to an SEM image during colourization are based on the creativity and common sense of the users. Figure 10 shows the colourization results using Canny optimization for SEM images of a wood surface.

Figure 10.

Wood sample image captured by JEOL 840A SEM.

Wood sample images

Figure 10(a), (c), (e), (g), (i) and (k) are the colour marking images from Canny optimization colourization method for different wood sample images, respectively. Figure 10(b), (d), (f), (h), (j) and (l) are the colourized SEM images using Canny optimization based on the colour marking images. By applying different colours onto the SEM images, different layers of the surfaces can be viewed clearly in the images.

Silver paint images

Figure 11 shows the colour silver paint SEM image after going through the process of Canny optimization. The image shows clearly different layers of the surface.

Figure 11.

Silver paint captured by SEM images.

Aluminium cracked surface images

Figure 12 shows the results for an aluminium cracked surface after using Canny optimization method. Figure 12(a) and (c) are the colour marking images. Figure 12(b) and (d) are the colourized SEM images using the colour marking images in Canny optimization. After applying different colours for different crack areas in the images, the cracks on the aluminium surface in SEM images can easily be seen.

Figure 12.

SEM images for an aluminium cracked surface.

IC wire bonding sample images

Figure 13 shows the SEM images for IC wire bonding samples. Figure 13(a) and (c) are the colour marking images. Figure 13(b) and (d) are the colourized SEM images after using the Canny optimization colourization method. The SEM images look more attractive after colourization.

Figure 13.

SEM images for IC wire bonding samples.

Gold on carbon sample images

Figure 14 shows the SEM images for gold on a carbon sample. Figure 14(a) is the colour marking image. Figure 14(b) is the colourized SEM images from Canny optimization. The colourization gives better illustration of the bumpiness of the material surface

Figure 14.

SEM images for gold on a carbon sample.

Charging images

Figure 15 shows the SEM images with charging effects. Figure 15(a) is the colour marking image. Figure 15(b) is the colourized image using Canny optimization. By assigning blue colour to the charging effect area in the image, the charging effects can be shown clearly.

Figure 15.

SEM images with charging effects.

Comparison between still colourization method and Canny optimization method

Figure 16 shows the ceramic image. The cracks clearly divide the image into three regions and marked with three different colours as shown in Fig. 16(a) and (c). In Fig. 16(b), red colour is crossing over the crack to mix with green and blue colours. Blue colour is mixing with green colour by crossing over the crack too. The three colours are separated well by Canny optimization algorithm shown in Fig. 16(d).

Figure 16.

Ceramic sample image with horizontal field-width = 500 μm. (a) Colour marking image for still colourization, (b) image after Still colourization process, (c) colour marking image for Canny optimization and (d) image after Canny optimization process.

Figure 17 shows the wood fibre image. Figure 17(a)and (c) show the colour markings onto the greyscale image. Still colourization algorithm can hardly differentiate between object and background as showed in Fig. 17(b). Brown colour from the fibre tends to flow out and mix with the background colour after the still colourization algorithm is performed. The problem is solved satisfactorily after performing Canny optimization algorithm in Fig. 17(d). More details about the comparison between the still colourization method and Canny optimization method are shown in Figs 18–19.

Figure 17.

Wood fibre sample image with horizontal field-width = 200 μm. (a) Colour marking image for Still colourization, (b) image after Still colourization process, (c) colour marking image for Canny optimization and (d) image after Canny optimization process.

Figure 18.

Comparison on details of ceramic sample image after applying still colourization and Canny optimization methods (from Fig. 16).

Figure 19.

Comparison on details of wood fibre sample image after applying still colourization and Canny optimization methods (from Fig. 17).

Comparison between optimization method and Canny optimization method

Figure 20 shows the silver paint image. Figure 20(a) shows the colour markings onto the greyscale image for optimization. In Fig. 20(b), colour from the background in the image tends to mix with the colour of the gold from the carbon after being coloured by optimization algorithm. The colours are better controlled within the segmented regions in the colourized result through the edge re-colouring technique in Canny optimization algorithm. The colour of the background in the image as shown in Fig. 20(d) clearly restored to its original colour after Canny optimization algorithm. The wood composite sample image in Fig. 21 shows similar characteristics. Figure 21(a) shows the colour markings onto the greyscale image. In Fig. 21(b), the yellow colour from the back of the fibre tends to flow out and mixes with the background colour after being coloured by optimization. It is seem that the yellow-green colour leaf shape tends to spread the colour to the fibre nearby. In Fig. 21(d), the colourized image using Canny optimization shows that the problem of colour leaking to the background is solved and all the fibres are clearly filled with appropriate colours. Figures 22 and 23 show the logic IC sample image. Figures 22(a) and 23(a) show the colour markings onto the greyscale image before processing through optimization algorithm. In Fig. 22(b), the colour of the IC wire bonding is influenced by the surrounding green colour of the pad after coloured by optimization algorithm whereas in Fig. 23(b), the wire bonding pad is also affected by nearby green and orange colour. In Figs 22(d) and 23(d), again Canny optimization solved the colour conflict problem. Another set of images in Fig. 24 shows clearer and better results in distinguishing between the optimization method and Canny optimization method. In Fig. 24(b), we can clearly observe that the pink colour region covers the background. The other colour regions such as yellow, blue and orange are influenced by surrounding colours like purple and grey colour. Figure 24(d) shows results after solving the conflict problem. More details about the comparison between the optimization method and Canny optimization method are shown in Figs 25–29.

Figure 20.

Silver paint image with horizontal field-width = 5 μm. (a) Colour marking image for optimization, (b) image after optimization process, (c) colour marking image for Canny optimization and (d) image after Canny optimization process.

Figure 21.

Wood composite material with horizontal field-width = 20 μm. (a) Colour marking image for optimization, (b) image after optimization process, (c) colour marking image for Canny optimization and (d) image after Canny optimization process.

Figure 22.

Logic IC sample image with horizontal field-width = 20 μm. (a) Colour marking image for optimization, (b) image after optimization process, (c) colour marking image for Canny optimization and (d) image after Canny optimization process.

Figure 23.

IC wire bonding sample image with horizontal field-width = 20 μm. (a) Colour marking image for optimization, (b) image after optimization process, (c) colour marking image for Canny optimization and (d) image after Canny optimization process.

Figure 24.

IC compound filler sample image with horizontal field-width = 100 μm. (a) Colour marking image for optimization, (b) image after optimization process, (c) colour marking image for Canny optimization and (d) image after Canny optimization process.

Figure 25.

Comparison on details of silver paint image after applying optimization and Canny optimization methods (from Fig. 20).

Figure 26.

Comparison on details of wood composite material image after applying optimization and Canny optimization methods (from Fig. 21).

Figure 27.

Comparison on details of logic IC sample image after applying optimization and Canny optimization methods (from Fig. 22).

Figure 28.

Comparison on details of IC wire bonding sample image after applying optimization and Canny optimization methods (from Fig. 23).

Figure 29.

Comparison on details of IC compound filler sample image after applying optimization and Canny optimization methods (from Fig. 24).

Benefits to SEM users

After evaluated the Canny optimization technique, we find that benefits can be elaborated as below.

Three-dimensional SEM images

SEM three-dimensional images can be observed clearly. Different layers and micro-structure of layout can be easily identified. Figure 30 shows wood sample images in three-dimensional details.

Figure 30.

Wood colour sample images captured by JEOL 840A SEM.

Charging effect

The problem of charging images is an artefact to SEM. Charging is commonly known problem when inspecting an insulating sample in SEM, and occurs when high-energy electrons landed on an insulator sample with charge built up rapidly on the insulator surface because there is no grounding path. In addition, charging will result in image distortion and create an unpleasant view, leading to wrong interpretation of results. With the colourization feature for SEM charging images, the location of charging effect can be easily identified. When comparison is made between the colour and non-colour charging images, the artefact can be detected. Figure 15 shows the effects of colour for images with charging effects.

SEM images with noise

Noise in the SEM images is a rather difficult issue to be handled. The SNR of the images depends on both the beam current as well as the materials present in the specimen and its topography. Reimer (1998) discussed the emission statistics of secondary electrons and backscattered electrons. Dubbeldam (1993) described the characteristic of shot noise, secondary emission noise as well as partition noise. If the SEM image is coloured with colours, the noise from the SEM can be diagnosed. Figure 31 clearly shows the SEM sample image with and without noise.

Figure 31.

Noisy IC wire bonding sample image with horizontal field-width = 20 μm. (a) Image without colourization and (b) image after Canny optimization process.

Layer of micro- and sub-structure images

Another feature that we can benefit from the SEM colourization method is clarification of the different layers of micro-structures. Figure 32 shows a good example where the micro-structures can be clearly seen.

Figure 32.

Logic IC sample image with horizontal field-width = 20 μm. (a) Image without Colourization and (b) image after Canny optimization process.

Materials science images

Microstructure of Resiten G7 glass silicone laminate compositeFigure 33 shows the image of a microstructure of Resiten G7 glass silicone laminate composite. This is another good example where micro-structure can be viewed clearly. Figure 33(a) has the horizontal field width = 100 μm and Fig. 33(b) is another sample image with horizontal field width = 50 μm.

Figure 33.

Microstructure of Resiten G7 glass silicone laminate composite. (a) Image captured with horizontal field width = 100 μm and (b) image captured with horizontal field-width = 50 μm.

Microstructure of wrought Ti – 8% Al – 1% Mo – 1% V The image of microstructure of wrought titanium alloys-Ti8Al1Mo1V is shown in Fig. 34. The sample is prepared using the three-step method and etched with Kroll's reagent to reveal primary alpha grains and a fine alpha-beta matrix structure. After colourization, the colour image is able to show the microstructure level clearly.

Figure 34.

Sample image of microstructure of wrought Ti – 8% Al – 1% Mo – 1% V prepared using the three-step method and etched with Kroll's reagent to reveal primary alpha grains and a fine alpha-beta matrix structure. Image horizontal field width = 20 μm.

Microstructure of carbon-reinforced polymer composite material The sample shown Fig. 35 is the microstructure of a carbon-reinforced polymer composite, a thermosetting polymer with a toughening agent. The horizontal field width is 100 μm.

Figure 35.

The microstructure of a carbon-reinforced polymer composite, a thermosetting polymer with a toughening agent. Image horizontal field width = 100 μm.

Conclusions

In this paper, the optimization algorithm is combined with the Canny edge segmentation technique to solve the problems in colour SEM images. This approach gives a better SEM image, which is useful for image analysis and mechanical contrast. The method is able to overcome the constraints of existing methods resulting in better colourization that can be produced with minimal colour marking pixels. Furthermore, it provides guidance to the user about the locations to do colour marking after overlaying the edges onto the image. The limitation of the manual colourization is that it is a tedious process that needs trial and error in order to get the preference sigma and threshold values. In future, it may be possible to study the conversion of this manual colourization method to an automatic method by using object recognition with precise edge detection.

Ancillary