Fault diagnosis of the bushing infrared images based on mask R‐CNN and improved PCNN joint algorithm

Bushings are served as an important component of the power transformers; it's of great significance to keep the bushings in good insulation condition. The infrared images of the bushing are proposed to diagnose the fault with the combination of image segmentation and deep learning, including object detection, fault region extraction, and fault diagnosis. By building an object detection system with the frame of Mask Region convolutional neural network (CNN), the bushing frame can be exactly extracted. To distinguish the fault region of bushings and the background, a simple linear iterative clustering-based pulse coupled neural network is proposed to improve the fault region segmentation performance. Then, two infrared image feature parameters, the relative position and area, are explored to classify fault type effectively based on the K-means cluster technique. With the proposed joint algorithm on bushing infrared images, the accuracy reaches 98%, compared with 44% by the conventional CNN classification method. The integrated algorithm provides a feasible and advantageous solution for the field application of bushing image-based diagnosis.


| INTRODUCTION
The transformers are one of the most important components of the power system, which contribute to voltage transformation and power delivery [1]. In which, bushings serve as the bridges that connect the transformers and the power grid, which help fix and lead wires out of the transformers [2][3][4]. According to the latest Cigre's transformer bushing reliability report [5], there can be more than ten bushings on one transformer, which is one of the most common causes of transformer failures, accounting for 5-50% of the total number of transformer failures. Any fault of the bushings would lead to the transformer failures, so it's of great significance to keep the bushings in good condition to guarantee the safe operation of the power apparatus.
There are many ways to detect bushing failures, including dielectric loss measurement, UHF (ultra-high frequency) method [6], FDS (frequency domain spectroscopy) technique [7], infrared image diagnosis [8], etc. The dielectric loss measurement requires off-line preventive tests, which would affect the stability of the power apparatus. The UHF method is difficult and complex to locate the exact fault, and the FDS method is only sensitive to the fault affected by damp. The infrared image diagnosis can diagnose fault on-line and it is sensitive to the heat caused by various faults, but it is limited by the development level of the image processing. With the further development and popularisation of intelligent substation and substation inspection robots, a large number of infrared fault images need to be analysed. The infrared image diagnosis method can identify and analyse the collected infrared images automatically, which can reduce the labour intensity and cut down the dependence on technicians. Thus, it is important to develop the infrared image diagnosis method to help deal with the great number of infrared image data.
In the early stage of infrared-based detection, it was greatly limited by the thermography and computer calculation level. The researchers focused on denoising the image noise with filtering algorithms including Gaussian filtering, adaptive median filtering, mean filtering, etc. Also, segmenting image technique, including segmentation of object or segmentation of fault region, is carried out simultaneously, with threshold segmentation represented by OTSU (named after Nobuyuki Otsu) method, grey histogram [9], K-means [10,11], the similarity of adjacent regions, growth algorithm based on area and morphological opening-and-closing operation [12], edge detection [13] and PCNN (pulse coupled neural network) [14,15]. However, the above methods are only applicable to images with a relatively simple background. For those infrared images with a complex background, the above mainstream image segmentation methods have a strong limitation.
Due to the rapid development of deep learning and machine learning, infrared imaging fault diagnosis brings in new ideas and methods. So far, the diagnosis of infrared images is mainly divided into two types. One is to rely on feature extraction, and further to filter and classify the extracted feature parameters, which can be named as feature extraction-based methods [16][17][18][19]. The second is based on the powerful model construction ability of convolutional neural networks, which can be named as CNN (convolutional neural network) based method. The CNN-based method can only classify different kinds of faults based on big data or plenty of images [20][21][22]. Once there are proper feature parameters selected, the feature-extraction-based method can greatly reduce the demand for faulty samples and provide better classification results under the condition of small samples. Therefore, compared with the CNN-based method, the feature-extraction-based method is more suitable for the diagnosis of bushing faults, and it can reasonably solve the problems of complicated background with the absence of enough fault samples in the infrared images of the bushing.
This paper provides a method to extract the bushing outline, then the fault region, and further fault diagnosis parameters regarding the field application of bushing infrared images, as shown in Figure 1. It does not rely on the conversion relationship between infrared images and temperature, but directly diagnose faults based on the infrared images. This kind of diagnosis method can help diagnose in the infrared images collected by different equipment, which reduces the limitation caused by the inconsistent conversion relationship between infrared images and temperature. The remaining of this paper is organised as follows. Section 2 briefly introduces the considerations of bushing region detection and segmentation. Section 3 presents the fault region extraction method and diagnosis. Section 4 analyses the experimental results. Section 5 concludes the paper.

| Infrared images of bushings
The bushing is an important component of the transformer and connects it to the power gird. Thus, the bushing should be required to have a good insulation property to keep the transformer and the power grid working safely. It mainly consists of HV joint, insulating oil paper and condensers, etc. As well, it is filled with insulating oil inside, as shown in -117 Figure 2. The condenser bodies are practically processed by turning (machining) and they mechanically adhere firmly and tightly to the flange as a whole. Moreover, multiple condensers are applied to reduce electric field gradients and make the electric field distribution more uniform, and then in a good insulation condition.
In the routine inspections, the infrared images of the bushing in substations are not so satisfying owing to the limitations of infrared image acquisition equipment. Especially, the complicated background is included in the bushing infrared image, which makes it hard to process the bushing region separately, as shown in Figure 3.
Besides, the bushing outline is in the shape of the umbrella-like, and this kind of structure makes the bushing surface temperature uneven. Then it's hard to get the accurate temperature information and segment the fault region. To be brief, there are two problems remaining to be solved: automatic bushing segmentation from the complicated background and further fault region extraction with the consideration of uneven surface temperature distribution.

| Object detection of bushing
Object detection is a computer technology related to computer vision and image processing. As far as the complicated background problem is concerned, the object detection methods can be applied to detect bushing in the infrared images and help process bushing separately.
Object detection can be divided into two types: detection of specific instances and classification of categories. The first type tries to detect the instance of an object, such as a person's face, is considered as a matching problem. The second type is designed to detect instances of certain tagged object categories. The first type is more concerned about object segmentation performance, and the second type is more useful to classify the to-be-determined image. Regarding the bushing diagnosis, the instance segmentation is more effective for the application.
In general, the spatial position and extent of an object can be roughly defined with a bounding box, as shown in Figure 4. An axis-aligned rectangle tightly fits the object, an accurate pixelated segmentation mask, or a closed boundary. Currently, bounding boxes are widely used to evaluate general object detection algorithms. However, the bounding box methods are not able to solve our problem perfectly since there is still background noise after R-CNNs being performed. Thus, pixelated target segmentation must be applied to help segment the bushing region, such as Mask R-CNN.

F I G U R E 2
The mechanical structure of OIP (oil-impregnated paper) bushing

| Mask R-CNN
Since it is difficult to ensure the purity of the background when taking the infrared images of the bushings at the substations. It is necessary to perform object detection on the image to segment the bushing area at first. The normal R-CNNs can only detect a bounding box. And the application of Mask R-CNN is introduced to achieve the goal of object detection, which can segment the object from the background in pixels.
Mask R-CNN [23] is an algorithm based on Faster R-CNN [24], ResNet [25] and FPN to detect and identify objects in candidate frames and further implement instance segmentation in a mask generation module. An image workflow is designed and illustrated in Figure 5. Firstly, a bushing image is processed by CNNs to extract features and generate a feature map. Then the RPN (region proposal network) is applied to process corresponding candidate regions. After the ROI (region-of-interests) extraction methods being performed, the different sizes of proposed candidate frames are resized on the same scale, and the losses of the extracted ROI can be calculated, including class loss, bounding box loss and the generated mask loss. The losses are combined to help train the segmentation neural networks. In other words, different ROI extraction methods and the extra loss of mask generation distinguish Mask R-CNN from Faster R-CNN.
The Mask R-CNN replaces the ROI pooling with the ROI align, which allows Mask R-CNN to segment objects in pixels because of the inaccuracy of the ROI pooling performance. Both the ROI pooling and the ROI align are applied to help convert different scales of images into the same scale at a certain stage in the network.
In the process of scale conversion, ROI pooling uses the nearest neighbour interpolation to adjust floating-point numbers caused by scale conversion. During adjustments, the nearest neighbour interpolation would generate an inevitable loss, that's the reason Faster R-CNN can only segment the object with bounding boxes. The ROI align improved based on ROI pooling, which replaces the nearest neighbour interpolation with bilinear interpolation. That is, even if the scaled coordinates may not be exact integers, the value at the floatingpoint number can be obtained by interpolation to process the pooled value. The application of bilinear interpolation enables the Mask R-CNN segment object more accurately. The extraction process of a typical bushing outline in a complex background can be seen in Figure 6.

| FAULT REGION EXTRACTION AND DIAGNOSIS
The purpose of the fault region extraction is to extract feature parameters from the fault region based on the bushing segmentation, and to take the extracted features as the input to the fault diagnosis module. The image of the bushings infrared image is segmented by the Mask R-CNN at the first step, and the fault region should be confirmed in the next step.

| Fault region extraction
Fault region extraction is an essential method of image segmentation, the main purpose of which is to select the appropriate threshold according to the colours in the infrared image to segment the fault region. The common methods are a binary method, OTSU method, K-means and so on. However, the method of segmenting images by threshold is not satisfying in the bushing fault region extraction, because of the uneven temperature distribution caused by umbrella shape construction. To make the method more robust, the PCNN algorithm is introduced since it segments images through a dynamic threshold, which makes it more adaptive.
PCNN is short for a pulse-coupled neural network that does not require pre-learning to extract information from complex backgrounds. However, for an adaptive image segmentation system, the parameters selection complexity of PCNN will reduce the robustness of image processing, so the structure needs to be simplified and improved. The simplified computing structure is calculated by the following equations: -119 where, β is the synaptic connection strength, g is the decay time constant, h is the amplitude coefficient of the regulator, F ij ½n� is the n-th feedback input signal of the neuron (i, j), I ij is the grey level of the pixel corresponding to the neuron (i, j), L ij ½n� is the link of the input signals, W ijkl is the linking weight coefficient of the feedback input, generally eight surrounding pixels, U ij ½n� denotes the internal activity, Y ij ½n� is the neuron output, and θ ij ½n� is the dynamic threshold.
In the simplified PCNN solution, the input signals are summed according to the connection weight, which both retains the advantages of PCNN and reduces the complexity of parameter determination. The input signals include two channels: the first channel called L channel transmits the output of the other neurons through the acceptance domain, and the second F channel transmits the input from the external input domain. The corresponding function of the F channel pulse is slower than others; the whole structure of PCN is shown in Figure 7.
The simplified PCNN algorithm has a point-to-point correspondence between the neurons and the image pixels when the image has been denoised. And the feedback input of each neuron, as well as the connection input, only accept the input of the external stimulus. When the input of the connection is greater or equal to half of the sum of the elements in the synaptic connection weight matrix, the synaptic connection strength would be inspired, or the pulse would be paused.
The segmentation effect of the PCNN algorithm is shown in Figure 8. The PCNN has an obvious effect on the extraction of the fault region, but it generates a large amount of interferences that are difficult to be de-noised, as shown in Figure 8b. The reason for the noises is that PCNN is very sensitive to the complex boundary of the bushing. Therefore, the SLIC (Simple Linear Iterative Clustering) super-pixel segmentation method is furthermore implemented here to preprocess the image, reducing the influence of boundaries on PCNN by averaging neighbouring regions' colour.
SLIC is proposed to generate regular lattices on the basis of colour and spatial proximity to help improve the performance of PCNN. The main idea of the SLIC algorithm is to convert the image from the RGB colour space to the CIE (Commission International Eclairage) lab colour space to help segment the image. The (l, a, b) colour value and the (x, y) coordinate of each pixel form a five-dimensional vector [l, a, b, x, y] T . For an image with N pixels, it can be split into K sets of super-pixels. By calculating the degree of similarity between each pixel point and the closest cluster centre. Then the new cluster centre is updated to the position nearer to the pixel points in the same category, and the process is iterated until each parameter converges. Since the cluster centre no longer changes, then the pixel is assigned a reasonable label. The measure of similarity degree is defined as follows: where d is the colour distance; i represents the spatial distance; s is the distance of the cluster centre; D (i, k) is the similarity degree between the pixel point and the kth cluster centre. As a super-pixel segmentation algorithm, SLIC is utilised for segmenting images from the perspective of region similarity. A common feature of the pixels in each cell is that they are in the closest colour, and then the average colour of the cell is selected to represent the whole cell. Owing to the infrared image contains the temperature information, the colour of the different cells represents the specific temperature value in the infrared image.
The process of the improved PCNN segmentation algorithm is shown in Figure 9, and the comparative results can be seen in Figure 10. Subgraph-a is a fault image with two fault areas, subgraph-b is a joint failure image, and subgraph-c is a normal image. The second column is the result processed by the OTSU algorithm, the third column is processed by the improved OTSU algorithm, the fourth column is processed by the PCNN algorithm, and the fifth column is processed by improved PCNN algorithm. Obviously, both of the OTSU and the improved OTSU algorithm are likely to be undersegmented. While PCNN just keeps the fault region with the noises, and the images processed by improved PCNN have the best performance.

| Typical bushing faults and diagnosis
According to DLT664-2016 Application rules of infrared diagnosis for live electrical equipment, the infrared image based images of the bushing can be divided into four types: dielectric loss fault, joint fault, oil leakage fault and partial discharge.
The dielectric loss fault occurs when there is moisture in the insulating oil or there is dielectric loss caused by partial discharges, and the dielectric loss fault may lead to the temperature increase of the whole bushing body. The joint fault is usually caused by the poor connection of the joint, which leads to overheating on the joint. The oil leakage due to the poor mechanical seal, results in the bottom temperature of the bushing higher than the top. The typical fault infrared images are shown in Figure 11. Since it is not a good approach to Image-based fault diagnosis is a data-driven diagnostic method. Its data is derived from the feature parameters extracted from the fault region, which is also obtained by analysing the infrared image of the faulty bushing. According to the analysis of the fault types, the fault infrared image features can be provided in Table 1. It is known that the fault type may be related to the specific position and area size. Therefore, the relative area and the relative vertical axis position are extracted as feature parameters. And the fault types are classified by the K-means method.
The relative position of the fault area, y, can be defined as the areal ordinate of the fault area, which can describe the position of the fault region. The relative area, S, is defined as the ratio of the number of points in the fault area to the number of points in the bushing area, which is useful to describe the relative size of the fault region to the bushing area.
where n is the number of points in the fault area, and N is the number of points in the bushing area, i.e. the size of the fault area and bushing area, respectively. The number of n is calculated by the region where the improved PCNN segmentation, and the number of N is calculated by the image synthesised by the Mask R-CNN segmentation.

| RESULTS AND ANALYSIS
In our case, the computing environment is provided in Table 2, and the dataset composition of the bushing infrared images from the field application is illustrated in Figure 12.  The object detection dataset is from 51 different substations of State Grid Jiangsu Electric Power Co. Ltd. between 2016-2018, and a total number of 430 available images from the bushing infrared image dataset is established. Wherein 400 pictures are taken as a training set and 30 images as a validation set. The number of fault sample images is only 54 in more than 2000 infrared bushing images. Besides, many fault images belong to joint failure with poor quality. Thus, there are 34 fault images with good quality being selected to form the fault diagnosis dataset. Other than diagnosing the fault images, the proposed method should also be able to classify the healthy images. So there are 16 healthy images being added to the diagnosis dataset.
After the fault region extraction, the feature parameters are extracted and applied to classify faults through K-Means. The result of the classification is shown in Figure 13, and the four kinds of bushing conditions can be well classified. The accuracy of the proposed method is calculated by the perdition of the K-means cluster and the dataset label. Through the calculation, the accuracy of the proposed method is 98%, which proves the effectiveness of the two proposed features.
The CNN classification method is applied to compare the accuracy of two methods. Typical VGG16 is used as the classifier to train and classify the images into four groups, using the same dataset. The result of the accuracy is merely 44%, much lower than the integrated technique, demonstrating the novelty of the proposed solution.

| CONCLUSION
Bushings are of great significance to be well maintained, since any fault of them leads to severe damages of the power transformers. An intelligent fault diagnosis technique integrating Mask R-CNN and improved PCNN joint algorithm is proposed in this study to improve the diagnosis of the bushing based on practical infrared images.
(i) The Mask R-CNN has been applied to segment infrared images to improve the performance of bushings' segmentation and reduce the interferences from the complicated background. (ii) The feature extraction performance is strongly related to the fault region extraction performance. The improved PCNN method, combining conventional PCNN with SLIC, is a good approach to extract the fault region within a bushing infrared image. (iii) Two feature parameters, the relative fault area value and its relative position, are proposed and defined to diagnose faults, especially for bushings based on the typical fault infrared images with small fault samples. With the actual images from the field, the integrated method can diagnose the fault infrared bushing images with high accuracy of 98%. (iv) The strategy based on infrared images without consideration of temperature information gets rid of the restriction that each equipment has a distinct conversion relationship between infrared point and temperature value, improves the applicability in the field.