Stent Deployment Detection Using Radio Frequency‐Based Sensor and Convolutional Neural Networks

A lack of sensory feedback often hinders minimally invasive operations. Although endoscopy has addressed this limitation to an extent, endovascular procedures such as angioplasty or stenting still face significant challenges. Sensors that rely on a clear line of sight cannot be used because it is unable to gather feedback in blood environments. During the stent deployment procedure, feedback on the deployed stent's state is critical because a partially open stent can affect the blood flow. Despite this, no robust and noninvasive clinical solutions that allow real‐time monitoring of the stent deployment exists. In recent years, radio frequency (RF)‐based sensors can detect the shape and material of an object that is hidden from the direct line of sight. Herein, the use of a 3D RF‐based imaging sensor and a novel Convolutional Neural Network (CNN) called StentNet is proposed for detecting the stent's state without a need for a clear line of sight. The StentNet achieves an overall accuracy of 90% in detecting the state of an occluded stent in the test dataset. Compared with an existing CNN model, the StentNet significantly outperforms the 3D LeNet in the evaluation metrics such as accuracy, precision, recall, and F1‐score.

DOI: 10.1002/aisy.202000092 A lack of sensory feedback often hinders minimally invasive operations. Although endoscopy has addressed this limitation to an extent, endovascular procedures such as angioplasty or stenting still face significant challenges. Sensors that rely on a clear line of sight cannot be used because it is unable to gather feedback in blood environments. During the stent deployment procedure, feedback on the deployed stent's state is critical because a partially open stent can affect the blood flow. Despite this, no robust and noninvasive clinical solutions that allow realtime monitoring of the stent deployment exists. In recent years, radio frequency (RF)-based sensors can detect the shape and material of an object that is hidden from the direct line of sight. Herein, the use of a 3D RF-based imaging sensor and a novel Convolutional Neural Network (CNN) called StentNet is proposed for detecting the stent's state without a need for a clear line of sight. The StentNet achieves an overall accuracy of 90% in detecting the state of an occluded stent in the test dataset. Compared with an existing CNN model, the StentNet significantly outperforms the 3D LeNet in the evaluation metrics such as accuracy, precision, recall, and F1-score.
sensors that do not require a clear line of sight to the stent. Here, a 3D RF-based sensor working within the bandwidth of 6.3-8.3 GHz, [3] Walabot by Vayyar Technologies, is used to detect the stent deployment. The sensor uses antennas to transmit amplitude-modulated signals and captures back the signals reflected from the surrounding. The obtained signal is then used to construct a 3D mapping of the radiated space. [3] To utilize the subtle changes captured by the sensor to detect the length of deployment stent and compressions in the stent, a novel deep learning model called StentNet is proposed and trained to perform multiclass classification.
Our contributions in this article are summarized as follows: 1) Detection method: A novel way to detect the deployed length of the stent and the presence of compression in the stent body without needing a clear line of sight using an RF-based sensor. 2) Network model: A novel convolutional neural network (CNN) model called StentNet that performs multiclass classification to classify the state of the deployed stent based on the sensory data captured by the RF-based sensor. 3) Dataset: Data of the signal reflectance collected under different stent deployment situations.
The structures of the article can be summarized as follows. Section 2 overviews the related work on detecting stent deployment and the use of RF-based sensors. Section 3 outlines the stent deployment detection strategy. The detailed design of the StentNet, the data collection process, and the evaluation results are presented in Section 4 and 5, respectively. The article then concludes the essential findings and briefly explains the limitation and future work in Section 6.

Related Work
To detect the opening of the stent, ref. [4] proposed a method to detect the expansion of a balloon-expandable stent. By using the singularity expansion method (SEM), it estimated the balloonexpandable stent's diameter during the angioplasty. In the SEM, poles and residues are used to represent the change in electromagnetic signals radiated from the bodies. The article proved that the stent's six different expansion lengths were relatable to the poles in the complex plane. Medical images such as images obtained from computed tomography (CT), conventional cineangiography, and intravascular ultrasound (IVUS) under ideal conditions were also used to detect the open state of the stent and the presence of any fractures in the stent. [5] Automated detection of the stent's outline and location during the IVUS pullback was demonstrated in ref. [6]. However, the proposed method only demonstrated a binary classification of the presence of a stent in the pullback. Ref. [7] used decision trees to achieve automated detection of stent struts using the features extracted from the images obtained from the intravascular optical coherence tomography (ICOT). Stent detection in ref. [8] employed the Bayesian network and graph search using 3D structural knowledge of the stent. Most of the methods proposed earlier either warrant an intravascular approach or expose the patients to radiation. The use of machine learning in the medical domain has also been reported. Conventionally, deep neural networks such as the CNNs have been typically used for classification tasks. An autoencoding CNN has been used to classify fluorescent images of single cell. [9] Ref. [10] proposed a novel, CNN-based multiclass classification model to classify brain tumors. Ref. [11] utilized a 3D CNN to perform neurodevelopmental age classification based on magnetic resonance imaging (MRI) data. CNNs have been widely used for object detection, semantic segmentation, [12] and monocularbased depth estimation. [13] CNNs have shown much potential in medical image processing. Ref. [14] proposed a deep learning model that utilizes CT images to automate the delineation of organs at risk in the head and neck. RF-based sensors have also seen its widespread use recently due to its ability to see beyond a clear line of sight. In ref. [15], a new medical imaging technique based on microwaves for detecting breast pathology was proposed. RF-based data and CNN were utilized for the segmentation of ovarian structures in ref. [16]. Detection of breathing and sleeping patterns has also been demonstrated in ref. [17] using the transient signals obtained from the RF-based sensor. The RF-based sensor, used in this article, has reported its use in other applications in recent times. Ref. [3] presented a material classification system by implementing CNN on a 3D radiance map obtained from the RF-based sensor. Ref. [18] analyzed the signal of the radar sensor to detect the human motion and objects through the wall.

Stent Deployment Detection Strategy
In this article, the change in signal reflectance of the target location before and after the deployment of the stent is used to detect the degree of stent opening and to detect the presence of compression in the middle section of the stent. An RF-based sensor is used for capturing this change in reflectance, as shown in Figure 1a. The sensor captures the reflectance of the surrounding space by transmitting and receiving amplitude-modulated signals using a series of linearly polarized broadband antenna. [3] For image construction of the reflectance, the sensor first performs calibration by capturing the background reflection. Upon capturing the background reflection, the sensor then compares it with the new reflectance of the surrounding and performs image construction of the radiated space. The sensor is first calibrated by capturing the surrounding reflectance before the stent deployment to detect and classify the state of the deployed stent. The captured image of the radiated space upon deployment of the stent, which changes depending on the shape of the stent, is then used for classification. The RF-based sensor's output represents the change in reflectance of the sensing space in stacked 2D images. The change in reflectance due to the stent's state can be represented within any part of the stacked images depending on the stent's location with respect to the sensor. Deep neural network (DNN) can perform 3D convolution to extract features with robustness in classification problems. Therefore, a novel network, called StentNet, is proposed for performing a multiclass classification of the stent's state based on the changes in the 3D images generated by the RF-based sensor. Here, the stent deployment was classified into four classes (0, 1, 2, and 3) based on the length (0, 1, and 3 cm) of the deployed stent and the presence of compression in the middle of the stent (Figure 1c). The class of 0 cm indicates that the stent has not been deployed; the class of 1 cm represents that the stent has been partially deployed; and the class of 3 cm indicates that the stent has been fully deployed. Compression in the middle part represents that the stent is crimped or compressed in the center. The workflow of this study is shown in Figure 1b. The data collection algorithm based on the application program interface (API) is built to collect the four classes of data, which are shown in Figure 1d. A novel deep learning model called StentNet is proposed for detecting stent deployment.

Sensor Setup
The RF-based sensor profile was configured to represent the 3D space in the cartesian coordinate system (Figure 1a) in which the coordinate system is represented in terms of X, Y, and Z plane. Furthermore, the sensor was configured for very short-range scanning to capture the subtle changes in reflectance at near distances. The scanning range of the sensor was set to AE5 cm in the X direction, AE5 cm in the Y direction, and 3-12 cm in the Z direction. The resolution of the scan was set of 0.3 cm in all three axes ( Table 1). The threshold for the sensor in detecting the changes in reflectance was set to 5 for detecting the stent's state through an empty cardboard box and was set to 2 to detect through a slice of pork and chicken.

Data Collection
Dataset representing the stent state in the form of 3D radiated space images was generated using the RF-based sensor. We aim to test the reliability and robustness of the proposed approach in www.advancedsciencenews.com www.advintellsyst.com detecting the stent state without a clear line of sight. The dataset was generated using the empty cardboard box, a slice of pork, and a slice of chicken as occluders between the stent and the sensor. The empty cardboard box is used for the feasibility study. The slice of pork and chicken are used to simulate tissues. The heights of the empty cardboard box used here were 4 and 5.3 cm respectively, and the thickness of a slice of pork and chicken was 0.4 cm. A stent that opens up to a diameter of 0.6 cm and a length of 3 cm upon deployment is used for the whole dataset, as shown in Figure 2a.
The stent is fixed on a piece of black paper, as shown in Figure 2b. The occluders and the sensor are placed under this paper. During each iteration of the data collection process, the stent was placed at different locations from different angles by rotating the black paper within the sensor's scanning zone. One of the three occluders was placed between the sensor and the stent, as shown in Figure 2a. Initially, the extension of the stent was set to 0 cm and the sensor was calibrated to capture the background reflection. Upon completion of the calibration process, the new 3D reflectance image constructed by the sensor was recorded under class 0. The stent was then extended to 1 and 3 cm, and the sensor's output was saved under class 1 and class 2, respectively. Finally, the middle section of the stent that expands to 0.6 cm was compressed to 0.3 cm using a straw of 1 cm length, and the sensory data was recorded under class 3. As it is difficult for the stent to enter a narrow space from a naturally open state to simulate the situation of being compressed in the middle without external compression, we adopted few steps for collecting the data for class 3 using two straws, as shown in Figure 2b. First, the sensor was calibrated by placing the straw A at the opening of the stent and placing the straw B in the middle portion. Second, straw A is slid toward straw B to guide the stent into a narrow space. Third, the stent is deployed to  www.advancedsciencenews.com www.advintellsyst.com its full length of 3 cm. During this time, due to the presence of straw B, the middle section of the stent remains compressed. Finally, straw A is slid back to its original calibration position. The reflectance image constructed by the sensor is then collected under class 3. These steps can ensure that the data collected reflect the change in the shape of the stent due to compression.

Training and Testing Dataset
In total, a dataset with 600 samples (150 samples per class) was collected. The dataset was then randomly separated to train (80%) and test (20%) dataset on the class basis. In total, 478 samples were used as the training set, and 122 samples were used as the test set. Figure 2c shows the 3D linear discriminant analysis (LDA) map of the entire dataset. Figure 2d shows the 3D LDA map of samples in the training set, and Figure 2e shows the 3D LDA map of samples in the test set. In these maps, different colors represent different classes. It can be observed that samples from the same class come together to form a subgroup.

Data Augmentation
In the AI domain, overfitting is mainly addressed by using a large dataset during the training stage. However, generating a large dataset is time-consuming and, at times, physically impractical. As the dataset used here is self-generated, data augmentation technique was adopted to expand the dataset's size virtually. When a DNN is trained for multiple iterations, the network may try to overfit the training data, for instance, look for exact values at the specific pixel locations in the image. In this situation, data augmentation techniques such as cropping, rescaling, and random flipping ensure that the model does not overfit to pixel location, thereby improving the model's generalization. In the past, ref. [10] reported the use of data augmentation to address the lack of data problems. In this work, during the data loading process, each train data was set to have a 30% probability of being flipped horizontally and a 30% probability of being flipped up and down. This random data augmentation technique ensures that the deep learning model does not overfit the small dataset in the training stage, and its performance is generalized.

Architecture of the StentNet
The StentNet proposed in the article is characterized by a series of down-convolution layers, feature bypass layers, and a series of linear layers. The architecture of the StentNet used to classify the stent's state is shown in Figure 3. The network first takes in the 3D image of size [1,31,34,34], generated by the sensor, and performs double convolution. The double convolution layer consists of a pair of 3D convolution and batch normalization.  The convolved data then passes through three layers of downconvolution, each performing a max-pooling and a double convolution. The features extracted in each down-convolution layer is then bypassed through a linear layer that reduces the feature dimension to a size of 256. Bypass layers are added to the network to ensure that the network utilizes both global and local features to perform classification. The extracted features from these layers are then concatenated, which then goes through two more linear layers, each reducing the dimension to 128 and 64, respectively. The features then pass through a final linear output layer that reduces the dimension to 4, corresponding to the four classes used in this article to indicate the stent's state. To avoid overfitting during the training stage, dropout layers are added after each down-convolution layer (dropout probability ¼ 0.2) and after the first linear layer (dropout probability ¼ 0.3) that extracts features from the concatenated bypass features. The network was trained on 478 training data using the Adagrad optimizer with a learning rate of 5 Â 10 À5 , a learning rate decay of 1 Â 10 À3 , and a weighted decay of 0.3. Crossentropy loss was used to calculate the network loss. The network was trained for 800 epochs with a batch size of 40. The network was implemented using the PyTorch framework and trained in the NVIDIA RTX 2080 Ti GPU.

Performance Metrics
The performance of the proposed Stentnet was evaluated based on accuracy, precision, recall, and f1-score. The multiclass DNN managed to achieve an overall train accuracy of 89.8% and test accuracy of 90.0% in classifying the state of the deployed stent based on 3D data of the radiated space obtained from the RF-based sensor. The network's performance on both the training and test data was analyzed during the training stage. As shown in Figure 4a, it was observed that the network started overfitting after the 515 th epoch. After the 515th epoch, the test loss remained almost the same, whereas the training loss kept decreasing. Therefore, the weights of the network at the 515th epoch were chosen as final weights in which the model achieved a training accuracy of 89.8% and a test accuracy of 90.0% ( Table 2).
Apart from the accuracy, the performance of the StentNet was further evaluated using confusion matrix, precision, recall, and F1-score. The confusion matrix represents the model performance at the class level. From the confusion matrix shown in Figure 4b and Table 3, it is observed that the model achieved 100% recall for class 0, 94% recall for class 1, 93% recall for class 2, and 73% recall for class 3. As high recall relates to low false positive, it is proven that the proposed StentNet makes very few false positives for the classes 0, 1, and 2. Furthermore, the results in Table 3 indicate that the proposed model achieved a precision of above 85% for every class, proving that the model can detect the right class most of the time. The model also achieved a high F1-score of 0.8, the weighted average of precision, and recall.

Comparison with 3D LeNet
The multiclass classification performance of the proposed StentNet model was compared with LeNet model. [19] A comparison with LeNet architecture was done mainly to highlight the   improvement in accuracy due to additional down-convolutional layers together with feature bypass. In the 3D LeNet model, the input data are initially propagated through two 3D convolutional layers, each performing a 3D convolution and a 3D max-pooling.
In the initial two layers of 3D convolution, the single-channel input is expanded to 6 and 16 channels, respectively. The output of the final max-pooling layer is then flattened and passed through four hidden fully connected layers with 200, 120, 120, and 84 nodes each. The final hidden layer is connected to the output layers, with four nodes representing four classes. 3D LeNet was trained on the same training data and evaluated on the same testing data as the StentNet model. Adam optimizer, with a learning rate of 2 Â 10 À5 and a weighted decay of 1.2 Â 10 À2 , was used to train the network. The model was trained with a batch size of 40 for 2000 epochs. The LeNet model achieved the highest accuracy of 80.0% in the test data. Compared with the 3D LeNet model, StentNet outperforms all the evaluation metrics according to Table 3 and 4.

Ablation Study
The ablation study of the proposed StentNet was performed by removing the feature bypass from the down-convolution layers. Without the bypass layers, the model achieved the highest test accuracy of 80.0% at the 756th epoch ( Table 5). This overall performance is similar to the LeNet model. When compared with the LeNet model on class-based performance, the StentNet without the bypass layers has higher precision for classes 0, 1, and 3, as shown in Table 4, 5, and 6. This study proves that the feature bypass layers added after every down-convolution layers help preserve both the global and local features and contribute to a significant increase in model accuracy.

Conclusion
This article presented a study on detecting different deployment lengths and compression degree of the stent without a clear line of sight using RF-based sensor and a novel CNN called StentNet. The StentNet model used the 3D data of the radiant space constructed by an RF-based sensor in performing multiclass classification. The model achieved an overall accuracy of 90% to predict the state of the stent that was beyond the direct line of sight. The high precision of above 85.0%, recall of above 73.0%, and F1-score of above 0.8 prove that the model can detect the state of the stent with less false positives. The article also proved that the StentNet model, characterized by a series of down-convolution layers, bypass layers, and series of linear layers, outperformed LeNet, an existing CNN network. This proves that the novel StentNet, together with the use of an RF-based sensor, could be used to detect the state of a deployed stent during the stenting procedures.
The main limitation of the proposed approach stems from the RF-based sensor used. During the data collection process, different occluders placed between the stent and the RF-based sensor influenced the sensor's output. While occluders such as an empty cardboard box had very less influence over the sensor's ability in detecting the change in reflectance, other occluders such as a thick slab of tissue had considerable influence. This warranted the sensor to operate at lower thresholds to capture the subtle changes in the reflectance to detect different stent state. Furthermore, change to sensor resolutions changed the shape of the sensor's 3D matrix output. The need for high resolutions and size limitation of the sensor's output data constrained the maximum distance between the sensor and the stent. Another limitation is that the dataset used here for StentNet training does not include the actual surgical scenes. This is because the experiments conducted in this article focus on studying the feasibility of using ultra-wideband (UWB) RF-based sensors to detect the state of the deployed stent and serves as a proof of concept.
Future work will focus on effectively detecting the deployed state of different types of stents inside a pulsatile flow model and subsequently in swine coronary or cerebrovascular vessels mimicking stent deployment inside the human heart or brain. In the follow-on cadaver experiment, the dataset closer to the actual surgical scene will be collected to validate, further improve, and refine the detection algorithm and process.