Assessing kidney stone composition using smartphone microscopy and deep neural networks

Abstract Objectives To propose a point‐of‐care image recognition system for kidney stone composition classification using smartphone microscopy and deep convolutional neural networks. Materials and methods A total of 37 surgically extracted human kidney stones consisting of calcium oxalate (CaOx), cystine, uric acid (UA) and struvite stones were included in the study. All of the stones were fragmented from percutaneous nephrolithotomy (PCNL). The stones were classified using Fourier transform infrared spectroscopy (FTIR) analysis before obtaining smartphone microscope images. The size of the stones ranged from 5 to 10 mm in diameter. Nurugo 400× smartphone microscope (Nurugo, Seoul, Republic of Korea) was functionalized to acquire microscopic images (magnification = 25×) of dry kidney stones using iPhone 6s+ (Apple, Cupertino, CA, USA). Each kidney stone was imaged in six different locations. In total, 222 images were captured from 37 stones. A novel convolutional neural network architecture was built for classification, and the model was assessed using accuracy, positive predictive value, sensitivity and F1 scores. Results We achieved an overall and weighted accuracy of 88% and 87%, respectively, with an average F1 score of 0.84. The positive predictive value, sensitivity and F1 score for each stone type were respectively reported as follows: CaOx (0.82, 0.83, 0.82), cystine (0.80, 0.88, 0.84), UA (0.92, 0.77, 0.85) and struvite (0.86, 0.84, 0.85). Conclusion We demonstrate a rapid and accurate point of care diagnostics method for classifying the four types of kidney stones. In the future, diagnostic tools that combine smartphone microscopy with artificial intelligence (AI) can provide accessible health care that can support physicians in their decision‐making process.


| INTRODUCTION
Kidney stones (calculi) are mineral deposits of crystalline and organic components formed when urine is supersaturated with minerals and/or organic components. 1 The formation of kidney stones (nephrolithiasis) is a common condition. According to the most recent National Health and Nutrition Evaluation Survey, the prevalence of self-reported kidney stones between 2013 and 2014 was 10.1%; the weighted prevalence of kidney stones was 10.9% (9.1-12.6) for males and 9.4% (7.6-11.1) for females. 2 There are several underlying risk factors for the formation of kidney stones; family history, race/ ethnicity, systemic disorders, environmental factors, dietary factors and urinary factors seem to all play a role in the development of kidney stones. 3 The abundance of each type of kidney stone varies with type: calcium stones comprise around 60-80% of all kidney stones, uric acid 8-10%, struvite 7-8%, and cystine 0.1-1%. 1,3 Kidney stones can be treated with minimally invasive surgical techniques such as Extracorporeal shock wave lithotripsy, ureteroscopic lithotripsy, and percutaneous nephrolithotomy (PCNL). 4 Following the kidney stone extraction and stone analysis, clinicians determine stone composition to prescribe treatment or diet for preventative measures. 5 The recurrence rate is approximately 50% in 5-10 years and 75% in 20 years without preventive treatment. 6 Hence, understanding and detecting the formation of specific types of kidney stone is crucial for prescribing treatment to prevent recurrence.
The gold standard for kidney stone analysis in physical analytic methods are Fourier transform infrared spectroscopy (FTIR) and X-ray diffraction. 7 Both methods are widely used and considered to be very accurate and reliable for kidney stone analysis. However, there are certain limitations; both FTIR and X-ray diffraction need trained laboratory personnel and specialized equipment in a laboratory setting. Due to these limitations, clinicians need to send kidney stones to specific testing centres, and this practice is both time consuming and costly for each analysis. Therefore, there is a need for a rapid and accurate pointof-care device for kidney stone analysis in daily clinical practice.
Over the past couple of decades, artificial intelligence (AI) has become a significant research area in medical diagnostics and analytics. 8 There has been ongoing research in building image-based diagnosis systems for many medical specialties. 9 Similarly, the adoption of AI in urology is a growing field of interest. 10,11 Recently, Black et al. reported kidney stone classification using deep learning and digital camera images. 12 The aim of the present study was to propose an image recognition system that can accurately detect the type of kidney stones using a data set of smartphone-based microscopic images. To our knowledge, this is the first study for combining smartphone microscopy with deep learning to classify kidney stone types.

| MATERIALS AND METHODS
A total of 37 surgically extracted human kidney stones consisting of calcium oxalate (CaOx) (7 calcium CaOx-monohydrate, 7 CaOx-dihydrate, 6 CaOx-monohydrate + CaOx-dihydrate) (n = 20, CaOx), cystine (n = 10), uric acid (n = 4, UA), and struvite stones (n = 3) were included in the study. All of the stones were fragmented from PCNL. The stones were classified using FTIR analysis before obtaining smartphone microscope images. The size of the stones ranged from 5 to 10 mm in diameter. The stones were obtained between 2018 and 2020 from Bulent Onal MD, Istanbul University-Cerrahpasa, Cerrahpasa School of Medicine, Istanbul, Turkey. All of the stones were preserved in a dry state and as extracted from the patients. Nurugo 400Â smartphone microscope (Nurugo, Seoul, Republic of Korea) was functionalized to acquire microscopic images (magnification = 25Â) of dry kidney stones using iPhone 6s+ (Apple, Cupertino, CA, USA). The smartphone-microscope was hand-held to obtain images of the kidney stones. Distance from objective and focus were manually adjusted by the user to mirror real-life point-of-care usage conditions. Each kidney stone was imaged in six different locations. In total, 222 images were captured from 37 stones. A summary of our data set for different stone types is presented in Table 1.
Microscopic image samples of different types of kidney stones are shown in Figure 1.
We propose a machine learning pipeline that can be used for kidney stone classification. Our approach consisted of multiple stages, starting with data acquisition and ending with outputting the correct kidney stone type from our classifier. Each stage output in our approach was provided as input to the next stage. Our pipeline and approach for this kidney stone problem was summarized in Figure 2.
Upon the creation of our data set for microscopic images, we utilized image pre-processing for classification. Each microscopic image was resized into 224 Â 224 pixels by cropping at the centre. Then, random horizontal split and random 5 rotation was applied to random sets of images to increase generalization. Finally, each image was normalized with specific mean and standard deviation values for each RGB channel. These specific values were selected after many years of image processing research for convolutional neural networks (CNNs). 13 Table 2 presents the image processing parameters in detail.
For all of our image processing operations, we used PyTorch's 13 transforms package. Some sample images after image pre-processing are presented in Figure 1. After the images are processed, the data set is split into training and testing sets with a ratio of 70% and 30%, respectively. Thus, it is important to note that 26 stones' images are used for training, and 11 stones' images are used for testing.

| RESULTS
We achieved an overall and weighted accuracy of 88% and 87%, respectively, with an average F1 score of 0.84. Training and validation accuracies for the number of epochs are shown in Figure 4. One epoch corresponds to all training images processed once.
The positive predictive value, sensitivity and F1 score for each stone type were respectively reported in Table 4 as follows: CaOx The confusion matrix in Figure 5 summarizes the predictions made by our classifier, where each column represents the predicted stone type, and each row shows the actual stone type.

| DISCUSSION
This study demonstrates a bench-to-bedside platform for rapid classification of kidney stones using smartphone microscopic images. Our pro- for testing our classifier and tested on 30% of the unseen kidney stones images, unlike the leave one out approach from Black et al.
Last, we believe that our technique has the potential to use AI for linking stone observations with patient data. In most cases such data will be recent data, acquired from the patient at the time when the stone problem presented or a few years before that in the case of recurrence. In that respect, the characteristics of the external parts of a stone will be linked to the current situation of the patient while the origin of the stone may be decades older and related to events that occurred many years earlier in a time period of during which probably no data are available for that patient. 14 Our current study has certain limitations. The low natural frequency of occurrence of some types of kidney stones was an obstacle in collecting data for representing all types of kidney stones in our data set. Our data set only represents around 90% of kidney stone cases, and the current data set has imbalances in the number of stones for each respective type. In the future, we want to include other kidney stone types such as brushite and mixed stones to expand the reach of our classification and improve our data imbalance. The collection of a larger image data will also improve CNN accuracy. Although our current study has limitations, it serves as a good first step towards demonstrating an automated kidney stone classification method using smartphones.

CONCLUSIONS
We demonstrate a rapid and accurate point of care diagnostics method for classifying the four main types of kidney stones. Our work demonstrates the significance of smartphone microscopy and deep learning for future medical diagnostics platforms. In the future, diagnostic tools that combine smartphone microscopy with AI can provide accessible health care that can support physicians in their decisionmaking process.

ACKNOWLEDGEMENTS
This project was supported in part by the Scientific and Technological Research Council of Turkey (TUBITAK).

DISCLOSURE OF INTERESTS
None.