First experience in clinical application of hyperspectral endoscopy for evaluation of colonic polyps

Early detection and resection of adenomatous polyps prevents their progression to colorectal cancer (CRC), significantly improving patient outcomes. Polyps are typically identified and removed during white‐light colonoscopy. Unfortunately, the rate of interval cancers that arise between CRC screening events remains high, linked to poor visualization of polyps during screening and incomplete polyp removal. Here, we sought to evaluate the potential of a hyperspectral endoscope (HySE) to enhance polyp discrimination for detection and resection. We designed, built and tested a new compact HySE in a proof‐of‐concept clinical study. We successfully collected spectra from three tissue types in seven patients undergoing routine colonoscopy screening. The acquired spectral data from normal tissue and polyps, both pre‐ and post‐ resection, were subjected to quantitative analysis using spectral angle mapping and machine learning, which discriminated the data by tissue type, meriting further investigation of HySE as a clinical tool.

purpose in a range of screening programmes, where polyps are identified and removed via polypectomy during the colonoscopy procedure. An interval cancer is defined as CRC diagnosed after a colorectal screening examination or test in which no cancer is detected, and before the date of the next recommended exam [3]. The rate of interval cancer has been found to be 1.1 to 2.7 per 1000 person-years of follow-up [4] and recent studies have suggested that many interval cancers occur because of missed lesions or incomplete removal of adenomatous polyps at prior colonoscopy [4,5]. At present, colonoscopy provides only morphological colour images of the colon, which presents challenges for identification of flatter polyps or residual polyp tissue after resection, particularly for larger polyps, as they are difficult to appreciate relative to the background normal tissue.
To improve lesion visualization and prevent interval cancers, many advanced colonoscopies have been introduced [6], including: narrow-band imaging [7]; confocal microendoscopy [8]; Raman spectroscopy [9]; and hyperspectral endoscopy (HySE) [10,11]. Narrow-band imaging illuminates the tissue with a restricted wavelength range of blue and green lights alternately, which enhances the visibility of vascular and mucosal structures due to strong haemoglobin absorption at the selected wavelengths. Nonetheless, detection of small or flat lesions remains challenging and bleeding during resection can obscure polyp margins. Confocal microendoscopy provides morphological information at subcellular resolution, but fluorescence labelling is needed and the field-of-view is very small, meaning that identification of suspicious regions by white-light colonoscopy is still needed prior to application. Raman spectroscopy measures vibrational signatures from the tissue with good discrimination between diseased and healthy colonic tissue, however, it requires a very sensitive optical setup and high-resolution spectrometer due to the measurement of inelastic scattering light and similar to confocal microendoscopy is still dependent on white-light colonoscopy to guide placement of the Raman probe.
HySE relies on the optical absorption spectra of tissue to discriminate disease based on changes in both morphological and biochemical characteristics. These spectral profiles could enable both detection and grading of disease in the gastrointestinal tract [11][12][13], including for CRC diagnosis based on different spectral profiles between normal mucosa and lesions [10,14,15]. By integrating line-scan hyperspectral imaging with a standard white-light imaging approach, spectral profiles can be obtained across a wide field-of-view during endoscopy [13]. Here, we sought to evaluate the potential of HySE as a visualization aid to enhance polyp discrimination for detection and resection in a proof-of-concept clinical study.

| Compact and mobile hyperspectral endoscopy system
A HySE system reported previously [13] was optimized and redesigned to make it more compact to increase mobility for the clinical trial. The HySE system is designed and constructed using an accessory channel "babyscope" (Polyscope, Polydiagnost, Germany) that can be introduced through the accessory channel of a standard colonoscope. The commercial CE-marked babyscope consists of a 10 000-fibrelet imaging bundle (PD-PS-0095, PolyDiagnost) introduced into a disposable catheter (PD-PS-0144p, PolyDiagnost) to relay diffusely reflected light from the colon to custom external detection optics. The proximal end of Polyscope was imaged and magnified by an infinity corrected objective lens (Â40, NA 0.6, Nikon). To create a more compact optical system compared to the previous report, two tube lenses were replaced with a single tube lens (f = 150 mm), which was put in front of non-polarizing beam splitter cube (BS025, Thorlabs). Thus, wide-field and spectral imaging shared the objective lens, tube lens, and beam splitter, which minimised image distortions between the wide-field colour (2D x,y) and spectral images (2D y, wavelength λ) ( Figure 1A). For hyperspectral imaging, a spectrograph (IsoPlane 160, Princeton Instrument) with grating (150 lines/mm) and electron-multiplied CCD camera (ProEM 1024, Princeton Instrument) were used. The spectral range of HySE was 400 to 800 nm with a spectral resolution of~2.5 nm. Lightfield v6.7 (Princeton Instruments) was used to control the spectrograph and the EMCCD camera. Labview 2017 (National Instruments) was used for synchronized control of the wide-field camera, spectrograph and EMCCD.
The HySE system was built on an aluminium breadboard (600 mm Â 600 mm, Thorlabs) and enclosed by black hardboards to protect sensitive image sensors from ambient light in the theatre ( Figure 1B,C). A self-centering mount was used to make it easy to align and release the proximal end of the fibre bundle from the optical imaging system between patients. A supercontinuum light source (SuperK Compact, NKT Photonics) was coupled to the illumination fibre of the polyscope catheter for the fine alignment of the hyperspectral endoscope and the guidance of endoscope position during patient measurements. Hyperspectral data capture was performed using the white light provided by the standard-of-care colonoscope (Olympus CF-H290) as the excitation source. The HySE system was subjected to careful photothermal and photosafety evaluations ( Figure 1D-F) to ensure compliance with investigational device exemption regulations. The methods used for photobiological safety evaluation are detailed in the Supporting Information. The compact hyperspectral imaging system was placed on the top of a trolley to allow the system to be moved to the theatre for patient measurements. In addition, the light source, laptop, and optical components, including reference targets, power meter, alignment tools, were mounted on the trolley, making it convenient and simple to prepare the imaging system ready for patient measurement within a limited time frame.

| Patient recruitment and consent
We undertook a single-site proof-of-concept study (ClinicalTrials.gov NCT04172493, first posted November 2019) at the Oregon Health and Sciences University (OHSU), United States, to determine whether the use of HySE can improve visualization of abnormal colonic tissue. The HySE protocol is shown in Figure 1G. Eligible participants were adults undergoing screening colonoscopy that consented to the study. All 13 patients included in the study were ASA:1 (American Society of Anesthesiology score). The trial was reviewed by the OHSU Institutional Review Board (IRB18947) following confirmation by the FDA that HySE did not meet the definition of a significant risk device under 21 CFR 812.3(m) of the investigational device exemptions regulation and was approved in June 2019. Participants underwent their screening colonoscopy and polypectomy (as required) according to the institutional standard-of-care. During the procedure, HySE was used to image normal tissue, F I G U R E 1 Optical system and pilot clinical trial design. A, Schematics of the compact HySE (Left) and the method for capturing hyperspectral images via a conventional colonoscope (Right). The HySE was assembled by exploiting a spectrograph, EMCCD camera, CMOS colour camera, and minimum numbers of optical components. A sterilized endoscopic fibre with a disposable catheter was aligned before a patient measurement. The endoscope was introduced through the accessory channel of the clinical colonoscope, and hyperspectral imaging of patient colon was performed by sweeping the clinical colonoscope. B, C, Photos of the HySE system which was assembled on the top of the trolley, carrying a light source, reference target, and calibration tools. D, E, Spectral irradiance and illumination angle of HySE and the clinical colonoscope were compared to check photobiological safety of the proposed system. W.D.: working distance; D: diameter. F, Surface temperature of the endoscope was assessed for 1 hour to test photothermal safety. G, Pilot trial design and experimental procedures colonic polyps prior to removal and residual tissue after polyp removal (post-resection tissue). Successful acquisition of HySE data was possible in n = 7 patients. Initial study recruitment was paused after n = 2 patients due to institutional concerns with the sterility of the reusable fibre bundle of the polyscope device. Subsequently, an additional sterilisation procedure was introduced. A further n = 11 patients were recruited, of which 8 underwent the full trial protocol. In one patient measurement, it was found that the accessory cap supporting the introduction of surgical tools prevented HySE from reaching to the distal end of the clinical colonoscope, and thus the accessory cap was removed before introducing HySE in other measurements. In 3 patients, their preparation was inadequate to continue with the study. Therefore, there were useable data from seven patients in the second phase of the trial, five of which had polyps (Table S1). Recruitment was completed in January 2020. This was a noninterventional study, so HySE was not used to inform on any aspects of medical management. Acquired data were analysed post-procedure in a video review with the treating endoscopist.

| Hyperspectral endoscopy procedure
The hyperspectral endoscopy procedure is summarized in a flow chart ( Figure S3) and details of the procedure are given below:

Optical fibre bundle was prepared for imaging
First, in order to avoid patient to patient cross-infection, the fibre bundle was sterilized according to the manufacturer cleaning protocol before aligning with the optical system. Briefly, the cleaning agents were injected through the input channel of the fibre container, and it was released at the output channel. After washing the entire fibre with the same high-level disinfectants used for the sterile processing of the endoscopes the compressed air was injected through the input channel of the fibre container to remove the cleaning agents from the surface of the optics fibre. The cleaning process was completed in the Sterile Processing Department of OHSU within 40 to 60 minutes.

Informed consent was obtained to recruit a patient
Clinicians obtained an informed consent from an eligible participant for the clinical study. 3. The sterile endoscope was aligned with the HySE system Next, the trolley conveying the HySE system was brought into the endoscopy suite and the polyscope catheter and fibre bundle were aligned within 15 minutes, just before each patient measurement.
The illumination fibre coupled to the supercontinuum light source was exploited for the fine adjustment of the fibre bundle position to measure a perfect focused image.

Colonoscopy and hyperspectral imaging was performed
The clinician first completed the screening colonoscopy of the patients via the standard-of-care colonoscopy, including white-light and narrow-band imaging. After the examination, the operating clinician decided on appropriate imaging regions for application of HySE, including normal colonic mucosa and visually detectable polyps. HySE was then introduced into the accessory channel of the colonoscope. Light from the supercontinuum source in the polyscope illumination fibre was used to aid in positioning the polyscope at the distal end of the colonoscope. Once complete, the supercontinuum source was turned off. The clinician collected hyperspectral and wide-field colour images simultaneously under white light illumination from the colonoscope, acquiring 100 spectral and wide-field images at each location while controlling the positioning of HySE with the colonoscope. Exposure times of the spectral and wide-field imaging cameras were set at 57 and 35 ms, respectively. Polyp areas were examined again after resection, with regions denoted as "postresection tissue." The entire image acquisition process was performed by the clinician. 5. The used optical fibre bundle was sent for cleaning.
After the patient measurement, HySE was removed from the colonoscope. The disposable catheter was removed and discarded. The fibre bundle was put in a specialized carrier and sent to the sterilization team. Two optical fibre bundles were used in the pilot study, therefore, each fibre measured 6 and 4 patients, respectively (Table S1). 6. Data analysis was performed after completion of the procedure. Data analysis methods are detailed in the Supporting Information.

| Wide-area hyperspectral image reconstruction was successfully applied in the colon
HySE operates on the basis of line-scan hyperspectral imaging, which means that a 2D camera is included in the spectrograph to record spectral profiles as a function of position along one dimension. The second spatial dimension is recorded by laterally scanning the endoscope across the colon lumen (Figure 2A,B). The full 3D data set, or hypercube, (spatial x, spatial y, wavelength λ) is then stitched together using tracked spatial features in the simultaneously recorded images from the colour camera ( Figure 2C). This feature tracking also compensates for image distortions that arise due to patient motion. We successfully stitched together panoramic images from the colour camera ( Figure 2D) and an associated panoramic hyperspectral data cube ( Figure 2E, Movies S2, S3) for each acquisition. A synthetic RGB image composed from the hyperspectral data cube as described previously ( Figure 2F) shows a clear lumen structure and obvious vascular features and were used to extract tissue-specific absorbance spectra ( Figure 2G). These results confirm that the HySE system and associated hyperspectral image reconstruction method can successfully obtain hyperspectral data under clinical conditions despite the low contrast images and distortions arising due to patient motion.

| Coefficient of variation for hyperspectral data was within 10%
In order to obtain tissue type-specific spectral profiles, hyperspectral signals were investigated in areas clearly identified by clinicians belonging to one of three groups: normal colonic mucosa; polyp; or post-resection tissue, present after polyp removal. In addition, hyperspectral signals arising from specular reflections were also investigated to ensure the accurate segmentation of spectral data. The simultaneously recorded colour images were used by the clinician to identify regions for quantitative spectral analysis (Figures 3A and S2). The average spectral profiles of each group clearly showed different shapes with small SD. In addition to the investigation of spectral profiles of each group, intra-and inter-patient variations were also assessed to check the quality of hyperspectral images. A coefficient of variation (CV) was calculated as a function of wavelength from spectra obtained in clearly identified normal mucosa and polyps. Intra-and inter-patient variations show overall CVs were less than 10%, which is an acceptable variability given the handheld nature of the imaging device ( Figure 3E,F).

| Absorbance spectra showed promising differences between tissue types
As high-dimensional hyperspectral data would be challenging for an operating clinician to interpret in real-time, we investigated the use of spectral angle mapping (SAM) to provide a quantitative comparison between the spectra recorded from different tissue types. SAM calculates the angle between a target and a reference spectrum; the resulting spectral angle values are small if the target and reference signals are similar. SAM analysis conducted using the average spectra from normal mucosa, polyp and post-resection tissues shows statistically significant differences between tissue types ( Figure 4A). Visualizing these SAM scores in 3D clearly shows 4 clusters associated with the 3 tissue types and specular reflection artefacts, indicating that SAM scores could potentially be used for tissue identification and hence image segmentation. We therefore developed a SAMbased segmentation method that could be applied to the real-time segmentation of line-scan hyperspectral data. The line-areas corresponding to the spectrograph slit position were extracted for sequential images from both the colour camera (RGB) and spectral camera; these lineareas were then placed adjacent to each other (Movie S3). An associated SAM image was synthesized, by assigning SAM scores for normal mucosa, polyps, and postresection tissue as green, blue, and red colours, respectively ( Figures 4C and S1). Encouragingly, the regions identified by the SAM scores were consistent with the tissue types annotated by the clinician.

| Machine learning enabled classification of spectral images
We next tested a k-nearest neighbour (kNN) algorithm as a classifier to predict the tissue type of newly observed data. kNN does this by choosing the most frequent tissue types of k-nearest neighbour data points in the feature space ( Figure 5A). Seventy percentage of the representative data displayed in Figure 4C were used for training the kNN classifier, then the remaining 30% of data were used as test dataset to validate the classifier via k-fold cross-validation. The segmentation results based on the established classifier were again consistent with the tissue types confirmed by the clinician (Figure 5B, Movie S3). It is worth noting that in the image representing post-resection tissue only a small area of the wide-field stitched image was occupied by that tissue type, but kNN segmented result determined almost all of the tissue as post-resection (Figures 5B and S2). These results indicate that the kNN predicts areas of bleeding or blood-rich tissue into the same tissue type and suggests a more comprehensive dataset is required for improved classification.
The kNN segmentation method was further applied to other spectral imaging data in which the clinician found it challenging to clearly identify polyps in our wide-field images, either due to the small size of polyps or incomplete scanning of the polyp during the HySE measurement. Spectral data sets, containing 100 spectral images each, were selected from 8 regions of normal mucosa and 5 regions known to contain polyps that could not be spatially located within our images. These data were used to determine whether our segmentation showed any difference between healthy and diseased colonic tissue based on a bulk classification. For these data, we only had the knowledge of whether the measured colon contained polyps or not, rather than their specific location in our spatial data. We evaluated the ratio of the overall area segmented as polyp compared to normal in these data sets. The mean and SD of the polyp ratio in normal mucosa and polyps were 3.17 ± 3.25 and 27.2 ± 13.9 respectively, indicating a statistically significant increase (P value: .0169) in area classified as polyp in the colons that contained polyps compared to the normal colon ( Figure 5C). These results suggest that a machine learning based method could become a versatile tool for the segmentation of hyperspectral data.

| DISCUSSION
Early detection and resection of colonic polyps prevents progression to CRC, yet interval cancers still arise between screening visits, that can be in part attributed to a combination of missed polyps and incomplete resections during standard colonoscopy. Using advanced optical imaging methods in colonoscopy has been proposed to enhance polyp visualization and reduce the occurrence of interval cancers. Here, we conducted a proof-ofconcept clinical study of one such technology, hyperspectral endoscopy. A mobile HySE system was constructed and hyperspectral images of the colon of 10 patients, including spatial and spectral information, were successfully reconstructed by scanning HySE laterally across the colon. Intrinsically co-registered wide-field colour images were used to enable reconstruction of the full hyperspectral datacube. Having obtained the reconstructions, clinician-annotated regions of normal mucosa, polyps and post-resection tissue were subjected to quantitative analyses. Firstly, SAM analysis with average reference spectra obtained from the normal mucosa, polyps, and post-resection tissue was exploited to convert the complex spectral signals into three dimensional-data that showed good discrimination between the tissue types and also distinction from artefacts arising from specular reflections. Secondly, a kNN algorithm was trained using the SAM results together with the information about the corresponding tissue types identified by the clinician. The trained classifier was applied for the direct segmentation of the line-scan spectral images, which in future could provide a route for online classification of polyps within the line area of the hyperspectral system, without the need for panoramic reconstruction.
While HySE has shown promise under clinical conditions in the present study, this was a first experience of applying the technology in patients and as such, two key limitations emerged, which need to be resolved in future studies. First, hyperspectral reconstruction was only possible for colour images that showed clear structural features of the colon. In order to acquire high-quality colour images, the position of HySE and the scanning procedure were carefully controlled during data acquisition. If HySE was placed too close to the tissue or the measured colour image were blurred due to water flow or bleeding, the computer vision algorithm used to compute the geometric transformation matrices was unable to find significant structural features, which compromised the accuracy of wide-field image registration and subsequent hyperspectral image reconstruction. This issue could be overcome in future by exploiting advanced computational tools for image registration, such as deep convolutional neural networks [16]. Even in conditions where the wide-field registration failed, the wide-field colour information was still helpful to compare the segmentation results from spectral machine-learning with conventional white-light colonoscopy methods.
A second limitation was the lack of gold standard histopathological annotation of the in vivo tissue types. The operating clinicians in the study were established experts in their field and annotated the tissue types using the wide-field colour images provided by our HySE system. These images are low-resolution compared to the images obtained by conventional high definition white-light colonoscopy, making accurate annotation of the tissue type challenging even for expert clinicians. The ground-truth technique for the diagnosis of disease is histopathology, but it would require additional biopsies to confirm the tissue type for the normal mucosa and deeper resections to take a biopsy sample following polyp resection. We were therefore unable to include these steps in our pilot study, but encouraged by our findings in this limited patient cohort, a future clinical study could be designed to validate our results and ensure histopathologic data of measured tissue could be obtained co-localised with spectral data, which is expected to improve the accuracy of hyperspectral imaging based-disease diagnosis [17,18].
In spite of these limitations, HySE has demonstrated potential for clinical application. Given the promise in the present pilot clinical study, the next steps are to expand clinical trials of HySE to encompass other tissue types (eg, sessile serrated polyps, flat polyp, and colorectal cancer) and link the HySE data to the underlying histopathological diagnoses. Future trials will help us to test whether HySE could accurately define normal and disease tissue in cases that are challenging with conventional endoscopic methods, such as for flat polyps, incomplete resections and also regular polyps arising in patients with inflammatory bowel disease. If our preliminary results are validated in a larger patient cohort, HySE could open up a new era for medical hyperspectral imaging.