Intraoperative multispectral and hyperspectral label‐free imaging: A systematic review of in vivo clinical studies

Multispectral and hyperspectral imaging (HSI) are emerging optical imaging techniques with the potential to transform the way surgery is performed but it is not clear whether current systems are capable of delivering real‐time tissue characterization and surgical guidance. We conducted a systematic review of surgical in vivo label‐free multispectral and HSI systems that have been assessed intraoperatively in adult patients, published over a 10‐year period to May 2018. We analysed 14 studies including 8 different HSI systems. Current in‐vivo HSI systems generate an intraoperative tissue oxygenation map or enable tumour detection. Intraoperative tissue oxygenation measurements may help to predict those patients at risk of postoperative complications and in‐vivo intraoperative tissue characterization may be performed with high specificity and sensitivity. All systems utilized a line‐scanning or wavelength‐scanning method but the spectral range and number of spectral bands employed varied significantly between studies and according to the system's clinical aim. The time to acquire a hyperspectral cube dataset ranged between 5 and 30 seconds. No safety concerns were reported in any studies. A small number of studies have demonstrated the capabilities of intraoperative in‐vivo label‐free HSI but further work is needed to fully integrate it into the current surgical workflow.


| INTRODUCTION
Multispectral and hyperspectral imaging (HSI) are emerging optical imaging techniques with the potential to radically transform the way surgery is performed. These optical imaging modalities are ideally suited for clinical use because they are non-contact, non-ionizing and non-invasive. As such, in vivo multispectral and HSI systems can potentially provide significant advantages over existing, more invasive intraoperative imaging techniques. Multispectral and HSI exploit the ability to split light into multiple narrow bands beyond the three conventional red, green and blue (RGB) visible spectral bands, enabling analysis of images not seen with the naked eye.
Several medical research multispectral and HSI systems have focused on the technique's diagnostic capabilities. The ability to quantitatively measure oxygenation levels in tissue has been tested in various pathologies and clinical contexts, including; peripheral vascular disease [1,2], retinal eye disease [3,4], diabetic foot disease [5,6] and wound healing [7,8]. Numerous groups have also published work demonstrating the effectiveness of using HSI for cancer detection [9][10][11][12][13][14][15][16][17][18][19][20]. With the development of high-resolution and fast frame rate multispectral and HSI cameras, enabling the capture of a rich imaging dataset at a video rate, these imaging methods offer great potential for real-time non-invasive tissue characterization and surgical guidance [21].
In this paper, we present a brief summary of the physical basis of HSI followed by a systematic review evaluating the intraoperative surgical applications of in vivo label-free, non-fluorescence multispectral and HSI. We report the different types of imaging systems used in surgery from a clinical perspective, review their key technical features and specifications and evaluate their clinical aims, reported effectiveness and safety.

| Light and the hyperspectral cube
Every wavelength of light within the visible light spectrum is perceived as a spectral colour also referred to as monochromatic light. Standard cameras capture images comprising three broad spectral bands, namely red, green and blue whereas multispectral or hyperspectral systems capture images from multiple narrow spectral bands (typically some tens to hundreds) covering a well-defined range of the electromagnetic spectrum. Consequently, each pixel in the multispectral/hyperspectral image provides a rich representation of the full colour spectrum (in contrast to restricting three values for RGB), which can characterize tissue with greater colour, detail, and accuracy. Increasing number of spectral bands drives the transition from multispectral to HSI but the distinction is loosely defined. For the remainder of this article we shall thus refer to all types as HSI.
Single timepoint HSI datasets are termed a hyperspectral cube (HS cube). Hyperspectral cube data spans two dimensions (2D) in space (field of view) and one dimension in spectra (captured wavelength/quasi-monochromatic colours) ( Figure 1). In label-free imaging, the spectral reflectance signature captured by each pixel enables HSI systems to quantitatively assess various tissue properties to provide an assessment of a tissue's microstructure, perfusion and presence of disease [21,22].

| Optical properties of tissues
All tissues reflect light allowing them to be seen (or imaged) in the visible spectrum. As light propagates across biological tissue it undergoes multiple absorption and scattering events [23]. Tissue optical characteristics are determined by the type of tissue and its constituents but may be altered by presence of disease and by surgery [22]. Tissue absorption is determined by molecular composition and can serve as a spectral fingerprint for diagnostic purposes. Tissue components absorbing light are called chromophores and may either be converted to heat or radiated in the form of luminescence (bioluminescence), including fluorescence and phosphorescence but play a limited role in label-free HSI. Scattering is principally caused by the heterogenous subcellular composition of biological tissue but it also reflects a tissue's structural variation due to its extracellular membranous proteins.

| Hyperspectral acquisition methods
There are three main methods of acquiring HSI data; namely, spatial scanning, spectral scanning or snapshot imaging ( Figure 2). Spatial scanning methods typically F I G U R E 1 Hyperspectral cube. Note how spectral signatures may be generated for specific tissues. A.U.: Aribitary Units acquire HSI data by concurrently capturing the complete wavelength spectrum for each pixel (point-scanning) or for a line of pixels (line-scanning) and scanning through the field of view. Point-scanning and line-scanning methods are also referred to as "Whiskbroom" or "Pushbroom" techniques, respectively, and are currently unable to provide a live display of spectral images.
Spectral scanning methods typically use a single exposure to concurrently acquire the whole scene at a single wavelength band with a 2D detector array; they then "step" through each wavelength sequentially to complete the datacube. Spectral scanning methods may also be termed "wavelength scanning" or "wide-field imaging methods" [24]. Advanced image detectors such as charged couple devices (CCDs) and scientific complementary metal-oxide semiconductors (sCMOS) can now enable rapid hyperspectral wide-field imaging by swiftly acquiring 2D images at multiple wavelengths over time. Wavelength scanning has the advantage of achieving high spectral resolution by using monochromatic filter sets such as rotating filter wheels, liquid crystal tunable filters (LCTF) and acousto-optic tunable filters (AOTFs) but the increased spectral selectivity of using a filter is tempered by low light throughput or prolonged imaging times.
Snapshot imaging approaches utilize image-diversion and dispersive elements to acquire the entire three dimensional datacube in a single exposure without using any scanning mechanism. Snapshot cameras may be divided into two categories according to their acquisition and image reconstruction strategies and include direct-measurement systems such as a snapshot mosaic system, or those utilizing computational systems. Similar to modern RGB image sensors that are arranged in a regular, mosaic-like Bayer 2 × 2 pixel filter pattern, the snapshot mosaic systems utilize refined n × n pixel filter arrays and may sometimes be referred to as snapshot mosaic cameras. Computational systems can either perform direct image reconstruction or iterative image reconstruction [25]. Snapshot systems have the potential to F I G U R E 2 Hyperspectral imaging acquisition methods. A and B, Spatial scanning. A, Point Scan. B, Line scan. C, Spectral scanning, for example, wavelength scan. D, Snapshot. Adapted from Wang et al [24] provide real-time HSI; however, at present, computational methods are often disadvantaged by their computational complexity which significantly slows datacube processing and direct-measurement systems currently require large-format cameras which typically have a low frame rate resulting in a lower spatial and spectral resolution [26].

| Search strategy
A structured search of the Pubmed and Web of Science databases was undertaken over a 10-year period. The last date of the search was 11 May 2018 with no initial language restriction. Two independent researchers (JS and YX) applied the search criteria using the Boolean search terms: To ensure all eligible publications were identified, reference lists of included articles were also reviewed, and expert opinion sought.

| Study selection
All citations were evaluated for eligibility; the selection scheme is reported in a Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram [27] (Figure 3).
Titles and abstracts were screened to identify articles that satisfied the following eligibility criteria: 1. Clinical studies in which in vivo label-free nonfluorescence HSI techniques had been used intraoperatively in adult patients undergoing surgery 2. Studies that reported on the technique's safety or effectiveness All peer-reviewed publications in which Englishlanguage manuscripts were available through electronic indexing comprising were eligible for inclusion in the systematic review. Full articles were obtained and further assessed for eligibility and any discrepancy was resolved through mutual review and involvement of the senior author.

| Data extraction and outcomes
A data extraction template was used by two independent reviewers (JS and YX) to collect study-level information, including: 1. System name 2. Stated clinical aim of the HSI device, including the target tissue, patient group and operation 3. Key technical features of the device including hyperspectral cube acquisition time 4. Technical specifications, including; camera(s) used, imaged wavelength spectrum, light source, size and compatibility (if designed to be used with current surgical hardware) 5. Safety and effectiveness of the device Corresponding authors and device manufacturers were contacted to provide supplemental data when required.

| Quality assessment
We assessed methodological quality by using the Methodological Index for Non-Randomised Studies (MINORS) scoring system [28]. All studies are scored on the following criteria: (1) a stated aim of the study; (2) inclusion of consecutive patients; (3) prospective collection of data; (4) endpoint appropriate to the study aim; (5) unbiased evaluation of endpoints; (6) follow-up period appropriate to the major endpoint; (7) loss to follow-up not exceeding 5%; (8) prospective calculation of the study sample size. Comparative studies are also scored with respect to: (9) an adequate control group; (10) contemporary groups; (11) baseline equivalence of groups; (12) adequate statistical analyses. Rating scores out of 16 and 24 for non-comparative and comparative studies, respectively, were generated by the lead author. Studies of greater quality, that is, those with a higher MINORS score, were given greater weighting in the subsequent qualitative synthesis.

| Study analysis
A descriptive analysis of the included studies is performed, summarizing the extracted data and comparing the clinical and technical features of the various HSI devices. The principal summary measure is the hyperspectral cube acquisition time.

| Studies included in final analysis
In total, 2473 articles were identified through the database searches and an additional record was identified through other sources. Following removal of duplicate and non-English language studies, 1837 manuscript titles and abstracts were screened. After applying the eligibility criteria 16 full text articles were reviewed. A further two articles were excluded because HSI analysis was only performed on tissue ex vivo in these studies. In all, 14 studies satisfied the inclusion criteria and were included in further qualitative analysis ( Figure 3).

| Study quality
Outcome reporting across studies with extremely variable ( Table 2). Using the MINORS assessment tool, most studies scored between 7 and 10, out of a possible 16, with one notable exception being the only comparative study scoring 22 out of 24 [33]. None of the studies included in this review prospectively calculated their study size and with the exception of the study by Liu et al none included consecutive patients. Prospective data collection was inferred by the study design in most studies but was not explicitly mentioned in any.

| Clinical aims
To date, HSI has only been used for intraoperative tumour detection in brain tumour surgery [37][38][39][40] whereas a variety of different surgeries have used HSI to assess tissue perfusion and oxygenation; including, an assessment of reconstructive skin flaps in breast [29] and oral cancer [34] surgery, to assess renal oxygenation during partial nephrectomy [30][31][32][33], and to assess brain perfusion within the epileptogenic zone during epilepsy surgery [35,42] and following cerebrovascular anastomosis [36].

| Acquisition methods
The reported HSI systems used either a wavelength scanning [29][30][31][32][33][34][35][36], line-scanning [37][38][39][40][41] or snapshot mosaic [42] acquisition mode. Two systems imaged a selection of discrete wavelengths [29,35], whereas the remainder preferred to image a spectral range with 0.4 to 5 nm wavelength steps. A wide spectral wavelength range was employed by the HELICoiD group for the purposes of tumour detection (400-1700 nm) [37][38][39][40] whereas a narrower spectrum was used in studies generating an intraoperative tissue oxygenation map. A very narrow wavelength range of 520 to 645 nm used by the Dallas group [30][31][32][33] but other HSI oxygenation studies included the infrared wavelength spectrum. Different bespoke configurations of cameras, and light sources were used in the construction of the reported HSI systems and these are listed in Table 1. The size of each system also varied considerably; some systems were designed to be compatible with existing surgical microscopes [30-33, 35, 42] whereas others were large free-standing machines [29,34,[37][38][39][40][41][42]. No safety concerns relating to the use of intraoperative HSI were reported in any of the studies.

| Clinical correlation of HSI data
All studies generating a tissue oxygenation map reported good correlation between the HSI data and the operative procedure performed although this was frequently a subjective assessment [29,34,35,41]. However, the Dallas group quantitatively analysed their HSI data by correlating intraoperative renal oxygenation measurements with patients' eGFR results in both open [30,31,33] and laparoscopic [32] renal surgery. Their pilot studies demonstrated that HSI renal oxygenation data may help to predict those patients at risk of postoperative renal insufficiency and may distinguish between patients with otherwise similar baseline characteristics such as eGFR [31,32]. To date, the HELI-CoiD group are the only ones to have used HSI to perform in vivo intraoperative tissue characterization [37][38][39][40] and successfully demonstrated that HSI may be used to discriminate between normal brain tissue, tumour tissue, hypervascularised tissue, and background with high specificity and sensitivity using an intra-patient 10-fold cross-validation method.
The most frequently reported limitation of the current HSI systems relate to the time-consuming calibration required and the slow image acquisition and processing time. Based on the available data, the time taken to acquire a complete hyperspectral cube dataset was calculated to take between 5 and 30 seconds. The size and bulkiness of some of the standalone systems, such as the one used in HELI-CoiD studies [37][38][39][40], also made it difficult to easily integrate its use into the standard surgical workflow which in turn reduced the amount of training data acquired and the image processing speed.

| Clinical functions of HSI
The ability to deliver real-time hyperspectral images at a video-rate makes it ideally suited for intraoperative surgical use. HSI has the potential to enhance a surgeon's vision at the molecular, cellular and tissue level in four different ways; including (a) the identification of residual tumour tissue; (b) the classification of critical anatomical structures within the surgical field; (c) the ability to monitor tissue oxygenation during surgery and (d) by enabling surgeons visualize target structures under blood.

| Identification of residual tumour tissue
It is often very difficult to differentiate a line of demarcation between a tumour and surrounding healthy tissue using the human eye. Panasyuk et al were one of the first to demonstrate that HSI may be used to guide surgery in vivo by using HSI to distinguish tumour tissue from normal breast and other tissues in an experimental rat breast tumour model [49]. Others have also evaluated the effectiveness of HSI to examine tumour resection margins in ex vivo surgical specimens in the operating room including patients undergoing head and neck cancer [20,50], colon [51], and skin cancer [52] surgery and in vivo fluorescence-based HSI systems have been reported for guiding urological [53] and brain tumour surgery [54].

| Identification of critical anatomical structures
In many operations, the target tissue or tumour is intricately associated with other critical anatomical structures which should be preserved during surgery. HSI offers the potential to create real-time anatomical maps that may be overlaid onto the surgeon's visualization system to highlight the location of such critical structures. Several animal studies have demonstrated this application of HSI for use in a variety of surgical procedures including; cholecystectomy [55,56], hysterectomy [57] and parotidectomy [57].

| Monitoring tissue oxygenation
Tissue perfusion and oxygenation are positive indicators of a tissue's viability. Assessing dynamic changes in tissue perfusion and oxygenation aids patient management in various clinical settings and several studies have already demonstrated its potential for guiding surgical procedures. The Dallas group developed HSI technologies to quantify tissue oxygenation in renal surgery [58,59] whilst others have applied the technology to guide flap reconstruction [60][61][62] and assess vessel [63][64][65] and bowel [66,67] anastomoses.

| Visualizing structures beneath blood
Finally, HSI in the near infrared wavelength spectrum has been shown to successfully characterize target tissues submerged in a blood layer that could not be seen by the naked eye [68].

| Intraoperative in vivo HSI functions
To date, intraoperative in vivo surgical HSI devices have either focused on generating an intraoperative tissue oxygenation map during surgery [29][30][31][32][33][34][35][36]41] or by enabling tumour detection [37][38][39][40]. The intraoperative assessment of tissue and perfusion can assist the surgeon in various ways. The Dallas group developed a HSI system capable of imaging the entire kidney during in both open and laparoscopic surgery [30][31][32][33] and is a continuation of the preclinical work performed in a pig model [58,59]. The customized DLP system used in their studies consists of a wavelength scanning system in the 520 to 645 nm range. An "optical biopsy" is subsequently performed on an 81 to 100 pixel square area to generate a representative sample of the kidney's oxygenation status. Their work demonstrated that oxygenation maps can successfully characterize the dynamic changes in renal perfusion observed during surgery. It also showed that intraoperatively acquired HSI data may be used as an independent predictor of post-operative renal function [31,32]. Three other studies have evaluated how intraoperative tissue oxygenation maps generated from HSI data might guide skin flap reconstruction in breast [29], oral cancer [34] and vascular [41] surgery. These studies all demonstrated a correlation between the HSI-oxygenation data and the expected dynamic surgical changes however post-operative follow-up data relating to the end-organ's viability was not available. Thus, in future work, it would be helpful to compare the viability of skin flaps performed with and without the intraoperative use of HSI-data. Following on from existing pre-clinical work [63][64][65][66][67], it will also be exciting to see whether HSI-generated tissue oxygenation maps of vessel and bowel anastomoses may be acquired in human patients and whether its use helps to improve surgical technique and outcomes.
The second clinical focus of using intraoperative HSI has been to guide tumour resection. The feasibility of using HSI as a diagnostic technique for cancer has been comprehensively evaluated in multiple preclinical animal models and several recent studies have also demonstrated its use in assessing tumour margins in ex vivo tissue in the operating room [9][10][11][12][13][14][15][16][17][18][19][20][44][45][46][47][48]. Fewer studies have reported the in vivo use of HSI during surgery but several groups have used HSI to highlight subtle changes in exogenous fluorescence [53,54,69]. Bravo et al demonstrated that hyperspectral data process improves PpIX contrast during fluorescence guided surgery of human brain tumours [54] and other laboratory studies have confirmed that HSI can quantitatively measure and visualize low concentrations of PpIX and that there is a significant correlation with the tissue's cellularity and nuclear-cytoplasmic ratio [70].
Recently, the HELICoiD group have been demonstrating the potential of label-free HSI in guiding brain tumour surgery [37][38][39][40]. Their customized supervised and unsupervised machine learning classifier method provided excellent results with specificity and sensitivity reaching 100% in most cases using a 10-fold intra-patient cross-validation method [39]. However, despite these very positive results, it is very difficult to validate in vivo HSI results. This limitation is particularly true in brain tumour surgery where the margins between tumour and normal brain are blurred and where normal brain tissue cannot be excised for routine testing.
Another limitation of the HELICoiD system was its size. The system's hardware was very large and cumbersome [39,40] which made it difficult to integrate easily into the standard neurosurgical workflow. This, in turn, limited its use and the amount of data captured. High-speed computational processing is required to deliver real-time intraoperative image analysis but is reliant on a large training dataset and only 22 patients (36 captures) were obtained [37][38][39][40]. Consequently, the total image processing time, including acquisition and computational processing, took between 35.53 and 68.76 seconds even with the use of an accelerated processing algorithm [40]. A slow hyperspectral cube acquisition and processing time was common to most studies (5-30 seconds) and this remains the key obstacle to delivering a truly realtime HSI system suitable for intraoperative use.
Future work should focus on developing optimal, yet practical intraoperative HSI systems compatible with existing, commonly used surgical visualization systems that may then be tailored for use in different surgical procedures.
Researchers should determine key design specifications to enable the easy integration of such a device into their desired surgical workflow in order to facilitate its widespread use, enabling the acquisition of larger training datasets required to generate reliable real-time models.

| Limitations
The present systematic review was limited by various factors. Firstly, given the variety of ways HSI data was presented and the small numbers of available studies, it was not possible to perform a meta-analysis and quantitatively analyse the data. Consequently, we were unable to draw any firm conclusions concerning the effectiveness of the described HSI techniques. Secondly, the studies were of mixed methodological quality and reported a variety of customized HSI systems, limiting our discussion to a qualitative discussion of a small number of higher quality studies.

| CONCLUSION
HSI has the potential to transform intraoperative surgical guidance. Numerous studies have evaluated its use in preclinical in vivo animal models and others have demonstrated the usefulness of intraoperative ex vivo tissue analysis. The use of in vivo label-free HSI systems has not been widely tested during surgery but the small number of studies demonstrate its current capabilities and highlight avenues for future research. However, in order to translate this promising imaging technique into regular surgical use, it must be seamlessly integrated into the surgical workflow. Hardware development therefore needs to leverage recent advances in the miniaturization of hyperspectral sensors and robust realtime image models must be developed.