A low‐cost multimodal head‐mounted display system for neuroendoscopic surgery

Abstract Background With rapid advances in technology, wearable devices as head‐mounted display (HMD) have been adopted for various uses in medical science, ranging from simply aiding in fitness to assisting surgery. We aimed to investigate the feasibility and practicability of a low‐cost multimodal HMD system in neuroendoscopic surgery. Methods A multimodal HMD system, mainly consisted of a HMD with two built‐in displays, an action camera, and a laptop computer displaying reconstructed medical images, was developed to assist neuroendoscopic surgery. With this intensively integrated system, the neurosurgeon could freely switch between endoscopic image, three‐dimensional (3D) reconstructed virtual endoscopy images, and surrounding environment images. Using a leap motion controller, the neurosurgeon could adjust or rotate the 3D virtual endoscopic images at a distance to better understand the positional relation between lesions and normal tissues at will. Results A total of 21 consecutive patients with ventricular system diseases underwent neuroendoscopic surgery with the aid of this system. All operations were accomplished successfully, and no system‐related complications occurred. The HMD was comfortable to wear and easy to operate. Screen resolution of the HMD was high enough for the neurosurgeon to operate carefully. With the system, the neurosurgeon might get a better comprehension on lesions by freely switching among images of different modalities. The system had a steep learning curve, which meant a quick increment of skill with it. Compared with commercially available surgical assistant instruments, this system was relatively low‐cost. Conclusions The multimodal HMD system is feasible, practical, helpful, and relatively cost efficient in neuroendoscopic surgery.


| INTRODUCTION
Neuroendoscopic surgery becomes widely accepted with the publication of results in large clinical series in the late 1990s and has been an expanding field of neurosurgery (Cinalli et al., 1999;Hidalgo, Ali, Weiner, & Harter, 2016;Hopf, Grunert, Fries, Resch, & Perneczky, 1999). Neuroendoscopic surgery reflects the tendency of modern neurosurgery toward minimum injury, which represents access and visualization through the narrowest practical corridor and maximum effective action at the target area with minimal disruption of normal tissues (Elbabaa, 2016). With the advantages of a wider vision of surgical field, less traumatism to normal brain tissues, and reduced complications, neuroendoscopic surgery is gaining more and more popularity in neurosurgeons and patients. It has become the first choice of management for pituitary adenoma and intraventricular pathologies such as obstructive hydrocephalus, intracranial cysts, and small intraventricular tumors (Di Rocco, Yoshino, & Oi, 2005;Rocque, 2016).
However, everything has its limitations. For one thing, neuroendoscope generally offers two-dimensional (2D) image and has no depth of field compared with microscope. For another, the learning curve of neuroendoscopic surgery is long and shallow, which means a longer learning time (Bokhari, Davies, & Diamond, 2013). Furthermore, the monitor of a neuroendoscope is usually connected together with xenon light source and digital video recording system, which is large and ponderous, making it inconvenient to adjust the location freely during operation. Using a standard monitor sometimes requires surgeons to move into unpleasant postures.
In neurosurgery, it is of key importance to have a thorough, accurate, and detailed knowledge of the anatomical structure of the surgical target because the surgical outcomes are directly connected to patient's neurological functions. Compared with 2D methods, 3D computer simulation enables more accurate, realistic, and intuitive diagnosis and surgical analysis (Khor et al., 2016). With rapid advances in technology, virtual reality, augmented reality, and wearable devices have been adopted for various uses in medicine, ranging from simply facilitating fitness to more complex devices used in surgery (Liu, Jenkins, Sanderson, Fabian, & Russell, 2010;Maithel, Villegas, Stylopoulos, Dawson, & Jones, 2015;Shao et al., 2014). In medicine, wearables can be broadly divided into body sensors and head-mounted displays (HMD). During operative procedures, HMDs can provide surgeons with 3D radiographic information while maintaining sterility.
To overcome the deficiencies mentioned above, we developed a low-cost, easy to use multimodal HMD system on the strength of wearable devices and computer image processing technology.
Feasibility and practicability of such a system were investigated in neuroendoscopic surgeries involving ventricular system diseases.
A detailed introduction of the system was made in this study.

| Constitution of the multimodal HMD system
The system mainly contained three components: a HMD, an action camera, and a laptop computer, all of which were commendably connected together. Concrete connection pattern and detailed specification information of instruments were presented in Figure 1 and Table 1. The HMD, a HMZ-T1 Personal 3D Viewer (Sony, Japan) to be specific, was the critical part of this system. HMZ-T1 is a visor style HMD composed of a visor and an external processor unit. It contained two miniature organic light emitting displays (size: 0.7 inch, resolution: 1,280 × 720px per eye) providing video. It could display any proprietary video compatible with the high-definition multimedia interface (HDMI) standard. By connection with the neuroendoscope, the neurosurgeon could get a high-definition or 3D image of surgical field, getting rid of the restriction of monitor position. By combining with computer image processing technique, we could realize 3D display and virtual endoscopy (VE) simulation of ventricular diseases.
To realize real-time visualization of surrounding environment on the premise not taking off the HMD repeatedly, an action camera (HDR-AS20, Sony, Japan) was bounded to the HMZ-T1 viewer. With a 170° ultrawide lens, the HDR-AS20 could record video of 1,080p with intense contrast and sharpness. Fastened to side arm of HMZ-T1, the action camera was like a pair of eyes observing the surrounding environment. With the action camera, the neurosurgeon could perceive the surroundings whenever needed by just switching the input sources rather than putting on and taking off the HMD repeatedly.
The HMZ-T1 processor was a small set-top box with only one input and one output. But there were three different image sources (the endoscope image, the 3D VE image, and real-time video of surroundings) to display. To overcome this defect, both HDMI switcher (Ugreen, China) and HDMI splitter (Ugreen, China) were used at the same time. Through the switcher, neuroendoscope, laptop computer, and action camera were connected to the HMD all together. With the splitter, collected images were displayed on both the HMD and the monitor of neuroendoscope simultaneously so that the assistants and nurses could get the same view with the operator. Combining all these instruments together, the neurosurgeon could freely switch among different image sources using a sterilized remote controller of the switcher without moving cables around.

| Acquisition and processing of radiological images
All image data processing was performed with 3D Slicer (https:// www.slicer.org, version 4.5.0), an open source software platform for medical image informatics, image processing, and 3D visualization. To acquire appropriate MRI data for processing, all patients routinely underwent both conventional and 3D sampling perfection with application optimized contrast (3D-SPACE) using different flip angle evolution MRI examination performed on the same 1.5-T MRI scanner (Siemens Espree, Germany) (Li et al., 2016).
The imaging protocol started with a T1-weighted, 3D magnetization prepared rapid acquisition gradient echo (MPRAGE) sequence

| Applications in neuroendoscopic surgery
It took just several minutes to assemble the multimodal HMD system in a surgery. The neurosurgeon first employed the action camera view to accomplish surgical procedures that did not need the neuroendoscope, such as scalp incision and trepanning. Once the neuroendoscope was placed into the ventricle, endoscopic image would be displayed on the HMD. When the neurosurgeon wanted to look through patient's preoperative MRI image or reconstructed 3D VE image to better understand the positional relation between lesions and surrounding normal tissues, all he needed to do was to switch the input source to laptop computer. Details on how the system worked in a neuroendoscopic surgery are shown in Figure 3.
The study was approved by the Institutional Review Board of Chines PLA General Hospital and was conducted in conformity to the Declaration of Helsinki. All patients gave their written informed consent prior to participation in this study.

| Statistical analysis
The quadratic regression analysis of operative time and number of cases performed was used to determine the learning curve for a F I G U R E 1 Schematic view of the overall configuration of the multimodal head-mounted display system for neuroendoscopic surgery. DVI, digital video interactive; DICOM, digital imaging and communications in medicine

| Demographic and clinical characteristics
Between March 2015 and December 2016, a total of 21 consecutive patients with ventricular diseases underwent neuroendoscopic surgeries with the help of this multimodal HMD system in our department. Their average age was 20.1 years (range, 1-53 years).
Interestingly, five patients with suprasellar arachnoid cyst had been previously misdiagnosed with other hydrocephalus-causing diseases, and four had been previously treated with ventriculoperitoneal shunt with no symptom relief. All MRI scans and medical image processing were performed prior to surgery. It took 12 min on average to reconstruct a 3D VE image of one patient while assembly of the system cost 5-10 min.
Demographic and clinical characteristics of included patients are summarized in Table 2

| Feasibility and comfort
Neuroendoscopic surgery was successfully performed under the assistance of the multimodal HMD display system in all patients, with no system-related technical problem happened. The neurosurgeon (Chen, X. L.) experienced this multimodal HMD as a helpful, comfortable, lightweighted, and easily adjustable device. Switching the display to the action camera modality, the neurosurgeon could perform surgical procedures and make contact with the environment easily as usual.
When switched to endoscope modality, the endoscopic image was always in the direct line of sight, allowing the neurosurgeon to work in a relaxed position. When the neurosurgeon would like to look over patient's medical images, all he needed is to switch the input source to computer using a sterilized remote controller.
Eye fatigue occurred in three patients whose operation time exceeded 80 min at the early stage of this study. No complaints of blurred vision, dizziness, or other forms of simulation sickness arose.
During operation, the neurosurgeon could adjust or rotate the reconstructed 3D VE image whenever necessary with the assistance of a leap motion controller connected to the computer ( Figure 3C). The VE view during operation helped the surgeon to get a better understanding of spatial relationships between lesions and surrounding anatomic structures. The ability of freely switching among endoscope images, VE images, and surrounding images was a key feature of this system.

| Displays and learning curve
Screen resolution of the HMD was high enough for the neurosurgeon to judge lesions and peripheral structures and to operate surgical equipment accurately. The range and depth of vision of the system seemed comparable to or even better than those of the video monitor. performed, which meant that the system was easy to master. The multimodal HMD system was feasible and easy to use with sufficient safety for neuroendoscopic application.

| DISCUSSION
This study proposed an innovative wearable multimodal display system for easy switching between endoscopic, surrounding, and computerized image modalities in neuroendoscopic surgery. Wearable technology is an emerging industry which has made possible the integration of technology into healthcare and surgery as well as daily life. The trend has been supported by the miniaturization of the components necessary for the collection, storage, processing, sharing, and presentation of data (Ortega, Wolff, Baumgaertner, & Kendoff, 2008;Rawassizadeh et al., 2015). Several HMD devices have been identified, which are currently utilized in surgery, imaging, simulation, education, and as a navi- F I G U R E 4 Learning curve of the multimodal head-mounted display system. Quadratic regression analysis of the operative time and number of cases experienced

| Multimodality medical image
Multimodality medical image fusion and processing have developed rapidly in recent years. Multimodality medical image and postprocessing have proved to be clinically useful for accurate diagnosis and effective surgical treatment of brain disease (Giordano, Wrede, Stieglitz, Samii, & Lüdemann, 2007;Ito et al., 2010). Reconstructed images can be adjusted and rotated freely, not only providing a better representation of spatial relationships between lesions and surrounding structures but also predicting the patterns and variants that will be encountered on the trajectory to lesions (Kakizawa et al., 2003). Moreover, reconstructed images allow the surgeon to depict the actual operative window from angles imitating surgical approaches and to understand the neurovascular complex by viewing from angles that are impossible to obtain intraoperatively (Neubauer & Wolfsberger, 2013;Satoh, Onoda, & Date, 2007;Xu et al., 2014). 3D visualization can provide the surgeon with individualized, objective anatomic information. In this study, the multimodal HMD system realized free switching between different modal images. The reconstructed VE images provide subtle visual cues such as shadow effects, depth perception, and highlighting results, which allow for an interactive evaluation by zooming, panning, and rotating (Haerle et al., 2015;Zhao et al., 2015). By comparing VE image with actual endoscope image, the neurosurgeon can get a better comprehension of the positional relation between lesions and normal brain tissues, enabling more accurate, realistic, and intuitive surgical analysis.

| Working condition in surgery
Working condition in surgery should enable the surgeon to perform an operation under relaxed position with minimal muscular strain (Cutolo et al., 2017;Hanna, Shimi, & Cuschieri, 1998). In our opinion, the use of such a multimodal display system seemed superior to standard monitor in neuroendoscopic surgery. The orientation of the head and eyes had a natural inclination toward gaze-down viewing in the direction of the hand. Wearing the HMD, the neurosurgeon could stand in his/her favorable position. When input source was switched to surrounding image, what the action camera recorded was adequate enough for the neurosurgeon to operate the instruments and communicate with assistants or nurses. There was no more need to constantly move the screen out of the way and plug it back later on. It was an advantage that with this system the monitor no longer needed to be moved repeatedly to find a better view for the surgeon when his/ her position is changed.
Comfort and expenditure of the system were also important factors to be considered. The HMZ-T1 was not very heavy (14.8 ounces) and accommodated glasses very well. To our experience, it was pleasant to wear that weight on the head for 1.5 hr or a shorter time, after that, it might make the host discomfort or annoyed. The weight of HMD required improvement. Luckily, most ventricular neuroendoscopic surgeries could be finished within 2 hr. Compared with most commercially available surgical assistant systems, the cost of our system is much less (total cost, exclusive of the computer, approximately $1,200). Moreover, the system had a steep learning curve, which meant that it was easy and quick to learn and master such a multimodal HMD system.

| Limitations
There are some limitations that should be acknowledged. The application of wearable devices in neurosurgery is relatively new, and the true usefulness of such a multimodal HMD system needs to be better validated and assessed with prospective controlled study.

| CONCLUSIONS
In this study, we present a low-cost multimodal HMD system for neuroendoscopic surgery. As shown by the study, the system is feasible, practical, helpful, and relatively cost efficient. Through free switching among different image modalities, the system enables a more relaxed position, a better comprehension of lesion's 3D positional relations, and a better view of the operative field. With advances in technology and instruments, wearable technology would play a more significant role in neuroendoscopic surgery and other surgeries in the near future.

ACKNOWLEDGMENTS
The study was partly funded by the National Natural Science Foundation of China (81771481). The authors would like to express their sincere appreciation to all participating patients.