Towards development of a tele‐mentoring framework for minimally invasive surgeries

Tele‐mentoring facilitates the transfer of surgical knowledge. The objective of this work is to develop a tele‐mentoring framework that enables a specialist surgeon to mentor an operating surgeon by transferring information in a form of surgical instruments' motion required during a minimally invasive surgery.

directly controls the movements of the tooltips, whereas in the case of robotic MIS, the surgeon indirectly controls the movement of robotically actuated tooltips via an interface on the console. In both cases of MIS, the surgical field exhibits the complex interaction of highly articulated surgical instrument tooltips with the tissue to be operated. With the current existing tele-mentoring technologies, the expert surgeon can assist the operating surgeon by providing guidance information in the form of either markings or hand gestures.
However, this information is limited because of its two-dimensional and static nature. As a result, it is difficult for the operating surgeon to visualise, comprehend and perform the required surgical tooltip movements. The notion of overlaying minimally invasive surgical instruments motion onto the surgical field is advantageous in mentoring scenarios. For example, augmented reality tele-mentoring (ART) platform proposed by Vera et al. 6 showed faster skill acquisition in laparoscopic suturing and knot-tying task. Preliminary studies conducted by Jarc et al. (using the ghost tool platform with da Vinci surgical system) demonstrated effectiveness for both trainees and proctors during robot-assisted dry-lab training exercises, 7 and robot-assisted tissue dissection and suturing tasks on a live porcine model. 8 In both industry [9][10][11][12][13][14][15][16][17][18][19] and academia, [6][7][8][20][21][22][23][24][25][26][27] tele-mentoring solutions have been developed. These solutions facilitate transfer of information from a specialist surgeon to the operating surgeon via a communication channel and enables tele-mentoring during different types of surgeries. All these solutions include basic capabilities to share the live video feed of the surgical view over a network, provide verbal guidance and perform screen markings (2D screen annotations).  Examples of these commercially available technologies includes VISITOR1 from KARL STORZ, [9][10][11][12] Connect TM -Intuitive Surgical 26,27 and RP Vantage from InTouch. [13][14][15][16][17] Some of these telementoring technologies (e.g., research prototypes, such as STAR from Purdue University [20][21][22] and VIPAR from University of Alabama, [23][24][25] and commercial systems from HELPlightning 18 and Proximie 19 ) also display augmented hands gestures of the remote surgeon. It uses computer vision and image segmentation techniques to segment the operator's hand (captured on a video through a web-camera) and overlays it onto the surgical view. This allows the remote surgeon to virtually put his/her hand in the surgical view and provide assistance to point out different anatomical structures, incision positions and surgical instrument placements. Augmented ghost tools have also been proposed for robotic surgery. 7,8 Though dynamic in nature, it renders only the tooltips without complete body of the surgical instrument. Apart from visualisation, this also limits the realism of augmented surgical tools' motion (as the constraints imposed by the incision points and the tools' kinematic are not taken into consideration). Second, the user interfaces are not applicable to laparoscopic instruments used in manual MIS. The ART platform 6 requires a similar surgical setup (with same configuration of incision points and surgical instruments) at both the remote and local site, thus limiting its application to laparoscopic simulated training scenarios only.
Although the aforementioned solutions are sufficient for open surgeries, and in some cases for MIS, a more sophisticated mechanism is required for MIS (either manual-laparoscopic or robotic), which involves complex interaction between the highly articulated surgical instrument tooltips and tissues in the surgical field. During MIS, by just analysing the hand gestures or markings provided by remote surgeon, it is difficult for the operating surgeon to visualise, comprehend and perform the required tooltip movements. The proposed tele-mentoring framework aims to overcome these limitations of existing solutions 6-27 by transforming hand gestures of the remote surgeon into virtual surgical instrument movements and superimposing them on the local surgeon's view of the surgical field.
The objective of this work is to develop a framework that would facilitate tele-mentoring between an operating surgeon and a remote surgeon for MISs. An architecture of the tele-mentoring framework is proposed, and the hardware/software modules required for implementation of the framework are described. The framework is assessed for simulated laparoscopic and robotic surgical scenarios. The work also analyzes plurality of parameters to assess the functioning of the tele-mentoring framework over a local area network.

| Architecture of the tele-mentoring framework
The architecture (Figure 1) of the tele-mentoring framework is implemented as a distributed system with one setup inside the operating room and another at the remote location. The subsequent two sections describe each setup in detail.

| Operating room setup
The operating room setup of the proposed tele-mentoring framework consists of an operating room workstation, visualisation screens, an optical tracking system, tracking frames to be used with optical tracking system, a scope system, an input device (mouse and keyboard) and a network router (as illustrated in Figure 1). The operating room workstation consists of six software modules (as illustrated in Figure 2) interfacing with the hardware units, processing the data and continuously communicating with each other. The functionality of each module is described as follows: (i) Core Processing Module: The core processing module acts as a central core for processing data at the operating room workstation. The module receives data from the graphical user interface (GUI) module, video module, tracking module and network module and sends data to the graphical rendering module and network module.
(ii) Video Module: The video module receives video stream of the surgical field from the scope system, processes it frame-byframe and sends the video frames to the core processing module. A video frame at time instant 't' is denoted by F SurgicalView (t).     Figure 3B). The mapping inside S Instrument (t) data is used by the graphical rendering module during rendering.

| Remote location setup
The remote location setup of the tele-mentoring framework consists of a remote location workstation, visualisation screens, a user interface, an input device (mouse and keyboard) and a network router (as shown in Figure 1). The remote location workstation consists of five software modules (as shown in Figure 5) interfacing with the hardware units, processing the data and continuously communicating with each other. The functionality of each module is described as follows: (i) Core processing Module: The core processing module acts as a central core for processing data at the remote location workstation. The module receives data from the GUI module, user interface module and network module and sends data to graphical rendering module and network module.

| Implementation details of the tele-mentoring framework
The framework was implemented in C++. The graphical rendering was performed using VTK 28 whereas the GUI was implemented using Qt. 29 The threaded implementation of the modules was performed using Boost. 30 Interfacing with user interfaces (connected to remote location workstation) was achieved using openHaptics 31

| Assessment of tele-mentoring framework on a simulated surgical setup
The implemented tele-mentoring framework (architecture depicted in Figure 1) was tested on a surgical phantom for a minimally invasive manual surgery ( Figure 6) as well as robotic surgery ( Figure 7). The hemispherical surgical phantom with five incision points simulated pneumoperitoneum during surgery and a silica gel structure inside the phantom mimic the surgical field when observed using a scope.
In manual surgical setup ( Figure 6A At the remote location workstation, SpaceMouse® devices (3DConnexion) were used as the user-interface to control virtual models of EndoWrist instrument tooltips ( Figure 6C).
The robotic surgical setup of our tele-mentoring framework was tested on Da Vinci Xi surgical robot-Intuitive Surgical Inc.
( Figure 7A). The output video stream from the vision cart was connected to the operating room workstation of the tele-mentoring framework using an adaptor (Magewell USB Capture HDMI 4K Plus). The augmented view from the operating room workstation of the tele-mentoring framework was rendered in tile-pro on the surgeon's console mode side-by-side with the view from the scope ( Figure 7B). The surgical instruments comprised of 30-degree angulated scope and EndoWrist instruments (470006 large needle drivers) as shown in Figure 7C. At the remote location workstation of the tele-mentoring framework, Touch TM devices (3D Systems) were used as user-interface to control virtual models of EndoWrist instrument ( Figure 7D). Figure 4A) is shown for manual and robotic surgery in Figure 8A,B.

The view of the surgical setup (schematically depicted in
Similarly, the augmented view (schematically depicted in Figure 4B) for manual and robotic surgery is shown in Figures 6A and 7B, respectively. The motion of the virtual tools performed by the operator at the remote location workstation was observed by the operator inside the operating room workstation on the augmented view.

| Evaluation of the tele-mentoring framework over local area network
To evaluate the performance of the implemented tele-mentoring framework, the system was tested for different time periods (varying for 8, 10 and 12 min) multiple times (three trials per time period).
The clocks on the remote and operating room workstations were 6 of 14 -SHABIR ET AL. synchronised from a common server (windows time service). The data sent and received over the network at both ends was logged and processed to evaluate the functioning of the tele-mentoring framework over the network.
The surgical state S SurgicalState (t), comprising of incision points P Incisions (t), scope pose M ScopeCamera (t) and surgical view F SurgicalView (t) was sent over the network from the operating room to the remote location workstation. In the current implementation of the telementoring framework, the position of the incision points P Incisions (t) was marked using a tracking tool. The position remained stationary during the study (as the surgical phantom was not moved). The pose of the scope's camera M ScopeCamera (t) was continuously sent over the network from the operating room to the remote location. Figure 9 presents M ScopeCamera (t) decomposed into position (translations along X, Y and Z axis) and orientation (rotations along X, Y and Z axis) measured with respect to optical tracking system.   frames after decoding. 37,38 The computed values of the video image quality metrics were the average mean square error (MSE) of 31.28, the average peak signal-to-noise ratio of 33.18 and the average structural similarity index measure of 98.24% (shown in Figure 10B).
The heat map presented in Figure 11 shows the relative values with respect to each other for the MSE of the surgical view frames received and sent.
When the virtual instruments were selected by the operator at the remote location workstation, tooltip poses M Tooltips (t) were sent over the network from the operating room to the remote location workstation. Figure 12 shows M Tooltips (t) for the movements of the left and right augmented tools. An average delay of 0.089 ± 0.017 s was observed while transferring M Tooltips (t) from the remote location to the operating room workstation. The delay was computed by taking difference of the logged timestamps for the received and send M Tooltips (t) at the operating room and remote workstations, respectively It was observed that the packets sent from the remote location workstation were received in batches at the operating room workstation (shown in Figure 12). Due to this behaviour, a buffer was required to consume the packets at a uniform rate. Whenever there is an update in the instrument state S Instument (t), it is sent asynchronously over the network between the operating room and the remote location workstations.

| DISCUSSION
The proposed tele-mentoring framework facilitates communication between an operating surgeon and a mentor surgeon via displaying motion of augmented surgical instruments during a minimally invasive manual surgery ( Figure 6) and robotic surgery (Figure 7). With dynamic virtual surgical instruments overlaid on the surgical field, the mentor surgeon is able to guide an operating surgeon with surgical tool motion required during the particular surgical step. While the previous studies 6-8 laid the foundation of using augmented tool motion for mentoring during the MIS, this work presents a framework that enables its usage over a network and in an operating room settings.
The information pertaining to the surgical field is transferred over the network from the operating room to the remote location with an average delay of 1.560 ± 0.426 s. At the remote location, the mentor surgeon performs the motion of augmented tools, which is sent to the operating room at an average delay of 0.089 ± 0.017 s (which is within the limit of 0.20 s recommended by Xu et al. 39 ). This

F I G U R E 9
Graphical representation of the delay in receiving information at remote location from the operating room. The pose of scope camera M ScopeCamera (t) is acquired at the operating room and sent to the remote location workstation. The remote location workstation receives the M ScopeCamera (t) with a delay. The poses are expressed as translation (in X, Y, and Z axis) and rotations (roll, yaw, pitch) with respect to the time and are measured in the optical tracking system coordinate system F I G U R E 1 0 (A) Delay in receiving surgical state S SurgicalState (t) at remote location workstation from operating room workstation over a time period of 8 min and (B) Video quality metric comparing sent frames before encoding and received frames after decoding. MSE, mean square error; PSNR, peak signal-to-noise ratio; SSIM, structural similarity index measure SHABIR ET AL.
-9 of 14 F I G U R E 1 1 Heat map of the mean square error (MSE) for a sample of 50 video frames sent from the operating room versus the 50 video frames received at the remote location. The heat map is generated to understand the relative value of MSE for video frames with respect to each other. The value is minimum for the same video frame number sent and received and is seen along the diagonal of the heat map F I G U R E 1 2 Graphical representation of delay in receiving information at operating room workstation from the remote location for a time period of 7 s. The pose of augmented instrument tooltip M Tooltips (t) for left M Tooltips [1] (t) and right M Tooltips [2] (t) hand is acquired at remote location workstation and send to operating room workstation. The operating room workstation receives the M Tooltips (t) with a delay delay is acceptable, when the surgical field to be operated is stable.
The recommendation provided by SAGES requires a latency of less than 0.45 s for live tele-mentoring. 40 Low latency is crucial especially during live surgery to ensure the remote surgeon is aware of the operating field and can mentor back as complications evolve intraoperatively. Also, the tissue motion induced by breathing or beating heart would require F SurgicalView (t) received at the remote location to be synchronised with M Tooltips (t) and sent back to the operating room to be visualised on a separate visualisation screen. As a result, the proposed framework is suitable only for simulated training scenarios and surgeries, where the operating field is stable.
The setup of the implemented tele-mentoring framework has certain limitations. First, the setup was only tested on a local area network instead of using it on the Internet. To test the setup on the Internet, it would require the RTMP server to be hosted on a cloud hosting service and access to network ports by the service providers. This may affect the delays in transferring the information.
An alternative method is to use low latency live streaming protocols An MIS usually has a high complication rates unless the procedure is performed by a specialised surgeon experienced in the field. 43,44 To gain experience in the usage of new surgical instruments or new surgical techniques for MIS, the surgeon has to go through a learning curve. [45][46][47] This requires the local surgeon to travel and get trained or to invite a specialist surgeon to the hospital as a mentor.
As a result, it imposes a burden in terms of time (scheduling patients only when the specialist surgeon is available) and logistics (such as travel, stay and cost-per-day). A tele-mentoring technology for MIS could address the associated problems as both the operating and specialist surgeons need not to be present in the same place. It is also worth noting that in developing economies and small countries, a regional shortage of a surgical sub-speciality may arise within a country due to uncontrollable geo-political factors. 48 An imbalance of surgeons' inflow and outflow may affect surgical services. 49,50 In such cases, tele-mentoring technology for MIS could facilitate surgical knowledge transfer across geographical boundaries. 51 For surgical tele-mentoring, there are several conceptual frameworks and learning theories. 52 The future work for further improving the tele-mentoring framework will be geared towards three main aspects. First, the tele-mentoring framework tracks the scope poses and incision points and uses the information to generate a virtual 3D environment of the surgical field. However, in certain MISs, such as NOTES 57,58   and visual cues transferred over the network in synchronisation. This could be achieved by using audio codecs such as advanced audio coding with RTMP server. Another option is to replace RTMP with webRTC, which internally uses Secure Real-time Transport Protocol. 66 The protocol adds sequence numbers/time stamps/unique stream IDs, which is used to ensure synchronisation between audio and video streams. We also plan to optimise the network components and test it across multiple networks. Finally, clinical studies will be required to assess the knowledge transferred using the telementoring framework, especially with respect to the motion of augmented surgical tools, and its applicability in different surgical sub-specialities. 51

CONFLICT OF INTEREST
The authors of this submission have no affiliations with or involvement in any organisation or entity with any financial interest (such as honoraria; educational grants; participation in speakers' bureaus; membership, employment, consultancies, stock ownership, or other equity interest and expert testimony or patent-licencing arrangements) or non-financial interest (such as personal or professional relationships, affiliations, knowledge or beliefs) in the subject matter or materials discussed in this manuscript.

AUTHORS CONTRIBUTION
The engineering team developed the prototype of the tele-mentoring framework. Jhasketan Padhan designed the architecture of the telementoring framework, integrated the modules together into the prototype and contributed to manuscript editing. Nihal Abdurahiman conceptualised, developed and tested the core processing and GUI modules of the framework. May Trinh and Zhigang Deng conceptualised, developed and tested the user interface module and contributed to manuscript editing. Elias Yaacoub, Aiman Erbad and Amr Mohammed conceptualised, developed and tested the networking modules and contributed to manuscript editing. The clinical team comprising of Shidin Balakrishnan, Mohamed Kurer, Omar Ali and Abdulla Al-Ansari provided input from the surgical point-of-view to the manuscript, perform literature search and provided revisions of the manuscript. Dehlela Shabir and Nikhil V. Navkar led the manuscript writing, data collection, testing of the framework and revisions.

DATA AVAILABILITY STATEMENT
Research data not sharable.