A comprehensive review of haptic feedback in minimally invasive robotic liver surgery: Advancements and challenges

Liver medical procedures are considered one of the most challenging because of the liver's complex geometry, heterogeneity, mechanical properties, and movement due to respiration. Haptic features integrated into needle insertion systems and other medical devices could support physicians but are uncommon. Additional training time and safety concerns make it difficult to implement in robot‐assisted surgery. The main challenges of any haptic device in a teleoperated system are the stability and transparency levels required to develop a safe and efficient system that suits the physician's needs.


| INTRODUCTION 1.| Clinicians' cognitive overload
Clinicians should make critical decisions in a short period of time during surgical interventions.They can make wrong decisions because they are overwhelmed with information from multiple sources, leading to cognitive overload.
The amount of data that physicians have to process has grown exponentially in recent decades due to the digitalisation of the health centre experience. 1 Although digitalisation has enabled greater collaboration and sharing of information between multiple institutions, surgeons have been dealing with problems in understanding and analysing large amounts of data stored in electronic medical and health records.This developed some errors and caused patient dissatisfaction in some cases. 2,3Researchers were motivated to establish objective techniques to monitor surgeons' cognitive load almost in real-time during work and compare that with error occurrences, 4 which helps to deepen our understanding of surgeon struggles.Based on this, they developed sophisticated cognitive support systems to meet their needs.The effects of information technology in the workplace and how they affect cognitive load were investigated by Rutkowski and Saunders. 5They focused on understanding how the brain processes new information to gain more knowledge and understanding of what is happening in reality.The study concluded that employees could make better decisions when information is filtered and divided into batches according to their priority.In, 6 a cardiac surgery team conducted a study that monitored cognitive load indicators during different surgery times.They recorded a preventable error caused by a distraction from one of the team members.Root cause analysis was used to thoroughly investigate any physiological indicator of cognitive overload rather than concluding that errors occurred due to a mere lack of experience or ineffective supervision.The results suggest that the anger that erupts in the surgery room and the lack of consistent guidance from the experienced mentor caused a temporary cognitive overload for the amateur surgeon.The team proposed implementing coping mechanisms to help the physician manage cognitive overload by monitoring heart rate variability, which indicates the level of cognitive load.These studies provided objective empirical data based on physiological indicators of the biological systems of medical personnel, which show the need to optimise the data delivered to medical workers.
Minimally invasive surgeries such as laparoscopy and endoscopy are based on imaging as primary data for the physician to take action.However, the significant amount of imaging data such as target organ imaging, Computed Tomography and Magnetic Resonance imaging, and patient vital signs can be overwhelming and not digestible efficiently by the physician, leading to potentially fatal decisions. 4lemanipulation can help the physician perform more efficiently with less cognitive overload by processing data from multiple sources to produce more compact information before the physician takes action.The physician controls the motion of a robotic manipulator through a source (master) device inside a cockpit that can be located outside the surgery room.The sense of touch is crucial for easily manipulating tissues and feeling their consistency. 7However, the physician loses it when operating remotely if haptic feedback is not integrated into the teleoperation system. 7In addition, needle insertion procedures are guided by imaging modalities, such as CT, which expose the patient to harmful radiation.Therefore, the objective is to find other technologies such as haptic feedback to mitigate the overuse of CT and keep physicians outside the CT room.
For that reason, we introduce haptic feedback technology and investigate how it affects the physician's cognitive overload and performance to overcome some of the clinician's challenges.This is assessed by reviewing training simulators and teleoperation systems that incorporate haptic feedback.Since, liver cancer is one of the leading causes of cancer-related deaths, [8][9][10][11] we find it crucial to investigate the requirements and challenges of needle insertions for the liver and how haptic technology can meet the physician's needs for such procedures.

| Related work and contribution
A variety of review papers have studied haptic technologies focusing on different aspects.
Giri et al., 12 and Rane and Sutar 13 investigated in their review articles the various aspects of haptic technology in multiple applications: medical, gaming, augmented, and virtual reality (AR and VR).
5][16] These reviews did not exploit the main aspects of surgical interventions such as the following.
Patel et al., El Rassi and El Rassi, Abdi et al., and Van der Meijeden et al. [17][18][19][20] exploited haptic feedback teleoperation technology and its benefits in surgical interventions addressing the challenges of implementing such systems in a medical environment.These studies did not specifically target a challenging organ to operate on such as the liver but took a more general approach to highlight the potential benefits of haptic feedback in medical procedures.Other review articles focus on more specific surgical interventions such as haptic feedback for needle insertion, 21,22 and haptic teleoperation for cardiovascular intervention. 23e reviews that are closest to ours focus on the different aspects of needle insertion procedures.Yang et al. systematically reviewed force measurement, modelling, and control in needle insertion teleoperation systems. 24Ravali and Manivannan focused on needle insertion modelling and simulation for haptic feedback.They discuss the main challenges and constraints of multiple needle deflection and deformation models. 25Although these studies introduce relevant literature incorporating haptic feedback solutions to needle insertions, our review paper takes a more holistic approach by also benefiting from haptic technology used in other applications to solve problems of operating on a specific organ, which is the liver.This review focuses on hepatic needle insertion procedures as the main application due to their challenges and the potential benefits of incorporating haptic feedback during training and teleoperation.This has not yet been thoroughly discussed in a single review article, to our knowledge.

| Outline and approach
This review study investigates the potential added value of teleoperation systems integrated with haptic feedback for needle insertions into the liver.It overviews the various attributes, challenges, and technological solutions of a typical teleoperation system.Then, we present multiple haptic systems designed for surgical interventions.Presenting systems from different applications provide alternative solutions using haptic technology to the challenges of liver procedures.Finally, we thoroughly review source and replica needle insertion technologies for the liver while exploiting multiple mixed-reality and force estimation models for the accurate haptic rendering of the liver.The main topics discussed in the paper are illustrated in the diagram shown in Figure 1.We collected articles from 2002 to 2023 that highlight the most innovative work that could be inspiring for liver procedures.

| Potential benefits of haptic feedback in surgical interventions
Several studies show clear indications that the integration of haptic features into surgical robotic systems increases physician performance. 20 this section, we investigate the effect of using haptic feedback to feel the interaction forces applied by surgical tools on tissues and for needle guidance.

| Feeling
Haptic feedback systems provide a sense of touch to the user who controls a robotic system remotely.It can be characterised into two main types: kinaesthetic and tactile (cutaneous) feedback.Haptic feedback is absent in most teleoperated minimally invasive surgeries.
Consequently, the physician loses haptic capabilities compared with manual operation, making it harder to manipulate tissues in laparoscopic or needle insertion procedures.Furthermore, haptic feedback enables the physician to be more aware of tool-tissue interaction while operating on the patient to avoid the destruction of healthy tissues and perform more efficiently. 7,26An example of a remote surgical system that provides haptic feedback is the state-of-the-art Senhance surgical system (TransEnterix Surgical Inc, Morrisville, NC, USA).It is considered the leading contender with the most popular haptic-incapable Da Vinci XI (Intuitive Surgical, Mountain View). 27e Senhance surgical system provides kinaesthetic haptic feedback while the user's eyes control the camera's orientation.The latest versions of both systems are shown in Figure 2.
Tholey et al. conducted three tissue characterisation experiments of different hardness with a laparoscopic grasper that provides feedback: haptic, visual, or both. 28They confirmed that force feedback combined with visual feedback provides the best characterisation results in laparoscopic surgeries.In, 29 haptic feedback has been argued to enhance surgeon performance and mitigate the effect of cognitive load.Novice surgeons who experienced haptic feedback in F I G U R E 1 Review paper highlighting the needs of physicians during needle insertion procedures and the added value of haptics in remote hepatic procedures.Consequently, the main topics related to haptic technology are presented for different applications.
the ProMIS simulator, operated faster and with more efficiency by 36% and 97%, respectively, compared to the MIST-VR (Mentice AB, Gothenburg, Sweden) simulator, which lacks haptic features.Furthermore, surgical tools apply less force to the subject when haptic capabilities are incorporated into the system, leading to a safer surgical intervention.In, 7 surgical residents and medical students used graspers with enhanced haptic feedback that improved their characterisation of tissue consistency and reduced force by a mean factor of 3.1 when palpation experiments were performed in various organs, including the liver.Additionally, Saracino et al. showed with their setup in Figure 3 that minor tissue damage was experienced when palpation and incision procedures were performed with the 7 degrees of freedom (DoF) Sigma.7 haptic device (Force Dimension, Nyon, Switzerland) that provides kinaesthetic feedback. 30Furthermore, Miller et al. developed a surgical robotic system named FLEXMIN with kinaesthetic haptic features to teleoperate two instruments. 31The team noticed twice the applied intracorporeal force measurement when the instrument was teleoperated without haptic feedback compared with teleoperating with haptic feedback.
On the other hand, a study showed that tactile feedback did not add value to laparoscopic tasks. 32The study used a VR simulator (VRS) called Simbionix Lap Mentor II to perform laparoscopic tasks.
Researchers concluded that more research is needed to improve its fidelity.Another study showed that force feedback has a limited effect on surgeon performance during laparoscopy. 33It showed that force transmissions through tissues are very hard to estimate robustly due to too many factors that affect force reading, such as the speed and depth of insertion affecting friction forces.

| Guidance
Shared control algorithms that use haptic feedback can increase the autonomy of the needle insertion procedure to guide the physician to follow a particular trajectory.Howard and Szewczyk showed the effectiveness of relatively low-cost tactile feedback when combined with visual feedback to inform the subject of current deviation from the desired trajectory while navigating a laparoscopic instrument. 34e results showed that the subjects, especially the experienced ones, were more precise but performed with an increased task completion time compared to laparoscopy without feedback.Another

F I G U R E 3
The experimental setup consists of the da Vinci research kit (dVRK) replica system equipped with a large needle driver endowrist tool, a high-resolution webcam (Logitech Europe, Lausanne, CH) for visual feedback, and a Sigma.7 haptic device replacing the already integrated source of the dVRK. 30A silicon phantom is also shown to be used for palpation tests.study implemented a force and torque feedback controller to align the ultrasound (US) probe in order to maintain the quality of the US image. 35The study successfully employed a force-torque sensor (ATI Nano-17, Industrial Automation, USA) to maintain steady contact as the probe moves along the different surface profiles of the subject.
The system guided a needle to reach a target inside a phantom tracked by the US with mean target errors between 0.49 and 1.12 mm.A similar study provided vibratory navigation cues combined with visual feedback to the user who guided the needle to the desired orientation with 9 times the accuracy of manual insertions. 36nally, Meli et al. implemented a passivity-based controller to guide the operator using force feedback to the desired needle insertion inclination with improved performance compared to direct hand interaction. 37

| Clinical needs for liver needle insertion
In addition to the aforementioned benefits of haptic feedback for mitigating the challenges of surgical procedures in general, there are multiple challenges that the clinician particularly faces in detecting and treating liver cancer when performing percutaneous procedures that involve needle insertion.We introduce the main physician needs as follows: 1. High characterisation of tissue mechanical properties to detect tumours.
2. Avoid excessive forces, which lead to the destruction of healthy tissues.
3. Accurate alignment of the needle to reach the targeted tumour efficiently.

Safe operation by avoiding contact with critical structures.
Consequently, the next section highlights the implications for the haptic system design requirements of needle insertion that would mitigate some of these clinician's challenges.

| Haptic feedback requirements for liver needle insertion
Magnetic Resonance and US Elastography are considered noninvasive methods to detect liver tumours by measuring the stiffness of their tissues. 38,39However, liver biopsies which involve needle insertion are the gold standard to identify the severity of liver cancer due to their higher accuracy and reliability. 40,41Needle insertion into human organs can be very critical and misalignment leads to ineffective treatment or destruction of sensitive tissues that can cause fatal complications. 26Abolhassani et al. claimed in their extensive survey on needle insertion into the soft tissue that for liver biopsies it is necessary to achieve millimetre placement accuracy. 42rthermore, they concluded that the main challenges of manual needle insertion into the soft tissue are delays in identifying changes in force measurement and the classification of forces from different sources, such as friction, cutting, and stiffness.Furthermore, physicians may find it hard to detect minute changes in forces when the needle is inserted into small tumours.Therefore, sensitive force feedback with high bandwidth could help the physician to characterise the source of the measured force and to accurately distinguish between the different layers of tissues and identify tumours. 43tegrating sensitive force feedback helps mitigate the effects of time delays and system uncertainties, which compromise the stability and transparency of the bilateral system. 44ver needle intervention is especially challenging during manual insertion due to its continuous movement during breathing as it is in close vicinity to the diaphragm and the complex structure of the liver, which increases the chances of needle deflection when the liver is deformed. 45,46Ravali and Manivannan classified the main sources of needle misplacement in soft tissues that are applicable to liver needle interventions. 25They are tissue deformation, needle orientation, and needle bending.These issues can be alleviated by appropriate haptic guidance of the needle taking into account needle bending and force estimation models.

Wang et al. conducted a study investigating differences in needle
behaviours while penetrating a porcine liver. 47It has been concluded that a maximum force of 5 N is produced when conic tip needles are used and less than 2 N when bevel tip needles are used in manual and robotic-assisted operations.For a biopsy insertion system, the teleoperation system should handle higher forces, since more forces are expected as the needle penetrates the skin and other tissues until it reaches the liver.Moreover, the changes of forces while penetrating different layers of tissue until the tumour is reached can be hard to identify by the surgeon, therefore these forces can be amplified to clearly distinguish between the different tissue textures, especially for small tumours. 37e mechanical properties of the different segments of the liver differ due to the heterogeneous nature of the liver.Therefore, a variety of forces can be produced while penetrating segments of the liver with different mechanical properties.This can give an estimate of the depth of the needle in the soft tissue 43 and its relative position to the tumour.

| TELEOPERATION SYSTEMS TECHNOLOGY
After we have discussed the potential benefits of haptic feedback for liver procedures, we provide in this section an overview of the available haptic technologies used for different purposes and their suitability for liver needle insertion requirements.

| State-of-the-art haptic devices
Over the past 30 years, companies have produced various haptic devices for various teleoperation applications.Table 1 classifies  workspace, DoF, maximum force and torque produced, and spatial resolution.This presents some of the high-end haptic devices that are used in the reviewed research regarding haptic teleoperation.Some of the main disadvantages of the available haptic devices are the price and the use of heavy metallic structures.It is also sometimes challenging to customise them for specific applications that do not require all the DoF of the device.For that reason, stateof-the-art haptic interface prototypes developed by researchers are presented to explore the design possibilities to build an efficient design personalised to the needs of liver needle insertion procedures.

| Electrical haptic devices
Pediredla et al. developed a novel 3-DOF haptic device that combines cutaneous and kinaesthetic feedback. 48The spherical platform shown in Figure 4 consists of multiple sections of different textures to illustrate the shape and consistency of a virtual environment.The design of the haptic device is based on a semi-compliant four-bar mechanism with flexure-in-tension presented in. 49The flexure-intension mechanism allows the user to exert a tensile force on the haptic interface.Two servomotors control the pitch and roll to adapt to the user's movement.The third degree of freedom provides stiffness feedback perpendicular to the spherical segment.The device showed accurate performance relative to commercial haptic devices, rendering shape and shear using position control, and stiffness using impedance control.This system can be used in liver palpation to enable the physician to feel the texture of its different tissue layers while experiencing the axial force acting on the tissues.
Other haptic devices can be much more sophisticated and their design can be very similar to that of commercial haptic devices.For instance, Jin et al. designed a novel parallel mechanism with a nonrigid platform for a 7 DOF (3 translation, 3 rotation,1 grasping) general-purpose haptic device. 50The design accomplished the objectives (large workspace, partial decoupling, and fixed actuators) by using parameter optimisation techniques suitable for minimally invasive surgeries to use fewer mechanical structures to build the system.Moreover, the adopted mechanism reduces the weight of the moving frames compared to the stacking mechanism used in some commercial haptic devices.Therefore, the user experiences a more precise feeling for the force with an energy-efficient system.Finally, the design offers a wide range of force magnitudes and low inertia for a more intuitive user experience.
A skin-stretch haptic device can be used to provide proprioceptive information to a prosthesis user, such as the system developed by Collela et al. 51 The device consists of active rollers actuated by gear motors that rotate tangentially to the skin of the user in either the same direction (unidirectional skin-stretch) or opposite directions (pinch).It is placed on the forearm to provide sensible cutaneous feedback without occupying the physician's hands during the procedure.This system showed great potential to be used in teleoperation after conducting experiments on both able and prosthetic users to identify object sizes.For needle insertion, the skin displacement could be related to the extent to which the needle is away from the target.The pinch feature could be used to demonstrate that the needle is diverging from the correct path or approaching critical regions.
Electrical haptic devices are clean, fast, and precise relative to pneumatically actuated ones.However, the simplicity, low cost, and higher power of pneumatically actuated haptic devices make it more convenient to develop novel designs.

| Pneumatically actuated haptic devices
Pneumatically actuated wearable haptic devices have the great advantage of being light and having low encumbrance, allowing the user to move freely while providing the required feedback. 52,53chierotti et al. claim that haptic devices are not commonly used as portable devices because wearability was not seriously considered.
Therefore, due to the mechanical rounding of most commercial haptic devices, they have been used more in laboratories and research centres. 54However, the advantages of pneumatic actuation would make portability possible for designing wearable haptic devices.
Young et al. designed, manufactured, and controlled a "Bellowband", a light wristband with eight independent pneumatically actuated bellows with a maximum pressure of 1 bar. 55The "Bellowband" is capable of providing kinaesthetic and vibrotactile feedback to the user by dynamically pressurising the bellows, producing a maximum of over 10 N normal force on the wrist and 10 mm maximum displacement.Yoshida et al. developed a similar haptic device using three linear pneumatic actuators combined with a rotating housing F I G U R E 4 3 DOF haptic device consisting of a multiple segment spherical platform. 48owered by a DC motor to provide vibration, shear, normal and torsion deformation of the skin. 56The device aims to relay directional signals to the user by moving along the skin with minimum skin contact when the device is oriented.This feature provided a more intuitive means of navigation compared to the previous design by Kanjanapas et al. 57 Other studies have developed active wearable devices to teleoperate a replica rather than merely receiving information.
Li et al. developed a pneumatically actuated haptic glove designed to control a robotic manipulator with a modified version of Open Bionics V1.1 Ada Hand. 58They integrated optical curvature sensors and IMUs on the glove to monitor the movement of the user and teleoperate the robotic manipulator while feeding back the forces acting on the end-effector through the pneumatic muscles pressurising the user's hand.Multiple users effectively completed grasping tasks using the system, which assessed the quality of force feedback to distinguish between the shapes and sizes of different objects.They concluded that more types of haptic feedback should be incorporated into the glove to convey object size and stiffness to enhance the user's experience.

Pachierotti et al. developed a 3 DoF cutaneous device that can be integrated into commercial grounded haptic devices such as
Omega 3 to complement kinaesthetic with cutaneous feedback for a more transparent system without compromising system stability. 59e intrinsically stable wearable device consists of three servomotors that control a platform providing a stimulus to fingertips without hindering users' dexterity due to its lightweight design.They conducted multiple experiments to assess the effect of employing this cutaneous device on the master device to complete a virtual needle insertion task of 1 DoF.Then, the study compared the performance of the user to complete the insertion task using full haptic feedback, cutaneous feedback, and visual feedback.The best performance occurred when the user placed the cutaneous device on the same hand that controls the end-effector of the kinaesthetic haptic interface experiencing full haptic feedback.

| Hydraulic haptic devices
Thai et al. developed a skin-stretch device (SSD) controlled by hydraulically actuated soft microtubule muscles. 60The device in Figure 5 shows the finger-worn SSD with adjustable tactor position in 3 DoF.The device can generate forces up to 1.8 N and move with a maximum displacement of 4.5 mm.The device also showed greater speed and durability compared to similar skin-stretch devices developed earlier using electric motors. 61,62However, the analytical model should be modified to account for the hysteresis of the soft material used.In addition, non-linear force and position feedback algorithms should be implemented to close the controller loop for more efficient performance.Wearable devices can limit the free movement of the user's hands to perform other tasks.In addition, they are usually only feedback haptic devices, which require another device to remotely control a robot to execute a certain task.However, they can provide navigation cues for the user to reach a target and avoid obstacles, which is very beneficial for liver needle insertions.
Haptic feedback technology used in a variety of applications satisfies the main requirements for the physicians to operate safely and efficiently during liver needle insertion procedures to feel the tissues and guide the needle to the target.However, additional requirements arise when teleoperating in a medical environment where sterility and compatibility are also necessary.

| HAPTIC SYSTEMS IN THE MEDICAL FIELD
This chapter presents the state-of-the-art technology of haptic feedback systems used particularly in medical procedures.We first present haptic systems integrated into laparoscopy and then introduce a variety of teleoperation systems that are used in surgical interventions and palpation.Integrating haptic feedback features into the medical field can facilitate surgical tasks such as knot ties 63 and blunt dissection. 64Thus, operating with much more efficiency and less error. 20,65Furthermore, when integrated into surgical simulators, it can open the door for more ethical training sessions for medical students and researchers to study anatomy, as it will reduce the use of animals for training. 66

| Haptic systems in laparoscopy
In, 28 an empirical study was conducted to test a teleoperated laparoscopic grasper with force and visual feedback using a haptic feedback device called PHANToM (SensAble Technologies, Woburn, F I G U R E 5 3 DOF skin-stretch device (SSD) with adjustable soft tactor. 60assachusetts, United States) and a CCD camera.The objective of the study was to determine whether visual feedback, force feedback, or both provide accurate results for the characterisation of tissue hardness.After extensive tests on several subjects, the researchers concluded that force feedback produces higher accuracy than visual feedback in identifying the hardness of the tissue and that combining both feedback systems produces the best results.Similar results obtained by 67 concluded that novice surgeons performed more efficiently on a cholecystectomy laparoscopic simulator with haptic feedback compared to a simulator without haptic feedback.

| Surgical and clinical palpation teleoperation systems
Haptic feedback can be used to manipulate surgical tools to feel interaction forces during interventions and can also be used to palpate different organs to assess their condition by checking the texture and stiffness.

| Surgical systems
In, 68 a robotic system with kinaesthetic feedback was designed to allow the teleoperation of CT-guided interventions to distance the surgeon from harmful X-ray radiations.The robot-assisted system was developed to help the physicians perform needle insertion procedures such as biopsies and ablations with greater precision and prevent critical tissue injury.The system design was developed to track the forces acting on the needle up to 20 N with a high resolution and an accuracy of less than 2 mm.MRI-guided surgeries have become prevalent among clinicians as they can distinguish between soft tissues, do not produce harmful radiation and create tomographic images without repositioning the patient. 69,70However, due to the confined space of the MRI bore, the clinician is restricted during the operation.This problem can be solved by remotely operating on the patient, but the haptic information is consequently lost, 69 which raises the need to review the different haptic teleoperation technologies that use devices compatible with MRI. 70Devices consisting of ferro-magnetic materials are not allowed in the MRI room due to the strong magnetic fields and radio frequency pulses. 71Tse et al. explicitly developed a haptic needle unit for MRI-guided biopsy. 72Antiferromagnetic ultrasonic actuators and piezoelectric ceramic actuators (PiezoLEGS) were used to accommodate the MRI compatibility requirements.
They used a neural network based on the back-propagation technique to estimate the nonlinearity of the motor model.During the experiments, they asked a physician to distinguish between two different models, one containing a tumour and another that does not.
The physician successfully differentiated between the two models in all trials with a realistic sensation, but it was recommended that the design become more ergonomic.
In, 73 a teleoperation source-replica system was also used for percutaneous interventional MRI procedures that focus on prostate biopsy as the target procedure.A 2 DOF pneumatically actuated haptic robot controls the translational and rotational positions of the 6 DOF insertion device and renders the exerted axial force on the needle while being inserted into the tissues.The system also decouples the translation and rotary motions using two angular ball bearings placed against each other, providing better axial force support.A piezoelectric motor is used for the replica to position the needle at the desired location and orientation.A fibre optic force sensor (FPI) is mounted on the motor to monitor the forces exerted on the needle.The system was tested for MRI compati- to monitor a minimum of 0.1 N force changes exerted on a phantom membrane inside the MRI. 74The 3 DOF biopsy device targets transperineal prostate needle insertions that exert a maximum force of 18N, which is 2 N less than the device's capabilities.The results show that pressure measurement can detect the event of a sheath puncture that is hardly identified by a clinician.

Critical emergency needle decompression must be performed
to extract trapped air between the lungs and the thoracic wall.
Reyes et al. 75 developed a haptic telementoring system for the mentor to guide the trainee to perform the needle compression procedure efficiently by providing axial force feedback to the finger from the two sides, as shown in Figure 6.A Geomagic Touch haptic device provides force feedback to the mentor, similar to the force applied by the trainee.The trainee achieves nearly perfect performance by combining haptic and real-time graphical displays showing the needle during insertion with a vector representing the force acting on the needle through the VR environment CHAI3D.
In endovascular catheterisation surgery, Li et al. 76 and Shi et al. 77 developed 2 DoF force feedback teleoperation systems to efficiently control guidewire and catheter insertion.Li et al. 76 used hydrogel and solid magnetorheological fluid for haptic feedback on two handles, one for catheter insertion and the other for the guidewire.The study claims that catheterisation consumed less time and effort with translation and rotation tracking errors of less than 1 mm and 1°, respectively.Shi et al. 77 used a spring-based haptic interface to provide accurate kinaesthetic feedback suitable for the elasticity of blood vessels.The integration of the collision avoidance feature qualifies such a system to be used in robot-assisted teleinterventional surgery.7 that transfers soft tissue properties of a patient to the physician through a haptic interface. 78The haptic sensor consists of multiple indenters mounted on a plane in a parallel arrangement.
Each includes a spring and a linear resistive sensor to measure the displacement of the indentation and the force of the tissue reaction.These measurements are used to calculate the stiffness of the soft tissue, which is recreated by the silicon-rubber haptic interface driven by servo motors to provide kinaesthetic and tactile feedback to the physician.In addition, a webcam provided visual feedback for a more immersive physician experience.Li et al. developed another soft tissue palpation device based on granular jamming and pneumatic air actuation to vary actuator stiffness.The study showed that a multifinger haptic method that combines granular jamming and pneumatic air actuation reduces hysteresis by up to 65% and increases the range of stiffness variation. 79These results show that the wide range of stiffness variation of these palpation devices can be used to identify cirrhotic and cystic tissues of the liver from healthy ones due to their different textures.
Table 2 characterise the different surgical teleoperation systems based on feedback type, actuation for source and replica, force measurement method, clinical procedure and image modality.

F I G U R E 6
The full haptic telementoring system consists of two needle decompression units: one for the trainee and the other for the mentor.The trainee performs the insertion by applying an axial force and sensing the applied force of the mentor from the other side to have more control during the procedure.The same mechanism is applied on the mentor side to feel the force applied by the trainee and regulate it by applying an opposite force. 75I G U R E 7 Teletaction system diagram.

| Liver interventions and palpation haptic systems
Frishman et al. developed a highly precise teleoperation system driven by hydrostatic actuators for MRI-guided liver biopsy. 88Other researchers developed a pneumatic needle guidance manipulator for MRI-guided liver ablation. 89However, the robot is only capable of alignment, and the insertion had to be done manually, which decreases accuracy and safety inside a confined space.Frishman et al.
solved the problem of confined space by using incremental pneumatic insertion rather than single stroke insertion, which was deemed impractical according to Franko et al. 89 Furthermore, the inherent passivity and transparency of the utilised hydrostatic system make it extremely safe in liver biopsies under MRI guidance.Other systems used a hydrostatic transmission to accommodate MRI requirements to avoid using ferro-magnetic materials inside the bore.Burkhard et al. relied on air and water in a paired rolling diaphragm actuator connected to a capstan drive to translate the needle in 1 DOF for biopsies. 90Although the hydrostatic system was able to transmit forces transparently with a 77% success rate for membrane punctures, the system needs to be improved to produce higher stiffness and allow multi-DOF actuation.In addition, to allow for faster needle movement, the inertia of the device should be decreased.
Mastmeyer et al. 91 and Fortmeier et al. 92  The simulator also shows the range of forces applied by the user to not damage critical tissues.In addition, there is a possibility to show the preoperative CT images side by side with the real-time 3D model of the liver.
Table 3 characterises a variety of haptic systems specific for liver interventions and palpation based on the type of feedback, the actuation of the source and replica, the force measurement method, the clinical procedure, and the modality of the image.
The next section discusses a variety of force measurement approaches for a teleoperation system that can be implemented for liver needle insertion.In addition, we introduce multiple modelling techniques for virtual livers developed for haptic rendering.

| FORCE MEASUREMENT AND MODELLING TECHNIQUES FOR LIVER HAPTIC RENDERING
To evaluate whether a remote needle insertion device using haptic feedback meets the requirements of needle insertion procedures for the liver, it must be tested on accurate liver models.This avoids operating on human or animal subjects, raising ethical concerns.In addition, force estimation techniques must convey the correct information to the physician to achieve transparency between the replica insertion device that operates on the patient and the source controlled by the physician.This section briefly presents the latest force estimation techniques that can be used for the required procedure.It also introduces various virtual liver models, such as finite element models (FEM), 3D models, mass-spring models (MSM), and machine learning models (ML) to establish fairly realistic haptic rendering experiments.

| Force measurement
The force sensors used for needle insertions should provide the correct information for the physician to estimate the location of the tip of the needle 43 and its deflection. 94These force sensors can be categorised into two groups: distal and proximal.Distal sensors are integrated nearer to the tip of the needle, whereas proximal sensors are integrated at the base of the needle.This section outlines the differences between the two types of sensors in the context of teleoperated needle insertion.

| Proximal force sensors
A design developed by Washio and Chinzei measured the forces acting during needle puncture using coaxial force sensors. 95The developed sensor was able to detect surface punctures up to one second earlier than video detection.This happens due to the phase lag of the surface motion detected by the camera compared to the source of the motion, which is the interaction forces of the needle and tissue.The sensor distinguished between friction forces acting on the shaft of the needle and forces acting on the tip that are significantly affected during tissue puncture.This sensor could be integrated into a teleoperation system for remote haptic-rendered needle insertion, since it is claimed that the sensor is more sensitive than experienced physicians.De Lorenzo et al. developed a coaxial needle insertion device consisting of a hollow needle (outer) with another needle (inner) embedded in it. 96The inner needle tip protrudes from the outer needle to measure the interaction forces of the tip, while the friction forces are exerted only on the shaft of the outer needle.This arrangement separates the tip and friction forces using customised proximal force sensors.In addition, a linear actuator compensates for friction forces.In this way, the user only experiences multiples of tip force signals based on a programed amplification constant.Users successfully feel puncture instances with higher clarity compared to force feedback that includes friction forces.

| Distal force sensors
Chadda et al. developed a needle tip force sensor that can be used in biopsies and brachytherapy for haptic feedback during needle insertion. 97The sensor can measure up to 10 N of force using siliconbased semiconductor strain gauges with an average sensitivity of Frishman et al. 88   This study shows that the force acting on the robot end-effector holding a needle can be estimated without using a distal sensor.
Mounting a sensor to the end-effector complicates the design considerations related to sterility and optimal integration to ensure given tissue. 102The study used ex-vivo bovine liver since it has properties similar to those of the human liver.The study also demonstrated differences in force profiles when using different types and diameters of needle tips, concluding that larger needle diameters induce more friction and cutting forces.
Force estimation models can be used to develop virtual models of the liver and to simulate insertion procedures, such as percutaneous biopsy or ablation, for the diagnosis and treatment of tumours.
The next section presents different methodologies on how these virtual models can be instigated.

| Virtual liver models
In order to simulate a liver needle insertion procedure, virtual models of the liver can be developed that illustrate its changing deformations and mechanical properties.These models could be used to train physicians to use the teleoperation system and integrate that system with augmented reality to achieve a more immersive experience for physicians during remote operations.Multiple methods are presented to implement virtual liver models based on FEM, mass spring modelling (MSM), ML, and other 3D modelling algorithms such as Haptics3D (H3D).

| Finite element models
Moghimi Zand et al. simulated the behaviour of the liver tumour during respiration using the end-inhale and end-exhale as boundary conditions for the finite element model. 103The location and deformation of the healthy liver and tumour were predicted based on that model and their mechanical properties were produced using a quasilinear hyperviscoelastic constitutive model.The study relied on CT images at different breathing phases and a tumour motion trajectory was predicted when embedded in different segments of the liver.

| 3D modelling algorithms
Hamza-Lup produced a haptic rendering simulator for different types of liver, healthy, cirrhotic, and cystic, that could be used to train novice physicians on palpation to check the texture of the liver. 104e simulator uses RGB colour maps to improve H3D to produce visual and haptic cues to simulate liver stiffness and its deformation.
Furthermore, the forces that act on the liver were monitored and shown on the simulator screen for a more controlled palpation procedure similar to the expert touch, as illustrated in Figure 9.

| Mass-spring models
Sulaiman et al. preferred to use MSM over FEM, since FEM requires extensive computation time to produce a sophisticated model that makes it difficult to implement in some real-time applications. 105The main challenge with MSM is to accurately define the parameters (mass, spring stiffness, and damping coefficient).To obtain these parameters for use in real-time surgeries, Barycentric mass lumping, Lloyd's approach, Rayleigh formula, and the Fourth-order Runge-Kutta integration method were used to determine node mass, spring stiffness, and damping coefficient.This study laid a foundation for a more realistic simulation of liver needle insertion with haptic feedback to replicate a biopsy or ablation that included tissue removal.

| Machine learning
Deo and De combined highly precise but computationally expensive FEM with nonlinear physical models in a ML neural networks algorithm to virtually simulate soft tissues. 106The algorithm is able to give haptic and visual cues to render the stiffness of tissues and their deformation at extremely high speed and accuracy since the algorithm linearises the nonlinear physical model to make it simpler while maintaining its accuracy.of 21 deformation of ex-vivo human livers modelled using finite elements. 107The different ML models produced a mean error of less than 1 mm, and particularly the extremely randomised trees showed the best results with a mean error of 0.07 mm.These models are more suitable for applications that require real-time communication such as remote liver biopsies with haptic feedback, since it also accounts for the movement of the organ during breathing.Pellicer-Valero et al. implemented a ML algorithm on a huge number of liver models with multiple geometries developed by the finite element method. 108The algorithm produced outstanding results with the ability to perform real-time force rendering on any liver with less than 1 mm Euclidean error.

| CONCLUSION
The aim of this review paper is to investigate the benefits of haptic technology in teleoperated needle insertion procedures targeting the liver compared with conventional manual insertion.We introduced the challenges facing physicians in targeting liver tumours due to the continuous movement of the liver, the inaccurate steering of the needle, and the change of tumour location after the needle is inserted due to deformation.We then reviewed studies that compared the efficiency of physicians performing medical procedures with and without haptic feedback.Most studies concluded that integrating haptic features improves physician performance by some means: less force applied by the tool, easier tissue manipulation and characterisation, and more efficiency to complete the task.Some of these studies concluded that the inclusion of haptic features in surgical equipment reduced cognitive overload during the performance of specific tasks.We inferred that the type of haptic feedback, the design of the teleoperation system, and the control methodology are the main factors that contribute to the success of the bilateral teleoperation system in improving the needle insertion procedures of the liver.
Therefore, we first presented the state-of-the-art haptic technology used in various applications and its potential appositeness for liver needle interventions.This comprises a variety of system designs with different types of actuators, sensors, and control architectures.
Despite the fact that the target application is liver procedures, this study covers the main components of teleoperation systems that could be considered for various medical applications.

| Actuation
There is a variety of haptic-based teleoperation technology for actuation: rotary, pneumatic, and hydraulic.In order to perform surgical procedures, some aspects must be considered, such as sterility, compatibility, and safety.Hydraulic and pneumatic actuation are commonly used in MRI-guided robotic interventions since they do not contain ferromagnetic materials close to the bore of the MRI room.One can estimate the forces on the needle based on the pressure of fluids flowing in these systems instead of integrating force sensors on the surgical robot, which need regular sterilisation.This can help the physician identify small tumours that are hardly felt by the physician when performing a manual insertion.In addition, the physician would be more aware when the needle penetrates different layers of tissue.Friction forces can be used to estimate the depth of the needle in the tissue.

| Implications on image-guided liver needle insertions
Firstly, for CT-guided procedures, haptic-guided teleoperation can benefit the patient and the physician.The physician can operate outside the CT room and consequently avoid harmful radiation.
Furthermore, the physician will experience enhanced force and tactile feedback in real-time to compensate for the lack of real-time CT imaging.Finally, sensitive data of interaction forces can be complemented with CT imaging to localise the needle relative to the tumour, which can potentially limit the CT imaging needed to be mostly used preoperatively.Therefore, radiation to the patient can be minimised.Secondly, in MRI-guided procedures, a haptic feedback teleoperation system can give the physician more space to operate away from the confined space of the closed bore without compromising the sense of touch due to the interaction of the needle with the liver.Finally, a teleoperation system can incorporate obstacle avoidance algorithms to ensure that the needle does not get into critical tissues.This can be realised using virtual walls to push the haptic interface end-effector away from a certain region.In addition, the same concept can also be used to guide the user to the desired target location using force feedback.In this way, the physician can always be in control and at the same time assisted by haptic technology to reach the target safely in the least amount of time.
We argue that integrating haptic features with augmented reality into a teleoperation system would improve physicians' efficiency and the overall safety of needle insertion procedures in the liver.
Consequently, the patient would suffer fewer complications after surgery and the clinicians' cognitive overload would be minimised.
a variety of the most used commercial haptic devices according to their 191 D mm, (rotation: 297°yaw, 260°pitch, 335°roll) Translation: 0.03 mm (HF: 0.007 mm), (rotation: 0.0023°yaw, 022 N), 3 rotations (515 mNm yaw, 170 mNm for pitch and roll) Translation: 838 W � 584 H � 406 D mm, (rotation: 297°yaw, 260°pitch, 335°roll) Translation: 0.02 mm, (rotation: 0.0023°yaw, 0.0080°pitch and roll) Haption SA (Soulg'e-sur-Ouette, France) (https://www.haption.com) bility and the results showed that the user could position the needle with an root mean square (RMS) error < 4 mm.It experienced RMS errors of about 2.227 and 2.58 N force feedback for sinusoidal and chirp signals, respectively.The device shows the potential to reach small liver tumours down to 4 mm in diameter, but advanced force control algorithms should be implemented to render more accurate force measurements of the different layers of the liver.Mendoza and Whitney used transmission hydraulic pressure sensors and modified hydrostatic rotary actuators (MRI compatible) et al. developed a palpation haptic device shown in

Figure
Figure 7 that transfers soft tissue properties of a patient to the developed a visuohaptic VR environment (AcusVR-4D) with US and X-ray imaging capabilities to perform a percutaneous transhepatic cholangio-drainage training procedure.Medical experts were able to identify the difference between soft and hard tissues using the 6 DOF Phantom Premium 1.5 haptic device.The haptic rendering algorithm relied solely on segmented CT images to produce linear (based on Hooke's law) and nonlinear (second-degree polynomial) spring models for needle insertion force estimation.The VR setup allowed the user to see real-time deformations of the segmented part of the patient during palpation and needle insertion.Although the resolution of the CT images and the haptic or imaging devices used were deemed inadequate to recognise a wide range of tissue hardness, the experiments that included 10 patients showed low errors when comparing the output forces of the system with the gold standard manually segmented data as ground-truth force measurements.Hamza-Lup et al.45,93 developed a visuo-haptic 3D simulator to detect liver enlargement, cirrhosis, cysts, and tumours by palpation.The force and tactile cues produced depend on a spring-damper model producing kinaesthetic feedback to the user through a Phantom Omni.

6 .
25 mV/N at a current of 1 mA.The sensor guaranteed almost linear performance with less than 5% hysteresis error.This has great potential to differentiate between the different layers of the liver while inserting the needle, allowing accurate localisation into the liver tissue.Kumar et al. used a fibre Bragg grating sensor (FBG) embedded in an 18G bevel-tipped needle used for biopsies and brachytherapy.43This sensor has major advantages in endoscopic procedures, which require accurate estimates of the position of the needle tip at different stages of the insertion.Sudrais et al.98 simulated the liver needle insertion biopsy procedure to evaluate the forces experienced by the needle through three different layers of a liver phantom.The study used a Fabry-Perrot load cell integrated into a 16G needle that is inserted at a constant speed into the gelatin phantom.Figure8shows the comparison of force measurement between distal and proximal sensors used in this study.It shows that the performances of distal sensors are repeatable at different needle insertion speeds, and that puncture events are more easily detectable compared with proximal sensors.From studies evaluating distal and proximal sensors, one can conclude that the former has an advantage over the latter in measuring the forces corresponding to needle puncture events in different tissues of the liver.The reason for the superiority of distal sensors is that proximal sensors measure the friction forces on the needle shaft, which are superimposed on the cutting force measurements of the tip.This leads to inconsistent measurements of the proximal sensor when the needle penetrates the tissue at different speeds and needle insertion depths, impacting friction forces on the needle shaft.However, using proximal coaxial force sensors95 or a coaxial needle insertion device96 separates tip forces from friction forces.Based on that, the user experiences amplified tip force signals to detect puncture events and friction forces are utilised to estimate the depth of the needle into the tissue.5.1.3| Force estimationAnother way to measure forces on replicas of teleoperation systems is to estimate the force by studying the dynamic model of the robotic manipulator or to train a model based on a large amount of force measurement data.Vo et al. developed a force estimation scheme for uncertain environments in a bilateral pneumatic artificial muscle (PAM) system.99Instead of using force sensors to measure the interaction between the environment and the end-effector, an adaptive force observer strategy based on the nonlinear dynamic model of the PAM was implemented.The observer showed high noise suppression T A B L E 3 Liver procedures haptic systems characterisation.
capabilities and the least RMS errors (0.076) compared to other observers such as the nonlinear disturbance observer and the reaction torque observer, with RMS errors of 0.211 and 0.192 respectively.Kruzic et al. estimated the forces exerted on the end-effector using three different deep neural networks: multilayer perceptron, convolutional neural network, and long-short-term memory.100The method adapted in the experiments was based on a Kistler 9257A force sensor (Kistler Instrumente AG) mounted at the base of a Franka Emika Panda robot.Long-short-term memory showed the least significant root mean squared error (RMSE) compared to other neural networks.During the simulation, the force estimation error was 0.1533 N and the joint torque estimation error was 0.5115 Nm.
reliable force measurements.Boabang et al. also developed a training model (Hidden Markov Model) to predict and transmit force feedback signals to the physician in less than 1 ms during 5G remote needle insertion. 101For the reproduction of the predicted force/ torque profiles, a Gaussian mixture regression is utilised to maintain force feedback messages to the physician with approximately 0.007 N RMSE between the sent (lost) and received (predicted) data.This technology enables physicians to operate remotely on the patient with little delay between the source and the replica, ensuring sufficient transparency for a liver needle insertion procedure.Okamura et al. classified the needle insertion forces in the liver into three different models: capsule stiffness, a nonlinear spring model, (2) friction, a modified Karnopp model, and (3) cutting, a constant for a Multiple methods were used to produce virtual liver models incorporating the mechanical properties of the liver based on force measurements of tool-tissue interaction.They are imported into remote surgical intervention simulators to be used for training.Introducing augmented reality or real-time imaging complements the user's haptic experience.Therefore, many studies have shown that combining haptic with real-time visual feedback from the field of operation maximises the efficiency of physicians and their ability to take action.We also reviewed studies investigating the interaction forces between soft tissues and surgical needles.It was clear that identifying interaction forces gives more information to the user than F I G U R E 8 Force measurements for distal and proximal sensors through the liver phantom at different speeds of needle insertion.98 the force they experience during manual insertion.It was also observed that the needle experiences a variety of forces, which can be classified into; friction on the needle shaft, cutting, and stiffness on the needle tip.These forces can be individually determined based on coaxial sensing or force estimation models such as nonlinear spring models for stiffness and the modified Karnopp model for friction.We concluded that it benefits the physician to perceive an amplified signal of subtle changes in needle-tip force measurement. 78
T A B L E 2

Type of haptic feedback Actuation Force measurement Type of procedure Image modality
-SELIM ET AL.