A Multidirectional External Perception Soft Actuator Based on Flexible Optical Waveguide for Underwater Teleoperation

As a friendly tool, the soft gripper can be used directly for grasping vulnerable objects in underwater environments. These tasks are generally performed by teleoperation based on the visual feedback, rather than the soft actuators’ sensing information. However, vision sensors’ function may be restricted in some complex underwater environments with poor visibility and narrow spaces. This will greatly reduce the efficiency of the underwater operations. Therefore, soft actuators strongly require an organism‐like perception system to sense environmental stimuli and can be applied to complex underwater environments. To address the problem, a multidirectional external perception soft actuator based on two flexible optical waveguide sensors is developed and machine learning methods are utilized to build its perceptual model herein. The experimental results indicate that the soft actuator can recognize 12 contact positions based on the sensing model, and the identifying accuracy is up to 99.82%. Additionally, according to the contact location feedback, teleoperation can be more efficiently completed in unknown underwater environments.


Introduction
[11][12] This has greatly facilitated the exploration and exploitation of marine resources.In general, existing underwater operations are conducted on land [7,8] or in underwater submersibles [9,10] through visual feedback information to remotely control soft grippers.The vision cameras can directly feedback the underwater environment and the status of the soft actuators in a relatively clear environment.However, their function can be limited in complicated environments, such as dark and murky underwater environments with low visibility, cramped working space, and others.This will increase the difficulty of underwater operations and some cumbersome algorithms are adopted to repair these blurry images. [13]Hence, to promote operational efficiency in the unknown underwater environment, it is necessary to enrich the ability of soft actuators to sense external stimuli and build their perceptual processing system like the human hand.
External stimuli may cause large deformations on all sides of the soft actuator in underwater complex environments.This requires soft actuators that can detect obstacles and identify their contact location to remotely control the direction of the soft actuators' movement.Now, there are many studies on soft actuators with various sensory units to perceive multiple external stimuli.Z.Sun et al. [14] integrated two multimodal hydrogel sensors into a soft actuator to realize the multimodal sensing of the soft actuator, including thermal and mechanical perception.A machine learning method was also used to distinguish the five stimuli: bending, touch, contact with temperature changes, twisting, and stretching.K. Galloway et al. [15] demonstrated a soft actuator with a commercial fiber optic shape sensor that can detect object shape, distortion, surface roughness, and material stiffness.Z. Xie et al. [16] As a friendly tool, the soft gripper can be used directly for grasping vulnerable objects in underwater environments.These tasks are generally performed by teleoperation based on the visual feedback, rather than the soft actuators' sensing information.However, vision sensors' function may be restricted in some complex underwater environments with poor visibility and narrow spaces.This will greatly reduce the efficiency of the underwater operations.Therefore, soft actuators strongly require an organism-like perception system to sense environmental stimuli and can be applied to complex underwater environments.To address the problem, a multidirectional external perception soft actuator based on two flexible optical waveguide sensors is developed and machine learning methods are utilized to build its perceptual model herein.The experimental results indicate that the soft actuator can recognize 12 contact positions based on the sensing model, and the identifying accuracy is up to 99.82%.Additionally, according to the contact location feedback, teleoperation can be more efficiently completed in unknown underwater environments.designed a soft actuator with three crosswise elongationexpansion sensors that can distinguish internal and exterior bending.A simple grasping in a restricted environment was also done.Besides, some sensing origami soft actuators based on different perceptual mechanisms have been applied to underwater environments. [17,18]These studies have demonstrated that soft actuators can respond to external stimuli and achieve more functionality based on intelligent methods.Nevertheless, they lack a comprehensive perception system that can completely integrate and utilize sensors' nonlinear data in actual environments.
The human hand, as the most dexterous actuator, can easily complete a specific grasping task in an unknown environment due to its rich sensing units. [19,20]However, it is challenging for a soft actuator to assemble many sensors due to some inevitable issues that can be produced by increasing the number of sensors.On the one hand, the whole hardware system will become larger and more complex.This will increase the difficulty of integration, application, and analysis.On the other hand, the compliance and bending range of soft actuators will decrease.Therefore, it is necessary to balance the perception of the soft actuator with the performance of the entire system.
Moreover, selecting a suitable sensor to realize the sensing soft actuator is critical owing to the complexity of underwater environments.Conductive liquid sensors [16,[21][22][23][24] and optical sensors [15,[25][26][27][28][29] are usually used to achieve the sensing capability of soft actuators, compared with piezoresistive sensors, [30,31] capacitive sensors, [32] and magnetic sensors.Although conductive liquid sensors are easier to be fabricated, they are prone to leakage and oxidation.Optical sensors have some notable advantages of fast responsiveness, low hysteresis, and immunity to electromagnetic interference.They are divided into two types: optical fiber sensors made of commercially available fiber optic materials [15,26,27] and flexible optical waveguide sensors made of soft silicone and polyurethane materials. [25,28,29]Optical fiber sensors are difficult to integrate with soft actuators and only can bend due to their high hardness.On the contrary, flexible optical waveguide sensors are sensitive to various deformations (stretching, pressing, bending) and have little compression under hydrostatic pressure. [25]In summary, the flexible optical waveguide sensors are more suitable for realizing the multisensory of soft actuators by comparison.
The main contributions of this article are as follows.Inspired by the fact that the human hand can grasp objects based on its ability to perceive external stimuli in unknown environments, a multidirectional sensing soft actuator has been developed for operations in complex underwater environments without visual sensors.Thanks to its sensing ability, a contact location recognition model based on the intelligent method is built for the soft actuator to identify 12 contact positions.Finally, the teleoperation of the soft actuator is further performed in a complex underwater environment without visual assistance (Figure 1).From the relevant literature on the underwater operation by soft actuators, it can be seen that there is no underwater remote operation relying on the soft actuators' perception ability at present.This article's structure can be summarized as follows.Section 2 describes the design and fabrication of the flexible optical waveguide sensor and the multidirectional sensing soft actuator.The soft actuator's proprioceptive and multidirectional sensing ability, as well as its application in a complicated underwater environment, are presented in Section 3. Section 4 is the discussion.Section 5 summarizes the conclusions.

Design and Fabrication of the Flexible Optical Waveguide Sensor
As seen in Figure 2a, the strip optical waveguide sensor consists of a light source, a photosensitive device, and an optical waveguide.The optical waveguide consists of a core transmitting light and a shell protecting the core.They are made of transparent soft polyurethane materials (Clear Flex 30 A/B with a refractive index of 1.47, Smooth on Inc.) and untransparent soft silicone materials (Dragon Skin 20 A/B with a refractive index of 1.41, Smooth on Inc.), respectively.Moreover, the infrared LED with a wavelength of 940 nm and the infrared receiving diode with a sensitive wavelength of 940 nm are selected as the light source and the photodetector, respectively.
The optical waveguide sensor is sensitive to various deformations, including stretching, pressing, bending, and twisting, since the light deviates from the optical waveguide when it suffers from deformations. [25,28]As a result, the perception soft actuator can be achieved by properly integrating the optical waveguide sensor.Additionally, the machine learning approach can be used to overcome the perceptual coupling problem. [33]e power loss of optical waveguide sensors can be quantificationally calculated as where P 0 represents the output power of the sensor in a straight state, and P is the output power of the sensor suffering from various deformations.L > 0 when the power loss increases.To facilitate the calculation of the output energy loss of the sensor, the light power of the sensor (P) is converted into an analog voltage through the optical conduction mode conversion circuit.Therefore, the output power loss of the sensor can also be calculated as where V 0 is the initial output voltage of the sensor when the sensor is not deformed, and V is the output voltage at any time when the sensor is deformed.
According to the components of the optical waveguide sensor, the sensor can be fabricated in three steps.First, the shell of the sensor (Figure 2b) was fabricated.Dragon skin 20 A/B added with a little black Silc Pig were mixed and poured into the shell mold.After curing, the black shell was removed from the shell mold.Second, the transparent core of the sensor (Figure 2c) was fabricated.The infrared LED and infrared receiving diode were placed on both ends of the black shell, respectively.Then, the mixed Clear Flex 30 A/B solution was poured into the black shell.Finally, after curing, it was put into the packaging mold and covered with the pigmented Dragon Skin 20 solution (Figure 2d).After curing, the prototype of the flexible optical waveguide sensor was formed, as shown in Figure 2e.The sensor was 95 mm in length, 3.5 mm in width, and 3.5 mm in height.The cross-sectional area and length of the core are 1.5 mm Â 1.5 mm and 80 mm, respectively.

Design and Fabrication of the Multidirectional External Perception Soft Actuator
A bellows structure soft actuator similar to the fast pneumatic networks [34] was designed, as shown in Figure 3a.To realize its sensing ability, two factors need to be taken into account: the bending performance of the soft actuator and the lifetime of the sensor.There are three kinds of locations on the soft actuator where the flexible optical waveguide sensors can be embedded, including the bottom, the back, and the sides of the soft actuator.The stress-strain distribution of each position is different when the soft actuator is driven, which will affect the sensing performance of the soft actuator.Therefore, the finite-element analysis tool (ABAQUS) was used to analyze this.The result is shown in Figure 3a.
The stress and strain gradually increase from the bottom to the back of the soft actuator when it is driven.Embedding a sensor into the bottom of the actuator will result in an increase in its thickness, which limits the bending motion of the soft actuator.Moreover, there are two drawbacks when the sensor is integrated into the back of the soft actuator, including irreparable damage to the sensor and constraints on the bending angle of the actuator.Finally, when the sensor is embedded in the side of the soft actuator, the bending performance of the actuator is hardly compromised, while the sensitivity of the sensor is improved due to the slight deformation (Figure 3b).Overall, both sides of the soft actuator are suitable for integrating sensors.
To promote the molding rate and simplify the production steps, the lost wax method was employed to fabricate the soft actuator.The soft actuator can be fabricated in the following steps.First, the wax core was fabricated (Figure 3c).Paraffin wax was melted on a heater and then poured into the wax mold with two iron wires fixed at wax core.After curing, the wax core was taken out of the wax mold.Second, the soft actuator was fabricated (Figure 3d).The wax core was put into the soft actuator mold, and the mixed Dragon Skin 30 solution was put into an injector simultaneously.Then, the solution was injected into the soft actuator mold under the push of high air pressure until the solution overflowed from the mold.The soft actuator mold was placed in an oven to speed up the curing of the solution.After curing, the soft actuator was removed from the mold and placed in an oven to melt the wax core.Finally, the multisensory soft actuator (Figure 3e) was assembled.We combined the soft actuator with a pipe joint, an actuator connector, and two optical waveguide sensors using the silicone adhesive (ELASTOSIL E43, WACKER Inc.).

Bending Performance of the Soft Actuator
The soft actuator's complete drive and data acquisition system are shown in Figure 4a.It was driven by air pressure regulated by a proportional valve and converted by an air pressure sensor (FESTO Inc.).A 16-bit data acquisition card (ZISHUTECH Inc.) is used to control the proportional valve and collect sensors' values (air pressure, two optical waveguide sensors' voltage).These sensors' values are converted by a four-channel voltage following amplifier at 20 Hz.The left and right optical waveguide sensors were defined as L-sensor and R-sensor, respectively.
First, to measure the influence of sensor integration on the bending movement of the soft actuator, the bending angle of the sensor-less soft actuator and the sensing soft actuator was calibrated.Furthermore, the relationship between the curvature and the sensors' power loss was calibrated to characterize the soft actuator's bending performance.The calibration method is shown in Figure 4b; the soft actuator with three marks was fixed on a board.The position of three marks can be tracked by three Opti Track cameras.The lengths of a, b, and c can be calculated according to the position of the three marks.Hence, the central angle (φ) of the soft actuator can be calculated as In this article, the supplementary angle (θ) of φ is used as the bending angle of the soft actuator.The experimental results are shown in Figure 4c-e.
Figure 4c displays the bending performance of the soft actuator before and after with two optical waveguide sensors.On the whole, there is a small gap between the imperceptible soft actuator and the sensing soft actuator.Additionally, there is no difference when the soft actuator is driven from 0 kPa to 55 kPa.An obvious difference can be observed when the soft actuator is driven at 60 kPa, and the maximum angle difference of about 3°occurs at 75 kPa.This is probably affected by the hard diodes.The soft actuator without sensors has a greater hysteresis than the sensing soft actuator due to the slight increase in hardness of the soft actuator.
The test results of the soft actuator's proprioceptive performance are also given in Figure 4d,e.They show the sensors' raw data and their fourth-order polynomial fitting curves.First, good linearity and small hysteresis can be observed.Second, the R-sensor has a greater hysteresis and a higher sensitivity than L-sensor.The difference between the two sensors is due to the handmade error.
In summary, two conclusions can be drawn from this experiment.First, the bending performance of the soft actuator is slightly affected and has a lower hysteresis by embedding two optical waveguide sensors.Second, the bending performance of the soft actuator can be characterized by the optical waveguide sensor.

Environmental Stimuli Response
The soft actuator can sense external stimulation at different locations due to the multisensing ability of the optical waveguide sensor.Therefore, the experimental platform depicted in Figure 5a was set up to test the response of the soft actuator to external stimuli.It is not possible to test every position of the soft actuator.Hence, we have defined %12 positions for the soft actuator, as shown in Figure 5b.These locations can almost cover the body of the soft actuator.In this experiment, the round indenter with a 13 mm diameter is treated as an obstacle.It moved forward 15 mm at 5 mm s À1 speed to press the center of each spot.Each location was subject to 60 trials, a total of 720 trials for the soft actuator.Each trial period took about 10 s.
We extracted one cycle of data from the original data and calculated the two sensors' power loss, as shown in Figure 5c.The distinct power loss of the two sensors can be observed overall.Additionally, the R-sensor is more sensitive than the L-sensor due to the fabrication error.Third, because the two sensors are initially lightly bent, the power loss of the two sensors is quite minimal, and negative values appear when the front of the soft actuator undergoes external stimuli.Fourth, when the left side of the soft actuator suffers from external stimuli, the L-sensor has a larger power loss than the R-sensor.Similarly, the R-sensor has a greater power loss than the L-sensor when the right side of the soft actuator suffers from external stimuli.It is difficult to distinguish the exact position of the soft actuator using a threshold method.Therefore, an intelligent method is necessarily used to recognize the 12 force positions.
Inspired by the external perceptual recognition pathway of the human hand (Figure 6a), an external perceptual recognition pathway of the soft actuator based on machine learning was established to classify the 12 positions (Figure 6b).Each force position was regarded as a class, so there are 12 classes.They were classified using a highly integrated machine learning model (light gradient boosting machine, LightGBM) and a traditional machine learning model (k-nearest neighbor, KNN).

Figure 6b depicts the classification process in detail.
Step 1: Processing data.A sliding window was utilized to extract usable data from the 12 groups of labeled sensor data.Then, 12 frequently used features were extracted using the tsfresh toolkit and prioritized by importance (Figure 6c).Next, the first six features of the data, including the median, maximum, sum of reoccurring data points, mean absolute change, standard deviation, and absolute energy, were chosen to train and test the model.The processed data were proportionally split into the training dataset and test dataset according to a ratio of 9:1.
Step 2: Training and testing model.The training dataset and test dataset were sent into the LightGBM model to train and test, respectively.Step 3: The classification results were evaluated using the accuracy of test results, and tenfold crossvalidation was used to validate the classification rate.
In the process of classification, the two models with three different inputs were used to predict the outputs, including six features of the L-sensor, six features of the R-sensor, and 12 features of the L-sensor and the R-sensor.Table 1 presents the classification results.The classification accuracy of the two models is close to 100% and the LightGBM model has higher accuracy by inputting 12 features of two sensors.Furthermore, the classification accuracy of the two models exceeds 90% and the accuracy of the LightGBM model is more than 98% by inputting six features of one sensor.Consequently, the external perceptual recognition pathway of the soft actuator can be set up based on the LightGBM model.

Application in Complex Underwater Environments
In this section, to demonstrate the role of the soft actuators' perception capabilities in complex underwater environments, a  simple underwater environment (Figure 7①②③) and a remote operation scheme based on the LightGBM model was built (Figure 7).The scheme mainly includes three aspects: obstacle detection, contact location recognition, and the movement of soft actuators controlled by humans.The details are as follows.
The first step was collecting a set of sensor values and calculating their average values.Then, a threshold was set to detect if the soft hand encounters an obstacle.When the average results are all higher than this threshold, the robotic arm will keep moving.Instead, the two sensors' values will be sent to the LightGBM model to predict the contact location in real time.Therefore, according to this, the next moving direction of the soft hand can be judged.The movement of the robotic arm was remotely manipulated by the human using the teach pendant.
The actual grasping process can be carried out in the following two stages (Figure 8 and Video, Supporting Information).Figure 8b-d shows the posture of the soft hand at different moments, and the corresponding sensors' power loss can be seen in Figure 8a.

Stage 1: Detecting Obstacles and Determining the Task Space
The initial position of the soft hand was arbitrarily set (Figure 7①).First, the soft hand was moved down without detecting any obstructions (Figure 8b①).Then, the soft hand was moved backward at intervals of 30 mm.The "L-m" of force position was recognized when the soft hand was moved backward for the second time (Figure 8b②).Therefore, the soft hand was moved forward at the same spacing.The "R-m" of force position was detected when the soft hand continued to move forward about 150 mm (Figure 8b④).The range of front and rear movement can be preliminarily judged to be 60-120 mm by analyzing  the entire moving process.Hence, the grasping task can be initially executed in this range.

Stage 2: Grasping Objects
The soft hand was moved back about 70 mm and tentatively grasped objects.The whole grasping process is shown in Figure 8c,d.First, heavier object 1 was successfully gripped in water and lifted out of the water by the soft hand pressurized at %0.45 bar (Figure 8c).However, object 1 slipped out of the soft hand due to the heavy object 1 and the low coefficient of friction between them.Moreover, a lighter object 2 was grabbed by the soft hand.The sensors' power loss in Figure 8a⑨-⑫ shows a stable grasping process.

Tiny Fluctuation Perception
Interestingly, some tiny fluctuations can be perceived by the soft actuator owing to the high sensitivity of the flexible sensors (Figure 8a⑥⑦⑧⑩).First, the sensors' power loss increased marginally when the soft hand was driven (Figure 8a⑥⑩).Additionally, Figure 8a⑦ presents a slight increase due to the changed pose of object 1 when it ran into an obstruction.There are tiny fluctuations in the two sensors in Figure 8a⑧ when object 1 slipped out of the soft hand.After that, the two sensors' power loss increases sharply due to the larger bending angle of the soft hand.Consequently, sliding detection, vibration detection, and other various meaningful tasks can be carried out utilizing the unique perception of the soft actuator.

Discussion
This article presents a multidirectional sensing soft actuator for complex underwater environment teleoperation.An external perceptual recognition pathway for the soft actuator was built inspired by the human hand perception system.Based on this approach, the soft actuator can recognize contact positions and the remote tasks can be further performed in complex underwater environments without visual feedback.
Soft grippers are increasingly utilized in underwater environments due to their safe interactions with organisms.Table 2 compares our soft actuator with other previous underwater soft grippers.The comparison shows that our soft actuator not only has external perception capabilities but also uses the sensing results to assist in underwater remote operation tasks.Moreover, flexible sensors commonly used to realize the external perception ability of soft hands are summarized in Table 3.It indicates that our soft actuator's sensing ability plays its obstacle avoidance function in the application, while other research mostly stays in the prediction of external stimuli and has not been applied.Furthermore, our multidirectional external perception soft actuator offers an effective approach for grasping tasks   [16,36]  Liquid metal sensor [16]   Obstacles and objects [36]   Force and pose Easy fabrication; Easy leakage and oxidation [16] Autonomous grasping (air) [36]   None [30]  Conductive  The study on the optical sensing soft actuator also has some limitations.First, the soft actuator can only distinguish the defined 12 positions due to the limited perception.Besides, detecting objects with the soft actuator remains challenging owing to the weak haptic perception of the soft actuator.In future work, the perception of the soft actuator will be enriched and a more intelligent model will be built to fusion multisensor information.We will commit to realizing both accurate contact location prediction and force amplitude estimation and further to reconstructing the whole posture of the soft actuator.Finally, we envision that the soft actuator can be utilized to effectively perform teleoperation or autonomous grasping tasks in practical underwater environments without cameras.

Conclusion
To improve the efficiency of remote operation in complex underwater environments, this study presents an optical sensing soft actuator with multidirectional external perception and its external perceptual recognition pathway inspired by a biosensing system.We only used two flexible optical waveguide sensors to achieve the sensing soft actuator.The experimental results demonstrate that the bending performance of the soft actuator is slightly influenced by the two sensors' integration and can be characterized by the two sensors' power loss.Moreover, the soft actuator can distinguish 12 force locations based on machine learning, and the classification accuracy is up to 99.82%.Finally, the teleoperation of the soft hand can be conducted in underwater environments without the help of visual sensors.

Figure 1 .
Figure 1.Remotely controlled soft actuators with multidirectional external perception to conduct tasks in complex underwater environments.a) The soft actuator integrated two flexible optical waveguide sensors.b) Machine learning methods used for distinguishing soft actuators' contact force positions.c) The diagram of 12 different contact positions (the blue arrows present the direction of the force on the soft actuator).d,e) Underwater grasping tasks by teleoperation method according to the sensors' feedback.

Figure 2 .
Figure 2. The fabrication process of the flexible optical waveguide sensor.a) 3D structure diagram of the sensor.b,c) Fabrication of the black shell and the transparent core, respectively.d) Covering of the transparent core.e) Prototype of the flexible optical waveguide sensor.

Figure 3 .
Figure 3. Design and fabrication process of the multidirectional external perception soft actuator.a) 3D model and stress-strain distribution of the bellows structure soft actuator, respectively.b) 3D model and stress-strain distribution of the soft actuator for embedding two sensors, respectively (the soft actuator was driven at an air pressure of 50 kPa).c,d) Fabricated wax core and soft actuator, respectively.e) Assembled soft actuator with a pipe joint, an actuator connector, and two sensors.

Figure 4 .
Figure 4. a) The complete drive and data acquisition system.b) Calibration method of the soft actuator.c) Calibration results of the soft actuator before and after with two optical waveguide sensors.(N-sensor and W-sensor represent soft actuators without and with two sensors, respectively.P and DP represent soft actuators in the pressurized and depressurized states, respectively.)d,e) The power loss in L-sensor and R-sensor, respectively, when the sensing soft actuator is driven from 0 to 75 kPa.

Figure 5 .
Figure 5. a) Experimental setup of external stimulation (blue arrow represents the direction of movement of the indenter).b) Diagram of three different stress areas, a total of 12 different force positions.c) The two sensors' power loss in one cycle when the soft actuator suffers from the same stimuli.

Figure 6 .
Figure 6.a) The external perceptual recognition pathway of the human hand to perceive external stimuli.b) The external perceptual recognition pathway of the soft actuator to identify 12 contact positions via machine learning methods ① The two sensors' raw data; ② The recognition process; and ③ Classification results).c) Ranking map of features' importance.

Figure 7 .
Figure 7. Underwater remote operation scheme without visual sensors; ①②③ represent the front view, top view, and side view of the experimental environment, respectively.

Figure 8 .
Figure 8.The whole grasping process of the soft hand in an underwater environment.a) The two optical waveguide sensors' power loss and the air pressure inside the soft hand.b-d) The different states of the soft hand correspond to specific moments.(The numerals in (a) correspond to the numerals in (b)(c)(d)).
in complex underwater environments by analyzing and comparing the advantages and disadvantages of the existing perception soft grippers.

Table 1 .
Experimental results of classification.

Table 3 .
Flexible sensors used for external stimulus perception.