A Flexible Bidirectional Interface with Integrated Multimodal Sensing and Haptic Feedback for Closed‐Loop Human–Machine Interaction

Human–machine interaction (HMI) establishes an interconnected bridge between humans and robots and plays a significant role in industry and medical fields. However, the conventional interface is bulky and rigid with a single function. It lacks feedback information to be provided to the user, limiting its application scenarios and the ability to complete complex tasks in a dynamic environment. Herein, a flexible bidirectional (FBD) interface with integrated multimodal sensing and haptic feedback and the development of an FBD‐based closed‐loop HMI platform are presented. The proposed FBD interface offers programmable haptic feedback through a coin vibration motor array to endow the user with better performance in missions. The multimodal sensing array can perceive signals of joint movement and the pressure imposed on the skin. In addition, the FBD interface exhibits significant advantages of flexibility and a compact structure. The experimental tests verify that the multimodal sensing module can accurately perceive various static and dynamic movements of a human. Furthermore, the actuation module can provide diverse haptic vibrations to convey different information to the user. Moreover, successful applications in the scenarios of bilateral robotic teleoperation and assistance of disabled people indicate great potential of the FBD interface for broad applications.


Introduction
[3] Traditionally, desktop-level linktype interactive devices were adopted to collect hand movement information and provide force feedback. [4,5]They served as the master devices to teleoperate robotic manipulators for executing particular tasks, such as dangerous work [6,7] and microscale operations. [8]In addition, an optimal motion capture system was developed to record and process motion by tracking target reflective spheres on the target. [9,10]owever, the conventional interfaces suffer from rigid and bulky devices, which limit the application scenarios of the interfaces.Alternatively, portable interfaces exhibit great potential to meet the urgent demand.In the past decade, rapid progress has been achieved in developing wearable and flexible interfaces for human-machine interaction (HMI).These include glovetype interfaces that recognize hand gestures and provide haptic stimuli for the fingertips [11][12][13][14][15] and brain-computer interfaces that offer a potential tool to detect electroencephalograms for exploring brain functional mechanisms. [16,17]20] With the development of flexible sensors, various interfaces have been designed and fabricated to perceive electrophysiological signals (e.g., electroencephalograms, [21][22][23] electro-oculograms, [24][25][26] electromyograms [27][28][29] ), body temperature, [30,31] sweat, [32][33][34] or human motions [35][36][37][38] ).[41] However, when the precise sensing ability of the sensing materials is solely focused on, the constructed device is often an open-loop system, which lacks a feedback mechanism for users to utilize their intelligence and experience in a dynamic environment.To provide realistic feedback to the user, various actuators have been deployed at the interaction interface, [42][43][44] including dielectric elastomer actuators, [45][46][47][48] pneumatic elastomer actuators, [45,[49][50][51] electromagnetic actuators, [52][53][54][55][56] thermal-tactile actuators, [57] electro-tactile actuators, [58,59] and so on.Existing flexible wearable interactive interfaces can acquire signals from particular body parts.However, if feedback functionality is simultaneously desired, then the overall device may become exceptionally bulky (with pneumatic actuators) or necessitate high-voltage driving (with piezoelectric or dielectric actuators).Consequently, developing a compact bidirectional interface that enables the sending of control commands and receiving of feedback in a dynamic environment becomes essential to integrate the user into the closed-loop HMI (CL-HMI) system.
To this end, this article proposes a new flexible bidirectional (FBD) interface for achieving CL-HMI to execute complex tasks in a dynamic environment.The proposed FBD interface provides programmable haptic feedback through a coin vibration motor array, which enables the user to achieve better performance in missions.The multimodal sensing array can precisely perceive movement signals of joints and the pressure applied to the skin.A modular design of a multilayer stack is introduced to achieve a thin and compact structure of 3.9 mm thickness.The excellence of the materials and components give ideal interface flexibility, which helps the interface fit nicely on the skin to ensure the reliability of information transmission and wearing comfort.The experimental tests verify the satisfactory performance of the FBD interface with the actuation and sensing modules in both static and dynamic conditions.The CL-HMI platform with the FBD interface exhibits great advantages in a series of scenarios, including bilateral robotic teleoperation, unmanned ground vehicle (UGV) remote control with collision reminder, haptic dialogue, and haptic navigation, which demonstrate its remarkable universality and broad application potential in the fields of industry, rehabilitation, and entertainment.

Design of the FBD Interface
The CL-HMI platform is developed, as illustrated in Figure 1.The user and the FBD interface attached to the skin constitute the master side of the system, while the end-effectors in the environment act as the slave side.The user's movement acts as the instructions to control the slave machine through the communication channel.The dynamic interaction information between the slave side and the environment can be fed back to the user in the opposite direction.Inspired by the characteristics of the human skin mechanism, the FBD interface is designed and developed.Figure 2 A exhibits the representative layout of the proposed FBD interface, which takes the form of a modular multilayer stack.The actuation module is a distributed array of mechanical vibratory actuators that provides haptic sensations.A customized, flexible sensing module provides the multiple functions of bidirectional bending and stress sensing, which can precisely distinguish human motions.The actuation module and sensing module are staggered in the horizontal direction to ensure that the two modules do not interfere with each other (Figure 2B).The upper layer is a thin, soft film to cover the interface.The proposed interface utilizes Ecoflex silicone rubber as the packaging material, providing exceptional stretchability and waterproof performance.The use of a specialized silicone adhesive ensures effective resistance to the sweat generated on the human skin.The core components are encapsulated in silicone rubber, preventing water infiltration during daily use.At the same time, the haptic spots are designed in the lower layer to enhance the haptic sensation for the skin.The support module is composed of upper and lower frames.Moreover, the sensor module is sandwiched between these two structures to ensure stability.The interface is powered by a standard lithium coin cell battery.

Working Principle of the FBD Interface
The primary principle of the actuation module is to provide haptic sensations that approximate the actual interaction.Figure 2C illustrates the structure of the actuation module, which is a Figure 1.Schematic diagram of the FBD-based CL-HMI platform, including the user, the application scenario, the communication channel, and the FBD interface.The user and the FBD interface serve as the master, while the end-effectors in the environment serve as the slave side.The user's body movements are used as instructions to control the slave through the communication channel, and the dynamic interaction information is fed back to the user.distributed actuator array.The vibration intensity of each actuator is independently controlled, which can be programmatically controlled to combine the vibrations into abundant haptic sensations.The chosen actuators are thin coin vibration motors, which exhibit the advantages of compact dimensions (with a diameter of 8 mm, a thickness of 2.7 mm, and a weight of 0.3 g), low-voltage direct current (DC) driving and a vigorous vibration intensity (up to 1.4 G).These features enable easy mounting at the HMI interface and offer a low-power solution for vibration.These coin motors can be directly driven by a maximum of 3 V DC and do not require an additional integrated circuit (IC) driver, whereas the majority of linear resonant actuators (LRAs) require  an IC driver and are not compatible with a DC voltage.The right subfigure in Figure 2C demonstrates the architecture of the actuator, where a nonsymmetric mass is attached to the motor shaft to produce an asymmetric centripetal force under the driving of a DC voltage.At a high revolutions per minute (rpm) speed, the resulting displacements appear on the skin as vibratory forms of actuation.The vibration pattern of the actuators in the array can be precisely controlled by modulating the waveform of the supplied pulse-width-modulated (PWM) signal.The duty cycle of a PWM signal affects the output voltage, which in turn controls the vibration intensity and frequency of the motor.Therefore, the actuation module enhances the user experience by providing a more comprehensive range of tactile feedback through a combination of vibrations.
In reality, many mechanoreceptors are distributed on the skin to sense mechanical stimuli and convert them into neural signals to the brain.Inspired by this mechanism, the sensing module is designed to accurately sense multimodal mechanical stimuli and general nonoverlapping electrical signals.The layout of the sensing module is shown in Figure 2D.It is a multifunctional sensor array that can detect the strain and pressure on the skin in parallel.The custom-made sensing module consists of an internal pressure sensor array and an external strain sensor array.The sensors in the sensing module operate independently, eliminating potential crosstalk issues.As depicted in Figure S1, Supporting Information, the experimental results illustrate minimal crosstalk between the pressure and strain sensors.Separate strain and pressure tests are conducted, in which the signals from both the pressure and strain sensors are concurrently acquired. [60]The results indicate that only a specific sensor produces a signal change when an action occurs during normal usage, whereas the remaining sensors exhibit a minimal response.Due to the nonoverlapping feature of the strain sensors, each sensor generates a positive or negative signal when it bends in different directions.Hence, the external strain sensor array distributed around the internal pressure sensor array can precisely detect the deformation of the skin.
The working principle of the sensing module is depicted in Figure 2E.The sensors are resistive sensors made of conductive materials with microcilia on the surface.The current conduction paths can be enhanced for effective electron transfer when a microcilium encounters the neighboring cilia due to the shape change of the substrate.The substrate and the microcilia form parallel circuits, and the resistance of the sensor can be expressed as where R refers to the sensor resistance.R s and R c represent the resistances of the substrate and the microcilia, respectively.When the sensors are pressed or bent inward, more microcilia bridges are constructed, resulting in a reduction in the resistance.In contrast, the resistance increases when the microcilia bridges are broken by outward bending.

Fabrication and Assembly
The material and fabrication of the microcilia substrate are detailed in our previous work. [61,62]Figure 2F demonstrates the fabrication process of the FBD interface.First, a mixture of carbonyl iron particles (CIPs) and polydimethylsiloxane (PDMS) as a magnetically responsive gel was coated on a cured PDMS thin film.Then, the sample was placed in a magnetic field to automatically form a microcilia array due to the synergistic effect of the intrinsic gravity of the materials, the surface tension, and the magnetic force.Afterward, the sample swelled in a prepared cyclohexane solvent containing carbon black nanoparticles to adsorb the conductive materials to uniformly form a conductive path.After being removed from the solvent, the sample was cut (by a laser) into rectangular and square shapes as a strain sensor unit and a pressure sensor unit, respectively.Then, the molds of the supporting frame and lower encapsulation layer were fabricated through the 3D printing method.After A and B of Ecoflex-030 were evenly mixed (typical weight ratio of 1:1) and the bubbles were discharged in a vacuum machine for 5 min, the mixed solution was poured into the prepared molds, which were then placed in a normal temperature environment for curing.After 24 h, demolding of the cured samples from the molds was conducted to obtain the supporting frame and the lower encapsulation layer.The top encapsulation layer was a thin film produced by coating and curing the Ecoflex-030 mixture.After that, the circuit and electrode were drawn with conductive silver paste on the supporting frame, and the sensors and actuators were arranged in a preset position.Then, the sensors and actuators were sandwiched by another supporting structure.Finally, the device was packaged with the encapsulation layers, which finished the fabrication process.

Integration of the FBD Interface into the CL-HMI Platform
The CL-HMI platform was then developed to integrate the human operator into the robotic system through the FBD interface.A schematic diagram depicting the architecture layout is illustrated in Figure 3.In Scenario 1, a bilaterally teleoperated robot system is constructed.Attachment of the FBD interface to the joints converts the user's motion into the control commands through motion mapping.Meanwhile, the contact force between the end-effector of the robot and the environment detected by the sensors (equipped on the robot) can be converted into vibration of the haptic motors to provide feedback for the user.In addition, the sensors on a UGV can detect surrounding objects and give the operator reminders to avoid collisions.Similarly, the interface can assist in the navigation scenario for a blind person (Scenario 2).Moreover, the FBD interface allows disabled people to communicate remotely through Morse code, that is, haptic dialogue (Scenario 3).

Actuator Performance
Figure 4A exhibits the flexibility of the FBD interface, which has a compact structure with a thickness of 5.2 mm.It can rapidly recover the initial morphological characteristics after the release of mechanical deformations (such as stretching, twisting, curling, or bending), guaranteeing the robustness of the interface and wearing comfort.According to the mobility of different body parts, three FBD interfaces were manufactured (Figure 4B).The right subfigure of Figure 4B shows a circular interface (with a diameter of 150 mm) suitable for large areas of the skin, such as the chest and abdomen.This interface was constructed by arranging sensors and actuators (Figure 2A) to transmit richer information bidirectionally.The upper-left part of Figure 4B depicts a strip interface (with a length of 80 mm and a width of 20 mm) for joints, which comprises a 2 Â 4 array of actuators and a strip of bending sensors.The lower-left portion of Figure 4B shows a blocky interface (20 mm in both length and width) for the palm, comprising a 2 Â 2 actuator array and a pressure sensor.The microcontroller unit (MCU) outputs the PWM signal with a specific duty cycle.Figure 4C illustrates the correlation between the PWM signal and the corresponding voltage that drives the actuator.The driving voltage is positively correlated with the PWM signal, and the output reaches the maximum driving voltage when the PWM signal has a duty cycle of 65%. Figure 4D depicts the results of the vibration intensity under different driving voltages, where the amplitude of displacement in the vertical cross-section direction represents the vibration intensity.The interface was placed on a vibrationisolation table to eliminate external influencing factors from the environment and ground.A precision laser sensor was employed to accurately measure the displacement of the interface surface.
The larger the displacement, the greater the vibration intensity, and the higher the required driving voltage.To explore the user's somatosensory feeling, [63,64] a volunteer (unaware of the test conditions) was asked to select the option that aligned with the subjective sensation.This test was carried out during application of vibration stimulation to the forearm, using various voltage levels ranging from 0 to 3 V.The vibration for a driving voltage below 1 V is imperceptible, the vibration for a driving voltage between 1 and 2 V is gentle, and the vibration for a driving voltage between 2 and 3 V is strong.In the range of 1-3 V, a gradual increase in the voltage results in a proportional increase in the vibration intensity, which is perceptible.Upon exceeding 2 V, the vibration can serve as a reminder by conveying appropriate information.To test the sensitivity of different regions of the body to haptic vibration, the FBD interface was divided into four parts (Zones A-D in Figure S2, Supporting Information).Then, a driving voltage of 2.5 V was applied to different vibration units to test the recognition accuracy in distinguishing the vibration zone.Vibrations were randomly generated in the four zones for the user to make judgments.The results in Figure 4E demonstrate that the accuracy rate is the highest when 2 Â 2 vibration motors are adopted as a vibration unit attached to the chest and abdomen.The preferred method to receive messages is to deploy the interface on the chest or abdomen, with 2 Â 2 vibration motors functioning as a vibration unit.The above results provide the basis and guidance for haptic vibration feedback settings for conveying complex information to the user.

Static Perception Performance of the Sensing Module
The excellent sensing performance of the microcilia structure was evaluated in previous work, including the sensitivity, recoverability, periodic response, long-term stability, and repeatability. [61,62]The setup for testing the pressure array is shown in Figure S3, Supporting Information, where a thin-film pressure sensor with a measurement range of 20 N and an accuracy of 0.1 N is laid over the pressure array.A static test was first conducted by placing building blocks (Figure S4, Supporting Information) with different contact surfaces and weights on the pressure sensor array.Figure 4F presents the weight distribution of building blocks placed on the sensing module, in which the depth of the color represents the weight.Figure 4G shows the detection results of the pressure sensing array, which indicate that the force distribution due to real contact can be accurately represented under static conditions.Furthermore, knocking and sliding tests were conducted to detect the dynamic sensing performance.Figure 4H reveals that knocking can be sensitively perceived, whether it is single knock, a double knock, or random knocking.Figure 4J depicts the signals measured when sliding a finger on the surface of the flexible interface following the trajectory given in Figure 4I.The above static and dynamic tests demonstrate excellent interface performance, in which complex contact in natural environments can be detected.In a static environment, the actuator module was driven by different driving voltages while the responses of the strain and sensor modules were simultaneously monitored.The experimental results (Figure S5, Supporting Information) demonstrate that the vibration motors have little effect on the sensor module under the driving voltage range of 0-3 V.

Dynamic Perception Performance of the Sensing Module
Figure 5A shows the complete set of FBD interfaces equipped on the user, including on the chest, palm, wrist, elbow, and shoulder.The user can choose a suitable layout according to the desired application.Figure 5B illustrates that the circular FBD interface on the chest successfully detects the user's unhealthy posture.In the perception tests of two unhealthy postures, the first one is a hunchback posture, in which both the left and right measurement units detect the deformation signal.The second one is the uneven-shoulder posture, in which the chest skin on the side of the lower shoulder produces a large deformation while the skin on other side is less deformed.Figure 5C shows that the blocky interface attached to the palm can capture finger presses.Figure 5D verifies that elbow bending can be recognized through the strip interface.Moreover, Figure 5E,F present the detection results of wrist and shoulder joint movements through the strip interfaces that bend on both sides.Because of the advantage of the microcilia structure of the sensing material, inward bending and outward bending motion of the joints can be discriminated, such as turning left or right of the wrist and moving upward or downward of the shoulder.

Bilateral Teleoperated Robotic Remote Control with Haptic Feedback
As a typical application of the fabricated FBD interface, a bilaterally teleoperated robot platform was developed, as depicted in Figure 6A.The movements of the human body captured by FBD interfaces were converted into robot control instructions through motion mapping.Then, remote control of the robot was realized via scheduling of the internal controller of the robot.The interaction information between the robot and the environment can be fed back to the human operator through haptic vibration to enhance the sense of the presence of an object.The setup and motion mapping of the bilaterally teleoperated robot system is shown in Figure S6, Supporting Information, where FBD interfaces are attached to the shoulder, elbow, and wrist.Motion mapping converts the body movements from the human coordinate system into the goal coordinate system in the operating environment through the transformation matrix between the human and robot bases.A block FBD interface was attached to the palm to control the opening and closing of the mechanical gripper.Figure 6B shows the manipulation process of the robot, which was remotely controlled to grab a box on a desktop.The robot grabbed the package from the initial position and transferred it to the target location.The trajectory in Figure 6C proves that the robot can be smoothly remotely controlled through the reasonable motion mapping method.As shown in Figure 6D, a force/torque (F/T) sensor is equipped between the manipulator and the end-effector to perceive the robot-environment contact, and a pressure sensor is mounted on the gripper (end-effector) to measure the clamping force.The haptic mode was designed and tested to demonstrate the convenience and advantages of the haptic feedback capability.
The FBD interfaces attached to the wrist and elbow were arranged to feed back the contact pulling force and pushing force in the contact with the environment, respectively.The vibration of the FBD interface on the palm was used to reflect the gripping force.The motor driving voltage controls the motor vibration intensity to express the force magnitude.Figure 6E illustrates the contact test results, where the motor voltages of the FBD interfaces on the wrist and elbow can track the contact force to feed back the interaction information.The human operator can feel the contact state experienced at the robot end through the haptic feedback.Figure 6F gives the results of the end-effector clamping cylindrical silicone objects with different hardness values (Figure S7, Supporting Information), in which the pressure obtained when clamping the object with the same action is positively related to its hardness.Therefore, haptic feedback can assist operators in perceiving the hardness characteristics of the target object, which provides more extensive information regarding the object interacted with, hence conveying a realistic sense of the object to a greater extent.Movie S1, Supporting Information, is given to demonstrate the performance and stability of the developed interface.

Obstacle Reminder for Remote Control of a UGV
A UGV disinfector was deployed as a slave machine to highlight the wide application of the bilaterally teleoperated robot platform and the significance of the haptic feedback.A block FBD interface was placed on the palm to control the velocity of the UGV, and a strip FBD device was attached to the wrist to capture the bending motion as the turning commands.Ultrasonic radar was equipped on the UGV to provide the conditions of the mission site through haptic feedback (Figure 6G). Figure 6H shows a schematic diagram of the obstacle reminder.When obstacles are within a 1 m range of the UGV, the operator can receive a reminder via haptic vibration.The closer the obstacle is, the greater the vibration intensity (Figure 6I).

Haptic Navigation
In addition to equipping a robot to establish a bilaterally teleoperated system, the FBD interface can also provide convenience for physically disabled people in real-life scenarios, such as blind people navigation and haptic dialogue.Blind people heavily depend on their hearing to perceive the external environment.As depicted in Figure 7B, a haptic navigation system was designed to assist a blind person.An FBD interface was attached to the chest to provide reminder information through easy-todistinguish haptic vibrations.Users carry mobile phones to enable the device to determine their location and send navigation instructions to the MCU.The instructions represented by the partition of the vibration module are illustrated in Figure 7A.The FBD interface is divided into four 2 Â 2 vibration zones.The vibrations of the zones represent turn left, turn right, go straight, and stop.For safety reasons, an emergency stop vibration mode is set, in which all the motors are driven so that the user can quickly identify the emergency command.The tests were conducted along a comprehensive pathway with the user blindfolded (Figure 7C).The fusion of haptic and voice prompts obtained a better navigation effect, in which the time consumption was reduced by 4.7 min, and the distance was reduced by 0.15 km.Although the voice prompts are easy to perceive, the turning angle reminder each time is not sufficiently intuitive.Voice commands can only remind the user to turn left or right.After the fusion with haptic controls, the user starts to turn after hearing the voice command, and the corresponding haptic unit vibrates until the user reaches the desired turning angle, which significantly improves the efficiency and accuracy of navigation for a blind person.As such, it is crucial to minimize any auditory disturbance that may hinder the ability of the person to effectively receive environmental cues.

Haptic Communication for Individuals with Hearing and Speech Impairments
The sensing and vibration capabilities of the FBD interface can enable disabled people to communicate through Morse code (Figure S8, Supporting Information) via haptics.Users can receive information through vibration feedback and can transmit information by tapping the interface.The devices were kept in the palms of the two users participating in the conversation.Short vibrations (<0.1 s) and continuous vibrations (>0.5 s) were set to represent dots and dashes, respectively.As shown in Figure 7D, one user can type the message into Morse code on the FBD interface, and the user on the other side can receive the message in the form of specific and regular vibrations.As revealed in Figure 7E, a trigger threshold is set on the sender side.When the signal voltage exceeds the threshold, it will be judged as valid.Figure 7F depicts the result of the haptic dialogue test.The sender tapped out the code that stands for "ILU", that is, "I love you".The receiver received the message through the haptic vibration and then sent a reply of "MT", that is, "Me too".In summary, the above experimental tests demonstrate diverse applications of the proposed FBD interface.

Comparison and Analysis
In recent years, there has been significant progress in the research on bidirectional HMI interfaces.Table 1 compares recent representative HMI interfaces in terms of their type, flexibility, sensing, and feedback.A variety of HMI interfaces are available, such as electronic skin (e-skin), [30,42,45,65] rings, [58] armbands, [57,66] and gloves. [59]These interfaces are capable of producing effective interaction effects.In particular, e-skin [30,42,45,65] has the most extensive application range due to its ability to cover most of the body's skin.Tactile feedback can be delivered using different methods, such as vibration, [42,45,58,59,65,66] heat, [58,59] and electricity. [30,57]Vibration is the most prevalent and instinctive feedback.Realizing a wireless nature and miniaturization of the entire interface is impossible when using pneumatic vibration because this vibration requires an air pump. [45]Elastic dielectric materials can generate force feedback but require a high voltage to drive them. [47]Thus, coin vibration motors are the optimal choice for generating vibration tactile feedback.Electric and thermal feedback can reduce the thickness of the e-skin.However, thermal feedback is slow in response, and developing electric feedback requires stimulation with a microcurrent, making wide acceptance challenging.Human body signals can be detected using various signals such as EMG, [30,57] temperature, [30] bending, [42,59] and pressure [30,45,47] signals.Surface EMG signals are highly susceptible to noise and necessitate the use of machine learning algorithms to indirectly recognize them, resulting in a bulky and expensive interface.Temperature provides valuable information about the body's health status.However, its practical applications in robot systems are limited.In contrast, the bending of joints and pressure on the skin are the most instinctive signal sources in human skin.A flexible and bidirectional HMI interface applied to the skin is proposed in this work.Its excellent flexibility allows it to be placed on most of the body's skin.The interface incorporates programmable vibratory tactile sensations to provide rapid, intuitive, and safe tactile feedback.Moreover, a multimodal sensing module is integrated to obtain various human skin signals of bidirectional bending and pressure, acquiring more interaction information.

Conclusion
To enhance the intuitiveness and convenience of HMI and eliminate the requirement of heavy and expensive professional devices while providing the user with precious haptic feedback, the FBD interface is proposed for achieving CL-HMI to execute complex tasks in a dynamic environment.It is a skin-integrated interface that can generate haptic vibration to provide external feedback information for the user and precisely perceive the deformation of the skin to detect the user's motions.The sensor modules of the FBD interface adopt nonoverlapping, multimodal flexible resistance-variable materials, which can detect various forms of interactive information, including inward bending, outward bending, and pressure, to realize multiple functions of the interface.Furthermore, the haptic vibration motor array is programmable to generate abundant vibration combinations for information transmission, which is crucial to achieve complex interactions in dynamic movements.Furthermore, the excellent static and dynamic detection performance of the sensor module and the investigation of the vibration motor characteristics enable the establishment of a CL-HMI platform so that the operator's decision-making wisdom and experience can be fused with a bilateral robotic teleoperation system.Meanwhile, disabled people can overcome inconveniences with the assistance of the CL-HMI platform.The successful applications of the FBDbased CL-HMI platform in multiple scenarios demonstrate the great potential in robot control, disability care, and many other domains.

Experimental Section
Schematic Circuit: The schematic circuit is depicted in the upper part of Figure 3.A MCU was adopted as the lower computer to control the state of each actuator and collect the sensor signals.The MCU generates digital pulse-width-modulated (PWM) signals with programmable frequency and duty cycle to control the actuators in an array through the general-purpose input/output (GPI/O).The DC currents delivered from the MCU were insufficient to drive a single haptic motor.Thus, the PWM signals were transmitted to a motor driver (MD) and amplified to meet the power requirements.First, the resistance change of the sensor module was detected via a resistance-to-voltage converter (RVC).Then, the output signals were transmitted to the MCU through an analog-to-digital converter (ADC), regarded as the human motion signal.
CL-HMI Platform: The platform comprises software and circuitry that enable rapid and effortless deployment and configuration of the FBD interface.Additionally, it facilitates efficient connectivity with external execution devices.The software is constructed on a local server using the robot operating system (ROS).It serves the purpose of receiving information, processing it, and sending instructions.It also serves as the core of the entire platform for information processing.The circuit is responsible for driving the interface actuators and receiving sensor data.The platform can accommodate various external devices and FBD interfaces, which allows it to perform various tasks based on the scenario and requirement.For example, a robot and an FBD interface can be arranged at opposite ends of the platform to create a robotic teleoperation system.When FBD interfaces were installed on both ends, a haptic communication system was formed.In this work, the wearable devices were tested on the leading author of the article only.
Bilaterally Teleoperated Robot System: The system uses robots, such as manipulators (model: UR5e, from Universal Robots) or a UGV, [67] as the external device.The operator uses the FBD interface to control the robot movement and receive haptic feedback through vibration.A gripper (model: 2F-85, from Robotiq) and a film pressure sensor (model: RF-C18.3-ST,from Shenzhen Ligan) were installed at the end of the robotic manipulator to from the end-effector.A suitable motion mapping method was developed to facilitate remote control of the robot by the user.As the sensors in the sensing module are nonlinear, precise control cannot be achieved if the position of the robot is controlled based solely on the position of the sensor.Therefore, the methodology of position-velocity (PV) mapping was adopted, which enabled smooth control of the manipulator.The mapping method converts the position of the user's joint into the velocity of the slave robot, optimizing the manipulator overall speed and movement precision.The slave robot end velocity can be obtained by PV mapping as follows.
where V s ðtÞ is the current velocity of the slave robot end, s m K v is the PV mapping factor, and P m ðtÞ is the current position of the user's joints.The advantage of the PV mapping mode is that the slave robot can move at a continuous velocity under the human operator's control.Therefore, the human will feel that the manipulation of the slave robot is flexible in a large workspace to achieve good efficiency.In the software system of the CL-HMI platform, a virtual dead zone of the joint was set for the user.If the joint is within this angle range, then the robot will stop.Users can adjust the speed and stop of the robot in real time in combination with the robot current motion state and the distance between the current and target positions obtained by visual feedback to achieve precise control of the robot and operational flexibility.Approval of all ethical and experimental procedures and protocols was granted by the Research Ethics Committee of the University of Macau under application no.APP-ARE-057.Informed consents were signed by volunteers prior to their participation in this study.
Navigation System: The user's distance and time consumption during navigation were recorded through a smartwatch (model: WATCH GT Runner, from HUAWEI), which was worn on the wrist of the tester.The mobile application (model: Amap, from AutoNavi) provided positioning and navigation directions.

Figure 2 .
Figure 2. Design, working principle, and fabrication of the FBD interface.A) Layout of the FBD interface.B) Vertical section of the FBD interface.C) Structure of the actuation module and working principle of the haptic motor.D) Layout of the sensing module.E) Working principle of the sensing module.F) Fabrication and assembly of the FBD interface.

Figure 3 .
Figure 3. Schematic circuit of the FBD interface and information flow of the CL-HMI platform.

Figure 4 .
Figure 4. Characteristics of the FBD interface.A) Flexibility of the FBD interface under stretching, twisting, curling, and bending.B) Prototypes of FBD interfaces, including a circular interface (on the right), a strip interface (at the upper right), and a block interface (at the lower right).C) Output driving voltage under different duty cycles of the PWM signal.D) Vibration intensity of the haptic motor under different driving voltages.E) Results of the recognition accuracy in distinguishing the vibration zone in different regions of the body.F) Weight distribution of building blocks.G) Detection results of the pressure sensing array.H) Results of the dynamic knocking tests.I) Trajectory of the finger slid on the interface.J) Measured signals when sliding a finger on the surface of the flexible interface.

Figure 5 .
Figure5.Characteristics of the strain sensing array dynamically perceiving the user's movement.A) Complete set of FBD interfaces equipped on the user, including on the chest, palm, wrist, elbow, and shoulder.B-F) Results of the dynamic tests, in which the interfaces are equipped on the user, including on the chest, palm, wrist, elbow, and shoulder.The chest, palm, and elbow can only be bent to one side, while the wrist and the shoulder can be bent bidirectionally.

Figure 6 .
Figure 6.Application in the scenario of robotic teleoperation.A) Architecture and strategy of the bilaterally teleoperated robot platform.B) Manipulation process of the robot, which is remotely controlled to grab the box on the desktop.C) Trajectory of the robot.D) A force/torque (F/T) sensor is equipped between the manipulator and the end-effector to perceive the robot-environment contact.E) Results of the contact (pull and push) test.F) Results of the end-effector clamping cylindrical silicone objects with different hardness values.G) Disinfector UGV equipped with ultrasonic radar.H) Schematic diagram of the obstacle reminder.I) Result of the obstacle reminder.

Figure 7 .
Figure 7. Application in the scenario of disabled person assistance.A) Instructions are represented by the partition of the vibration module.B) Setup for the haptic navigation system.C) Navigation test results when using voice-only commands and a fusion of voice and haptic commands.D) Haptic dialogue.E) Trigger threshold on the sender side.F) Result of the haptic dialogue test.

Table 1 .
Comparisons of bidirectional HMI interface.