Optical‐Nanofiber‐Enabled Gesture‐Recognition Wristband for Human–Machine Interaction with the Assistance of Machine Learning

The metaverse, where the virtual and real world are fused, is currently under rapid development. Immersive and vivid experience in the metaverse requires human–machine interaction devices that, unlike those currently available, are simultaneously imperceptible, convenient to use, inexpensive, and safe. Herein, an optical‐nanofiber‐based gesture‐recognition wristband that can accurately recognize gestures and be used to interact with a robotic hand is proposed and realized. Requiring only three optical‐nanofiber‐based pressure sensors, the wristband is simple in structure, convenient to use, and remarkably imperceptible to the user. With the assistance of a machine‐learning algorithm, a maximum recognition accuracy of 94% is achieved for testers with different physiques. A robotic hand can be remotely controlled by the wristband through gestures. The wristband has broad application prospects and is a promising solution for advanced human–machine‐interaction devices. An interactive preprint version of the article can be found here: https://doi.org/10.22541/au.167022827.77928491/v1.


Introduction
The latest advances in artificial intelligence (AI) are leading to increased interaction between humans and smart robots; we are also witnessing the rapid development of the metaverse. [1] More immersive and convenient human-machine interaction (HMI) devices are, thus, urgently needed to facilitate natural and continuous interactions in both the real physical world and online virtual ones. [2][3][4] Because the human hand is capable of performing complex tasks and executing elaborate movements, hand-gesture recognition provides a natural arena for the development of more advanced HMI. [5] Many types of equipment, including cameras, [6,7] electromyographic (EMG) sensor-based armbands, [8,9] and data gloves, [10][11][12][13][14] have been used to acquire hand-movement information. With the help of these technologies, several real-time and accurate handgesture-recognition devices have been developed. [15] However, they have major drawbacks: vision-based devices require a line-of-site optical path between cameras and hands, surface EMG signals are difficult to detect and always overwhelmed by noise, and glove-type devices are uncomfortable to wear.
By contrast, wristband-type devices are comfortable and inexpensive. More importantly, they have no impact on a user's day-to-day operations and can significantly improve the user experience by their remarkable imperceptibility. [16] Shull et al. have described a gesture-recognition wristband (GRW) equipped with 10 modified barometric-pressure sensors that can recognize up to 10 different hand gestures recognition and estimate finger angles. [17] Liang et al. demonstrated a wristband consisting of five PDMS-encapsulated capacitive pressure sensors; three gestures could be correctly recognized with an accuracy higher than 90%. [18] Recently, Tan et al. developed a GRW with eight sensing units, and each sensing unit contained a triboelectric nanogenerator and piezoelectric nanogenerator. [16] Combined with a machine-learning algorithm, the wristband achieved letter-byletter recognition of sign-language actions with a maximum recognition accuracy of 92.6%. Although these wristbands show great promise for application in HMI, their practical use may be restricted by electromagnetic interference (EMI), crosstalk noise, and complexity in the data processing. The metaverse, where the virtual and real world are fused, is currently under rapid development. Immersive and vivid experience in the metaverse requires humanmachine interaction devices that, unlike those currently available, are simultaneously imperceptible, convenient to use, inexpensive, and safe. Herein, an opticalnanofiber-based gesture-recognition wristband that can accurately recognize gestures and be used to interact with a robotic hand is proposed and realized. Requiring only three optical-nanofiber-based pressure sensors, the wristband is simple in structure, convenient to use, and remarkably imperceptible to the user. With the assistance of a machine-learning algorithm, a maximum recognition accuracy of 94% is achieved for testers with different physiques. A robotic hand can be remotely controlled by the wristband through gestures. The wristband has broad application prospects and is a promising solution for advanced human-machineinteraction devices. An interactive preprint version of the article can be found here: https://doi.org/10.22541/au.167022827.77928491/v1.
Using photons instead of electrons as signal carriers is an ideal strategy to address the EMI and crosstalk issues. For this reason, flexible optical waveguides, including fiber Bragg gratings, polymer optical fibers, and optical-micro/nanofibers, have attracted increasing interest for use in tactile sensors, [19][20][21][22] data gloves, [23,24] and HMI devices. [25] Optical micro/nanofibers with diameters close to or below the vacuum wavelength of visible or near-infrared light can offer engineerable waveguiding properties, making them attractive for applications in ultrasensitive sensors with small footprints. [26] Already, optical-micro/ nanofiber-based sensors have demonstrated high sensitivity, fast response, and a tunable working range for pressure sensing, [27,28] strain sensing, [29,30] and bending-angle monitoring. [31][32][33] A highly sensitive sensor for obtaining mechanical information from the surface muscles of the hand would improve the accuracy of a GRW and reduce the number of required sensing units. A single gesture typically generates different signals at different sensors; the gesture can be recognized by combining basic signal-processing methods with a machine-learning algorithm. However, in a GRW, sensors are not attached directly to the user's skin: a wristband tends to slip as the fingers move, potentially resulting in a significant change in the sensing signals. Therefore, location insensitivity of the mechanical sensors is a key factor in the success of a GRW. Nevertheless, most previous optical-micro/nanofiber-based pressure sensors have been in filmy structures where the sensing signal experiences a critical-position-dependent response.
To overcome this problem, we propose and demonstrate a flexible optical-nanofiber sensor with a soft liquid sac structure that can effectively mitigate the impact of stimulus position on the response of the pressure sensor without sacrificing sensitivity. This sensor can be used for precise acquisition of the arterial pulse with negligible position drift. Furthermore, we develop a GRW with only three such optical-nanofiber pressure sensors. Using the support-vector machine (SVM) machine-learning model, we decode the signals from the three sensors; the proposed GRW achieves a maximum hand-gesture-recognition accuracy of 94% for testers with different physiques. As a proof of concept, a robotic hand was successfully controlled by different testers through hand gestures, indicating the excellent adaptability of the GRW. This study offers promise of major advances in the tactile interfaces needed for a comfortable and immersive user experience in the metaverse. Figure 1a shows the configuration of the proposed GRW. It consists of three nanofiber-based pressure-sensor units (NFPSUs) placed at different locations on a person's wrist to capture the mechanical signals generated by finger movements. Multiple groups of muscles control finger movements; most of the muscle bellies and tendons are near the superficial epidermis of the wrist. [34] Therefore, when a finger moves, the corresponding groups of finger muscle contract or relax, causing deformations in the skin surface of the wrist. The skin deformations associated with different hand gestures are collected by placing NFPSUs at different locations on the wrist. With the assistance of machinelearning algorithms, a hand gesture can be recognized by decoding the multichannel sensing signals corresponding to it.

Results and Discussion
One key to the success of the GRW in this work is its highly sensitive NFPSU, as shown in Figure 1b. It consists of a filmy optical-nanofiber sensor, a soft liquid sac, and a rigid 3D-printed resin shell. The optical nanofiber, serving as a light waveguide to detect skin deformation with high sensitivity, is U-shaped for increased compactness and is encapsulated by polydimethylsiloxane (PDMS) films (length: 21 mm; width: 14 mm; thickness: 200 μm) to form a filmy optical-nanofiber sensor. A soft liquid sac (length: 27 mm; width: 23 mm; thickness: 2.5 mm) filled with glycerol is sealed by the filmy optical-nanofiber sensor; it is used to transfer the applied pressure due to skin deformation to the optical nanofiber. The rigid 3D-printed resin shell is designed to isolate the sensor from mechanical stimuli other than those generated by finger movements. The sac base (length: 6 mm; width: 6 mm; thickness: 200 μm) penetrates the hole in the center of the resin shell to contact the skin surface and collect pressure signals. In this case, a liquid sac can cover 2-3 adjacent tendons, thus, the crosstalk of acquired signals and the position-sensitive The locations of the three sensors were arranged by considering the distribution of wrist tendons, subcutaneous fat, and the blood vessels. To avoid the impact of wrist pules on the hand gesture recognition, two NFPSUs on the dorsal wrist (back side) were used to detect the movements of thumb and middle fingers. One NFPSU on the volar wrist (palm side) was used to detect flexion of index and ring finger. As shown in Figure 1c, the three NFPSUs can recognize 12 hand gestures and the interference of wrist pulse can be neglected.
To make the GRW suitable for testers with different physiques and avoid random sliding between skins and NFPSUs, we used elastic straps to connect the three NFPSUs, as shown in Figure 1d. Note that we do not recommend wearing the wristband too much tightly. Each NFPSU has dimensions of 3 Â 2.5 Â 0.3 cm 3 (comparable to a coin) and can be comfortably worn on the wrist. Figure 2a illustrates the working principle of the proposed NFPSU. When skin deformations relating to finger movements occur, the pressure applied to the bottom of the attached soft liquid sac changes. According to Pascal's principle, the pressure applied to any point within an incompressible liquid can be transmitted to every point of the liquid in real time. [35] Therefore, the pressure applied to the bottom of the soft-liquid-sac base is transmitted to the top surface, causing the contact filmy opticalnanofiber sensor to deform. In this way, the external mechanical signals related to finger movements captured at any point of the soft liquid sac base are transmitted to the optical-nanofiber sensor with high fidelity, eliminating the impact of position drift on the sensing signals. As shown in Figure 2b, when the optical nanofiber is slightly bent under pressure, the well-confined symmetric mode of an 800 nm diameter nanofiber at the input port evolves into an asymmetric profile with clear optical leakage, making it highly sensitive to mechanical stimuli. The model between the nanofiber deformation and optical intensity is provided in Supporting Information. As shown in Figure S1, Supporting Information, the bending radius decreases with the increase of the applied normal force. In contrast, when an 800 nm diameter nanofiber is embedded in PDMS, it will be not sensitive to the change of temperature based on our previous studies. [36,37] In this study, the optical nanofiber was fabricated by heating and stretching a standard silica single-mode fiber (SMF). [38] The as-fabricated optical nanofiber showed excellent flexibility, significantly exceeding the performance of standard silica or polymer optical fibers. For example, the bending radius could be made less than 10 μm, as shown in Figure 2c. Owing to its smooth surface and geometric uniformity (Figure 2c), the as-fabricated fiber offered a transmission greater than 99% [38] and a tensile strength higher than that of spider silk. [39] Figure 2d shows a U-shaped nanofiber guiding 633 nm wavelength laser light. The bright red light along the fiber indicates the presence of an evanescent field outside the optical nanofiber. Generally, with the decrease of the fiber diameter, the fractional power of the light outside the optical fiber increases exponentially and a stronger evanescent field results in a higher sensitivity. However, a thinner optical nanofiber loses mechanical stability when it is manipulated to form a U-shaped structure. In this work, an 800 nm diameter optical nanofiber was chosen for the trade-off between high sensitivity and mechanical stability. For high compactness, the curved end of the U-shaped www.advancedsciencenews.com www.advintellsyst.com optical-nanofiber sensor was intentionally positioned slightly over the edge of the liquid sac to ensure that deformation would occur in the sensitive area of the nanofiber.
To investigate the sensor's pressure response, we used a mechanical testing system ( Figure S2, Supporting Information). Figure 2e shows the wavelength-dependent transmittance response to applied static pressure in the range of 0-31.8 kPa. With increasing wavelength, the transmittance of the nanofiber decreases and the sensitivity increases as a result of the increasing fractional power of the evanescent field. [19] By defining the sensitivity as S = ΔT/ΔP, where ΔT is the change of transmittance and ΔP is the change of pressure, the sensor shows a lower sensitivity of À0.01 kPa À1 when the applied pressure is less than 6.4 kPa. In the high-pressure range (6.4-31.8 kPa), the sensor achieves a higher sensitivity of À0.03 kPa À1 (inset of Figure 2e).
The NFPSU demonstrated high durability under a pressure of 31.8 kPa at a frequency of 0.55 Hz (Figure 2f ): after 6000 cycles, the sensor performance showed little change. In addition, a sinusoidal mechanical signal with a high frequency of 800 Hz was applied to evaluate the temporal frequency response of the NFPSU. The stable performance shown in Figure 2g demonstrates the sensor's ability to obtain mechanical signals related to finger movements in real time.
In addition, Figure S3, Supporting Information, gives the temperature response of a NFPSU in the range of 30-60°C. It can be clearly seen that the spectrum shows a temperature-independent change. Moreover, the skin temperature is relatively stable, which reduces the impact of temperature on our sensor.
To characterize its response to tiny mechanical stimuli, the NFPSU was used to monitor wrist pulses in real time (Figure 3a). Owing to its high sensitivity, it could readily obtain wrist-pulse waveforms with high resolution; the two distinguishable peaks and late systolic augmentation shoulder agree very well with the expected shape of a noninvasive radial-artery pressure wave. [40] The trace shows clearly that the pulse frequency before the exercise was %70 beats min À1 and the pulse shape was regular and repeatable. After exercise, the pulse frequency increased to %90 beats min À1 , and the shape and intensity were irregular (Figure 3b,c).
To prove that the NFPSU response was position independent, we drew a nine-point grid with an area of 1 Â 1 cm 2 on the wrist skin, centered on the wrist artery (Figure 3d). The NFPSU was positioned point-by-point on the grid to obtain pulse signals. Figure 3e presents the measured arterial-pulse signals recorded by the NFPSU at various positions in the grid (indicated by red dots in the figure). Because the whole experiment lasted more than 1 h, the heartbeat cycles show a slight fluctuation in the range of 60-70 min À1 . Each of the nine pulse waveforms in Figure 3e resembles a typical pulse waveform (Figure 3f ) consisting of a percussion wave (P-wave), a tidal wave (T-wave), and a diastolic wave (D-wave). Thus, we believe that high-fidelity pulse signals can be successfully obtained anywhere within a maximum distance of p (5 2 þ 5 2 ) = 7.07 mm from the arterial pulse. Such position-independent sensing ability of the NFPSU is crucial when the GRW is worn for a long time.
For gesture recognition, the proposed GRW with three NFPSUs was placed on the tester's wrist to obtain mechanical signals related to hand gestures. Data-processing methods were employed to calculate the characteristics of each gesture signal, as shown in Figure 4. Light was launched into the three sensors and collected using a CMOS camera. The time-varying output data-in the form of a CMOS image (1280 Â 720 pixels)-were then transferred to and processed by a computer. By extracting the change in the gray level of the CMOS images over time, we obtained the time-domain output light intensities of all three sensor channels as they varied with different gestures. (The time-domain signal was automatically collected using the change-point-finder algorithm or the threshold setting method.) To avoid the influence of ambient light, the CMOS was well packaged and shaded. In addition, an algorithm was used to remove the background noise. Subsequently, data consolidation was  achieved by an end-to-end merge of the time-domain data obtained from the three sensors. The consolidated data were then appended to a gesture label and collected in a dataset of integrated time-domain signals containing all gestures and their corresponding gesture labels. Because some degree of random motion is inevitable for a GRW when a person is wearing it, a machine-learning algorithm for support-vector classification (SVC) was introduced to relearn the tester's gestures every time the wearing condition changed. In this study, an SVM classification model was trained using the consolidated database and was subsequently used as a classifier to detect gestures. Once the real-time gesture-related data were collected, the trained support-vector classifier (SVC) was used to predict the real-time data and return the predicted gesture. Figure 5a shows the mechanical signals obtained by our GRW for 12 fundamental gestures. The effects of different gestures on the output intensity are readily visible. The cross-section of the wrist was altered by the gesture-related movement of even a single tendon, and the wearing conditions of the GRW changed accordingly. Even for similar gestures (e.g., Gestures 1 and 2), notable differences were observed in the corresponding timevarying output of NFPSU 2; this can be attributed to the high sensitivity of the NFPSUs. Though the introduction of machine-learning algorithm can effectively solve the inevitable problem of random wristband motion, disturbance in the output of GRW sensors caused by sliding between the sensors and the skin surface may occur during long-term wear, reducing the recognition accuracy. However, the position-independent response of the NFPSU means that the effects of sliding on the results are insignificant. Consequently, a stable output of the NFPSUs during long-term wear is achieved even with very few sensors.
To characterize the adaptability and gesture-recognition accuracy of the GRW, four testers with different physiques were employed (Figure 5b). Each tester performed each gesture 10 times to update the corresponding databases. The corresponding classifiers were obtained by training new SVM classification models with the updated databases and corresponding gesture labels. Figure 5c shows the classifier obtained from Tester 1. Twelve gestures were successfully recognized with an accuracy of 93.2%, which was comparable to or slightly higher than that reported for GRWs with more than five electrical sensors. [16][17][18] To prove the adaptivity of the wristband and save the testers' time, only five representative gestures were included for the other testers. The classifiers obtained from the other testers are shown in Figure 5d-f. The slight fluctuations in the recognition accuracy may be attributed to different physiques. Specifically, the subcutaneous fat of the chubby tester (Tester 2) reduced the degree of finger movement-related deformation, which slightly decreased the recognition accuracy. Nevertheless, the excellent adaptability of the proposed GRW can be seen in the high recognition accuracy (92%-94%) for all the testers, regardless of physique. Besides, calibration is required before each wear and can be performed by repeating each gesture 10 times, which takes about 10 min.
Robotic hands are widely used in modern industry, serving as an efficient method for improving productivity and working conditions. Humanoid robotic hands, which can perform more complicated tasks involving various gestures, have wide application prospects, for example, remote surgical operation, sign-language translation, and virtual/augmented-reality interactions. To demonstrate further the use of the proposed GRW as an HMI device, a robotic hand was used to perform specific movements based on the gesture-recognition results, as shown in Figure 6a. A schematic of the entire experiment is shown in Figure S4, Supporting Information. All four testers, wearing GRWs, controlled the robotic hand through gestures almost in real time (Video S1-S4, Supporting Information). Specifically, there is a time delay of %1 s between the tester's action and the response of the robotic hand in the videos. We attributed the time delay mainly to the robotic hand, which was driven by steering engines, because the response time of the NFPSU and the CMOS-based signal collection system is between 12 and 150 ms, respectively. Additionally, each tester played rock-paper-scissors with the robotic hand (Video S5, Supporting Information). Since the robotic hand recognized the tester's gesture before executing its own move, it always won, as shown in Figure 6b.

Conclusion
In this study, we proposed and demonstrated an opticalnanofiber-enabled GRW for HMI with the assistance of machine   machine-learning algorithm, the three-sensor GRW could successfully recognize 12 hand gestures with a maximum accuracy of 94%. Moreover, the GRW was able to control and play games with a robotic hand, demonstrating its significant potential for use as an immersive HMI terminal. To recognize more gestures with much higher accuracy, one can include more NFPSUs in a GRW or optimize the SVM model. With the assistance of micro/nanofiber angle sensors, [30,31] it is possible for the GRW to recognize wrist-related gestures. The proposed GRW seems to offer a promising solution for application scenarios in virtual/augmented reality and the metaverse requiring gesture recognition, for example, remote control of machines or translation of sign language.
Fabrication: First, the nanofiber was fabricated by heating and stretching the SMF. The uniform diameter and length of the waist area were 800 nm and %1.5 cm, respectively. By accurately measuring the time interval between transmission laser intensity drops during the fiber pulling process, we could precisely determine the time to stop the heating and pulling based on a target diameter. Both the accuracy and standard deviation of diameters can be less than 5 nm for expected micro/nanofiber diameters ranging from 800 to 1300 nm. [41] The stretched nanofiber was connected to an unstretched SMF at both ends by a conical tapered transition region. Subsequently, the PDMS monomer and curing agent were mixed in a ratio of 10:1. After degassing for 30 min, the uncured PDMS was cast on a glass slide, followed by spin-coating (500 rpm, 5 min). The glass slide coated with PDMS was heated at 80°C for 30 min to form a PDMS membrane with a thickness of 200 μm. The thickness can be readily adjusted by controlling the spin-coating speed.
The optical nanofiber was then formed into a U-shaped and embedded between the PDMS membranes. Therefore, a membrane-nanofibermembrane sandwiched optical-nanofiber sensor was realized. To form the soft liquid sac, we cast the degassed PDMS precursor into a custommade mold. The region of the sac contacting the skin surface, referred to as the sac base, was designed to have a thickness of %200 μm. After solidification, the sac was demolded and filled with glycerol solution to ensure its nontoxicity and chemical stability. Subsequently, the liquid-filled sac was sealed using a filmy optical-nanofiber sensor. Finally, the sealed optical-nanofiber sensor with the soft liquid sac was embedded and fixed in a rigid 3D-printed resin shell using ultraviolet solidification glue to complete the assembly of the NFPSU. The detailed process of NFPSU fabrication is shown in Video S6, Supporting Information.
Characterization and Measurement: A motorized force tester (ESM 303, Mark-10 Inc.) was used to control the applied pressure and maintain the operating cycle. A tungsten light source (SLS201L/M, Thorlabs Inc.) and a spectrometer (USB2000þ, Ocean Optics) were used to inject and collect the light signals for the characterization of NFPSU. In the gesturerecognition experiments, an LED (GCI-0604, Daheng Optics) was used as the light source, and a CMOS camera (ov5640, OmniVision Technologies) was used for collecting the output signals. A robotic hand (uHand 2.0, Hiwonder Inc.) controlled by STM 32 was used to perform the gestures recognized.
Experiments involving human subjects were performed with the full, informed consent of the volunteers, who are also the authors of the manuscript.

Supporting Information
Supporting Information is available from the Wiley Online Library or from the author.