Artificial Intelligence‐Enabled Sensing Technologies in the 5G/Internet of Things Era: From Virtual Reality/Augmented Reality to the Digital Twin

With the development of 5G and Internet of Things (IoT), the era of big data‐driven product design is booming. In addition, artificial intelligence (AI) is also emerging and evolving by recent breakthroughs in computing power and software architectures. In this regard, the digital twin, analyzing various sensor data with the help of AI algorithms, has become a cutting‐edge technology that connects the physical and virtual worlds, in which the various sensors are highly desirable to collect environmental information. However, although existing sensor technologies, including cameras, microphones, inertial measurement units, etc., are widely used as sensing elements for various applications, high‐power consumption and battery replacement of them is still a problem. Triboelectric nanogenerators (TENGs) as self‐powered sensors supply a feasible platform for realizing self‐sustainable and low‐power systems. Herein, the recent progress on TENG‐based intelligent systems, that is, wearable electronics, robot‐related systems, and smart homes, followed by prospective future development enabled by sensor fusion technology, is focused on. Finally, how to apply artificial intelligence to the design of intelligent sensor systems for the 5G and IoT era is discussed.

more complex and complete analysis on the gathered data sets than traditional methodologies. [75,76] The accuracy of prediction can be increased by selecting appropriate algorithms, fine tuning algorithm parameters, and combining various types of data from different sensors. Fundamentally, digital-twin based intelligent systems have the potential to revolutionize the way we sense and interact, with applications as diverse as enhanced identity recognition, personalized healthcare monitoring, rehabilitation, robotic control, smart building, and encrypted interactions in VR and AR space. [77][78][79][80] Hence, in this review, we first introduce the approaches for building a digital twin system and the application of digital twin. Then, the state of the art of AI-based systems will be highlighted. Then, we systematically introduce the tactile sensor and e-skin. Next, we summarize the latest TENG-based intelligent system in the different applications, that is, wearable electronics, robotic manipulators, smart homes, and self-sustainable AIoT systems. In the end, conclusions of how to use AI algorithms to analyze various sensor systems, perspectives on the existing challenges of current intelligent systems, and future trends of the digital twin-based system are provided, which give a glimpse of the further development in the 5 G/IoT era.

Digital Twin and Its Application
Taking wearable sensors as an example, the development of wearable sensors ranges from the smart watch proposed in 1975, [81] to smart glasses, [82] smart gloves, [83,84] even smart clothing, [85,86] and many wide-range applications. Therefore, the technological development of VR and AR has evolved from the initial head-mounted display to the possibility of various interactions with the real world. [14,87] As shown in Figure 1, using the smart home as an example, the realization of digital twins always needs three steps. The first step is collecting the information of the physical entities in real space. Then the virtual models can be built in cyber space. Finally, the connection between physical and virtual worlds will be visualized. [88,89] The collection of physical information relies on various sensors to detect the different characteristics, behaviors, and performance of the monitored system when they are under manufacturing, utilization, disposal, and other operations. In VR space, the virtual models will map these sensor data of physical products to reflect their lifecycle process, such as simulation, monitoring, diagnosis, and prediction. On the other hand, the parameters from virtual models will be transmitted and processed to control the real physical products. The virtual models can control and actuate the status and behaviors of the corresponding physical products using many actuators. Such interconnected data between physical data and virtual data experiencing sensor integration, data fusion, and AI analysis will bring about a piece of more specific information for monitoring the design and production. With the aid of AI analytics, machines will automatically generate a more reliable decision of product attributes, machine load, operating status, and fault detection. The digital twins will realize the closed-loop process of the system by building a model in the virtual space through AI and feeding back the digital model to the physical space. This process can continuously collect and accumulate life cycle data and knowledge of physical products through sensors. Therefore, using the dynamic perception, storage, and presentation of the entire sensory data in the digital twin-based intelligent system, the management, tracking, and consistency maintenance of smart home applications can be realized.
Here, for example, we will introduce the four steps for creating a fully functional digital twin from an existing physical scenario. The first step is to build a virtual representation of the physical product for digital twin needs. Hence, the computer-aided 3D modeling technology is highly desired to achieve more comprehensive and more realistic construction of real space. Second, data collected from different sensors need to be analyzed, integrated, and visualized in virtual space. Therefore, advanced data analytics are significant to deal with the abundant sensor data which will be converted to more concrete information for decision-making. In addition, AI algorithms will help analyze the sensor data collected from multiple sources. With the assistance of AI, it is easy to discover the intrinsic relationship of physical changes that cannot be revealed from a single data source. Generally, AI-based methods could visualize the physical changes in a more distinct fashion. In addition, it can also be combined with advanced AI algorithms to enhance the cognitive capabilities of the digital twin (such as reasoning, problem solving, and knowledge representation) and can be used to simulate product behavior in a virtual environment. Therefore, many classification and regression algorithms would help simulate the key behaviors and predict the further trend of physical products in the virtual world. The relevant technologies of sensor, data collection, data analytics, data transmission and security, AI, and simulation are highly desired in this step. The third step is the application of VR technology, whose role is to allow users to experience a more immersive interaction with virtual products in a simulated virtual environment. In recent years, VR technology has been widely used to support product design with virtual digital formats, allowing many VR hardware devices to be directly used in digital twins. Digital twins need to order physical products to perform recommended actions and establish a real-time and secure connection between reality and VR. Based on the recommended command of the digital twin, physical products will have the ability to adaptively adjust their functions, behaviors, and structures in the physical world through various actuators. In the whole process, sensors play a role in perceiving the outside world, and actuators play a role in performing the ideal adjustments required by the digital twins. Finally, another common technology used to build a digital twin is AR technology, aiming to achieve a more interactive experience of a real-world environment by reproducing a part of the product virtually. For example, AR can realize the application of observing the real-time status of products for users, virtual GPS map navigation, etc. VR and AR can finally allow designers and users to easily access from any Internet access point through wireless technology and finally realize the digital twin system.

AI-Based System
Digital twin-based systems require many sensors to detect changes in the target and the interaction with the target. AI can deal with the enormous sensor information from digital twins, which assists the various applications of modern society ranging from wearable devices to smart building. [90][91][92][93] Currently, AI algorithms are used to recognize image objects, transcribe speech into text, match news items, predict user interests, and select relevant results from web searches. Recently, deep learning (DL) has been widely used in various fields as a new type of AI algorithm. DL is a representation learning method with multiple representation levels. It is obtained by combining simple but nonlinear relations, each of which transforms the representation of the original input level into a higher and more abstract level. [94][95][96] After combining enough such transformations, complex functions will be learnt in the neural network structure. For classification tasks, higher-level representations amplify various aspects of input information, which is important for distinguishing and suppressing irrelevant changes. Therefore, DL-based data analysis approaches have made great achievements in image recognition and speech recognition. [97,98] In addition, DL has beaten other machine learning approaches, such as in predicting the activity of the epidemic disease, [99] analyzing particle accelerator data, [100] reconstructing the circuits of human brain, [101] and predicting the mutations of noncoding DNA on gene expression and disease. [102] Natural language processing as a ticklish question for machines, including particularly topic classification, sentiment analysis, question answering, and language translation, has been prompted with promising results by DL technology. [103][104][105][106][107] Benefiting from the assistance of DL, the most popular studies in AI-based systems mainly include the analysis of image information, tactile information, and voice information. [108,109] A few key examples of how AI algorithms optimize and augment the performance of the smart sensor and the smart system are discussed.

Vision Sensor Application
The vision information is experiencing rapid development along with the computation power improvements of graphic processing units (GPUs). The software and computer vision have gone through significant advances in the past decades. [110,111] Computer vision is a combination of concepts, technologies, and ideas in digital image processing, pattern recognition, artificial intelligence, and computer graphics. It has been most widely implemented in many applications, such as object detection, image recognition, and facial emotion decoding. Most of these tasks in computer vision are related to the process of obtaining information from digital images and feature extraction. As a way of emulating the human visual system, computer vision aims to develop an automated machine that can realize the tasks required for visual cognition. [112][113][114][115] In this article, we take a few examples on the recent progress of vision sensors, as shown in Figure 2.
The first popular application of computer vision is object detection, which is a computer technology related to visual computing and image processing. The AI algorithms of object detection can deal with classifying instances of semantic objects of a certain class, such as humans, buildings, or cars, in images and videos. [116][117][118] Furthermore, well-researched domains of object detection include face detection and medical image segmentation. Therefore, the object detection algorithm has been already applied in many areas of society, such as picture retrieval, security, observation, computerized vehicle systems, and machine investigation. For an instance, You Only Look Once (YOLO) is one of the object detection algorithms that uses neural networks to provide real-time indication. YOLO is much popular because of its high speed and accuracy of object detection. [115] YOLO and its updated version have been successfully used in various applications to detect traffic signals, people, parking meters, and animals. In addition, the algorithm based on object detection also can be used to help soft robotics recognize the grasping object. Saxena et al. present a learning algorithm that can build a 3D model of the object from the process of grasping objects. [113] Given two or more images of an object as input, their algorithm can identify several good location points corresponding to the captured object in each image. Then, triangulate the sparse point set to obtain the 3D position that you are trying to capture. In addition to object detection, image-to-image translation is another significant application of computer vision, which is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image using a training set of aligned image pairs. Generative adversarial network (GAN) is one of the algorithms, which is a class of DL frameworks designed by Ian Goodfellow and his colleagues. [119] GAN can learn to generate new data with the same statistics as the training set from the given dataset. For example, Zhu et al. capture the special features of an image collection and figure out how to transform these features into another image collection, all without any paired training examples. [114] With the amazing feats of AI and computer vision technology becoming more common in different industries, the future of computer vision-based intelligence seems to be full of unimaginable outcomes.

Voice Sensor Application
Voice information is also one of the most significant sensor information for intelligent systems. [120][121][122] As a type of speech recognition, speaker recognition has attracted much attention in various applications such as personalized voice assistants, smart home appliances, and biometric authentication. Traditional speaker recognition uses MEMS-based microphones to detect sound by measuring the capacitance between two conductive layers while providing continuous power. [123,124] Therefore, recently, many other approaches show the potential to be the complementary solutions of microphones. [125][126][127][128] For instance, Guo et al. propose a self-powered triboelectric auditory sensor (TAS) as the component of an electronic auditory system for an external hearing aid in intelligent robotic applications. [120] The proposed TAS has a broadband response from 100 Hz to 5 k Hz by the annular and sectorial inner boundary architecture with systematic optimization. TAS shows high performance of music recording and voice recognition in the application of intelligent human-robot interaction. In addition, Han et al. report a flexible piezoelectric acoustic sensor (f-PAS) with a highly sensitive multiresonant frequency band. [129] The f-PAS records the speech waveforms from the multichannel sound inputs. Then, Gaussian mixture model (GMM) is implemented to analyze the multichannel outputs for voice data processing after fast Fourier transform (FFT) and a short-time Fourier transform (STFT). The speaker recognition system exhibits an outstanding accuracy rate of 97.5% with an error rate reduction of 75% compared with that of the reference MEMS microphone. As a result, much more comprehensive voice information and optimal algorithms will help speaker recognition to achieve a high recognition rate. For example, recognizing speakers in a scene involves complex processes such as big data, massive training, speaker distance, and background noise. [130,131] Concretely, these systems rely on running voice classification models to recognize the user's voice and execute user commands, such as automatic speech recognition (ASR) and speech generation based on DL models. [132][133][134] For instance, the ASR systems can recognize and detect who is speaking in a visual scene for one or more speakers, which is an essential frontend for a wide range of multimodal applications such as audio-visual speech recognition. Although complex DL-based algorithms are already utilized to solve these issues, fundamental research based on the sensor of sound collection is still required to enhance. In recent years, voice-based interaction is one of the most effective approaches and is widely used as personal assistants (such as Siri, Google Assistant) of smart phones. In the future, after the integration of the various sensors to collect abundant voice information, the voice-based system will be more intelligent to apply in healthcare monitoring, HMI, robotic control, etc.

Tactile Sensor Application
Tactile sensors measure the information generated by the physical interaction with its environment, such as the biological sensation of skin touch. Tactile sensors can detect the stimulus generated by mechanical stimulation. [135][136][137] Tactile sensor is also an interaction between humans and external objects, which is suitable for detecting contact-based information, such as roughness, texture, temperature, and weight. There are different types of tactile sensors including piezoresistive, piezoelectric, capacitive, and triboelectric sensors. [135,[138][139][140][141][142][143] In recent years, considerable advances in material science have enabled the rapid development of tactile sensors. Plenty of tactile sensors are capable of detecting multidimensional environmental information, in which the data acquired are processed by comprehensive algorithms. [144,145] Kwak et al. introduce a soft, 3D design tactile sensor that is integrated into a thin, flexible battery-free, and wireless system. The system has a built-in temperature sensor, which can be operated directly at the skin-prosthesis interface in a noninvasive and imperceptible manner. [146] Sun et al. develop a TENG tactile (T-TENG) to realize the recognition of the grasped objects using a soft robotic manipulator. The final accuracy of 28 different shapes of objects is 97.143% with the assistance of ML for data processing. [147] In addition, Wang et al. propose an ultrathin and flexible sensor (UFS) with high flexibility, good shape adaptability, and wide-range pressure sensitivity. [148] When demonstrating it on a flexible phone, UFS can be used to unobtrusively monitor fingertip pulse waves, and the measured fingertip pulse can be shown on the phone's flexible screen. In addition to the application of the soft robotic and flexible touch pad, e-skin is becoming a popular research topic of tactile sensing due to its flexibility, wearability, portability, etc.

Electronic Skin and Glove Platforms
Apart from vision and voice bridging the human and digital world, e-skin, as a significant branch of wearable electronics, is also widely adopted to decode sensory information between humans and robots and build a human-centered intelligent system. [149,150] In the past decade, e-skin has experienced rapid progress owing to the development of materials and manufacturing techniques, being more stretchable, integrated, and bionic. [151][152][153][154][155] To mimic laminated human skin with different mechanoreceptors such as tactile, thermal, and feedback sensation, electronic skin is generally multilayered with embedded pressure, temperature, and even feedback units. [156][157][158] While the soft characteristic of e-skin is endowed by stretchable substrates, regarding applications ( Figure 3a), [159][160][161][162][163] e-skin has been explored for human motion sensing and humanoid robotics (e.g., HMI, tactile sensing, and texture recognition). Upon human motion decoding, the glove HMI has unique advantages when compared with e-skin. It provides excellent mechanical strength, effectively avoiding the damage on electronic components during putting on and taking off. In addition, glove can lower the risk of skin inflammation due to good breathability of textiles. [164] As shown in Figure 3a, [83] classical mechanisms including resistive, [165][166][167] capacitive, [168] and inertial measurement unit (IMU) are extensively used to fabricate smart gloves. [169,170] However, the battery is indispensable for the abovementioned glove types in which the signal cannot be triggered with battery absence. In recent years, the emerging self-powered technologies offer new potentials for realizing large-scale battery-less IoT by reducing the power consumption of massive sensor nodes. In addition to being energy sources, self-powered sensors such as triboelectric and piezoelectric nanogenerators enable dynamic sensing, which is complementary to static sensing of resistor and capacitor. Among them, TENG has been attractive owing to less limitation on materials and uncomplicated fabrication. On the other hand, AI can extract many imperceptible features and get an optimized learning outcome. [171] Combining a triboelectric glove with machine learning may give us inspiration for more intelligent applications (Figure 3b).

Self-Powered Glove HMIs
For instance, as shown in Figure 4a, Zhu et al. develop a low-cost 3D-printed glove HMI with triboelectric finger bending sensors, palm sensors for sliding detection, and piezoelectric stimulators for feedback. [84] Projecting hand motions to cyber space, the glove can achieve VR/AR control and human machine collaboration. With further integration with AI, their proposed glovebased system realizes advanced object recognition. In detail, the glove HMI contains three main components for multifunctions. They first show a basic control of mouse in Figure 4b by relying on the multiple-finger bending sensors which are assembled by semi-sphere Ecoflex. The triboelectric signals are generated when human skin contacts or separates with Ecoflex. Different gesture signals are defined as corresponding commands to control the virtual mouse for shopping. Similarly, the palm sensor is located in the center of four-quarter round electrodes. Varied sliding directions could induce output difference between four electrodes ( Figure 4c). In addition, the piezoelectric stimulator helps to provide feedback when touching virtual object in unity by tuning applied voltage (Figure 4

d).
As an alternative for optical/ultrasound and wearable tactile sensor-based 3D object recognition, the proposed glove has a minimalist design with fewer sensors. As provided in Figure 4e, it achieves recognition of six objects facilitated by machine learning with the accuracy of 96.88%. To demonstrate the benefit and practicality of such recognition in medical field, the authors show VR surgical training based on object recognition. As depicted in Figure 4f, the left glove is responsible for operation mode and switching into recognition mode. By switching into the recognition mode using the left index finger, the left glove is then disabled, and the right glove enters the recognition mode to record the signals of grabbing gestures for corresponding recognition of selected tools. In addition, Figure 4f also shows human-humanoid interactions in AR space. Overall, although there are only few sensors, the smart glove still verifies the potential applications in advanced multipurpose manipulation, in a simpler and more intuitive manner.
Except for advanced control and human machine interaction, wearable glove-based gesture recognition also plays a significant role in the care of speech/hearing impaired. For instance, sign language recognition is an indispensable part of the signers' daily communication. However, the majority of population has less understanding of sign language without prior learning, which generates a communication barrier and is unfavorable for the social participation of signers. As an intuitive interface, the glove can measure the hand motions in a comfortable and directcontact manner, which is suitable for sign language recognition. Generally, previous glove solutions mainly depend on amplitude, bandwidth, polarity, or peak number to achieve gesture translation. In this case, only a limited variety of gestures (i.e., number, letter, words) can be recognized, far from the practical need of signers' daily communication that involves sentences. Thus, the integration of AI data analytics could be an optimal choice to realize more comprehensive gesture recognition.
Correspondingly, Wen et al. show a sign language recognition and communication system with the aid of gloves, DL, and VR user interface (Figure 4g). [83] The representative sign gestures along with their corresponding triboelectric signals are shown in Figure 4h. As depicted in Figure 4i, to make AI recognize sentences of sign language, the sentence samples (800 data points per sentence signal) in training dataset are split into word units using a sliding window in the size of 200 data points and sliding step of 50 data points. Then the DL algorithm would identify these word elements first and then inversely reconstruct the original sentences, with the accuracy of 82.81% and 85.58%, respectively (Figure 4j,k). Furthermore, this segmentation method offers new possibilities for new/never-seen sentence recognition. Specifically, recognized word units can be arranged in a new/different order to create new sentences. In the meantime, the DL model will recognize all basic word elements in the new sentence and give a reasonable translation. In such a way, the new sentences that are not included in the training dataset can be recognized. Finally, the recognition results of sentences are projected into a virtual space in which the signer can use their . Tactile based intelligent system. a) Electronic skin and glove platforms for human motion decoding. [83,[159][160][161][162][163][167][168][169] b) With triboelectric and piezoelectric mechanisms, electronic skin and glove are endowed with self-powered capability. Further integrating with artificial intelligence toward more comprehensive applications. [83,84] Figures in panel (a): Tactile sensing. Reproduced with permission. [161] Copyright 2014, IEEE. Texture recognition. Reproduced with permission. [159] Copyright 2021, Wiley-VCH. HMI. Reproduced with permission. [163] Copyright 2014, Spring Nature. Gesture identification. Reproduced with permission. [160] Copyright 2015, IEEE. Electronic skin. Reproduced with permission. [162] Copyright 2020, Wiley-VCH. Glove. Reproduced with permission. [83] Copyright 2021, Springer Nature. Glove based on resistive. Reproduced with permission. [167] Copyright 2016, American Chemical Society. Glove based on capacitive: reproduced with permission. [168] Copyright 2020, IEEE. Glove based on IMU. Reproduced with permission. [169] Copyright 2016, IEEE. Figures in panel (b): Right: Reproduced with permission. [84] Copyright 2020, AAAS, and Left: reproduced with permission. [83] Copyright 2021, Springer Nature.
www.advancedsciencenews.com www.advintellsyst.com www.advancedsciencenews.com www.advintellsyst.com familiar sign language to communicate while the nonsigners directly type in their controlled VR interface (Figure 4i). This advancement of recognizing existing sentences and new sentences improves the practicality of the sign language recognition system, paving a way to lower the communication barrier between signers and nonsigners.

Self-Powered Robotic Interfaces
The progress of AI in recent years has injected impetus into the rapid development of intelligent robots. [172][173][174][175] Compared with those expensive rigid-body industry robots, soft robots, [176][177][178][179][180][181][182] usually made of flexible materials, that is, thermoplastic polyurethanes (TPU) or silicone rubber, have been frequently investigated with the advantages of lightweight, multidegree of freedom, low cost, etc., showing the potential to be further designed as anthropomorphic robots that are applied in the area of assembly and healthcare to enrich the corresponding ecology as well as save the cost. As the medium for soft robots to perceive the external world, sensors with high flexibility and stretchability that can follow the multidegree deformation of soft robots to real time monitor the bending angle or tactile stimuli have been developed recently, such as flexible resistance strain/tactile sensor, [183,184] Hall effect bending sensor, [185] and optical fiber strain sensor. [186,187] However, these sensors still reveal drawbacks in terms of power consumption, environmental influences, sensing range, etc. Thanks to the merits of wide material choices and selfpowered property, the emerging sensing technology based on TENG, which has been widely investigated as low-power sensory systems for wearable scenarios, also shows great compatibility to be utilized for soft robots' deformation/tactile monitoring and provides a new research direction. [188,189] As illustrated in Figure 5a, Lai et al. propose a highly sensitive triboskin made of silicone rubber for self-powered tactile perception. [190] The highly stretchable material endows the sensor with excellent compatibility and makes it be perfectly integrated with the soft gripper to detect different tactile stimuli during grasping tasks, including approaching, grabbing, lifting, and lowering. The constructed triangular microprisms on the sensor surface greatly enhance its sensitivity and make it have good performance in the low-pressure regime (<5 kPa). In addition, the triboskin can be also embedded onto crawling robots for real-time muscle motion and gait perception toward self-sustainable exploration robots. In addition to developing e-skin for soft robots' tactile-related sensory information collection, Chen et al. chose to integrate the TENG sensor inside the chamber of a soft actuator to achieve self-powered configuration sensing (Figure 5b). [191] The continuous deformation of the pneumatic actuator results in the gradual contact/separation of the embedded sponge electrodes and silicone rubber sidewall, thus generating the triboelectric output due to the difference in electron affinity of these two materials. By connecting all the electrodes in parallel, the open-circuit voltage shows good linearity with the actual bending angle of the soft actuator and can be further used to measure the size and weights of grasped objects. Furthermore, the developed sensor can also be utilized to monitor the gestures by programing the pneumatic actuators into different segments to achieve assisting gloves for healthcare applications.
Though the abovementioned works verify the feasibility of TENG-based sensors for soft robot tactile/deformation perception, they mainly focus on one sensory modality. To realize anthropomorphic robots with multimodal sensing capability, a highly integrated sensory system fusing sensors that respond to different kinds of stimuli is needed. [192][193][194] In addition, recent works on machine learning-enabled data analytics demonstrated the capability of artificial intelligence to enrich the sensor functions by automatically extracting the features from the sensor signal. [69,195,196] By bringing this kind of technology into the robotic sensory system, more advanced interactive functions, for example, object recognition, gait analysis, pose estimation, etc., could be realized to implement self-discriminable intelligent robots. Based on these considerations, Sun et al. report a smart soft robotic manipulator enabled by a multifunctional self-powered sensory system, as illustrated in Figure 5c, where two types of TENG sensors and a poly (vinylidene fluoride) (PVDF) pyroelectric sensor are integrated. [147] The gear structural TENG sensor is used to detect the continuous bending motion of the soft actuator. When the pneumatic finger is inflated and deforms, the strip mounted on the shaft of the gear will be stretched to drive the gear to rotate, resulting in the contact and separation movements between the gear teeth and the PTFE film, thus generating the triboelectric peaks. The tactile TENG sensor has distributed electrodes, with a silicone rubber thin film covered on top serving as the tribolayer. When the stimuli occur on the silicone rubber surface, the contact area and the position can be extracted from the generated output amplitude of the corresponding electrode and the voltage ratio between different electrodes, respectively. With a simple three-layer 1D convolutional neural network (CNN) for data fusion, a recognition accuracy of higher than 97% could be achieved for 28 grasped objects based on the features contributed by both two TENG sensors (Figure 5d), which is much higher than that just based on one sensor type, showing the effective complementary effect of multiple modalities. Moreover, selfpowered temperature sensing functions could also be realized by the PVDF sensor based on the pyroelectric mechanism, as shown in Figure 5e. With the temperature information further fused at the decision level, a more comprehensive perception of the grasped items could be achieved by this intelligent soft gripper, showing its great potential to complete complex decisionmaking tasks in virtual shop and other unmanned working scenarios.

Self-Powered Socks
Walking is a fundamental trait that enables people to lead their daily lives and function as productive members of society. A walking gait is characterized by a repetitive sequence of limb motions that propels the body forward while maintaining stability. Then, a typical walking gait is to have one foot forward and the other foot keep the same distance from the first foot. Normal walking gait is a very efficient walking pattern in terms of strength and walking speed, allowing a human to walk comfortably for long periods of time. Furthermore, in addition, a normal gait allows people to easily switch between running and walking, as well as to easily go up and down stairs and change the walking direction and even quickly avoid obstacles. However, some www.advancedsciencenews.com www.advintellsyst.com Figure 5. TENG-based robotic HMIs. a) A flexible/stretchable TENG skin for soft robot tactile perception. Reproduced with permission. [190] Copyright 2018, Wiley-VCH. b) A TENG-based soft actuator for measuring the size and weights of grasped objects. Reproduced with permission. [191] Copyright 2020, Elsevier. c) A TENG-enhanced smart soft robotic manipulator for AIoT virtual shop applications. d) The flow chart for the shape-size-temperature-fused sensory system. e) The real-time system interface with testing object of hot/cold coffee and room temperature/iced canned drink. Figures in panel (c-e): Reproduced with permission. [147] Copyright 2021, Wiley-VCH. patients with neurological or muscular diseases have difficulty maintaining a normal gait, so even if they may have been severely damaged, they still need to work hard to recover and return to a normal gait. [197][198][199] Therefore, it is significant to monitor the pattern of walking gait because of the useful information of healthcare and personal identification. [200][201][202][203][204][205] Meanwhile, the activity of footsteps is one of the major kinetic energy sources from the movement of human body; hence, there are many pieces of research on foot sensors and energy harvesters. [206,207] Therefore, TENG-based devices on foot can integrate the function of energy harvesting and gait analysis to realize the selfpowered system. Currently, many studies have invested a lot of energy to develop insoles with the functionalities of energy harvesting and gait sensing, but in terms of flexibility and wearing comfort, socks are the best choices for gait analysis. [208][209][210][211] In addition, socks can be more widely used in indoor scenes than the insoles, such as wearing socks at home is much more suitable than wearing shoes. [206,212] As shown in Figure 6a, Zhang et al. developed low-cost triboelectric intelligent socks (LTIS) for the collection of gait pattern. [86] LTIS can also wirelessly transmit the sensor signal using the waste energy harvested from low-frequency body motions. The whole system equips with self-powered functionality to deliver the users' information of identity, health status, and activity. As shown in Figure 6b, the sample triboelectric outputs from different participants show the differences of some important features, that is, amplitude, frequency, interval of stages in a gait cycle, etc. Such important information could represent the gait speed, contacting force between feet and shoes, sensors triggering sequence, and the individual walking operation manners. The gait signal from all participants in 500 s is collected when the gait is stable, which is then segmented by a 4 s sliding window to obtain two or three entire dynamic gait cycles. The whole data are divided into the training set, testing set, and validation set to evaluate the performance of AI. Then, the training set will be fed into the training 1D CNN model directly without the complicated preprocessing steps. The 1D CNN is composed of four convolutional layers, four max-pooling layers, and one fully connected layer. The results of predicting the identification of five participants are shown in Figure 6c. The accuracy of training set and validation set can be up to 100% and 98.4%, and the confusion map of testing set shows that 1D CNN method can assist the LTIS to achieve a high identity classification accuracy of 96%. When attaching the pieces of more sensor arrays on three optimized locations of sock, the LTIS is able to collect more comprehensive information from foot motions for a larger group of participants and improve the accuracy performance. As a result, a 13-participant gait dataset achieves a high accuracy of 93.54%. LTIS also gives two simple demonstrations of the future applications of digital twins in smart home and smart classroom. The LTIS-based smart building system realizes the automatic distinction between family members and strangers, and real-time monitoring of indoor activities of family members in a camera-free environment, by projecting the recognition results from real space into virtual space. Figure 6e summarizes the overall framework of the entire digital twin system, including the first stage of identification and the second stage of activity monitoring in real-time VR projection. Among them, LTIS has the possibility to become a solution using DL technology to provide a safer and smarter environment for smart buildings that do not require the support of cameras and microphones. Such wearable intelligent systems will promote the rapid development of digital twins with wearable formats in the future, in which real digital copies of humans can be created in virtual spaces.

Self-Powered Smart Floor
In addition to using smart socks to achieve gait analysis, another common approach is using floor mat-based sensors. [213][214][215][216][217][218] Traditional approaches used for surveillance and identification are usually based on cameras in offices and home areas, but this method can cause serious privacy issues. For instance, laser beam scanning as one of optical methods can better protect people's privacy than camera approaches. [219,220] As one of the most frequent interactive interfaces in our daily life, floor sensors can obtain rich sensory information from human walking through embedded sensors, including judging the user's indoor location, activity status, user identity, entering and leaving the room, etc. [221][222][223] Fall detection, position detection, home lighting automation, and safety monitoring realized by a carpet provide a more convenient and simple solution for elderly care. Among the various sensor mechanisms, the floor mats based on piezoelectric and triboelectric exhibit exceptional self-powered advantages, which will reduce system-level power consumption and potentially achieve self-sustainability. [224][225][226] Ma et al. propose a TENG-based smart carpet sensor fabricated by some singleelectrode flame-retardant TENG yarn. The authors also show the feasibility for the application of indicating the fire escape route in smart homes. [227] Then, Hao et al. report natural wood-based TENG sensors, which can be used as a smart floor system to track movements of human body, and such information can be recorded for further big data analysis. [228] However, the above-reported floor sensors only present the possibility for the application of intelligent systems. They do not use AI algorithms to realize the software part of data analytics, and their demonstration seems to be difficult to realize largearea floor sensing. Therefore, in the aspect of sensor design part, advanced electrode design is very promising to realize a triboelectric floor mat sensor with simple configuration, costeffectiveness, and high reliability. In the aspect of data analysis, it is easier to use advanced AI algorithms for practical large-area floor monitoring and interaction. Hence, Shi et al. propose a DLenabled smart mat (DLES-mat) array integrated with DL analytics to realize a smart home in Figure 7a. [229] The DLES-mat can realize not only position/activity sensing but also individual recognition. As each person's step frequency and foot force are different, when a person walks on the triboelectric DLES-mat array, it can generate a unique output voltage pattern. After the integration of data acquisition and data analysis, the overall structure of the smart floor monitoring system based on DLES-mat is shown in Figure 7b. Triboelectric output signals are generated on the two sensing electrodes due to the contact-separation motions between human steps and triboelectric surface. These generated signals are acquired by the signal acquisition module and built into a whole dataset from ten different users. As a result, the CNN-based DL model provides a high recognition performance www.advancedsciencenews.com www.advintellsyst.com Figure 6. TENG-based sock for gait analysis. a) Schematics of smart sock for the digital twin-based intelligent system. b) The process of data collection and the 4 s sliding window for the segment of dataset. c) Schematics of the process and parameters for constructing the 1D CNN structure. d) The confusion map of the prediction with the gait patterns of five participants. e) The structure of the whole digital twin system, including the first stage of identity recognition, second stage of activity monitoring, and real-time VR. Figures in panel (a-e): Reproduced with permission. [86] Copyright 2020, Spring Nature.  www.advancedsciencenews.com www.advintellsyst.com with the accuracy of 96.0%, as shown in Figure 7c. The trained model can provide a real-time control interface for identifying similar gait patterns. In addition, with the assistance of CNNbased DL model, the smart floor mat system can also recognize the different walking statuses of users, that is, normal walking, fast walking, and running, which is also applicable for the smart monitoring system in a wider environment. Then, the authors demonstrate a digital twin system using DLES-mat. First, a virtual corridor environment is established by measuring the size of actual sensor to imitate the real walking scene. Second, the realtime status of personnel will be reflected on the DLES-mat array, including position sensing through peak detection and individual identification generated from DL prediction. In general, unlike traditional camera-based surveillance that usually involves video shooting, the intelligent floor monitoring system builds a virtual environment through a digital twin, which can better reflect human activities and their identity in the physical world. The digital twin-based intelligent system shows the potential for automation, control, healthcare, and security applications. In addition, Shi et al. present another TENG floor mat sensor fabricated by screen printing, where the schematic diagram of another 4 Â 4 floor mat array includes polyvinyl chloride (PVC) substrate layer, silver (Ag) sensing electrode layer, and polyethylene terephthalate (PET) triboelectric layer, as shown in Figure 7d. [230] The floor mat sensor is composed of reference electrode, coding electrode 1, and coding electrode 2 for position sensing and sheet electrode for gait identification. The voltage output ratio calculated from each electrode can eliminate the humidity disturbance of output from floor mat sensor. As shown in Figure 7e, the entire data set is collected from 20 users and uses the 1D CNN model to demonstrate the versatility and scalability of smart home applications. To better show the performance of the smart floor mat sensor, the authors first randomly select six groups of five persons (group 1-6). The classification accuracy rates are 91.33%, 97.33%, 88.00%, 96.67%, 86.00%, and 93.33%, respectively. The proposed 1DCNN model also can perform the identification of different numbers of users (i.e., 5, 10, 15, and 20). As a result, the classification accuracy decreases slightly with the gradual increase in the number of users in the data set, while the 1D CNN model can maintain a classification accuracy of about 85% for the identification of 20 users, as shown in Figure 7f. In addition, the proposed floor monitoring system can also realize multiuser detection and diagonal walking detection by the modified electrode connection, as shown in Figure 7g. Compared with the first electrode design, the updated electrode connection can be also extended to a larger scale and realize the recognition of two-user walking, which will enable more practical applications in smart buildings. Overall, in these two demonstrations, the smart floor mat monitoring systems show the great potential of relative automatic control and security access in smart building.

Self-Sustainable System
With the numerous self-powered sensors developed in the IoT system, the various monitoring functions with low power consumption can be also achieved in our homes through equipping TENG-based temperature sensors, gas sensors, motion sensors, humidity sensors, etc. [53,[231][232][233][234] Zhang et al. develop a low-cost textile-based TENG (T-TENG) pressure sensor aiming for an AI-Toilet, as shown in Figure 8a. [235] The T-TENG contains four function layers: two conductive textile layers for charge collection and transfer, one nitrile thin film as the positive electrification layer, one silicone rubber film as the negative electrification layer, and the main sensing layer. The silicone rubber layer with inner frustum structures and a spacer is made by Ecoflex material through 3D printing molds, which can achieve different pressure-sensing ranges through modifying the height. With ten T-TENGs placed on a stool, the detailed pressure distributions and characteristics of a user sitting on and standing up from the stool can be comprehensively recorded. However, traditional data analysis method can only obtain shallow information from the collected signals, such as the frequency, hold time, peak gap, etc., which are not enough for complex recognition and advanced monitoring functions. Through introducing the AI technology to analyze the data, such a sensing array is able to distinguish ten different users and at the same time obtain their health status with AI-analyzed stool states images. Figure 8b shows contactless tracking for the motion of elderly and visually impaired people in the home achieved by noncontact TENG sensors. [236] Such a sensor consists of a negatively charged PDMS film and a flexible aluminum sheet, which can generate electron flow when a positively charged object approaches it. Therefore, this noncontact triboelectric sensor can tell the movements of the people in the room through the different output characteristics and at the same time, can act as an anticollision system, fall detector, and indoor position platform for blind and vision impaired persons. Further integrated with AI technology, such sensors can achieve a higher convenience for multiple AIoT smart home applications, acting as a caregiver for users' healthcare monitoring. [28,74] Aiming for the caregiving of elderly and motion-impaired people, the population of which has already exceeded one billion in the world and is still continuously increasing, a multifunctional walking stick has also been proposed, as shown in Figure 8c. [237] The proposed walking stick contains two main functional units, that is, the hybridized unit and the rotational unit. The hybridized unit consists of a top press TENG (P-TENG), a middle EMG, and a bottom rotational TENG (R-TENG), while the rotational unit only consists of the EMG part. The P-TENG contains two aluminum layers, one nitrile layer, and one Ecoflex layer, which can generate various output voltages regarding the applied pressure. The bottom aluminum has been divided into five electrodes which are able to record the entire process of a walking stick when contacting and separating the ground, including the contact point, contact force, contact time, and contact sequence. With DL with the 1D CNN structure to analyze the output from P-TENG, the walking stick is able to distinguish five different movements (stand up, sit down, walk, upstairs, and downstairs), evaluate three different statuses, and identify ten different users. At the same time, the R-TENG can also detect the gait abnormity like the fall down of the user through irregular patterns of output. To reflect the real-time motion status of the user, a virtual environment has been also built, mimicking real life. The output signals of P-TENG and R-TENG are collected by an micro-controller unit (MCU) module and further sent to the computer wirelessly for analysis. Through the DL model, the real-time motion status of the user in the home can be easily obtained and reflected in the virtual environment. When the user falls down in the home, that gait abnormity will also be acquired instantaneously and able to call for immediate help. This caregiving walking stick only monitors the user's motion status as an important well-being indicator to avoid privacy concerns from traditional camera-based Figure 8. Self-sustainable system for application of smart homed. a) A low-cost T-TENG pressure sensor integrated with vision sensor aiming for an AI-Toilet. Reproduced with permission. [235] Copyright 2021, Elsevier. b) A contactless tracking for elderly and visually impaired people in the home achieved by noncontact TENG sensors. Reproduced with permission. [236] Copyright 2021, Elsevier. c) A multifunctional walking stick aiming for the caregiving of elderly and motion-impaired people. Reproduced with permission. [237] Copyright 2021, American Chemical Society. d) Demonstration for the application of indoor monitoring with the caregiving walking stick. e) Schematics of the AIoT-based smart rehabilitation system with gait detection and waist motion capture. Reproduced with permission. [271] Copyright 2021, Wiley-VCH.
www.advancedsciencenews.com www.advintellsyst.com indoor healthcare monitoring. In the meantime, through the linear-to-rotary structure, which can transfer the low-frequency linear motion to high-speed rotation, the two units can efficiently harvest the ultralow frequency of the motion-impaired people. Maximum average power density of 0.595 mW cm À3 under 1 Hz driven frequency and the capability to charge a 4 mF capacitor to 5 V in 8 s has been successfully achieved. The harvested biomechanical energy can serve as the power supply for a self-sustainable IoT system with GPS location tracking and environmental temperature and humidity sensing, achieving omnimonitoring for users. In addition, Zhang and co-workers present a wearable TENGbased sensor for the lower-limb and waist rehabilitation based on the analysis of gait pattern and waist motion signal, as shown in Figure 8d. [230] For gait analysis, they used a TENG-based insole equipped with two triboelectric sensors for the detection of walking pattern, which can get a high accuracy of 98.4% for five different humans by leveraging ML technology. For waist rehabilitation, they used a belt equidistantly sewed with four triboelectric sensors to realize real-time robotic manipulation and immersionenhanced virtual game by the recognition of waist motion. With the above two functions, a lower-limb rehabilitation robot is realized in user recognition and motion monitoring. The lower-limb sensory system performs well in the application of robot and gaming-based rehabilitation, which shows good potential in IoT-based healthcare applications.

Prospects of Future Direction
The trend of integrating AI techniques with various sensor systems has been promising for various application scenarios in the past few years. The combined technologies of AI and the various sensors are transforming the conventional paradigms in many fields, such as healthcare (e.g., including disease diagnosis, physical health monitoring, healthcare tracking), smart wearable HMI, and multiple sensations-based VR/AR applications. [238][239][240][241] From the previous sections, we have already introduced several intelligent systems, especially based on TENG sensing mechanisms, to realize diversified applications in smart homes with the aid of AI analytics. In addition to the classification task as one of supervised learning, the AI applications can also incorporate into the scheme with supervised regression and unsupervised learning including clustering and dimensionality reduction, as shown in Figure 9. The main difference between supervised learning and unsupervised learning is whether the algorithms learn pattern from tagged data or untagged data, that is, whether the output data is given in various forms of labeled data. [129,[242][243][244][245] In this section, we will introduce some main applications of ML/DL analytics. First of all, as one of the common ML algorithms, dimensionality reduction is used to compress input datasets with high multidimensionality from various sensors, into lowerdimensional datasets. Dimensionality reduction is also used as the preprocessing for sensor data to reduce the computational power consumption. Common dimensionality reduction approaches are based on the principal component analysis (PCA), linear discriminant analysis (LDA), non-negative matrix factorization (NMF), and artificial neural network (ANN)-based approaches. [246] In addition, current state of art technologies such as GraphSAGE, [247] Network embeddings, [248] etc., have also made great progress in dimensionality reduction. Then, the resulting data can be used in subsequent data analysis. For instance, Kühner et al. demonstrate the reliable optical detection in the midinfrared spectral range of pure glucose and fructose solutions as well as mixtures of both in aqueous solution utilizing PCA for the extraction of vibrational data. [249] In addition, Zhu et al. report a volatile organic compound (VOCs) recognition system using a plasma-enhanced IR absorption spectroscopy with fast response, accurate quantization, and good selectivity. PCA is utilized to visualize the relationship of different VOCs in the mixture, which demonstrates the feasibility of VOC identification. [250] Second, clustering is the task of dividing the population or data points into a number of groups such that data points in the same groups are more similar to other data points in the same group than those in other groups. Because the training data are not labelled, the clustering results can better express the characteristics of the data itself. During clustering training, the system will learn to cluster the input data based on similar features and predict the relevant features of the new input data based on the association between the new input data and the recognition group. The trained clusters can be used as input in the form of features for further AI methods or to help users evaluate the potential meaning and consequences of the identified features at the data level. [251,252] As shown in Figure 9, Ji et al. propose a TENG-based smart electronic device with good sensitivity in detecting the mechanical inputs during handwriting and they use hierarchical cluster analysis to find the intrinsic relationship between 26 letters' handwriting. [145] In addition, He et al. introduce an unsupervised and annotation-free framework, named as ClusterMap. This algorithm can incorporate the physical location and gene identity of RNAs, formulate the task as a point pattern analysis problem and identify biologically meaningful structures by density peak clustering. [253] Third, regression is a group of algorithms, which can predict the output values based on input features from the data fed in the system. The go-to methodology is the algorithm that builds a model on the features of training data and uses the model to predict the value for new data. The regression algorithm is a type of algorithm for estimating or predicting continuous output variables. It trains the input data and the assigned output data to the AI system to obtain a known response to each input variable. Regression problems range from simple statistical regression techniques to complex DL techniques. The ultimate goal is to use various AI methods to make predictions in highly complex systems, such as predicting machine failures, sensor service life, gas concentration changes, and so on. [254][255][256] For instance, Wertheim et al. use the regression analysis to estimate the effect of viral load on clustering to provide an approximation of the impact of higher viral load on transmission fitness in the reconstructed HIV genetic transmission clusters. [257] Angenent-Mari et al. use multilayer perceptron (MLP) to build a comprehensive regression model to predict toehold switch function as a canonical riboswitch model in synthetic biology. [258] Finally, the most wide-range application of AI is classification task. Like in the aforementioned section, using AI algorithms to recognize grasped objects, translate hand gestures, predict gait www.advancedsciencenews.com www.advintellsyst.com movements, and determine identity are all classification tasks in the application of wearable electronics. Common ML-based classification algorithms are support vector machine (SVM), naïve bayes, k-nearest neighbor, decision tree, random forest (RF), etc. Based on these algorithms, the AI can also help to build many intelligent systems for various applications, such as disease diagnosis, chemical classifier, speech recognition, etc. [259][260][261] In addition to ML-based algorithms, DL-based algorithms, such as CNN, recurrent neural network (RNN), GAN, deep belief network (DBN), etc., can deal with more complicated classification tasks because they are inspired by the way the human brain processes information. [262]  (from left to right) Reproduced with permission. [249] Copyright 2019, American Chemical Society. Reproduced with permission. [250] Copyright 2020, American Chemical Society. Clustering: (from left to right) Reproduced with permission. [145] Copyright 2019, Wiley-VCH. Reproduced with permission. [253] Copyright 2021, Spring Nature. Regression: (from left to right) Reproduced with permission. [257] Copyright 2019, Spring Nature. Reproduced with permission. [258] Copyright 2020, Spring Nature. Classification: (top) Reproduced with permission. [261] Copyright 2021, Spring Nature. Classification: (bottom: from left to right) Reproduced with permission. [260] Copyright 2021, Wiley-VCH. Reproduced with permission. [259] Copyright 2021, American Chemical Society. Deep learning: Reproduced with permission. [262] Copyright 2020, Spring Nature.

Concluding Remarks
Due to recent breakthroughs in computing power, artificial intelligence technology, and software architecture, intelligent systems are undergoing rapid development. This makes it possible to better create connections between physical products and virtual products, and it will be also more conducive to the applicability of digital twins in various activities in industry and manufacturing. Various sensors play more and more important roles in digital twin applications. The TENG-based intelligent system provides a new possibility for realizing low-power/self-sustainable systems, which will be better applied on the digital twin. In addition, the feature direction of intelligent system relies on the sensor fusion with multimodality data. With the evolving AI technologies, more advanced sensors could be realized toward intelligent systems ranging from wearable sensors, for example, glove, sock, touchpad, exoskeleton, electronic skin, to the comprehensive applications of the healthcare, robot perception, smart homes, etc. Intelligent systems play indispensable roles on diversified occasions. The application of AI will bring process automation and process optimization to a new level, due to its ability for inaccurate feature classification and prediction of sensor data, efficient processing of massive and dimensional datasets, and multimodal physiological signal analysis. Intelligent systems also make it possible to predict the behavior of highly complex production systems. On the other hand, digital twin-based intelligent systems will promote and fascinate the smart future. The intrinsic properties make digital twins one of the fundamental building blocks of the metaverse. [263,264] Therefore, the metaverse may be able to bring us virtual worlds and experiences beyond imagination. In professional settings, a meeting in the metaverse would be more productive if participants could interact with an exact replica of the information system. In the future, the digital twin can recreate a complete smart home and factory in the metaverse, filled with digital machines to be serviced, all the necessary tools, and potentially connected with or mapped onto a real home and factory so that any kind of maintenance can be carried out remotely. Furthermore, the digital twin of the entire factory can be designed automatically, almost without manual intervention, which will greatly increase human resources, thereby reducing costs, reducing resource consumption, and creating new applications. In the future, digital twin-based intelligent systems will still face the challenge of enhancing algorithms or equipment or both in large-scale practical applications.