Convolutional-Generative Adversarial Network: Data-Driven Mechanical Inverse Method for Intelligent Tactile Perception

the key part for tactile robot to perceive object parameters. Tactile robot obtains tactile information by touching objects, and then realizes the perception of structural parameters through AI brain. The proposed algorithm helps to improve the perception ability of AI brain, which could predict structural parameters with high accuracy from incomplete tactile information.


Introduction
Reconstructing structural parameters of a system according to a known limited strain information is a typical mechanical inverse problem. [1]Among the many applications of the inverse problem, tactile perception has attracted a lot of attention from researchers recently. [2,3]Tactile perception plays a key role in physical interaction between human and the environment, and to ensure human's safety and stability. [4,5]8][9][10] Comparing with other perception methods, the mechanical signal extracted by tactile perception is limited, while the sensed structural parameters of the object is complex. [11][14][15] As shown in Figure 1, a conceptual robot of tactile perception consists of sensing and processing systems.Tactile sensors including strain gauges and data acquisition board are integrated into the fingers, which are used to evaluate the features of the target object, such as shape and structural parameters.Researches have shown that the state of the system can be evaluated by limited strain response. [16,17]However, when the features of the target object get more complex, limited strain response will bring difficulties to prediction.What is worse, in the long-term use of the tactile system, possible damage of strain sensors will lead to further loss of strain information. [18,19]The limited strain input results in significantly decreasing the accuracy rate of predicting structural parameters.Therefore, to create robots to perceive, explore, and manipulate their environments, an algorithm to process the tactile information like human needed to be developed, which is also the key factor that limits the development of robotic tactile perception.
For the tactile perception, it would be much more difficult to solve the mechanical inverse problem by utilizing the aforesaid classical methods, such as Tikhonov regularization [20] and TV regularization. [21]The difficulties originate from two aspects.On the one hand, the strain field of structure needs to satisfy the mechanical governing equations and boundary conditions, which is intricate owing to the complexity of the loading and structure under real working conditions.Besides, the shape and pattern of microstructures severely affect the localized spatial relations of the strain data.Therefore, we need to consider both the global and local relations of the mechanical field when predicting structures.On the other hand, the predicted structure contains numerous randomly distributed microstructures, which will engender large categories while using the traditional classification algorithm.Generally, the problem of low prediction accuracy arises when the strain data measured are limited and the categories need to be distinguished are diverse.To overcome the problems caused by above two aspects, a common process DOI: 10.1002/aisy.202100187Inverse problem method uses the results of observations to infer the model parameters of a given system, which can be used in the area of tactile perception.By integrating tactile perception in robotic systems, reconstructing structural parameters of target object can be achieved.However, with insufficient information, how to evaluate complex structural parameters accurately remains a challenge.A data-driven method is proposed for the structural perception based on convolutional-generative adversarial network (CGAN), which can precisely evaluate the structural parameters, notwithstanding missing a large quantity of sampled strains randomly in space domain.The CGAN model has been verified on a reconfigurable structure.Both the numerical calculations and experiments indicate that the structural accuracy rate can reach above 90% in spite of the strain loss ratio being 50%.Through inpainting the observations and discretizing the model parameters, a complete process is proposed to deal with the inverse problem of predicting continuous structure from the incomplete strain, which provides a new solution for applying machine learning method into intelligence tactile robot.
was proposed for solving the mechanical inverse problem in case of incomplete input.The process could be divided into two steps, viz., first to design an inpainting mapping which could map incomplete input to complete input, and then to design a reconstruction mapping which maps the complete input to the output.The feasibility of the effective and accurate method to design the suitable mappings for each step remains a key challenge.
[24][25] However, artificial intelligence in tactile research obviously lags behind other perception methods such as vision and hearing. [26]Here, we proposed a data-driven method based on convolutional-generative adversarial network (CGAN) to evaluate the structural parameters from a small amount of static strain response.The proposed CGAN model consisted of a well-designed generative adversarial network model (GAN) to inpaint strain information and a sophisticated convolutional neural network model (CNN) to predict structural parameters.A loss function was proposed to describe the global and local relations of strain field, and a multilabel classification algorithm has been integrated into the CNN model to improve the accuracy of predicting the structural parameters.The experimental results showed that even for the absence of 50% data points in the strain field, our proposed method could predict the structure with accuracy over 90%, which indicated its excellent generalization ability and accuracy.Our research not only improves the accuracy of sensing structure from strain, but also provides a new solution for applying machine learning method into robotic intelligence tactile and human-computer interaction devices.

Inpainting Incomplete Strain Fields by Training a GAN Model
Based on data generated by finite element method (details are in Supporting Information), we designed a functional GAN model to precisely inpaint the strain fields, which is shown in Figure 2. The GAN model was composed of two parts: one was a generator G, which could inpaint the incomplete strain field, and the other was a discriminator D, which discriminated between the real and fake input data.MaxAbsScaler normalization technique was first used here to rescale the complete strain fields x 0 to x, and all x form a training set p data (x).Subsequently, we randomly removed some data points (set their strain values as 0) of each x and mark it as z, for simulating the incomplete strain information.The percentage of removed data points varied from 0% to 60%.The set p z (z) was the input of G.The inputs of D consisted of the output of G and the complete data x.
We proposed a loss function to minimize the difference between the generated strain field G(z) and the complete strain field x min G E z∼p z ðzÞ ½GlobalðGðzÞ, xÞ þ λ ⋅ E z∼p z ðzÞ ½LocalðGðzÞ, xÞ The first part E z∼p z ðzÞ ½GlobalðGðzÞ, xÞ was the original loss function, [27] where GlobalðGðzÞ, xÞ ¼ logð1 À DðGðzÞ, xÞÞ, which guaranteed that G(z) was restricted by governing equations and boundary conditions.The second part λ ⋅ E z∼p z ðzÞ ½LocalðGðzÞ, xÞ was the local content loss that we have designed, where LocalðGðzÞ, xÞ ¼ GðzÞ À x.The second part guaranteed G(z) to satisfy the strain features in the local region to the best possible value.λ was a hyperparameter which governed the similarity between z and G(z).Further details about GAN model can be found in Supporting Information.

Sensing Structural Parameters by Training a CNN Model
For a general mechanical inverse problem, the single label classification algorithm was of low precision while predicting the structure with a large number of randomly distributed defects.Alternatively, multilabel classification algorithm can improve the accuracy under the condition that outputs must be discrete variables.To utilize the multilabel classification algorithm, we first proposed to mesh any structure into the same square unit cells (N D ).Each unit cell could be assigned one of two states, viz., the 1-state, which consists of homogeneous material, or the 0-state, which has a defect.Based on this digital method, the whole structure could be discretized into a binary sequence with N D elements (0/1), i.e., the structure was described by N D discrete unit cells.Thus, the number of variables, which was predicted by the machine learning model, could be reduced from The discrete structure combined with the multilabel classification method significantly reduced the computational complexity while predicting the structure, which is important for improving the accuracy of the solution to this mechanical inverse problem.Here, a structure, as shown in Figure S1, Supporting Information, was taken as an example with 9 Â 9 same square unit cells.Subsequent to discretizing the structure by the method mentioned above, our proposed multilabel classification algorithm combined with CNN model could predict each element of the binary matrix independently, which decreased the number of categories of labels from 2 81 to 81.As shown in Figure 2, the CNN model primarily contained five convolutional neural layers and two fully connected layers.Sigmoid was used as an activate function at the output layer.By adding the discrete structure K (i) to the loss function, we designed a modified binary cross entropy for CNN model as where N D is the number of discretized structural variables, GðzÞ ðiÞ and K (i) represent the input of the strain data and its corresponding structural parameter (0/1 bits), respectively.h θ is a sigmoid function, where θ represents the parameter set of the CNN model.More discussions of data-driven method for the physical inverse problem and the details about the CNN model could be found in Supporting Information.

Inpainting Strain Information with High Precision
First, the trained GAN model was used to inpaint incomplete strain fields.We defined a mean relative error (MRE) P N i¼1 j GðzÞÀx x j=N to quantify the precision of the inpainted complete strain fields from the GAN model, and the MREs of a single example and the test set were calculated as follows.Figure 3a shows a specific example, where 10 data points have been randomly removed, i.e., strain loss ratio is 10%.Our method could inpaint the incomplete strain field with a mean relative error of 3.6% and a maximal relative error of 7.8%.In contrast, the mean relative error of the strain inpainted by the linear interpolation method was 11.1%, and maximal relative error was 25.9%.For the strain field shown in Figure 3b, when the strain loss ratio was less than 30%, our method could inpaint the strain field with MRE less than 6.8%.The numerous missing strains made the linear interpolation method invalid because the remaining strains were not sufficient for parameters fitting.Therefore, our proposed method showed an excellent performance with different values of strain loss ratio.Besides the examples shown in Figure 3a,b, the average performances of the GAN model on all the samples in the test set have been conducted, as shown in Figure 3c.We introduced the energy of gray-level co-occurrence matrix (GLCM) to quantify the distribution of the holes in the structure, whereas the number of holes was used to describe the volume fraction of the structure.Scatter plot in Figure 3c shows the MRE of the inpainting strains from GAN model with different structures and strain loss ratios.As the strain loss ratio changed from 10% to 50%, the MRE increased from 3.1% to 7.8%.Even for the case of 50% strains randomly missing, our method could inpaint the strain fields with MRE less than 8%.These results demonstrated that the distribution of holes had negligible effect on MRE, whereas the strain loss ratio was the primary factor affecting MRE.

Predicting Structure from Strain Information with High Accuracy
Then, the trained CGAN was used to predict structural parameters from incomplete strains.A new index-structural accuracy rate (SAR), which was defined as the ratio of the number of precisely predicted cells to the number of all cells in the structure, has been employed to quantify the ability of predicting structures of CGAN model.As shown in Figure 4a, CGAN can predict the distribution of holes in structure when input strain data points were incomplete.As shown in Figure 4b, when the strain loss ratio continued to increase from 0% to 20%, the SAR would decrease gradually from 100% to 96%.Incomplete strain field, i.e., sparse strain field, has been directly used to predict the The strain loss ratio is 10%, and the mean relative error of the inpainted strain by GAN is 3.6%, whereas the strain loss ratio of the linear interpolation method is 11.1%.b) For a same strain field, the strain loss ratio changes from 0% to 100% while the position of the strain loss is random.The relative error of the inpainted strain to the original strain is plotted.c) The test set contains 10 000 different structures and their corresponding strain fields, and relative error of the inpainted strain from different incomplete strains under different loss ratios is plotted.The number of holes is used to describe the volume fraction of the structure, and the GLCM energy is used to describe the distribution of the holes in the structure.
structural parameters, and the SAR is plummeted from 100% to 66%, as shown in the orange scatter plot in Figure 4b.Furthermore, when the strain loss rate was in the range of 0-60%, the SAR from inpainted strain was significantly higher than the incomplete strain.Thus, restoring the missing strain information plays a critical role in evaluating the structural parameters with high accuracy.Besides the above examples, the average SAR on all the samples in the test set is shown in Figure 4c.Even when the strain loss ratio reached 50%, the CGAN model could predict the structure with an average SAR 90%.Furthermore, in the test set, each sample took an average of 7.86 Â 10 À4 s for a process, beginning with the inputting of incomplete strain fields up to the outputting of the structure by CGAN model.The quick response indicated that our proposed method had a huge potential for the applications in real-time parameters predicting of a structure with multiple and randomly distributed microstructures.

The CGAN Model Was Verified on a Reconfigurable Structure
Finally, to verify the proposed mechanical inverse method, we have designed an experimental system, which contained data acquisition board, controller, and the CGAN algorithm.The data acquisition board (DH3816N, Donghua Test, China) has been connected with a strain sensor array (10 Â 10 strain gauges), and the strain gauges were pasted on the back area of the red dotted box, as shown in Figure 5a.CGAN model has been built into a computer, which was utilized for controlling the whole experimental system and processing of data.Corresponding noise reduction algorithms are applied to reduce possible noise in the above procedure, which could be found in Supporting Information.As shown in Figure 5a, to verify the practical generalization ability of the proposed system, a 3D printed reconfigurable structure has been designed and fabricated with photosensitive resin (E ¼ 2460 MPa, ν ¼ 0.23), whose size was 544 mm Â 364 mm Â 9 mm.Inverted trapezoid thread holes have been regularly arranged in the structure (27 Â 27).The cell could be tuned by installing and uninstalling the stud.Based on the tunable cells, the test specimen could be restructured to simulate differently distributed holes.Therefore, multiple tensile tests of the structures with differently distributed holes could be carried out on a single structure.A uniform loading of 1980 N has been applied on the structure by using a stretching machine (WDW-2, Changchun Kexin, China).The strain rate was about 3 Â 10 À5 s À1 .By turning off the associated strain data acquisition board channels, the scenario of losing strain data at specific Figure 4. Accuracy rate of predicting the corresponding structure from incomplete strain is tested by using CGAN model.a) The strain loss ratio is 15%, and GAN model is used to strain inpainting, whereas CNN model is used to predict the corresponding structure from the strain inpainted by GAN.b) For a structure and its strain field, the SAR based on the strain inpainted by GAN and the incomplete strain are tested under different strain loss ratio.c) The test set contains 10 000 different structures and their corresponding strain fields.The number of holes is used to describe the volume fraction of the structure, and the GLCM energy is used to describe the distribution of holes in the structure.The mean SARs of different strains by CGAN model under different strain loss ratio is plotted.
points can be simulated.The SAR of structure (II in Figure 5a) changed with the strain loss ratio as shown in Figure 5b, where a strain field with a loss ratio of 20% was shown in the lower left.When the strain loss ratio increased from 0% to 50%, the SAR decreased from 100% to 90.5%.The SAR of further experimental structures predicted by their corresponding strain fields with loss ratio of 10% is shown in Figure 5c.The average SAR was 97.8%.Additional structures are shown in Figure S5, Supporting Information.Although the 3D printed reconfigurable structure used in the experiment had errors in assembly, our method gave a high accuracy in predicting the structures with different number of randomly distributed holes.Thus, our method was applicable, stable, and of high precision, which shows potential application of CGAN model in tactile robot.

Data-Driven Method for the Physical Inverse Problem
The process to overcome the two problems the mechanical inverse problem with an incomplete input could be divided into two steps, viz., first to design an inpainting mapping which could map incomplete input to complete input, and then to design a reconstruction mapping which maps the input to the output.
The first step could be concluded as a typical mathematical problem of matrix completion, i.e., recovering a matrix x with a matrix x À having incomplete observable elements.For the second step, the common ways to build a neural network from strains to structural parameters include single-label classification and multilabel classification.The single-label classification is suitable for problems with few categories. [28]owever, as the position and number of holes in the predicted structure contain a lot of categories, the single-label classification The 3D printed reconfigurable structure was employed as the experimental test specimen, and the back area of the red dotted box was pasted with strain gauges.Based on the tunable cell, the tunable structure was conducted to obtain the strain responses of different structures.b) The accuracy rate of the predicting structure (II) from its strain field under different loss ratio was plotted.As an example, a strain with a loss ratio of 20% is shown in the lower left.c) The accuracy rate of predicting the corresponding structures from experiment strains whose loss ratio was 10%.
algorithm showed weak performance here. [29]Formally, the multilabel classification was the problem of finding a model that mapped inputs to the discretized binary vectors (assigning a value of 0 or 1 for each element in vectors).Here, to compare the performance of the two algorithms, the corresponding gradient values of loss functions of both single-and multilabel classification algorithms have been calculated.The results showed that in the case of N D > 6, L D still maintained a higher gradient and lower memory requirements compared with the single-label classification algorithm.Finally, the proposed multilabel classification algorithm combined with CNN gives results of high accuracy, which solve the typical mechanical inverse problem of continuous structure with multiple categories.

Potential Application of CGAN in Tactile Robot
The ability of CGAN to predict the structure with high precision is undoubtedly important for tactile perception.For tactile robots, sensors are usually placed on the tips of human-like as shown in Figure 1.The limited number of fingers leads to a limited number of sensors, which further leads to less strain information.Besides, different types of tactile perception such as pressure and vibration may occur in real haptic applications.Our method still works well with high accuracy.Corresponding results could be found in Supporting Information.Generally, complex 3D structures are typically considered as the primary source for tactile perception in robotic systems, but some mechanical problems in 3D structures can be simplified to plane strain problems. [30,31]The proposed mechanical inverse method could also be applied to solve 3D structures.Benefit from the proposed CGAN model, the CGAN model can inpaint strain information with high precision and output structural parameters with high accuracy, solving the problems faced by intelligent tactile robots, such as insufficient input information and low accuracy of predicting structural parameters.The CGAN model can be well integrated in wireless devices and compact devices through online deployment or TensorFlow Lite.Furthermore, the tactile robot could manipulate the structure through reconfigurable components.As an example shown in Figure 5a, the robot first interprets the required structural parameters by CGAN model.Afterward, the robot self-regulates the tunable cell by installing or removing the stud.In this way, with the help of CGAN model, intelligent functions including obtaining structural strain response, sensing structural parameters, and tuning structural pattern could be integrated into tactile robots.In addition, by arranging encrypted and encoded geometric information inside, a structure can be used as a carrier of information.Tactile sensing system can act as a receiver to quickly decode the information carried by the structure.Therefore, the CGAN will develop the research of tactile perception and promote the practical application of tactile robots and human-computer interaction devices.

Conclusion
In conclusion, we have proposed a data-driven method for tactile perception based on CGAN, which could precisely evaluate the structural parameters even for the absence of a large quantity of sampled strains.The CGAN consisted of an inpainting mapping of strain field and a reconstruction mapping of structural parameters.The CGAN model considers both the global and local relations of strain field, possessing high accuracy by utilizing multilabel classification algorithm.Both numerical and experimental results show excellent performance on sensing structure from incomplete strains.The SAR could reach above 90% even for a sampled strain loss ratio of 50%.The proposed method established a standardized process for inverse problems, which can be further applied for the structural inverse prediction and intelligent tactile perception.

Figure 1 .
Figure1.AI brain is the key part for tactile robot to perceive object parameters.Tactile robot obtains tactile information by touching objects, and then realizes the perception of structural parameters through AI brain.The proposed algorithm helps to improve the perception ability of AI brain, which could predict structural parameters with high accuracy from incomplete tactile information.

Figure 2 .
Figure 2.Method to solve mechanical inverse problem of predicting structure from incomplete strains.Generative adversarial networks consisted of a discriminator and a generator, and the well-trained generator could inpaint the incomplete sampled strains to complete strains.Thereafter, the multilabel classification algorithm combined with the convolutional neural network was utilized to predict the structure from the inpainted sampled strains.

Figure 3 .
Figure 3. Strain inpainting tests of different strain loss ratio and different structure are conducted by the GAN model.a)The strain loss ratio is 10%, and the mean relative error of the inpainted strain by GAN is 3.6%, whereas the strain loss ratio of the linear interpolation method is 11.1%.b) For a same strain field, the strain loss ratio changes from 0% to 100% while the position of the strain loss is random.The relative error of the inpainted strain to the original strain is plotted.c) The test set contains 10 000 different structures and their corresponding strain fields, and relative error of the inpainted strain from different incomplete strains under different loss ratios is plotted.The number of holes is used to describe the volume fraction of the structure, and the GLCM energy is used to describe the distribution of the holes in the structure.
To drive GAN model to generate the matrices, a special loss function for generator needed to be designed.When the discriminator could not determine whether the recovered matrix was real or from the generator, the GAN model reached global optimality, which indicated that the global part of the loss function could drive the model to learn global relations.By minimizing the local part of the loss function to 0, G(z) would tend to be similar to x.Therefore, G(z) owned the same local relation as x, which indicated that the local relation could be learnt by the generator of GAN model.The loss function that took into account both the local and global relations could make the GAN model solve the matrix completion of strains.

Figure 5 .
Figure5.Experiments are carried out on a reconfigurable structure to verify CGAN model.a) The 3D printed reconfigurable structure was employed as the experimental test specimen, and the back area of the red dotted box was pasted with strain gauges.Based on the tunable cell, the tunable structure was conducted to obtain the strain responses of different structures.b) The accuracy rate of the predicting structure (II) from its strain field under different loss ratio was plotted.As an example, a strain with a loss ratio of 20% is shown in the lower left.c) The accuracy rate of predicting the corresponding structures from experiment strains whose loss ratio was 10%.