High‐Density Force and Temperature Sensing Skin Using Micropillar Array with Image Sensor

Driving toward the goal of gaining a high level of intelligence and agility that mimics or surpasses that of humans, sensing systems have been widely investigated. As a complex network, tactile sense converts environmental stimuli into electrical impulses through various sensory receptors, which has been exploited in a large number of revolutionary applications, including robotics, prosthetics, and health‐monitoring devices. However, it remains significantly difficult to mimic all the functionalities of human skin. Herein, a machine tactile sensing system is proposed based on machine vision, which is commonly referred to as “electronic skin” or “e‐skin.” With a high density of 625 sensing points per square centimeter similar to that of human skin, the proposed sensing system can successfully measure 3D force and temperature distribution simultaneously. Based on this information, the shape, weight, texture, stiffness, and viscosity of objects can be obtained, comprehensively mimicking the human tactile system. Moreover, the experimental results show that the proposed e‐skin achieves excellent repeatability, reproducibility, and stability compared to those based on other principles such as the piezoresistive effect and capacitive effect.


Introduction
The investigation of sensing systems is generally driving toward obtaining a high level of intelligence that mimics or surpasses that of humans. Although impressive progress has been made in computer vision, speech recognition, etc., thus enabling machine vision, hearing, and to a large degree, the remaining three senses of touch, smell, and taste are still lagging far behind compared to that of humans. A tactile sensing system is a complex network that converts environmental stimuli into electrical impulses using various sensory receptors (e.g., mechanoreceptors, thermoreceptors, nociceptors) and transmits these signals via neural pathways, enabling the sensations of pressure, temperature, shape, texture, softness/stiffness, and punching. Electronic skin (e-skin) [1][2][3][4][5][6][7] has been developed to mimic the human somatosensory system by detecting and quantifying various stimuli in the ambient environment. It has attracted tremendous attention due to its revolutionary application in robotics, [8,9] prosthetics, [4,10,11] and health-monitoring devices. [3,12] According to the information perceived by the human tactile system, three layers are abstracted and modeled for tactile sensing ( Figure 1) in this article. The first layer is the sensing perception layer including pressure force sensing, shear force sensing, and temperature sensing. The force sensors must be an array of a certain density, which is necessary in some common tactile sensations, for example, the sensing of shape. The second layer is the object property layer, including shape, softness, texture, adhesion, weight, and heat/cold. The third layer is the recognition layer, which indicates the function of recognizing an object. Thus, tactile sensations include multiple sensing in addition to pressure. To sense multiple stimuli, multiple kinds of sensors and sometimes sensor arrays of certain high density are required. For instance, 3D shape sensing needs a 3D force sensor array. These increase the difficulty of mimicking all the functionalities of human skin. To date, some devices with limited functions have been demonstrated, [13][14][15] as described subsequently.
The integration of individual pressure and temperature sensors on one device is a common approach for achieving dual-parameter-sensing functionality. [10,16,17] Someya et al. [16] demonstrated conformable, flexible, and large-area networks of pressure and temperature sensors with organic field-effect transistor (OFET) active matrices. Hua et al. [10] presented skin-inspired highly stretchable and conformable matrix DOI: 10.1002/aisy.202000280 Driving toward the goal of gaining a high level of intelligence and agility that mimics or surpasses that of humans, sensing systems have been widely investigated. As a complex network, tactile sense converts environmental stimuli into electrical impulses through various sensory receptors, which has been exploited in a large number of revolutionary applications, including robotics, prosthetics, and health-monitoring devices. However, it remains significantly difficult to mimic all the functionalities of human skin. Herein, a machine tactile sensing system is proposed based on machine vision, which is commonly referred to as "electronic skin" or "e-skin." With a high density of 625 sensing points per square centimeter similar to that of human skin, the proposed sensing system can successfully measure 3D force and temperature distribution simultaneously. Based on this information, the shape, weight, texture, stiffness, and viscosity of objects can be obtained, comprehensively mimicking the human tactile system. Moreover, the experimental results show that the proposed e-skin achieves excellent repeatability, reproducibility, and stability compared to those based on other principles such as the piezoresistive effect and capacitive effect.
networks for multifunctional sensing. Despite their pioneering works, widespread application of these sensing elements has been limited by the sophisticated fabrication processes involved in fabricating multiple types of functional devices and the fragile properties. A multifunction electronic skin based on thermosensation was developed by Zhao and Zhu. [18] It perceives temperature and pressure stimuli. Thermosensitive platinum (Pt) films deposited on a flexible polyimide substrate are used and the fabricating steps are very simple. However, the pixel is sparse and only 3 Â 3 pixels were fabricated in a 4 cm Â 3 cm polyimide substrate.
An alternative approach is making bimodal sensors that transduce different input stimuli into a coupled signal within a single device. [19][20][21] Taking advantage of the integration of piezopyroelectric and piezothermoresistive materials in organic transistors, simultaneous detection of pressure and temperature has been achieved in an OFET via an exquisite combination of an AC (alternating current) gate bias operation and a currentdecoupling analysis. However, highly sensitive detection of different stimuli has not been realized, which is due in part to the multiple-signal interference in the decoupling analysis. [22] A solution to minimize signal interference is using dualparameter sensors that transduce different stimuli into separated signals. Taking advantage of independent thermoelectric and piezoresistive effects in a single microstructure frame supported organic thermoelectric (MFSOTE) device, simultaneous monitoring of temperature and pressure is realized by transducing external stimuli into separate electrical signals. However, 3D force measurement based on this method is difficult to integrate. [13,14,17] Several kinds of sensors for individual 3D force measurement based on embedded multiple capacitors [23][24][25] have been reported. However, the temperature sensor was hard to integrate. Moreover, the coupling between shear force and pressure was also very large. Sensitivity, especially sensitivity to shear force, is relatively low for these sensors.
From the aforementioned, we can see that challenges still remain for an e-skin to mimic all the sensing functions of real human skin shown in Figure 1. It is relatively simple to implement some of them separately. The realization of all functions, including pressure sensing, shear force sensing, temperature sensing, and sensing in a high density, is rather difficult.
In this study, a novel high-density (625 sensing points per square centimeter) machine tactile sensor based on machine vision is demonstrated. The sensor contains a 3D force-sensing array and a temperature-sensing array. The sensor mainly consists of three parts: 1) a transparent flexible layer integrating a micropillar array with a temperature liquid-crystal (TLC) composite working as the dermis with 3D force-sensing nerves and temperature-sensing nerves; 2) an image sensor working similarly to the peripheral nerve bundle of human skin; and 3) a white membrane working as the epidermis, which can protect the nerves from ambient light and mechanical damage. The 3D force distribution and temperature distribution can be obtained from monitoring the deformation of the micropillars and color of the TLC composite. Due to the fast development of the machine vision, the information can be processed in a fast and real-time manner. The sensor possesses excellent stability and hardly drifts, which is very important for long-term use. Because of machine vision, complex electrical wiring is avoided for a tactile sensor with high density. It also can avoid electromagnetic disturbance. The sensor can almost mimic all the functions, as shown in Figure 1, of real human skin. Based on the information, an object will be finally identified, which is the ultimate goal of a smart machine with tactile sensing. Moreover, due to the low cost, stability, and multifunctionality of the sensor, it could be massively manufactured and used in smart robots.

Principle and Structure
To realize all the functions of skin, the machine vision-based tactile sensor mainly consists of three parts-namely, a micropillar array with a TLC composite; an image sensor; and a white membrane-as shown in Figure 2A. The pillars work as the 3D force-sensing nerves, as shown in Figure 2B. The radius of the micropillar will expand when a pressure is applied on the tactile sensor, while the micropillar will tilt when a shear force is applied. The image sensor can capture the deformation of the micropillars through the transparent substrate. The 3D force is then measured based on the captured images using an image-processing algorithm. To measure the temperature, a TLC composite, which changes colors with temperature, was added on the pillars. The temperature distribution is then obtained by monitoring the color distribution on the top of pillars through the transparent pillar and substrate. Thus, the TLC composite works similarly to electric temperature nerves. To avoid disturbance from ambient light and mechanical damages, a thin white flexible layer of 0.2 mm was added, which is similar to the epidermis of human skin. All the image information is captured through the image sensor, which is similar to the peripheral nerve bundle of human skin. Light emitting diodes (LEDs) were added to offer sufficient and stable light for sensing images.

Simulation
The deformations of the micropillars were simulated with different pressures (0, 1, 2, and 3 mN) and shear forces (0, 0.02, 0.04, and 0.06 mN) to study the relationship between them. The www.advancedsciencenews.com www.advintellsyst.com simulation results are shown in Figure 3A. The results in the same row share the same shear force and the results in the same column share the same pressure. It can be seen that under different 3D forces, obvious deformation occurs in the micropillars. Several parameters are defined to quantify the deformation. The expansion is defined as the increments of the radius of the section. The expansion ratio K is defined as the ratio of the pillar radii after and before applying pressure, and the expansion is the absolute increase in the radius of the pillar. The offset is defined as the displacement of the geometric center of the micropillar cross-section, and the relative offset is the ratio of the offset and radius. The offset and relative offset are used to indicate the magnitude of the micropillars by default. The curves showing the relationship between the expansion rate and the pressure under different shear forces were extracted from the simulated results, as shown in Figure 3B. We can see  www.advancedsciencenews.com www.advintellsyst.com that the expansion rate has a linear relationship with the pressure force. The curves under different shear forces almost overlap, which means that shear forces do not have any effect on the expansion rate. Thus, the expansion rate or expansion is an ideal parameter for quantifying the pressure force without any coupling of the shear force. We also plotted the curves showing the relationship between the relative offset and the shear force under different pressures and got the same conclusion ( Figure 3C). Thus, the 3D force can be fully decoupled using the machine vision-based method; however, it is not an easy task for resistive and capacitive 3D force sensors. [23,24,26] The shear force and pressure can be differentiated by the resistive and capacitive force sensors on the condition that one of them is fixed. If varied tangential force and varied normal pressure were applied simultaneously, the values of the forces could not be accurately obtained.

Optimization Experiment
Obviously, the micropillars with different radii and heights have different sensitivities to 3D force. The relation between them was further simulated using finite element analysis. The relationship between the expansion and pressure as well as the relationship between the offset and shear force using micropillars with different parameters was simulated; the results are shown in Figure 4. The sensitivity is defined as the slope of the output characteristic curves, such as those shown in Figure 4A,B,D,E,G,H. Figure 4A-C shows that sensitivity to both pressure and shear force increases with decrease of the radius. It also can be seen that the sensitivity to shear force increases more rapidly than that to pressure. Although we can benefit from a higher sensitivity, the radius cannot be too small. Because a small radius can cause a small measure range, and the expansion limit will be too small to be detected by an image sensor if the measuring range is too small. Considering both the sensing array of a real tactile sensor (100 sensing points per square centimeter) and the detection limit, the radius was set to 200 μm. The interval is 100-200 μm to offer enough space for tilting and expanding. This leads to a density of 625 sensing points per square centimeter on the e-skin, which is much higher than that of human finger skin (%100 sensing points per square centimeter). In addition, the influences of the height of the micropillar were studied. The results are shown in Figure 4D-F. The pressure sensitivity of the sensor with micropillars of different heights does not change, but the sensor with higher micropillars possesses a higher sensitivity for shear force. So, the sensitivity of the shear www.advancedsciencenews.com www.advintellsyst.com force can be tuned by changing the height and radius of the micropillar according to application scenarios. However, the tuning pressure sensitivity is limited because the radius is limited to a small range. An alternative solution is to change the Young modulus of the micropillar. Thus, the influence of the Young modulus of the micropillar on pressure sensitivity and shear force sensitivity was studied. The output characteristics of the 3D force sensor with materials of different Young's moduli (550, 750, and 950 κ) are shown in Figure 4G-I. The results indicate that if the stiffness of the material is decreased, the sensitivity to both pressure and shear force is increased, and the increments are the same. The Young's modulus can be tuned by changing the material or using composite materials with tunable stiffness in practice. [27] 3. Characteristics of the Sensor

Output Characteristics of the 3D Sensor for Pressure and Shear Force
To achieve accurate calibration, we first calibrated the output characteristics of the 3D sensor for pressure and shear force using a microscope. The pressure-sensing performance of the sensor was tested using different precise weights. The testing equipment is shown in Figure 5A. The deformations, including expansion and tilt, were monitored by a microscope with a magnification of 4. The images of the micropillars under different pressure forces (0, 0.92, 1.84, 2.75, and 3.67 mN for one point) are shown in Figure 5B. The relationship between the expansion and pressure force was plotted ( Figure 5C) and we can see that as the pressure increases, the expansion increases linearly, which is in accordance with the simulated results. The sensitivity to pressure force was %0.063. The linear measurement range of pressure force is from 0 to 8 mN. And the resolution of pressure force is 0.18 mN.  Figure 6B, and the relative offset changes against the shear force were plotted ( Figure 6C). We can see that it has an excellent sensitivity of 0.102 when measuring shear forces. The linear measurement range of shear force is from 0 to 4 mN. And the resolution of shear force is 0.048 mN. In the following practical experiments, a portable image sensor and an image processing algorithm were used   for directly and automatically outputting the 3D force. The algorithm is supplied in the Supporting Information in detail.

Sensor Durability
Long-term stability is very important for a sensor, yet challenging to obtain. To assess long-term stability, we used a material testing machine to apply over 10 000 loading-unloading cycles of periodic pressure and shear force (at a frequency of 2.7 Hz) under a pressure of 2 mN for one pillar. The change of the pillar diameter is shown in Figure 7A. It shows that the pillar diameter achieves reasonable changes (within 0.3%), which indicates excellent reproducibility and reliability for pressure force. The relative offset of the pillar diameter is shown in Figure 7B. With smaller than 0.5% changes in relative offset, the sensor shows excellent reproducibility and reliability for shear force. Thus, the proposed e-skin possesses good stability, which is also very important for commercialization.

Temperature Stability Test
To evaluate the effect of the temperature on the zero point drift and sensitivity of the sensor, the device was subjected to different pressures, and meanwhile its strain was monitored using the material testing machine over the temperatures of 25, 35, 45, 55, and 65 C, respectively. The zero drift was so tiny that we could hardly measure it over the temperature range from 25 to 65 C tested using a microscope. The sensitivity change over different temperatures was also investigated, as shown in Figure 8. As can be seen, the sensitivity hardly changes until the temperature reaches 45 C, and the increment is less than 4% for a temperature below 65 C.

Temperature Sensing
Calibration of the colorimetric temperature sensor was conducted by a continuous heating process in a temperature chamber over the temperature range 28-43 C with an accuracy of AE0.1 C. The e-skin was first attached to the image sensor in the temperature chamber and then heated from 28 to 43 C, followed by cooling to 28 C. Images of the e-skin were captured at different temperatures, and the average red-green-blue (RGB) value against temperature was plotted in Figure 9A. The colors of the micropillars at different temperatures are listed under the curve correspondingly. The RGB value has an obvious blueshift with increasing temperature. The change of the RGB value shows great repeatability. However, the temperature-sensing layer may not be uniform due to the fabrication process. We listed 50 points of colors for each temperature from 28 to 43 C at an interval of 1 C in the RGB space as training samples ( Figure 9B). The K-nearest-neighbor (KNN) algorithm was used for calculating the temperature from the newly measured data. The temperature measurement resolution is 1 C. Thus, its detection accuracy is AE0.5 C. We further verified the calculating method using temperatures of 28-42 C with an interval of 2 C. A shown in Figure 9C, the temperature can be measured accurately based on the KNN algorithm. As a result, it is possible to measure the temperature distribution using our e-skin. As shown in Figure 10A, when a paper clip heated on a hot plate at 45 C was placed on the proposed e-skin, the colors of the micropillars changed quickly. Figure 10B shows the temperature distribution extracted from the color. The region near the clip tip was heated up to around 40 C, while the surrounding region heated up slightly. We then removed the paper clip, let the e-skin cool to room temperature, and placed the e-skin on a hot plate at 45 C. After 5 min of the e-skin reaching  www.advancedsciencenews.com www.advintellsyst.com 45 C, the cooled paper clip was again placed on the e-skin ( Figure 10C). Obvious color change was observed, which indicates a temperature decrease near the tip of the paper clip ( Figure 10D).

Pressure Mapping
Pressure mapping is the foundation of shape sensing for the eskin. To demonstrate the pressure mapping functionality of the integrated e-skin, the e-skin was used to touch a triangular prism and a quadrangular prism with side lengths of 1 mm, respectively. The shape of the section can be clearly mapped using the e-skin, as shown in Figure 11. This indicates that the e-skin can map the contact area pressure with a high resolution of (625 pixels cm À2 ).

Object Weight Sensing
The ability to measure and discriminate between normal and shear forces in real time is necessary to provide the weight information of a grabbed object, [7] which allows a robot to intelligently apply grasping force accordingly to ensure that the object does not slip or get damaged. As shown in Figure 12A, we used a robotic hand equipped with the e-skin to grab objects of different weights under the same grabbing pressure. The shear forces were monitored and the results are shown in Figure 12B. It shows that the shear force increases with the weight of the grabbed object. Thus, our proposed e-skin is effective for object weight sensing.

Stiffness Distinguishing
Distinguishing the stiffness of an object is one of the key functions of human skin. We demonstrated the stiffness-distinguishing function of the proposed e-skin using a robotic hand.
In this experiment, a sponge, polydimethylsiloxane cube, and iron cube were grabbed by the robotic hand. During the grabbing, the pressure was monitored, and the pressed displacement given by control instructions was recorded simultaneously. The curve of the pressure against the displacement was plotted, as shown in Figure 13A. The stiffness index was defined to quantify the inability to be forced out of shape; its value is calculated by the ratio of the applied pressure and the corresponding pressed displacement in the range 0-0.5 mm using the least squares method ( Figure 13B). The results show that the e-skin can feel the relative stiffness of an object.

Texture Distinguishing
The 3D force-sensing function also facilitates the distinguishing of surface textures. To verify this, the e-skin was attached to a microstage and scanned over surface textures at different scanning speeds, as shown in Figure 14A. Three types of surface textures with different distances between two peaks were first tested using the e-skin at a speed of 1 mm s À1 . The distances between the two peaks of the scanned surface were 2, 1.5, and 1 mm, respectively. The shear force was monitored, and one typical point of the shear force was transformed to the frequency    Figure 14B. The base frequency indicated using a red arrow in each figure increases with decrease of the distance between the two peaks. Then, we doubled the speed when scanning the triangular peaks at a 250 μm interval; the result is shown in Figure 14C. The base frequency was doubled accordingly.
In addition, distinguishing of surface textures can be conducted in static mode instead of scanning using our e-skin. This principle is shown in Figure 15, where a texture was attached to the e-skin. The micropillar tilted due to its high sensitivity for shear force when the humps of the texture were pressed on. Fingerprints can be easily mapped using this method even when the finger is wet. However, the capacitive fingerprint sensors do not work well for a wet finger.

Adhesion Distinguishing
The adhesion of glue can be distinguished by a real finger. However, previous works on e-skins seldom discussed adhesion distinguishing. Here, we demonstrate an adhesion-distinguishing experiment for the proposed e-skin.
Glue was first poured onto a fixed workbench. After 30 s, we pressed the e-skin onto the glue. The e-skin was then pulled up, and the change of the section radius of the micropillar was monitored. Figure 16 shows that the radius decreased during the pulling, which indicates that the micropillar was under tension because the top of the micropillar was stuck to the workbench. Continuing to pull the e-skin, the top of the micropillar suddenly got released from the workbench; thus, the radius was recovered.

Arterial Pulse Wave Sensing
During a period of diastole and contraction of the heart, human blood generates pulse waves that propagate along the walls of arterial blood vessels. The pulse contains rich physiological information such as heart disease, respiratory disease, and degree of vascular sclerosis. [28,29] In traditional Chinese medicine, it is a useful symptom to diagnose a disease. [18] Sampling and analysis of the pulse wave signals are practical tasks. Sampling the pressure of a pulse in an array with a high density can provide more information for further analysis. Our e-skin could be used to map the signal of a pulse. In this article, the arterial pulse at the wrist was measured. The mapped results of the arterial pulse at   Figure 16. Detection of the adhesion of glue using the e-skin.
www.advancedsciencenews.com www.advintellsyst.com different times during one cycle are shown in Figure 17. We put the relationship of the average force of the tactile sensor with time in the upper left corner of the figure. We can see that the period of the arterial pulse was 0.64 s, and the time when the pressure reaches the maximum point is 0.32 s. It indicates that the proposed e-skin could be used in a smart arterial pulsation diagnostic instrument or a health-monitoring device.

Conclusion
In this article, a novel machine tactile system with 3D force-and temperature-sensing arrays was proposed based on machine vision. Similar to human skin, the proposed tactile system achieves a high density of 625 sensing points per square centimeter. Specifically, the tactile system consists of three parts: a transparent flexible layer integrating a micropillar array attached with a TLC composite, which acts as the dermis with 3D force-and temperature-sensing nerves, respectively; an image sensor working similarly to the peripheral nerve bundle of human skin; and a white membrane working as the epidermis, which protects the nerves from ambient light and mechanical damage. The 3D force and temperature distributions are obtained from the deformation of the micropillar and the color of the TLC composite on the micropillar (assisted by imageprocessing algorithms). Because of machine vision, complex electrical wiring can be avoided for a tactile sensor with a high density. Electromagnetic disturbance, which is a key problem for resistive sensors, capacitive sensors, piezoelectric sensors, etc., is not a serious problem for our e-skin. The e-skin can mimic almost all the functionalities of human skin. Based on the information, an object will be finally identified by touching, which is the ultimate goal of a smart machine tactile sensing system. As the proposed sensor possesses excellent repeatability, reproducibility, and stability, it could be massively manufactured and used in smart robots.