A Highly Sensitive Multimodal Tactile Sensing Module with Planar Structure for Dexterous Manipulation of Robots

Herein, a multimodal tactile sensing module that can improve the dexterous manipulation capabilities of robots with parallel grippers is reported. The tactile sensor consists of a 4 × 4 matrix 3‐axis force sensor array (with spacings of 2 mm) and a single‐temperature sensor. The tactile sensor uses inorganic silicon and gold as materials for the detection of strain and temperature and a polymer‐based elastomer to encapsulate the sensing layer. The sensing module is equipped with a readout circuitry for signal processing. Within the measurable force range (≈1.5 N) of the sensor cell, a prototype module exhibits a repeatability error within 2%, a hysteresis error within 3%, and a resolution as small as 10 mN. Furthermore, each sensor cell independently measures 3‐axis forces with a cross‐talk error of approximately 3%. The temperature sensor exhibits linear output properties in the approximate range of 5–75 °C. Experiments are performed by mounting the module on a parallel gripper to grasp a paper cup and a reflex hammer toy. According to the experimental results, the sensor can accurately detect the state of contact with an object by analyzing three tactile modalities (i.e., 3‐axis force distribution, vibration, and temperature).


Introduction
During the recent pandemic, robots became more important than in the past due to the global effort to minimize physical contact between people, signaling a new turning point in the field of robotics. The global robotics market is rapidly shifting its focus from robots for automated manufacturing purposes to various forms of service robots that can help average consumers in everyday tasks.
For example, robots are used in healthcare facilities to help transport necessary items to patients or medical staff members. [1] An increasing number of robots are also being employed in restaurants to help deliver food from the kitchen to customer tables. [2] However, conventional service robots are limited in the roles that they can perform. This is because robots, unlike humans, struggle to manipulate easily deforming objects with a dynamic center of mass (e.g., blood packs or Hartmann's solution packs), and they also struggle with handling plates or bowls that contain food. When humans handle objects, we simultaneously use our sight and tactile sense, but robots usually only depend on visual sensing capabilities. To overcome this limitation, the idea of mounting tactile sensors on robots is gaining traction as a potential solution. [3][4][5][6][7][8][9][10] Human fingertips have numerous sensory receptors capable of detecting multiple tactile modalities (e.g., pressure distribution, slippage, depth, and thermal conductivity). These receptors allow us to recognize if an object has slipped from our hand, whether the object has been damaged by our grip, and whether the object is hot or cold. Based on these traits, humans are able to handle objects with dexterity. To develop robots that are also capable of dexterous manipulation, it will be important to develop multimodal tactile sensors that mimic the sensing capabilities of our skin. Therefore, many research groups are in the process of developing such sensors.
Gong et al. fabricated a pneumatic tactile sensor using an air bladder. [11] This sensor exhibited low hysteresis error characteristics in the proposed force range and was able to detect object slippage. Also, it could measure the roughness of an object by applying a fingerprint to the sensor surface. However, the sensor is limited in that it is difficult to detect the object's contact location and the slippage direction as the sensor is composed of a single cell, and thus, 3-axis force detection is not possible. Chathuranga et al. demonstrated a tactile sensor capable of 3-axis force detection based on a simple structure that places disks of permanent magnets on Hallsensor-chip arrays distributed inside a fingertip-shaped silicon rubber piece. [12] Nevertheless, due to the sensing principle, it is vulnerable to external magnetic fields. The Pirozzi group developed an array-type optoelectronic tactile sensor based on LED-phototransistor couples. [13,14] The developed sensor is capable of measuring 3-axis force and 3-axis torque, but it has a drawback in that it requires complex modeling and numerical calculation methods to measure external force distributions. The Duchaine group introduced multimodal tactile sensors that can simultaneously detect static and dynamic pressure using the fringe capacitance effect with a device design incorporating micro-structured dielectric and comb-shaped taxels (tactile pixels). [15,16] The modular design, where a circuit board, capacitive sensing arrays, a cover layer, and a metallic protecting case are integrated into one device, allows users to easily install tactile sensors on robotic grippers or replace them. Moreover, the developed design addressed the noise issue of the capacitive driving method by providing a proper electrical shield with covers of conductive materials (e.g., a metallic case and conductive fabric). Despite its merits, these tactile sensors exhibit repeatability and hysteresis errors as high as 10% and 20%, respectively. Moreover, the sensors are highly temperature dependent and are incapable of 3-axis force and temperature detection.
In this study, we present a highly sensitive multimodal tactile sensing module based on a 3-axis force sensor with a hybrid structure that grants the sensor the high-performance measurement capabilities of inorganic silicon-based sensors and the mechanical flexibility and robustness of polymer-based elastomers. This module was also designed so that it could be installed on a robot's parallel grippers. The tactile sensor consists of a 3-axis force sensor array, which has a 4 Â 4 array arrangement of taxels composed of four single-crystal silicon-based strain gauges, and a temperature sensor with a thin, serpentine-shaped gold electrode. Additionally, we designed a circuit board capable of minimizing cross-talk error pertaining to all resistance elements of the tactile sensor for the sensing module. Through a series of experiments, we evaluated the performance (resolution, hysteresis, repeatability, 3-axis force, and temperature detection) of the tactile sensing module. Furthermore, to demonstrate the feasibility of the sensing module, the module was installed on a parallel gripper to perform measurement experiments involving two specific gripping scenarios.

Design of the Multimodal Tactile Sensing Device
Our goal was to develop a tactile sensor with excellent sensitivity and repeatability properties as well as multimodal detecting capabilities. Therefore, our research team implemented a hybrid structure that takes advantage of the excellent and stable detecting properties of inorganic silicon-based sensors and the mechanical flexibility of polymer-based elastomers to design a tactile sensor consisting of an array-type 3-axis force sensor and a temperature sensor. [17][18][19][20] For the 3-axis force sensor, we used a strain gauge-based design as it is less susceptible to external noise compared to capacitive sensors while also being capable of quantitatively measuring forces applied on the sensor. The sensitivity of strain gauge-based sensors is dependent on the gauge factor of the gauge material. The gauge factor is usually in the range of 2-5 for metals, and graphene has an approximate value of 6. [21,22] In the case of single-crystal silicon (which we used in our design), the gauge factor range is approximately 30 to 50 times greater (À120 % 180) than the two aforementioned materials. For this material, the gauge factor value varies depending on the doping material and its concentration: in our design, we achieved a gauge factor of 85 by doping the material with boron at a concentration of 9.0 Â 10 18 ions cm À3 . [19] As shown in Figure 1a, we arranged four horseshoe-shaped strain gauges based on single-crystal silicon nanomembranes in an NESW (north-east-south-west) arrangement separated by 90°to form a single taxel. We then arranged these taxels in a 4 Â 4 array to form an array-type sensor consisting of 16 taxels. To minimize the number of signal lines, we implemented a matrix-type wiring method with a total of 16 signal lines consisting of eight rows (X ) and eight columns (Y ) that are used to readout the resistance readings of 64 strain gauges (Inset [i] in Figure 1a). The taxels were evenly spaced with intervals of 2 mm in the horizontal and vertical directions, and 3-axis forces can be detected by the strain gauges that were arranged in the NESW arrangement in each taxel in combination with the bumps. [23,24] The bumps serve to concentrate loads onto the strain gauges to enable 3-axis force detection. To make it easier to align each bump to the center of each taxel, we created alignment marks that aligned with the corners of the square-shaped base of each bump (Inset [ii] in Figure 1a). Additionally, the 3-axis force sensor array was surrounded by a temperature sensor based on a thin gold strip to enable the tactile sensing module to detect temperature along www.advancedsciencenews.com www.advintellsyst.com with the 3-axis force distribution when it came into contact with the object. Here, we utilized the proportional relationship between the electrical resistance of the gold strip and the temperature. To ensure the temperature sensor experienced minimal resistance changes due to the strain resulting from the substrate deforming, the gold strip was designed with a serpentine shape. The force and temperature sensors are all based on inorganic materials, and thus, they have superior repeatability, lower hysteresis error, and high stability compared to sensors with organic-based detecting materials (e.g., polymer composite sensors). [17,19] However, the disadvantage of inorganic materials is that they are susceptible to damage from impacts. Therefore, we implemented a hybrid structure where the sensing layer (with the 3-axis force sensor and the temperature sensor) is placed in between two layers of a polymer-based elastomer material. Among the various types of polymer-based elastomers, we used polydimethylsiloxane (PDMS, Sylgard 184, Dow Corning, Midland, MI, USA) as it is a relatively superior elastomer material in terms of mechanical rigidity and also because it readily adheres to homogenous substances. The deformation layer, the encapsulation layer, and the bump layer were fabricated with PDMS. As we already described the role of each layer in detail in a previous study, [20] we will focus on other characteristics in this study. In terms of the design, the deformation and bump layers have changed from our previous version. We also adjusted the ratio of the PDMS base and curing agent from the general ratio of 10:1-5:1 to improve the mechanical rigidity of the deformation layer. The deformation layer was formed by pouring liquid PDMS (mixture of the base and curing agent) into the empty space inside the sensor case (W: 20 mm Â L: 30 mm Â t: 1.0 mm) and leaving it to cure. By using a more rigid PDMS material, we were able to fabricate a sensor with a greater force sensing range, superior elasticity, and improved repeatability and hysteresis properties, compared to the one using a PDMS deformation layer with 50:1 mixing ratio in our previous study. [20] We have chosen a PDMS substrate with 5:1 mixing ratio based on stress analysis results. [17] However, this also diminished the sensitivity of the sensor in terms of force sensing. The PDMS deformation layer was placed under the sensing layer and served to produce strains proportional to the force applied on the sensor; with a more rigid deformation layer, more force is required to produce the same strain value. We addressed this issue by designing a bump with an isosceles trapezoidal shape (S: 1 mm Â L: 1.5 mm Â H: 0.5 mm) to concentrate the load onto the strain gauges ( Figure 1b). We also added tabs that can be fixed with M 2.5 bolts on the four corners of the aluminum sensor case (outer dimensions: W: 26 mm Â L: 36 mm Â t: 2.0 mm). This allowed the module to be firmly fixed on a parallel gripper (which is generally used as an end effector) while also making it detachable ( Figure 1c).

Microfabrication of the Tactile Sensing Device with Multimodal Sensing Capabilities
The fabrication process of our sensor device (which has a hybrid structure) can be subdivided into two parts: a sensing layerfabrication process, which involves a microelectromechanical system (MEMS) process, and a casting process where polymer layers like the deformation layer, encapsulation layer, the bump layer are formed. Figure 2 presents a diagram that illustrates the overall process. The sensing layer-fabrication process begins with a dry-transfer process to transfer a 100 nm thick single-crystal silicon nanomembrane, which was obtained from a silicon on insulator (SOI, SOITEC, Isere, France) wafer that was doped with boron to a 25 um thick polyimide (PI, DuPont Kapton) substrate that was attached to a glass substrate using a PDMS stamp (Figure 2a). A total of 64 horseshoe-shaped strain gauges were patterned on the silicon nanomembrane on the PI substrate by patterning photoresists via the photolithography process. Afterward, an etcher (reactive ion etching (RIE) system, LAT Corp., Suwon, Korea) was used to etch out the remaining parts to leave only the gauge pattern ( Figure 2b). Next, we formed a metal layer (Cr: 5.7 nm Au À1 : 120 nm) on the substrate using a thermal evaporator (KVE-T2000, Vacuum Tech, Gimpo-si, Korea). The shapes of the first electrodes (row electrode lines) of the gauge and the temperature sensor were patterned via the photolithography process. The remaining sections of the metal layer were then etched via wet etching, leaving only the shape of the electrodes and the temperature sensor (first metallization process) ( Figure 2c). Prior to forming the secondary electrodes, insulating islands must be formed to ensure that the electrodes are not shorted at points where the first electrodes and second electrodes (column electrode lines) overlap. To this end, we used a Su-82 002 (MicroChem Corporation, MA, USA) to form insulating islands (dimensions: W: 500 μm Â L: 500 μm Â t: 2 μm) in regions where the first and second electrodes intersect (Figure 2d). To fabricate the second electrodes, the photoresists were coated and then patterned in the shape of electrodes via the photolithography process, and a metal layer (Cr: 5.7 nm Au À1 : 200 nm) was deposited on the electrode using a thermal evaporator. Following the deposition process, a liftoff process was employed to obtain the metal layer in the shape of the second electrodes, which concluded the sensing layer-fabrication process (Figure 2e). The encapsulation layer was formed by pouring 10:1 PDMS on the sensing layer, performing spin-coating (4000 rpm for 30 s), then curing in an oven at 70°C for 2 h (Figure 2f ). The bump layer was formed by pouring 5:1 PDMS into an aluminum mold, performing spin-coating (500 rpm for 30 s), and curing under the same conditions as the encapsulation layer. After fabricating the two PDMS layers (i.e., the encapsulation and bump layers), the following stage involved bonding the bump layer to the encapsulated device. To bond the two PDMS layers, we used a vacuum plasma system (COVANCE, Femto Science Co., Ltd., Hwaseong-si, Korea) to perform O 2 plasma treatment (Figure 2g). [25] A stereo microscope (HSZ-645 or 660, Huvitz Corp., Gunpo-si, Korea) was used to ensure that the 16 bumps on the bump layer were aligned with alignment marks on the taxels on the sensing layer ( Figure 2h). Once aligned, we attached the bump layer to the encapsulation layer and separated the sensing layer (which was now attached to the bump layer) from its mold (Figure 2i). Afterward, we attached the sensing layer to the deformation layer of the sensor case. This concluded the tactile sensing device-fabrication process.

Signal-Processing Electronics
To turn the tactile sensor into a module, a circuit board capable of processing the sensor signals is required. As shown in Figure 1a, www.advancedsciencenews.com www.advintellsyst.com the electric circuit of the tactile sensor consists of a matrix-type resistance array. Since the circuit does not include a transistor that can switch the current on or off for each resistor or a diode element that can control the direction of the current, the resistance value readings of each gauge are affected by the resistance values of the surrounding gauges. The zero-potential method was devised to minimize this cross-talk effect, [17] and thus, we also employed this method when developing our signal-processing board. Figure 3a illustrates the circuit developed in our study. To minimize the electromagnetic interference (EMI) noise that is generated when the robot is being operated, an analog switch is used to supply a voltage modulated at 40 kHz to a resistor connected to a specific column electrode. Simultaneously, the analog switch supplies a bias voltage (V bias ) to the remaining columns other than the selected column. The current flowing through the resistor that receives the modulated voltage flows along the row electrodes to a pre-amp consisting of an inverting amplifier.
Here, the current is converted into a voltage that is supplied to the input of the MUX (multiplexer). Since the positive input terminal of the inverting amplifier receives the bias voltage, theoretically, all rows have voltages equal to V bias , and since all rows and columns with the exception of the column that receives the modulated voltage have a voltage equal to V bias , the potential difference is zero. Therefore, as current only flows through the resistor connected to the selected column in the matrix and not through any other resistor, we can minimize cross talk. When the MUX selects a specific row to measure, the voltage of the selected row is sent as an output. The output voltage passes through a demodulator and a low-pass filter (LPF) to be converted into DC voltage, which is subsequently converted into a digital signal by an ADC (analog to digital converter). This allows us to measure the resistance of an element in the sensor matrix. The resistance of each element is sequentially scanned and measured using the analog switch and MUX to produce a resistance map of the 64 elements of the matrix. This resistance map serves as a single frame of the tactile image.
The nominal resistance of the designed strain gauges is 7 kΩ, and thus, the circuit is optimized for resistance measurements near 7 kΩ and has a resistance measurement range of 6-10 kΩ. The frame rate when scanning all 64 resistors is 30 Hz, but for situations that require a faster bandwidth (e.g., slip or vibration detection), we added a measurement mode that enables the www.advancedsciencenews.com www.advintellsyst.com sampling of a single taxel at a rate of 300 Hz. For data transfers from the circuit board to a PC (personal computer), we used UART (universal asynchronous receiver/transmitter) communication, a type of serial communication, with a baud rate set as 460 800 bit s À1 . It is more advantageous to minimize the length of wires connecting a sensor to a signal-processing circuit board as this minimizes noise. Therefore, our board unit was designed to be fixed to the body of the parallel gripper on which the tactile sensor was attached. To ensure that the circuit board was a suitable size to be attached to the gripper, we designed a circuit board with two layers, as shown in Figure 3b, with an analog circuit section (bottom) and a digital circuit section (top). The board had dimensions of W: 75 mm Â L: 40 mm Â t: 10 mm. Additionally, to shield the circuit board from external noise, the board was mounted in an aluminum case (size: W: 110 mm Â L: 54 mm Â H: 30 mm). The aluminum case included four holes in each corner so that it could be fixed to the gripper with M4 bolts. Figure 4a shows the tactile sensor unit and the signal-processing unit installed on the gripper. Once the tactile sensor and circuit board were installed on the robot, we performed noise measurements while the robot's motor was running. According to the measurement data obtained with sampling frequencies of 30 Hz for 30 s for a single-strain gauge of the sensor, the standard deviation and the peak-to-peak value were  www.advancedsciencenews.com www.advintellsyst.com 3.61 Ω (0.056%) and 0.68 Ω (0.011%), respectively, with the average resistance value of 6426.94 Ω (Figure 4b). In case of the 300 Hz sampling mode, the noise values in terms of the standard deviation and peak-to-peak increased to 1.25 Ω (0.019%) and 9.71 Ω (0.151%), respectively, with the average resistance value of 6425.44 Ω ( Figure S1, Supporting Information).

Testing the Prototype Tactile Sensing Module
Evaluation tests were performed to measure the performance of the developed sensing module. First, to measure the module's detection capabilities for forces in the vertical direction (along the Z-axis, F z ), we placed the sensor on a high-precision scale and applied a load by pressing on the center of the bump using a 2 mm sized metal tip. [20] A value proportional to the vertical force can be obtained by finding the average resistance change rate of the four strain gauges located in each taxel, as described in Equation (1).
Here, α z is the sensitivity coefficient and S z is the average relative resistance change of the four strain gauges that represents the output sensitivity for F z . i is an index number assigned to each strain gauge in a taxel (refer to Figure 1a), ðR i Þ 0 is the initial resistance value of strain gauge i, and ΔR i is the resistance change of strain gauge i and can be expressed as R i ¼ R i À ðR i Þ 0 . Figure 5a shows a graph of the sensor output results that were obtained in real time while applying a load range of 10 mN % 1.8 N in a stepwise manner. The maximum load was set as 1.8 N as we often observed certain strain gauges in the taxels being damaged when subjected to loads greater than this value ( Figure S2, Supporting Information). In other words, 1.8 N was the highest load that could be measured by the sensor without it being damaged. Although the sensor had a resolution that enabled it to distinguish between loads separated by 10 mN, we also noticed the sensor exhibited a creep error (where the output value decreased over time) that became more pronounced as the applied load increased. We believe that this phenomenon stemmed from the substrate being made of a viscoelastic ductile rubber (PDMS). [17] To evaluate the repeatability characteristics of the sensor outputs, we repeatedly applied the maximum load (1.8 N) on the sensor for a total of 3,000 times. As a result, the relative standard deviation of the output sensitivity (=the standard deviation of sensitivity/full scale output [FSO] *100) was found to be less than 2%, and the sensor output exhibited a response time of approximately 0.17 s to step loads (Figure 5b). In contrast, the linearity, sensitivity coefficient, and hysteresis error of the sensor outputs were obtained by measuring sensitivity while the applied load was increased and decreased in increments of 0.3 N from 0 to 1.8 N then back to 0 N (Figure 5c). The results showed that sensitivity increased in a linear manner for loads up to 1.5 N, but the trend became nonlinear beyond this point as sensitivity increased by a lesser degree when the applied load was 1.8 N compared to when smaller loads were applied. We deduced that this was due to the nonlinear mechanical properties of the PDMS substrate. [26][27][28][29] As a result of performing linear regression with the sensitivity and load data up to 1.5 N, the slope (¼ 1=α z , that is, the reciprocal of the sensitivity coefficient) was determined as 7.73% N À1 and the correlation coefficient was www.advancedsciencenews.com www.advintellsyst.com R 2 = 0.995. The variation of slopes among all taxels were measured to be 0.25% N À1 (approximately 3.4%) in terms of standard deviation with the mean value of 7.32% N À1 ( Figure S3, Supporting Information). The hysteresis error, which indicates the difference in the output value when the load is increased compared to the value when the load is decreased, was within 3%. Next, we evaluated the detection capabilities of the sensor for forces in the horizontal directions (F x and F y ). We used the same equipment that was used to evaluate the vertical force detection performance but added a commercial 3-axis load cell (SRI-M3701A, Sunrise Instruments, Canton, MI, USA) to measure horizontal forces. The force acting on a single bump along a horizontal axis is proportional to the resistance change rate difference between two strain gauges along the corresponding axis, as described in Equations (2) and (3).
Here, α x and α y are the respective sensitivity coefficients of F x and F y , respectively, and S x and S y represent the respective output sensitivities of F x and F y , respectively. To apply a horizontal load on a bump, preloading is first required in the vertical direction. In our experiment, we used a metal tip to apply a vertical force of 0.15 N on the bump, then we obtained the sensor output values as we subsequently moved the metal tip in the X and Y directions in increments of 10 μm (up to 30 μm). According to the measurement values of the 3-axis load cell, a displacement of 10 μm corresponded to a horizontal force of 30 mN, and the sensor output was proportional to the magnitude of the horizontal force. Using Equations (2) and (3), the sensitivity coefficients α x and α y were calculated as 0.077 N/% and 0.076 N/%, respectively. Figure 5d,e presents the force detection results in the X and Y directions, respectively. As shown in the graphs, the sensor had sufficient sensitivity for horizontal forces to clearly differentiate between increments of 30 mN, and the sensor only responded to forces along the corresponding axis. For example, when a force was applied along the X-axis, only S x was affected while S y and S z barely changed. This indicates that the sensor can measure forces in each direction with minimal cross-talk error. The cross-talk error with respect to forces along the X-axis caused by forces along the Y-axis or Z-axis (E xy , E xz ) can be calculated using Equations (4) and (5), respectively.
Similarly, the cross-talk error with respect to forces along the Y-axis caused by forces along the X-axis or Z-axis (E yx , E yz ) can be calculated using Equations (6) and (7), respectively.
As a result of calculating the cross-talk error values using the aforementioned equations, when a force of 90 mN was applied along the X-axis, E XY and E XZ were 1.7% and 3.4%, respectively. When a force of 90 mN was applied along the Y-axis, E YX and E YZ were 1.8% and 2.5%, respectively.
Additionally, we evaluated the properties of the temperature sensor by placing it in a chamber where the temperature and humidity could be controlled. Inside the chamber, the temperature was increased in increments of 5°C from 5 to 75°C. As the temperature increased, we observed the resistance changes exhibited by the temperature sensor, and the temperature inside the chamber was measured using a precision thermometer (model: F200, ASL). Figure 5f presents the experiment results. The graphs show that the resistance of the temperature sensor increased proportionally to the temperature within the chosen temperature range. Through linear regression, the temperature coefficient of resistance (TCR) of the temperature sensor was calculated as 0.29%°C À1 . This value is within the theoretical TCR range of the Au-based resistive-type temperature sensor that was fabricated via the deposition process. [30]

Demonstration of the Tactile Sensing Module
To verify whether the developed tactile sensing module, with its multimodal detection capabilities, could improve the object manipulation abilities of a robot, the module was installed on a parallel gripper (which is often used as an end-effector for robots) and subjected to several experiments. These experiments aimed to demonstrate whether the module could properly detect various physical quantities (e.g., 3-axis force distribution, slippage, and temperature) under certain scenarios with the gripper grasping objects. We used a 3D printer (SLA600, ProtoFab Co., Ltd.) and a servo motor (XM430-W350-R, ROBOTIS Co., Ltd.) to fabricate a parallel gripper on which the developed tactile sensing module could be installed. As previously described in Section 2.3, the tactile sensing modules were installed on each jaw of the gripper (Figure 4a). Two scenarios were devised for the demonstration tests, and we aimed to demonstrate that the sensing module outputs could be used to accurately analyze the contact condition for each scenario.
For the first scenario, the gripper was used to grasp a delicate and light object: a paper cup. As shown in Figure 6a, the scenario involved four sequential stages: before the paper cup is grasped, when the paper cup is grasped, when warm water is poured into the cup being held by the gripper, and when the gripper releases the cup holding water. Since the first stage did not have the gripper grasping the cup, we could obtain sensor outputs for the zero-load state that could serve as a standard to check whether the sensor outputs returned to their initial values after the final stage (when the paper cup is released). The next stage involved the gripper grasping the cup. Output changes were detected for the four taxels that were located at the points where the cup was in contact with the detection areas of the sensor. Using Equation (1), the normal force acting on each taxel was calculated. As shown in Figure 6b, the taxel located in the second row and fourth column (R2,C4) was subjected to the greatest normal force of 1.29 N, and the three taxels surrounding this taxel each detected normal forces of 0.86 N (R1,C4), 0.22 N (R2,C3), and 0.25 N (R3,C4). Therefore, the total normal force involved when the gripper grasps the cup was calculated as 2.62 N. In the following third stage, warm water (with an approximate temperature of 45°C) was poured into the cup that was still being held by the gripper. Here, we examined the 3-axis force outputs of the taxels that were in contact with the cup. As shown in Figure 6c-e, all taxels in contact with the cup detected force increases in the X direction, which was parallel to the direction of gravity (in terms of the gripper's coordinate system, refer to the third stage in Figure 6a) along which the water descended. Additionally, the temperature sensor of the tactile sensing module detected the temperature of the cup's surface increase as the warm water was being poured (Figure 6f ). Lastly, in the fourth stage, the gripper released the cup (with water inside), and so the force output readings of the taxels that were in contact with the cup returned to their initial values as measured during the first zero-load state. Conversely, the temperature sensor output was in the process of slowly decreasing toward the initial temperature value as the heat that was transferred from the object did not instantly dissipate.
The second scenario involved tilting the gripper as it grasped a wooden reflex hammer toy (weight: approximately 15.8 g). As illustrated in Figure 7a, the gripper softly held the head of the hammer, and a collaborative robot (UR-10, Universal Robotics) altered the angle of the wrist joint to tilt the hammer so that the hammer would slip from the contact points of the gripper (Video S1, Supporting Information). We then assumed a situation where the gripper detected the slip and regrasped the hammer. In the initial stage, with the gripper grasping the head of the hammer, we determined which taxels in the sensor detection area were in contact with the hammer. We then selected the taxel that indicated the largest normal force and adjusted the measurement mode (sampling frequency 30 Hz à 300 Hz) for the slip detection. When the gripper grasped the hammer, nine taxels detected output changes (Figure 7b), and the taxel in the fourth row and fourth column (R4,C4) produced the highest normal force measurement (0.23 N) ( Figure S4, Supporting Information). Once the measurement mode was subsequently changed so that the R4,C4 taxel had a sampling frequency of 300 Hz, the joint was rotated at a constant rate (approximately þ10°s À1 ). When the joint rotated, the center of gravity of the hammer shifted away from the pivot where the gripper was grasping the hammer, and thus, the torque due to gravity increased as the hammer began to tilt. Simultaneously, a tangential force was applied on the taxel to counteract this torque. [31] As shown in Figure 7c,d, the tangential force vector ! þ F y e y ! , ) began to increase in magnitude starting at 1.14 s as the gripper started to tilt downward from its initial horizontal position. In particular, the change was more profound for F x , which corresponds to the component along the direction of gravity (the þX direction in terms of the gripper's coordinate system, refer to Figure 7a). The tangential force increased up to a peak at approximately 4.37 s, after which it decreased until the www.advancedsciencenews.com www.advintellsyst.com end of the joint motion at 4.70 s. Considering that both the normal force and tangential force decreased in this final section, we deduced that there was a slip that altered the contact state. [32] A decrease in normal force can cause slippage, and Figure 7d indicates that the direction of the tangential force vector drastically changed in the period from 4.37 to 4.70 s. This observation supports the aforementioned conclusion that there was a change in the contact state. By the time the joint ceased its rotation at 4.70 s, the normal force had returned to its value at 4.37 s (the point at which the maximum tangential force was achieved) whereas the tangential force saw little change. However, the hammer continued to rotate due to inertia, resulting in a gross slip at 4.85 s. As the gross slip occurred, the gripper detected major fluctuations in the contact forces and initiated a regrasping motion to halt the slip and stabilize the contact forces. Furthermore, Figure 7d shows that, at 5.22 s (which is after the regrasping motion), the direction of the tangential force vector became similar to its initial direction. This indicates that the contact state had stabilized.
The stick-slip phenomenon, which occurs during incipient (or micro) slips, cause vibrations that can affect the outputs of the taxels. [33] To verify whether the taxel output signals indicated any signs of vibrations, outputs obtained over time were subjected to a discrete wavelet transform (DWT) to check for the presence of high-frequency components. [34][35][36] As a result of performing single-level DWT analysis using the Haar wavelet [37] (Figure 7e), a 30 Hz frequency component distinguishable from the noise levels (AE0.009 N) was detected from 3.41 s and onward ( Figure S5, Supporting Information). Given that this component was present up to the regrasping point, it was clear that a slip had occurred in this interval. At 4.85 s (the moment of the gross slip), the sensor detected the 30 Hz component with the greatest magnitude. Conversely, after the regrasping motion at 5.10 s, the 30 Hz component was no longer detected. The dominant frequency component of the vibration signal that appears after the regrasping motion was around 10 Hz, which was a lower frequency than the frequency that was detected when the slip occurred. In addition, the magnitude of the 10 Hz component was approximately a quarter of the magnitude of the 30 Hz component. We believe that this 10 Hz vibration signal was not a result of a slip but rather due to vibrations from the collaborative robot as the joint came to a halt.

Conclusion
Our research team developed a multimodal tactile sensing module with a planar structure that was designed to be installed on robot grippers. The tactile sensor consisted of a 4 Â 4 3-axis force sensor array (with silicon nanomembrane-based strain gauges and polymer-based deformation and bump layers) and a single-metal-based temperature sensor. To turn the sensor into a module, we developed a readout circuit that was tailored to our sensor. Upon testing the sensor module, the 3-axis force sensor www.advancedsciencenews.com www.advintellsyst.com array exhibited a repeatability error within 2%, a hysteresis error within 3%, and a resolution of 10 mN for vertical forces up to 1.8 N. The module was also capable of detecting horizontal forces as small as 30 mN. Furthermore, the cross-talk error of the 3-axis force sensor array was within 4%, meaning that it could independently detect forces along the three axes. The temperature sensor exhibited linear sensitivity within the temperature range of 5-75°C. To perform feasibility tests to see whether the module could be applied to robots, we installed the tactile sensor module on a 3D-printed parallel gripper and performed grasping experiments involving two distinct scenarios. Through these experiments, we demonstrated that it is possible to determine various contact states that occur when manipulating objects by analyzing the outputs of the sensing module. We expect that the tactile sensing module developed in this study could be used to help improve a robot's ability to manipulate objects by allowing the robot to detect contact states and use that information to control gripper feedback. In a future study, we will develop technologies that enhance the grasping abilities of robots by implementing the tactile sensor developed in this study. We will also develop tactile sensing modules with curved shapes like human fingers that can be applied to anthropomorphic robotic hands.

Supporting Information
Supporting Information is available from the Wiley Online Library or from the author.