Monitoring Soft Shape Surface Deformation via Optical Images for the Distinction of Contact State

The deformability of elastomer surfaces offers the opportunity to capture multimodal physical information, which opens up possibilities for applications in intelligent interactive systems and flexible electronics. Accurately identifying the surface deformation of elastomers has been a key issue for understanding the contact state in a soft touch. However, effective monitoring of the deformed surface topography during soft‐touch processes is difficult due to the complexity of interfacial deformation. Herein, the optical detection method based on an ideal diffuse reflection model is proposed to monitor the deformation of the elastomer, enabling the acquisition of a spatially continuous deformation of the surface in a soft touch. This optical system with the parallel incident light can distinguish the elastomers’ surface deformation field by constructing the correlation between the reflected light irradiance and the surface deformation angle. This method allows for the perception of the contact area. Through the deep learning method, the recognition rate can reach more than 84.24% when there are slight differences in the contact of objects with different geometric shapes. The design offers valuable insights for contact state detection tasks and facilitates soft‐touch sensing functions through the high precision of optical signal acquisition.


Introduction
10][11][12][13][14][15] However, current progress on elastomer deformation in soft touch is hampered by many key issues.For example, the elastomer will deform around the contact area in soft touch, leading to a noncontact region in which the detailed deformation is notoriously difficult to examine. [16,17]Meanwhile, the elastomer deformation would be affected by contact depth, surface deformation angle, and other factors.20][21][22][23][24][25][26][27] At present, methods utilizing feature points are used to characterize the deformation of elastomers, such as robots' hand grasp tasks based on elastomer design [28] and slip detection. [29]n this method, the elastomer surface is divided into discrete regions by setting the feature points array on the elastomer. [30]he deformation occurring in each region is considered uniform and could be indicated by the variation of feature points' displacement. [31][34][35] Though the discrete feature points method can represent the overall trend of elastomer deformation, the local details are missing so it remains challenging to evaluate minor deformation features caused by contact behavior. [29]If the continuous deformation on the surface distribution could be introduced, it would be beneficial to realize elastomer deformation detection with high spatial resolution. [36]40] For example, the optical sensing strategy of the planar optical waveguide structure could detect the real contact area with high DOI: 10.1002/aisy.202300535 The deformability of elastomer surfaces offers the opportunity to capture multimodal physical information, which opens up possibilities for applications in intelligent interactive systems and flexible electronics.Accurately identifying the surface deformation of elastomers has been a key issue for understanding the contact state in a soft touch.However, effective monitoring of the deformed surface topography during soft-touch processes is difficult due to the complexity of interfacial deformation.Herein, the optical detection method based on an ideal diffuse reflection model is proposed to monitor the deformation of the elastomer, enabling the acquisition of a spatially continuous deformation of the surface in a soft touch.This optical system with the parallel incident light can distinguish the elastomers' surface deformation field by constructing the correlation between the reflected light irradiance and the surface deformation angle.This method allows for the perception of the contact area.Through the deep learning method, the recognition rate can reach more than 84.24% when there are slight differences in the contact of objects with different geometric shapes.The design offers valuable insights for contact state detection tasks and facilitates soft-touch sensing functions through the high precision of optical signal acquisition.
accuracy, which has great potential in the study of interface friction. [41]However, limited by the planar optical waveguide structure.It has a shortcoming in vertical contact deformation detection, so it is weak in decoupling the more complicated contact behavior between contact objects effectively. [42]Confocal microscopy can obtain high-resolution 3D contact deformation information of elastomers, which has been widely used in the study of surface tension and solid viscoelasticity at a microscopic scale. [36]Thus, although these optical sensing methods could capture high-resolution images of the deformation field distribution, there is still a lack of effective evaluation to restore the continuous deformation field distribution from the images.
In order to monitor the surface deformation from images, we propose an optical detection method based on the ideal diffuse reflection model.This optical system can distinguish the elastomers' surface deformation field by constructing the correlation between the reflected light irradiance and the surface deformation angle with the excellent directivity of the parallel incident light.It could not only help for extract the contact profile of the object but also the surface deformation angle of the elastomer from optical images directly.The system adopts parallel light incidence as a light source, which avoids the error of optical field reduction caused by the angle of incident light and could characterize the contact state.The deformation of the elastomer enables the parallel light transmission to be blocked and reflected into the complementary metal-oxide semiconductor (CMOS) when subject to contact.Combined with the convolutional neural network (CNN) and the feature of the optical image, the recognition rate can reach more than 84.24% in the contact of objects with different geometric shapes when the deflection angle interval is 1°in the XOZ plane deflection experiment.The proposed method can help understand the physical contact behavior occurring on elastomers and play a positive role in human-computer interaction platforms, such as electronic skin and flexible electronic devices.

Results and Discussion
To discuss the elastomer deformation in soft touch, a simple contact scene was built: A sphere was vertically loaded to a plane polydimethylsiloxane (PDMS) surface.Note that spheres with perfect axisymmetry have been well studied in model problems of contact mechanics. [43,44]This set-up thus can be used to verify the effectiveness of our monitoring method in soft touch.During the contact process, the deformed region of the elastomer can be divided into a contact area and a noncontact area.When the sphere is applied vertically to the PDMS surface, the PDMS produces extrusion deformation in the contact area affected by the contact depth (shown in Figure 1a).With the increase of the contact depth, the deformation angle at a certain point on the elastomer gradually increases and finally equals the tangent angle at the corresponding x-coordinate of the sphere.γrepresents the surface deformation degree of a certain point on the elastomer (Figure 1b).
In this process, the deformation angle of the elastomer is an important physical parameter, which can reflect the actual situation of the soft contact process.In particular, when a sphere is in soft contact with PDMS, the deformation angle of different positions of PDMS varies with the vertical depth of contact.For example, α, β, and γ, respectively, represent the surface deformation degree of three points on the elastomer, and the corresponding α s , β s , and γ s represents the tangent angle of the sphere position where the sphere comes into contact with the PDMS.The deformation angle of the elastomer (α, β, and γ) will vary until the object is brought into contact with PDMS.(visualization video 1, Supporting Information).Finally, the deformation angle (α, β, and γ) of each point on the PDMS surface would be identical to the object's tangent angle (α s , β s and γ s ) to the horizontal plane (Figure 1c).Therefore, the deformation of the PDMS surface is related to the contact states of the sphere, and monitoring the elastomer surface deformation is helpful in analyzing and decoupling the contact state, such as contact force, contact depth, and real contact radius, with the aid of supplemental measurements.In this context, monitoring methods that can achieve high-resolution deformation field distribution are the key to obtaining the complete elastomer surface deformation.

Principle of Optical Sensing Method for Surface Deformation Monitor
To extract the surface deformation angle, we propose a reflex optical sensing method based on parallel incident light to construct the correlation between the deformation angle and detection signal.The optical system consists of a pair of parallel incident lights, a pair of polarizers, and a flexible optical transmission multilayer.The parallel incident light sources with the 520 nm wavelength and the 2 mm line width and 58°beam divergence (see Figure S1, Supporting Information).The polarizers are used to adjust the irradiance of the incident light.The fabrication process of a multilayer elastomer structure is shown in Figure S2 (Supporting Information).The path of parallel light transmission is blocked and results in the variation of the optical field on the elastomer, which is captured by CMOS (Figure 2a).(The parallel incident light spital divergence is shown in Figure S3 (Supporting Information).)It is worth noting that in the process of incident light from the air medium to the PDMS, the incident light will be scattered at the boundary of the medium, which could be captured by CMOS and manifested as background noise in the image.It will cause the acquired surface deformation information to lose its authenticity.To eliminate the interference of scattered light on the contact image, the semi-transparent layer is made of copper that reduces the transmittance to enable the irradiance of scatted light lower than the imaging threshold (also shown in Figure S4, Supporting Information).
In the process of parallel light being reflected into CMOS, the elastomer surface deformation is translated into optical information.The contact state could be identified by decoupling the reflected optical information.As shown in Figure 2a, the optical image captured by CMOS plots a round, which turns from dark to light along the direction from the center to the edge.This regularity is similar to the rule of the sphere's tangent angle change along the diameter.We make the following assumptions to make this problem tractable.First, the four sides of the PDMS are always perpendicular to the horizontal plane.Second, the incident light energy is uniformly distributed over the space line width.Third, the reflection behavior of the incident light in elastomer follows ideal diffuse reflection regularity, thus the irradiance of the reflected light is consistent at any viewing angle.Fourth, the reflected light is detected by the CMOS at the angle perpendicular to the direction of the incident light.The connection between the irradiance of reflected light captured by CMOS and the surface deformation angle refers to the equation: [45] where I out represents the irradiance of reflected light in a certain direction, ρ represents surface reflectivity.The optical reflection model is supposed to ideal diffuse reflection, so the surface reflectivity ρ is constant.k represents the normal vector of the surface, L represents the vector direction of the incident light, and I 0 represents the irradiance of incident light.
According to Lambert's cosine law, the irradiance of diffuse light is proportional to the cosine of the angle between the direction of incident light and the surface normal.The reflected light irradiance of ideal diffuse reflection behavior in a certain direction could be obtained from the equation: where π 2 À θ À Á represents the angle between the reflection surface normal direction and the vector of the incident light, and α represents the angle between the reflection surface normal direction and the viewing point.Considering the assumption, α would not affect the irradiance received by the CMOS, so the influence of cos α to I out could be negligible.Therefore, the relation between the irradiance of reflected light in a certain direction and the surface deformation angle is shown as Equation ( 4) shows the correlation between the irradiance of reflected light and surface deformation angle.It suggests that the irradiance of reflected light is positively related to the surface deformation angle when the irradiance of incident light is fixed.For the sphere, the local tangent to the horizontal plane increases from the central to the edge along the diameter, causing the local irradiance to increase along the same direction (Figure 2b).The green circle shape represents the light behavior is diffuse reflection.The incident light is evenly reflected in all directions.The results from the ideal model fit well with our observation in Figure 2a.For further discussing the relationship between the deformation angle and the irradiance, the local irradiance was converted into the channel value of each pixel in the range from 0 to 255, as shown in Figure 2c.In this way, the elastomer deformation information could be converted to optical image brightness, which benefits analyzing and decoupling the contact state of the object.

Detection of Surface Angle
As discussed before, the local irradiance of the optical image is supposed to be positively correlated with the local deformation angle and the irradiance of the incident light.And the relationship could be proved experimentally.As shown in Figure 3a, a cube is fixed on a rotation displacement table to apply a specific deformation angle to the elastomer.In the contact area, the deformation angle of each point is equal due to the flat shape of the cube so it is easy to acquire the corresponding channel value to a specific deformation angle.The relationship between the channel value and the deformation angle is shown in Figure 3b.It proves the positive correlation between the local irradiance of the optical image and the irradiance of the incident light when irradiance is 10.41 lx to 70.27 lx.If the laser is not incident horizontally, this positive correlation is broken (see Figure S5, Supporting Information).
In the soft-touch process, deformation angle is not the only variable, the deformation depth is also a key value.With fixed deformation angle and irradiance of the incident light (Figure 3c), the relationship between the contact depth and the local irradiance of the optical image could be given.As shown in Figure 3d, the channel value does not typically change with the contact depth.Thus, for a contact point on the elastomer surface, the irradiance of the optical image is only decided by the object's local tangent angle to the horizontal plane.In other words, if a point's channel value does not change with the contact depth, it could be indicated that the elastomer has conformal contact with the object at this point.The local tangent angle of the object could be settled due to the given relationship in Figure 3b.According to the above relationship, it is hopeful to indicate the object's shape and the contact state in a soft touch.

Mechanical Model of Contact States
After the above discussion, the optical system proposed in this work is expected to be used to identify objects with different morphologies by fitting the tangent angle distribution of each contact point (see Figure S6, Supporting Information).Besides, our sensor systems allow the detection of the deformation of the elastomer in both the contact region and the noncontact region (Figure 4a).We find that with the increase of the contact depth, the boundary of the two regions appears more distinct (see Figure S7, Supporting Information).This is not surprising; however, the surface deformation angle in the contact region is almost linearly proportional to the distance from the center of the sphere (see Figure S6, Supporting Information).In addition, the deformation angle reaches the maximum at the boundary between the contact and noncontact region.Thus, the position of such a boundary should be hinted at by the coordinates where the maximum channel value is found in the image (Figure 4a).This can be further seen in Figure 4b, where it is shown that the greater the slope of the irradiance of the reflected light from the maximum to the minimum value in the direction away from the contact center when the contact depth is deeper as shown in Figure 4b.For a certain point on the elastomer, the rising rate of the channel value is proportional to the distance from the contact center when the contact depth is deeper.However, the upper limit of the channel value is inversely proportional to the distance from the contact center (Figure 4c).
We then examine our measurements by comparing them to the classical contact mechanics model. [17]In particular, the Hertzian contact model established a relation between the applied load F, the radius of sphere R, the radius of the contact a, and the contact depth h (Figure 4d), given as [17] F ¼ 4 3 where E Ã is the plane-strain Young's modulus of the PDMS.It is obvious that with the increase in the applied load, both contact radius and contact depth increase correspondingly.Steel spheres with radii of 2, 3, and 4 mm are placed on the surface of PDMS.
Considering that the elastomer will creep when subjected to load, we apply the load to the sphere slowly and record the contact image and load only after the dynamometer number is stabilized for 30 s.The radius a is obtained by the distance from the center of the circle to the border, i.e., calculating half the distance between the two brightest pixels on the diameter (the distance between the brightest pixels refers to the maximum value in the curve in Figure 4a,b).In Figure 4e,f, we find decent agreement between the Hertzian contact model and our measurements.However, there are some deviations between experimental results and the Hertzian contact model, particularly at small indentation forces.The sources of error could be that the experimental accuracy is not enough to capture these details at the initial contact state in which both the contact depth and radius are relatively small and that interfacial friction may play a role during the contact but is not considered in the simplified model.

Contact State Recognition
In order to verify the effectiveness of the optical system for deformation monitoring of elastomers, attitude detection of objects with small deflection angles is demonstrated here.Usually, the deflection of the object in the small-angle space is difficult to directly distinguish during the soft contact process because the elastomer surface deformation cannot accurately represent the actual geometric size of the object.In the optical system proposed in this work, the small-angle deformation of the elastomer surface will cause the irradiance of the reflected light to change, thereby affecting the brightness distribution in the optical image.By analyzing optical image features, surface deformations induced by contacting object deflection can be decoupled.In the XOZ plane deflection experiment, the objects with the bottom geometric shapes of circular, hexagonal, rectangular, and ring were in contact with PDMS, respectively.In the image, these characteristics are represented by tiny features contained in each region of the image.The CNN can perceive and extract these local features and then for category recognition and judgment.Therefore, the CNNs are used for image classification training.When the deflection angle interval is 1°, the recognition accuracy is 95.93%, 99.01%, 94.60%, and 84.24%, respectively, by using the CNN to train the optical image (Figure 5a-d).
Besides, the accuracy of each deflection direction is 97% when 22.5°was selected as the step size of 16 classification experiments in the recognition task of 360°deflection direction on the XOY plane (Figure 5e).In addition, this method is expected to be used for the detection of texture and crack (Figure S8 and visualization video2, Supporting Information), object hardness (Figure S9, Supporting Information) and be used to differentiate the small force applied from a large spherical shape and the large force from a small spherical shape (see Figure S10, Supporting Information).For single-target contact state detection, experiments show that the proposed elastomer deformation monitoring strategy can effectively obtain the surface deformation of elastomers and realize the identification of the contact state of objects with simple geometric shapes.

Conclusion
In summary, we propose an optical sensing method based on the ideal diffuse reflection model to construct the correlation between surface deformation degree and irradiance of reflected light.For single-target contact state detection, this method allows space-continuous deformation on the surface acquisition and enables the large information and high precision perception of soft touch, which enables the acquisition of contact profile and contact state.The method successfully monitors the process of deformation of the elastic surface from the deformation of the noncontact area to the deformation of the contact area.It has high accuracy with more than 84.24% in the recognition task between contact states with tiny differences through the deep learning method.Of course, this method also has room for improvement.First, it could only detect a limited number of contact areas.When multiple areas of the elastomer surface are in contact at the same time, there may be a problem that the optical path is blocked, which will lead to the loss of detection information.Second, system integration needs to be further improved in the future.For example, metasurface structures can be prepared to shape the incident light, and the equivalent parallel incident light is prepared to replace commercial lasers.Thus, it is expected to be applied to single-target detection in human-computer interaction, such as motion modes analysis and contact command recognition in electronic skin, flexible electronic devices, and soft robots.

Experimental Section
Fabrication of Multilayer Structure: A 40 mm*40 mm quartz was selected as the base.A layer of copper was deposited on quartz substrate by magnetron sputtering.The sputtering current was 30 mA, and the time was 4 min (high-resolution sputtering coater, Quorum, Q 150T S PLUS).To keep the sides of the elastomer smooth and flat, four pieces of quartz were pressed against the sides vertically.About 6 mL PDMS was prepared with a mass ratio of 10:1(SYLGARD TM 184 Silicone Elastomer), poured into a quartz container, and heated at 45°for 72 h.After PDMS was solidified, the quartz sheet around was removed.The upper surface of PDMS was treated with plasma for 180 s.The white propylene was spin-coated on the upper surface of PDMS and heated at 75 °C for 5 min (Figure S2, Supporting Information).
Acquisition of Elastomer Surface Deformation Angle: The highprecision electric rotary translation stage (Zolix, TBRK100) was used to control the contact angle between the object and the elastomer.A cube was selected as the contact object.The contact angle with the step of 5 degrees was changed and then the optical images were recorded.The average value of the channel depth of the contact area was calculated.
Experimental Configurations Regarding CNNs: Before the neural network training, we performed data augmentation operations on the image first.The pattern was translated, rotated, and added noise randomly.Then we used the ResNet18 [46] network to calculate the result.The official pretraining model was loaded.The dataset was divided into training set, verification set, and test set by 0.6:0.2:0.2.Each experimental group trained 30 epochs with a learning rate of 0.01.The weight with the lowest verification set loss value during training was selected for the experiment.

Figure 1 .
Figure 1.Schematic diagram of object contact elastomer.a) Schematic diagram and optical images of the elastomer surface being subjected to external vertical load.b) Schematic diagram of vertical contact process between sphere and elastomer.c) The elastomer surface deformation angle varies the contact depth of the sphere in soft contact.

Figure 2 .
Figure 2. The method and principle of optical sensing method for surface deformation monitor.a) Schematic diagram of the optical sensing system.b) The schematic diagram of the relation between surface deformation angle and reflected light.c) The irradiance is quantified by the channel value.

Figure 3 .
Figure 3. Surface deformation information obtained by capturing reflected light.a) Scheme of the deformation angle test.b) The relationship between the irradiance of reflected light and surface deformation angle.c) Scheme of the contact depth test.d) The relationship between the irradiance of reflected light and deformation angle in different contact depths.

Figure 4 .
Figure 4. Contact state and mechanical verification of soft-touch model.a) The contact area image and contact radius of the sphere (scale bar: 1 mm).b) The scheme of sphere vertical contact PDMS with different depths and distribution of irradiance of reflected light along the x-axis with different contact depths.c) The irradiance of reflected light for certain contact points varies with the change in contact depth.d) Axisymmetric mechanical model of semiinfinite space compressed by sphere.e) The curve about the force applied to the sphere and the radius of the circle in the image.f ) Dimensionless processing of force and the radius of the circle in the image.

Figure 5 .
Figure 5.The optical system can identify the state with a convolution neural network.The geometry contacts with PDMS, and the degree represents the angle between the long axis direction and the normal vector perpendicular to the horizontal plane.a) The geometry with a circular bottom shape touches the PDMS.b) The geometry with a hexagon bottom shape touches the PDMS.c) The geometry with a rectangle bottom shape touches the PDMS.d) The geometry with a circular bottom shape touches the PDMS.e) The convolutional neural network is used to recognize the pressing of a cylinder in 16 directions in the XOY plane at 22.5°steps.(Scale bar in (a)-(e): 2 mm).