EmoSense: Revealing True Emotions Through Microgestures

Stress is a universally ubiquitous emotional state that takes place everywhere and microgestures (MGs) have been verified to indicate more accurate hidden emotions. However, only limited studies attempted to explore how MGs could reflect stress levels. Herein, EmoSense, an emerging technology for wearable systems containing a three‐layer stress detection mechanism, is proposed: 1) converting the MGs into digital signals; 2) training a machine learning‐based MG detection model; and 3) configuring the stress level based on the MG frequency. To detect the MGs, the swept frequency capacitive sensing technology to is adopted capture the MG signals and the random forest model to detect the MGs effectively is applied. 16 participants are recruited in the pilot study to verify the correlation between stress level and MG frequency. The experimental results further verify that stress level is highly related to other negative emotions that should be studied while handling high stress levels.


Introduction
Stress is a universally ubiquitous emotional state and takes place everywhere.According to The American Institute of Stress, [1] the global average of the number of stressed people out of 143 countries is 35% and around half of American adults report their behaviors have been negatively affected due to the physical and emotional toll of increased stress.Stress has been closely linked to psychological illness and physical diseases, [2] such as depression, [3] hypertension, [4] cardiovascular disease, [4] infectious illness, [5] and even cancer. [6]Stress plays a vital role in the development of unwanted behaviors and detecting stress at an early stage can help prevent aggression. [7]ferring users' emotional states [8] is a long-standing problem in humancomputer interaction (HCI) community as it involves the UX design, [9,10] tracking methods, [11] security concerns, [12] and so on.To track the emotional stress changes, some previous research works have been proposed based on facial tracking [11] and touch sensing. [7,13]However, people tend to suppress or hide their feeling on most occasions. [11]As a special group of gestures, microgestures (MGs) have been verified to indicate more accurate people's hidden emotions. [7]Specially, when people experience high stress or nervousness, they may perform those physiological responses, [14] for example, hand wringing and wrist rubbing.[17][18][19][20] Besides, tracking facial expressions or acoustic signals can be highly intrusive to the participants, which may cause severe privacy leakage and security issues. [12]Although several works [21][22][23][24] focused on inferring emotions by sensor-based touch sensing technologies, no one has explored the issues like analyzing the correlations between MGs and stress.Tackling these issues could be beneficial for a deep understanding of the emotional changes represented by MGs, which further provides guidelines while detecting high stress levels.
Inspired by prior studies in MGs, [21,25] we focus on handbased MGs when people experience high stress, which can be beneficial for stress detection in wearable devices (like watchband and wristband). [2]Prior work [26] investigated the effect of suspicious and deceptive feelings on the frequency of hand movements based on video-recorded tapes.More frequencybased studies have been found on human bodily movements [27] based on strain sensors.Instead of classifying the MGs under surveillance, [11] we analyze the users' reactions toward the media stimuli and count the MGs simultaneously (we note the number of MGs between two stimuli as MG Frequency).To capture the MGs, we adopt the swept frequency capacitive sensing (SFCS) circuit [21,28] that links with a three-layer tangible object, which changes the signal flow regarding various MGs.To analyze the correlation between stress level and MG frequency, 16 participants were recruited to provide emotional reports toward the stimuli from the international affective picture system (IAPS). [29,30]Considering the variances of MGs, our study focuses on the MG frequency instead of investigating the effects of various MGs as a pilot study.This article yields a MG-based stress detection system, namely, EmoSense, and conducts empirical studies to investigate the correlation between the frequency of MGs and changes in stress level.Specifically, our contributions to the HCI community are as follows: 1) We embed SFCS technology with a three-layer tangible object to capture MGs; 2) We unpack the assumption that MGs link to stress through user studies; 3) We provide a pilot study on inferring the stress level based on the MG frequency; and 4) Our study delves deeper into the relationship between stress levels and specific emotions, and our finds suggest that other negative emotions need to be explored in order to better understand how high stress levels impact emotions.

Stress
Although the term "stress" is widely adopted in diverse scenarios, there is no unified definition.In general, we can consider stress from two perspectives: psychologically and physiologically, [2] where the former focuses on the emotional feelings and perception of stress, and the latter refers to the bodily responses to external events.
The traditional psychological definition of stress [31] indicates that stress occurs when a person perceives the demands of environmental stimuli to be greater than their ability to meet, mitigate, or alter those demands. [32]Perceived stress [33] typically includes several psychological components of the stress response or feelings of overwhelm, or anxiety, as well as cognitions that demands outweigh resources, or not having control.
In medicine, stress relates to the scientific and objective appraise by specific characteristic changes in the structural and chemical composition of the body, [34] controlled by the sympathetic nervous system (SNS) and hypothalamus-pituitaryadrenocortical axis (HPA axis).Existing research works have proposed several physiological measurements as stress indicators, such as electroencephalography (EEG), [35] heart rate variability (HRV), electrodermal activity (EDA), electromyogram (EMG), blood pressure, pupil diameter, salivary cortisol, and salivary alpha-amylase. [36]

Stress Detection and Assessment
To detect and assess human stress, subjective and objective measures have been used pervasively, [37,38] where the former tends to measure the current emotional state of the participant based on the standard questionnaires designed by field experts. [39]However, the latter includes physiological and physical measures. [40,41]or subjective stress assessment, standard stress measurement questionnaires are widely adopted, [39] including the perceived stress scale (PSS) questionnaire, [42] acute stress disorder (ASD) scale questionnaire, [43] relative stress scale (RSS), [44] and so on.Comparing to these existing old-fashioned methods, a graphic questionnaire, the self-assessment Manikin (SAM), [45] can be adopted to measure the emotional state on three dimensions, including affective valence, arousal, and dominance, where more attractive emojis could be added to show more specific emotions (e.g., "Happy", "Sad", "Tired"). [13]bjective measurements include physiological and physical measures.As mentioned above, physiological methods need extra sensors on human bodies at specific locations [46] to track physiological changes, like EEG, HRV, and blood pressure.For physical measures, nonwearable sensors (like acoustic sensors and cameras) can be adopted to keep a distance from the subject and track the stress based on the behavioral changes, including facial expressions, [11] gestures, [7,13] speeches, [7] and so on.As there is no extra human intervention in the objective measurements, they can avoid the intuitive bias caused by subjective questionnaires.Moreover, the research finding from the objective methods can help validate the results from the questionnaires. [46] Related Work

Linking Stress with MGs
Although previous studies mostly focused on gestures, [21,25] for example, hand-waving refers to the expression "hello" or "goodbye", which are intentionally expressed for each piece of certain information.Compared with them, MGs could be more appropriate to indicate the current inner emotional state of the subject, for example, hand-rubbing in an interview relates to stress.As MGs are unintentional behaviors elicited by humans' hidden feelings, detecting the MGs could be a promising way for researchers to consider if the subject feels stressed.
To detect the MGs, several research studies have proposed multiple sensor-based frameworks.In ref. [47], bioacoustic sensors equipped with piezoelectric elements are placed on the subject's hand, and a machine-learning classifier is based on the sensor data to detect MGs.In ref. [22], a thumb ring built with a flexible printed circuit board was designed, and a supportvector-machine (SVM) classifier was adopted to detect 12 microfinger poses.In ref. [48], electrodes attached to the fingertips of users' gloves were used to infer the real-time spatial relationship between fingers, coupling for interacting with a head-mounted device.In ref. [23], they designed a wristband for 3D finger tracking and pose estimation with four miniature thermal cameras by adopting a customized deep neural network.However, these technologies mainly focus on one-hand interaction and finger gestures, which could not directly apply to MGs with both hands.Moreover, Puchihar et al. [49] investigated Google Soli's radar sensing technology for MG detection, which led to low recognition accuracy.Therefore, exploring novel technologies and interactive ways for MG detection is essential.
Only limited works investigated the relationship between the MGs and stress levels.In ref. [11], the authors proposed an MG recognition framework by leveraging neural networks based on the videos captured from the competition interview and provided a hierarchy that allows exploration of the relationship between MGs and emotional states.In ref. [7], a multimodal framework was proposed to track the subject's stress based on acoustic and gesture features.A common limitation is that these methods mainly distill gesture features from the visual input (images or video frames), which can be invasive for users who care about privacy issues.
To satisfy the technological, interactive, and privacy-related needs, we adopt a flexible printed circuit board and propose a three-layer tangible object to capture the subject's MGs.Then we use the machine learning (ML) classifiers for MG detection and propose an analytical hierarchy mechanism to link MG frequency with various stress levels.

Stress Detection Approaches
Some previous works explored stress detection from the HCI perspective.In an office environment, interactive ways with the mouse and the keyboard can be analyzed to indicate the subject's emotional change. [50,51]Unfortunately, these ways may limit to a specific scenario.Besides, smartphones embedded with sensors are commonly used to detect stress based on human voices, [52] smartphone usage data, [13] touching activity, [13,53,54] and so on.However, these methods are constrained to smartphone platforms and may cause severe privacy issues.
In contrast, wearable devices embedded with sensors are more advantageous for daily stress detection, [2] as physiological data can be assessed in a real-time manner.For instance, a chest belt [9] was adopted to sense respiration data and electrocardiographs (ECGs) to detect stress.Besides, wearable devices from commercial companies also provide stress detection services by tracking HRV, respiration data, and blood pressure, including Huawei, Garmin, and Samsung.These products can also provide visualization interfaces for users to track their current stress levels.For example, Huawei "Health" app can be paired with a wearing device to measure current stress level every 30 min by providing a stress value s in refs.[1,55], where s ∈ ½1, 29 relates to the "relaxed" state, s ∈ ½30, 50 indicates the "normal" state, s ∈ ½60, 79 refers to the "medium stress" and s ∈ ½80, 99 shows the "high stress".However, the stress update frequency is still quite slow for practical use.Furthermore, stability and accuracy issues are frequently raised for such wearable products. [2]o overcome low frequency, stability, and accuracy issues, the user-based experiment is included as a preliminary study to analyze the correlation between stress level and MG frequency.Based on the method suggested in SAM report, [45] the emotional changes and the MG signals from the participant can be collected simultaneously from our validated MG tracking system during each experiment.Furthermore, based on the quantitative analytical results, a fit model is exploited to describe the linear function between MG frequency and stress level, which can be applied to the user interface and update the stress level frequently (per minute or even faster) for practical scenarios.

Swept Frequency Capacitive Sensing
Motivated by refs.[21,28], we adopted the SFCS technology to detect the MGs (as shown in Figure 1).The human body is relatively conductive, whereas the skin is highly resistive. [56]herefore, the skin generates a capacitive interface between the electrode and the ionic physiologic fluids inside the body, allowing alternating current (AC) to pass through. [57]The resistive and capacitive properties of the human body oppose the applied AC signal, changing the phase and amplitude of the AC signal.Besides, the amount of signal changes highly depends on the signal frequency, [21] as the AC signal will follow distinct pathways inside the body at various frequencies. [57]For instance, some tissues become more and others less resistant to the passage of charges when the frequency of the AC signal changes, changing the route of the signal flow.
Therefore, by sweeping the capacitive sensing through various frequencies, we can explore: 1) how material type influences the generation of the signal flow and 2) how MG affects the changes in AC signal.The whole architecture of the double-sided capacitive sensing PCB is shown in Figure 2: the SFCS circuit was placed on the front surface and the other smooth surface was used to connect with wearable conductive objects.The SFCSbased sensor worked with an update frequency of 20 Hz, and the minimum displacement can reach to a few micrometers. [28,58]Due to the minimal voltage resolution 3.3/256 V supported by Arduino, we further evaluated the minimal displacement of 2.08 μm in our case (For ease of reading, we leave the detailed evaluation process in the Appendix).Specifically, the conductive object is a three-layer synthetic material (as illustrated in Figure 1): the top layer is to adjust the conductivity (e.g., thermoplastic polyurethane [TPU]), the middle layer is fully conductive, whereas the bottom layer is fully resistive to insulating unwanted human contacts.Considering the wearable need, we adopted the bottom layer to prevent unnecessary changes of AC signals when placing the object on the human body.
In practical deployment(as shown in Figure 3), we can deploy the circuit based on Arduino Uno Board [59] and show a 160-point capacitive profile chart from the sweep frequency through the Processing Platform. [60]In our experiment (shown in Figure 4), the top layer of the tangible object was constructed using a thin piece of tissue (0.1 mm thickness) to reduce sensitivity, while the middle layer was composed of a fabric mesh woven from silver (0.2 mm thickness) and cotton threads (0.4 mm thickness), providing optimal conductivity and stretchability.The bottom layer consisted of a nonconductive felt ball (4 cm diameter) with a plausible shape and size for conducting experiments.From the size and shape perspectives, the tangible object was designed to avoid mistouching cases when participants put one hand on top of the object during the experiment, while the sensing area could be touched when various MGs occurred.This design was intended to reduce the likelihood of errors occurring during the experimental process.

ML-Based Stress Detection
As ML methods were verified with compelling performances in MG recognition, [22,47] we adopted ML methods with the SFCS application to detect MGs through the captured MG features.Furthermore, by collecting the emotional feedback from participants, we calculated the MG frequency within each media stimulus by counting MGs and linked the MG frequency to the corresponding specific stress level.Besides, a mathematical fit model between MG frequency and stress level can be generated through correlation analysis.Finally, we further designed the user interface embedded with the trained MG recognition model and the stress prediction fit model, which can present the current MG state and stress level by connecting with the SFCS circuit.The overall architecture of the ML-based stress detection system (ML-SDS) is illustrated in Figure 1.Step 1: Adopt ML models to detect MGs.When touching the conductive object with a MG, the output from the SFCS circuit is a 160-point profile chart that can keep stable for a while.Each MG can be recorded as a 160-point data sequence, and each point represents a signal value.To collect enough valuable data, we recruit four participants (Female: Male = 2:2) to touch the conductive object with various MGs (As illustrated in Figure 5).To improve the generalization and usability of the trained ML model, we allow the participant to touch every area of the object.Each participant is required to touch the object at least 100 times.For example, a participant can touch the object at a random area and repeat the process 100 times.Considering the fluctuation of MG signals, we further collected 400 sets of data sequences without touching the object.Finally, we collect 800 labeled data sequences with two labels separately: "Normal", "Detected" (with a ratio 400:400).
For the ML-based detection model, we select and compare some standard methods that are widely adopted in gesture recognition [22,47] : random forest (RF), SVM, naive Bayes classifier (NBC), decision tree (DT), k-nearest neighbor (KNN), adaptive boosting algorithm (AdaBoost).All details of these models are illustrated in Table 1.Note that we deploy all these models and conduct the experiments on PyCharm, [61] which is a Python integrated development environment (IDE) with builtin ML libraries.
We randomly split the whole dataset into train and test subsets with a ratio of 9:1 and train each model 10 times (standard validation).All the experimental findings are summarized in  A) The three-layer tangible object was designed with careful consideration for both functionality and practicality.B) The top layer was composed of a thin piece of tissue (0.1 mm thickness) to reduce sensitivity, while the middle layer was made of a fabric mesh woven from silver (0.2 mm thickness) and cotton threads (0.4 mm thickness), providing optimal conductivity and stretchability.Finally, the bottom layer consisted of a nonconductive felt ball (4 cm diameter) with a plausible shape and size for experiments.
Table 1: RF model performs the best among them with the perfect accuracy (100%) when distinguishing "Normal" and "Detected".To verify the robustness of adopted detection models, we adopt the four-time leave-one-out cross-validation (LOOCV), [62] in which we leave all the data from one volunteer as the test data while regarding the rest data as the training data.The experimental results further verify the effectiveness of the RF-based model over other methods.Considering the outstanding performance of the RF-based detection model, we adopt this model in our subsequent user studies.
Step 2: Link the MG frequency to the stress level.To investigate the correlations between the MG frequency and the stress level, participants were recruited to report feelings toward various figures from the IAPS. [29,30]IAPS is a database of pictures designed to provide a standardized set of pictures to a viewer with specific emotional labels (Note that these labels are anonymized during the experiment), for example, happiness, contentment, fear, anger, disgust, etc.According to the prior study, [63] higher stress levels would cause an increase in negative emotions in the practical environment.We randomly choose pictures labeled with positive and negative emotions to analyze various stress levels.As the emotional state is periodic and subjective, we modify the graphic self-assessment scale (SAM) [45] questionnaire with emotional emojis, and self-assessment stress scale [13] (as shown in Figure 6) to collect emotional reports from participants.Three dimensions of emotion (valence, arousal, dominance (VAD) dimensions) are measured by the SAM questionnaire: 1) Valence is the pleasure scale, which shows SAM smiling at the left end and unhappiness at the other; 2) Arousal represents the intensity scale, which uses an excited figure at the left end and a calm figure at the other; and 3) Dominance evaluates the degree of control, where a tiny portrait indicates a controlled state.Based on the collected MG signals and emotional reports, we can compute the MG frequency by counting MGs within each stimulus and further analyze the correlations between stress level and MG frequency.After collecting and preprocessing all the data sequences, we analyze the correlation between stress level and MG frequency and provide a fit model that predicts the stress level based on the captured MGs.

Experiments
Following the MG-based stress detection based on SFCS technology and ML model, user studies were conducted to verify the correlation between stress level and MG frequency.This section first introduces our experiment setup and how users interact with our stress detection model.Based on the MG data and questionnaire, we analyze the received user data and present the experimental results: 1) the correlation between the MG frequency and stress level and 2) the deeper correlation between these two variables in the VAD emotional space.

Participants
[66][67] None of them have any mental, visual, or hearing disorders.Participation in this study is voluntary, and all the collected information/data is solely utilized for research purposes with the participants' consent.Furthermore, our project has been approved by HKPolyU Institutional Review Board (PolyU IRB) (HSESC Reference Number: HSEARS20211006003).

Apparatus
To detect the gesture signals, we implement a gesture tracking system based on the processing platform [60] (similar to the setting in Figure 5) leveraging the capacitive sensing circuit, which can update and record the 160-point vector of touch sensing signals at a frequency of 20 Hz.That means we can capture around 20 sets of touching data per second.To analyze the experimental results more efficiently, we equip a high-performance computer with Intel Core i9-13900 K, GeForce RTX 3090 Ti, and 64 GB RAM.Note that we preprocess all the data sequences on PyCharm and complete all the correlation analysis in IBM Statistical Product Service Solutions (SPSS) Platform. [68]1.3.Stimuli 56 picture stimuli are provided from the database IAPS.[29,30] As shown in Figure 7, the stimulus pictures are colorful photographs ranging from everyday objects and scenes labeled with eight emotional labels: amusement, contentment, excitement, awe, anger, disgust, fear, and sad.In the following user study, seven pictures are provided representing each emotion.The stimuli were presented on a laptop at 1440 horizontal Â 960 vertical pixels (full screen). Particiants can adjust the visual angle to their satisfaction.

Procedure
The whole experimental layout is shown in Figure 8.We carry out the experiment with each participant separately in a soundproof room.The whole experiment lasts around 40 min.Upon arrival, participants are informed that the experiment involves evaluating emotional changes with a self-assessment report.As illustrated in Figure 8, each participant would place one hand on top of the tangible object, recording the initial state as the "Normal" state and counting the occurrences of MGs.In addition, the other hand is used to answer an online questionnaire on a tablet between the participant and the screen.Each participant is given about 5 min to adapt to the experimental room.In the beginning, 6 pictures were given for practice, which allowed all participants to get familiar with the experiment.Then, the other 50 stimuli were presented successively.We further pseudo-randomize the picture presentation orders across participants so that no one can predict the picture type before viewing it.According to existing experiments on IAPS, [69][70][71] each stimulus was allowed to display for 3 s for invoking emotional changes.The screen transforms into a blank page for participants to finish five questions within 15 s between two stimuli. [30]efore moving to the next picture, we allow 5 s for each participant to relax.After the experiments, a fast post-interview would be conducted for around 5 minutes, acquiring feedback about this experiment.According to feedback, all the participants reported it was enough to fill in the questionnaire in 15 s.Besides, they were satisfied about setting the relaxing time between two stimuli.

Data Preprocessing
After completing all the experiments, we collect 850 sets of data sequences in total, each of which contains the MG signals and the questionnaire output.Based on the RF-based MG detection model (100%), each MG data sequence is applied as the input, and the MG state can be obtained as the output ("Detected" or "Normal").As we care about the occurrence frequency of MG, we count the "Detected" states viewing the stimulus for each participant.Specifically, we regard MG counts in 18 s as the MG frequency (3 s for displaying the figure plus 15 s for showing the blank page).

Link Stress Level to MG Frequency Based on the Questionnaire Report
To investigate the correlations of MG frequency and stress level, we adopt a measure of linear correlation between two variables, namely, Pearson correlation coefficients ranging in ½À1, 1 with the general rules of thumb [72] : 1) if the value is near AE1, then it is said to be a perfect correlation; 2) if the coefficient value lies between AE0.7 and AE1, then it is said to be a strong correlation; 3) if the value lies between AE0.4 and AE0.7, then it is said to be a  moderate correlation; 4) when the value lies below AE0.4, then it is said to be a weak correlation; and 5) zero value means no correlation.As we notice that participants rate diverse stress levels to the same picture as emotional changes are intuitively personalized, we first analyze and create a user profile for each of them (as shown in Table 2).Table 2 shows that a strong correlation between these two variables can be obtained from 62.5% (10 out of 16) of subjects, 25%, a moderate correlation can be viewed from (4 out of 16) of them, and only 12.5% (2 out of 16) of subjects show a weak correlation.We add significance tests to verify the confidence level of the correlation between MG frequency and stress level.We adopt the T-test to calculate the p-value: if the p-value is 5%, there is only a 5% chance that MG frequency and stress level are unrelated.To give a T-test, a null hypothesis states that no relationship exists between the stress level and MG frequency being studied.A p-value less than 0.05 is typically considered statistically significant, in which we reject the null hypothesis. [73]As we are solely interested in the positive correlation between these two variables based on the Pearson correlation coefficients, we omit the negative correlation and adopt a one-tailed T-test in our experiment.Since all p-values are smaller than 0.05, a positive correlation exists between the stress level and the MG frequency.
The majority of statistics data from the subjects (14 out of 16) show a strong or moderate correlation between stress level and MG frequency.We received comments from them after the studies indicating that some pictures made them feel highly stressed.We note the tolerance variance toward various stress levels for various participants in our analysis.Regarding O10 and O16, the experimental results only reveal a weak correlation between stress level and MG frequency.Then a thorough conversation was held individually with O10 and O16 to investigate the bias.O10, who consistently displayed the same stress levels in response to various figure stimuli, explained: "These pictures are nearly the same for me as they are not very striking.I just lost interest and started fiddling with my fingers."Besides, O16, who consistently reported a stress level between 2 and 4, stated: "I was a bit tired of viewing these pictures.Furthermore, I have to say some pictures made me feel so bored."For O16, more MGs are detected when some emotional states are reported as "Boring" with lower stress levels.
To investigate the generalization of the correlation between these two variables, we consider the bias in the physical application scenario and provide the scatter plot based on all the data (Stress level vs. MG frequency) and the relationship map (as shown in Figure 9).We further obtained the Pearson correlation coefficient 0.625 and the p-value 7.36E À 137 ð<< 0:05Þ, which indicates a highly moderate correlation between the stress l evel and the MG frequency.As shown in Figure 9a, based on the linear correlation, we generate the mathematical fit model y ¼ 1.47 þ 1.22x ðx∶MGfrequency, y∶StresslevelÞ.As shown in Figure 9b, we note that higher counts occur in some slightly low stress levels, like 0, 1, and 2. Most counts occur when the MG frequency is between 0 and 3. Besides, more occurrences of lower stress levels are linked to lower MG frequency as well.

Analyzing Stress Levels in the 3D VAD Emotional Space
For each participant, we collect the stress level, MG frequency, and three emotional components simultaneously.We calculate the mean values and standard deviations (SD) regarding each stress level and summarize the results in Table 3.As evident from Table 3, a positive correlation exists between the stress level and the MG frequency, further validating our findings.Besides, each variable (stress level or MG frequency) tends to hold a positive correlation with the valence score and a negative correlation with the arousal or dominance score.To further investigate the emotional differences among various stress levels, we plot all 850 points in VAD space as shown in Figure 10: higher stress levels tend to bring higher valence scores (more unhappy), lower arousal scores (more excited), and lower dominance scores (more controlled), and vice versa.Specifically, when the stress level increases to 10 (highest level), the valence score is 5 (highest), the arousal score is 1 (lowest), and the dominance score is 1 (lowest).However, we also notice some outliers.For example, when the stress level is 4, there is no clear distribution difference considering the valence and dominance scores.Besides, the arousal/ dominance scores are incredibly high when the stress level is 6 and the arousal score is relatively small when the stress level is only 4. Considering the high SD values in arousal (1.242) or dominance (1.452) when the stress level is 6 in Table 1, it indicates a higher fluctuation when rating these two dimensions.One reason can be the presence of noise caused by participant bias.The data analysis in the VAD emotional space further provides evidence that there exists the correlation between the stress level and MG frequency.According to a prior study, [63] findings suggested that people who perceived higher stress levels would lead to increases in negative emotions.As part of our modified questionnaire, we collect emotional changes toward various stimuli for each participant.Therefore, we further explore the relationship between stress and other emotions.Considering the limited amount of data sequences for some stress levels in the VAD space (Figure 10), we adopt the k-means clustering [74] method and divide all stress levels into three classes: 1) low stress level (including Levels 0, 1, 2, and 3); 2) middle stress level (including Levels 4, 5, 6); and 3) high stress level (including Levels 7, 8, 9,  10).We summarize the occurrence probability of each emotional state in Table 4 and generate the top-5 word cloud for each stress class (as shown in Figure 11).From Table 4 and Figure 11, we observe the following: 1) The most emotional occurrences are "Surprised", "Shocked", and "Fearful" for low stress level, middle stress level, and high stress level separately; 2) "Happy" and "Tired" states only occur at low stress level; 3) "Sad" and "Shocked" states occur neutrally in all three stress classes; 4) "Stress" and "Disgusted" emotional states merely exist at low stress level; 5) Compared to the other two stress classes, "Fearful" and "Annoyed" emotional states are mostly viewed in high stress level; and 6) We find it interesting that participants tend to choose other specific negative emotions (e.g., "Fearful", "Disgusted", "Annoyed", etc.) instead of "Stressed" state even in high stress levels.

Novelties
In this article, we propose EmoSense and conduct empirical studies to investigate the correlation between stress level and MG frequency.We summarize our novelties in the following aspects.First, although SFCS technology has been verified with robust sensing capabilities on objects in a fixed position or on rigid surfaces [21,28] (e.g., door, sofa, mobile device), limited works explore the integration with wearable devices, such as wristbands and watchbands.In order to prevent mistouching cases and improve the performance of MG detection using SFCS, we employed a three-layer structure for the touchable object linked to the sensing area.The top layer is designed to adjust conductivity,   the middle layer is fully conductive, and the bottom layer is fully resistive to insulate unwanted human contacts.Besides, we adopt ML models for MG detection based on the received MG sequences, in which RF outperforms other models with the best accuracy (100%).

MG Frequency-Stress Level Correlation
Second, we analyze the correlation between stress level and MG frequency based on user studies.From the general perspective, the experimental results verify the moderate linear positive correlation between these two variables.As evident from Table 3 and Figure 10, we provide the inner correlation analysis between stress level and MG frequency in the VAD emotional space.

Stress-Other Negative Emotions Correlation
Finally, we further discuss and analyze the relationships between the "Stressed" and other emotions.As evident from Figure 11 and Table 4, people tend to use other negative emotions instead of "Stressed" state when people experience high stress levels, for example, "Fearful", "Disgusted", "Annoyed", "Schoked", and "Sad".We assume that people tend to use specific emotional words while suffering high stress levels.These findings can guide us in investigating the effects of various negative emotions on high stress levels.

Limitations and Future Work
As a comparatively small group of subjects were involved in our experiments to verify the correlation between these two variables, Table 5.We adopt six standard classification models for MG classification and summarize all the experimental results.Note we consider five classes: "Tap", "Pinch", "Grab", "Press", and "Normal".Figure 13.Stress detection interface contains three components: the detected state ("Normal", "Tap", "Pinch", "Grab," and "Press"), the current stress level, and the SFCS graph.To illustrate the phase and amplitude changes of signals, we further zoom up the red circular part in the SFCS graph for each state.A) "Normal", B) "Tap", C) "Pinch", D) "Grab", and E) "Press".some outliers exist due to participant bias.As part of future work, more participants will be included to improve the generalization of correlation analysis and the fit model.As a pilot study, this article discussed the correlation between MG frequency and stress level.Considering the variety of MG types, we further split most of the MG data sequences for MG detection with four MG labels as MG representatives [11] : "Tap", "Pinch", "Press," and "Grab" (as shown in Figure 12).Adding the existing "Normal" state, we follow the similar ML model training procedures mentioned in Step 1 of Section 4.2 and adopt those six standard ML models.To keep a balanced dataset for the proper model training, we selected 80 sets of data sequences for each MG (including the "Normal" state).As shown in Table 5, these experimental results have verified the effectiveness of the RF-based model on MG detection.We provide basic tests on the six standard ML models for MG classification.An intriguing research direction is investigating how to improve MG classification performance using cutting-edge AI/ML techniques. [11]In our user studies in Section 5, we only focus on the MG frequency rather than a specific MG and adopt the RF model (with 100% accuracy) to detect the "Normal" and "Detected" states during the experiment.More research studies on how specific MGs affect emotions will be investigated in our future work.As part of our research plan, more variables could be considered in our future work, like MG type and the duration of a specific MG.
Additionally, with the touch-sensing device and picture stimuli in the lab environment, recruited volunteers could be induced to generate a set of emotional reactions accompanied by MGs.To further apply the correlation function in real scenarios, we employ the fit model y ¼ 1.47 þ 1.22x ðx∶MG frequency, y∶ Stress levelÞ into a user interface, which shows the current stress level and further presents the MG state based on the trained RFbased detection model (Note that we use the wristband (Prototype C) as an example) (as shown in Figure 13).More practical user experience studies in real scenarios will be conducted in our future work, such as mocking a stressful interview and investigating the long-term emotional analysis when using our wearable devices.In accordance with the subsequent experimental requirements, a more user-friendly interface will be designed to fit the smartphone or wristwatch screen and fulfill numerous application scenarios.Finally, we are currently investigating the occurrence rate given the self-rating reports from all participants to evaluate the correlation between the "stress" state and other emotions.Furthermore, we believe that occurrence of other negative emotions when people suffer high stress levels can be significant for researchers to investigate the hidden correlations among those negative emotions.

Conclusion
In this article, we propose EmoSense, an emerging technology for wearable systems containing a three-layer stress detection mechanism.To capture the MGs, we design a three-layer structure for tangible objects embedded with the SFCS technology.To recognize the MGs, we adopt various ML models and verify the effectiveness of RF with the highest accuracy (100%).16 participants were recruited to verify the correlation between stress level and MG frequency.We further design a real-time user interface that shows the current stress level and MG state based on the generated correlation fit model and trained MG recognition model.Also, we observe that the "Stressed" state is highly related to other specific negative emotions (e.g., "Fearful", "Disgusted", "Annoyed").Finally, we summarize three novelties in this work and provide more promising directions that can benefit both researchers and designers in HCI.As future work, we will analyze the influences of specific MGs on emotions and conduct long-term user studies based on wearable devices.Solving the equation, we further obtain the minimal displacement By bringing in the known data, we can calculate Δd ¼ 2:08 μm.

Proof-of-Concept Wristbands
To conquer the oversensitive responses to human contacts, we further propose the three-layer wristband design and provide four proof-of-concept wristbands (as shown in Figure 14), which are designed for daily applications in real scenarios.The interaction way between the hand and the wristband is shown in Figure 15.More detailed and comparing characteristics will be evaluated as part of future work.To verify the effectiveness and robustness of those prototypes, we leave conducting user studies in field deployment for future work.

Figure 2 .Figure 1 .
Figure 2. A self-designed PCB of the LC circuit drawn based on the schematics from ref. [76].A): the layout design of the LC circuit; B): the front view of the circuit with mounted electronic components; C): the back side of the circuit to which the conductive material connects; and D): the SFCS circuit diagram.

Figure 3 .
Figure 3.The working mechanism of SFCS based on our self-designed PCB.The dashed line indicates the current flow between the conductive object and the human body.

Figure 4 .
Figure 4.A) The three-layer tangible object was designed with careful consideration for both functionality and practicality.B) The top layer was composed of a thin piece of tissue (0.1 mm thickness) to reduce sensitivity, while the middle layer was made of a fabric mesh woven from silver (0.2 mm thickness) and cotton threads (0.4 mm thickness), providing optimal conductivity and stretchability.Finally, the bottom layer consisted of a nonconductive felt ball (4 cm diameter) with a plausible shape and size for experiments.

Figure 5 .
Figure 5. MG data collection: A) The MG collection interface is developed on the processing platform.B) The touch-sensing object and circuit are connected to the laptop.C) The MG data example: Gesture type, 160-point MG sequence, and the timestamp.The first number represents a specific state (0: Normal, 1: Detected).

Figure 6 .
Figure 6.Graphic Questionnaire with 5 questions: 1) Q1 measures the pleasantness of the picture (Valence), 2) Q2 measures the intensity of emotion provoked by the picture (Arousal), 3) Q3 measures the degree of control exerted by the picture (Dominance), 4) Q4 gauges the stress level caused by the picture, 5) Q5 collects the user's emotions toward the picture.

Figure 8 .
Figure 8. Experimental layout: A) Image stimuli are shown on the laptop screen, a touch-sensing model is adopted to capture participants' MGs, and a tablet is used to answer the questionnaire.B) We show how participants interact with our devices.C) The touching-sensing circuit based on the Arduino board is hidden behind the laptop.

Figure 9 .
Figure 9. Correlation analysis based on all the data.a) Linear Correlation: Stress level versus.MG frequency: We generate a fit line between MG frequency and stress level through linear model generation on SPSS.Based on the fit model, we can predict the stress level due to the MG frequency.b) Relationship map: Stress level versus MG frequency: we note a larger dot area (Green: stress level, Blue: MG Frequency) indicates higher occurrences.A wider line means a higher count between the specific stress level and MG frequency due to the relationship table.

Figure 10 .
Figure 10.A) Stress level distribution in 3D emotional space: The X-axis shows the "Valence" score, Y-axis indicates the "Arousal" score, and Z-axis means the "Dominance" score.B,C,D) Stress level distribution between Valence, Arousal, and Dominance pairs: Diverse colors represent various stress levels.A larger dot area indicates higher occurrences of a specific stress level.

Table 1 .
We adopt six standard classification models for MG detection and summarize all the experimental results.We test and compare the MG detection accuracy performances of baselines based on two validation methods, and the RF-based model outperforms other baselines through our experimental results.

Table 2 .
Analysis of all experimental subjects: all participants are labeled from O1 to O16.We count data sequences for each participant and summarize all the Pearson correlation coefficients and p-values.

Table 3 .
Mean and SD of different stress levels on MG frequency and VAD dimensions.

Table 4 .
The occurrence probabilities of emotional states for each stress level.