A Simple, Inexpensive, Wearable Glove with Hybrid Resistive‐Pressure Sensors for Computational Sensing, Proprioception, and Task Identification

Wearable devices have many applications ranging from health analytics to virtual and mixed reality interaction, to industrial training. For wearable devices to be practical, they must be responsive, deformable to fit the wearer, and robust to the user's range of motion. Signals produced by the wearable must also be informative enough to infer the precise physical state or activity of the user. Herein, a fully soft, wearable glove is developed, which is capable of real‐time hand pose reconstruction, environment sensing, and task classification. The design is easy to fabricate using low cost, commercial off‐the‐shelf items in a manner that is amenable to automated manufacturing. To realize such capabilities, resisitive and fluidic sensing technologies with machine learning neural architectures are merged. The glove is formed from a conductive knit which is strain sensitive, providing information through a network of resistance measurements. Fluidic sensing captured via pressure changes in fibrous sewn‐in flexible tubes, measuring interactions with the environment. The system can reconstruct user hand pose and identify sensory inputs such as holding force, object temperature, conductability, material stiffness, and user heart rate, all with high accuracy. The ability to identify complex environmentally dependent tasks, including held object identification and handwriting recognition is demonstrated.

and complex environmental interactions, [9] while providing high sensitivity and repeatability in the sensors. Second, to fully understand human interactions with the environment, both kinematics (pose) and contact information are relevant. The ideal glove must be able to capture both independently, but reason about them jointly. This is challenging for many soft sensing technologies, and even for vision-based systems, as many sensors respond to both stimuli, making it hard to uniquely identify the cause of a sensor response. [10] Finally, for such a device to be practical for wide-scale deployment, such a device must be easily and inexpensively manufacturable. Although there has been a recent focus on the development of soft wearable systems which can be rapidly manufactured, [11] many state-of-the-art commercial gloves contain a complex assembly of embedded electronics. [3] We present a glove with a simple, straightforward recipe based purely on knitted and sewn fibers and interwoven tubes, with no in-hand electronics.
Our solution is rooted in a number of observations. First, resistive and fluidic sensors are naturally decoupled in input sensitivity and do not interfere with each other, meaning they can serve as building-blocks for pose and pressure sensing capabilities, respectively. Second, using a fully conductive glove as a base, we observe a smooth, learnable mapping between deformation and material resistance. With a small number of electrode points, we can estimate a large number of resistances, which is quadratic in relationship to the number of electrode points. Third, inexpensive materials can be used to achieve both resistive and fluidic sensors, providing a fabrication strategy that can take advantage of modern textile machine manufacturing methods, has high design flexibility, and is well suited for human motion.

Related Works
There have been a number of existing wearable glove and data capture systems which integrate sensing and inference to enable the identification of different stimuli. These sensing methods come in many forms.
Computer vision is one approach which provides an unobtrusive method of tracking hand motion. Early approaches from the 1990s used markers and neural networks to identify hand postures and locations. [12] These technologies have evolved to enable tracking of colored gloves, [13] or completely unmarked hands. [14] The precision and reliability of such camera systems is high, enabling applications such as sign language identification [15] and gestural interfaces for gaming. [16] Despite the precision of markerless vision systems, they have a limited field of view and suffer from occlusion. They also cannot provide information about complex tactile interactions between human hands and objects, such as holding force or object temperature. [17] Such limitations of vision-based systems motivate the adoption of in-hand wearable sensorization, which are not limited by field of view or occlusion, and can sense more than pose.
Soft skins and soft electronics provide a promising means for achieving such wearable sensors. Realization of such technologies has recently become more feasible, given low cost electronics and wireless communication devices. [18,19] These leverage a number of different sensing modalities including capacitative, [20,21] resistive, [22][23][24][25] knitted resistive sensors, [26,27] and stretchable electronics. [9,28] Many wearable devices focus on providing tactile information or hand pose information, advancing from previous work where the main focus was on the detection of inertial forces [29] or acceleration. [30] A key limitation of many of these sensors is the inability to decouple strain and normal force responses. In addition, the sensors are often challenging to fabricate, making it difficult to integrate multiple modalities of sensing.
Recent years have given rise to a number of high-performance gloves. First, [31] presented a glove system that leverages stretchable electronics to provide high-resolution strain sensing to measure hand postures. Fabrication is challenging, and there is no method for identifying applied forces [32] developed a glove that merged capacitive sensing with neural-network-based inference to reconstruct hand pose with 5.3 error per joint angle. [32] Contrasted with MemGlove, their approach achieves lower accuracy and cannot consider environmental interaction. HandSense is an alternative approach based on many small soft capacitative sensors to allow detection of microgestures focusing on detecting very small changes in posture. [33] In the study by Sundaram et al., [34] researchers developed a high-resolution, low-cost glove which applied learning techniques to identify grasp types and modalities. Contrasted with MemGlove, this approach required extremely complex manual assembly, and applications were limited to grasp classification, which is but one of our demonstrations. In addition to research platforms, there have also been a number of notable commercial wearable gloves, including the Manus VR systems, which provides pose and limited pressure information for VR applications; [3,35] these gloves are expensive, require intricate on-board electronics, and have been surpassed in pose-reconstruction accuracy by other state-of-the-art gloves, including [32] and ours.
A primary limitation of most current systems is the focus on using only a single sensing modality. [36] By comparison, human skin has many different receptor types allowing inference of pressure, pose, temperature, and other environmental stimuli. [1] Although some wearable glove systems have shown temperature monitoring [37] or heart rate sensing, [38] there is no one wearable system that begins to show the breadth of sensing capabilities that human skin possesses. Our glove is a first step toward achieving rich, multimodal stimulus identification from two simple sensing modalities.

Contributions
In this article, we present a wearable glove ( Figure 1) that incorporates two novel sensing technologies-a resistive sensing architecture and a fluidic sensing architecture. By applying machine learning techniques, the glove can be used to perform proprioception, identify a number of different stimuli and also perform task classification. The resistive sensing measures the spatially varying resistance of a conductive knitted glove as it deforms, allowing postural information to be inferred. This is achieved by multiplexing the measured voltage difference across all pairs of strategically chosen probe points along the glove surface. The fluidic sensing integrates flexible soft tubes sewn directly into the glove's surface. These tubes are attached to pressure transducers on one end, with the other end sealed. By measuring changes in pressure within the tube, forces applied to the tubes (and thus, force applied to the glove) can be identified and reasoned about. While fluidic sensing has been demonstrated in a few cases, [39][40][41] this article marks to our knowledge that for the first time fluidic sensing has been combined with a resistive sensing scheme for high-level computational sensing tasks. Further, unlike previous resistive sensors in wearable devices which have been limited to simple morphologies or tasks, [42,43] we present a novel resistive sensing architecture which is simple to fabricate and amenable to a wide range of data-driven sensing tasks.
MemGlove, which relies primarily on knitting and sewing technologies, can be rapidly (machine) manufactured. Further, it uses only existing, commercial-off-the-shelf (COTS) materials. In comparison with existing wearable systems, the dual-sensing modalities offered by this glove allows pressure and pose to be decoupled. This allows for a wide range of different sensory stimuli to be identified. In addition, by combining the two sensing modalities we can reason jointly about the two signals, using a machine learning approach. To our knowledge, ours is the first wearable glove which allows for such combined pose and pressure-based reasoning, allowing for state-of-the-art proprioception as well as high accuracy on the currently widest array of environment sensing tasks.
In this article, we contribute the following: 1) A novel fiberbased glove design that uses a resistive knit architecture with interwoven pressure sensors, which is simple to manufacture with COTS components, inexpensive, and sensorially powerful.
2) A combination of two sensor modalities (conductive and fluidic sensors) which are physically decoupled but jointly reasoned about in a neural machine learning architecture, allowing for a wide range of downstream sensing applications.
3) Demonstrations of pose estimation, environment sensing, human state sensing, and static and dynamic task identification.
The remainder of this article is organized as follows. First, we describe our design logic for the resistive and fluidic sensing of our glove, followed by the machine learning pipeline used to transform raw sensing data into higher-level reasoning. Next, we describe the physical fabrication procedure for our glove, along with how data were captured to train the various neural network models used. Finally, we present demonstrations of our glove on a wide range of both simple and complex tasks, and conclude with thoughts toward future work.

System Overview
MemGlove is based on the idea that resistive sensing can be used to measure in-hand (pose) deformation, and fluidic pressure sensing can be used to measure out-of-hand (tactile) deformation, i.e., contact pressures. Strategic design choices ensure that these readings are mostly disjointed in the aspects of the interaction they capture, and they do not interfere with each other. Thus, these readings provide maximum information to then be (jointly) reasoned about by a downstream neural network model. This neural network model is responsible for translating raw signals to task-specific inference, such as hand pose reconstruction, grasped object classification, and so on, and is trained offline in a supervised manner from labeled ground-truth data.
One of our major contributions is the coupling of two complementary sensors into the glove, unlocking powerful downstream inference applications. We use a fully conductive thread glove as a substrate for the resistive sensing and sew in thin, tube-based sensors for fluidic pressure sensing. Electrodes are placed in strategic locations so as to maximize deformation information provided by resistance measurements. The resistance between all unique pairs of electrodes is Figure 1. Our wearable glove system. Resistive sensing and fluidic sensing combine to provide signals for strain and contact force information, respectively. These signals are fed into a pretrained neural network architecture for inference, which is trained from captured data labeled with ground-truth knowledge. Depending on the neural network used, the output of the network provides inference for a wide variety of downstream tasks, which we demonstrate. Electrode points which are usually not visible have been colored and enlarged in the left-hand images for clarity.
www.advancedsciencenews.com www.advintellsyst.com measured to allow large area measurements of strain information. The tube-based fluidic sensors provide pressure signals induced by compression, as per the ideal gas law. The fluidic tube sensor's fibrous nature allows it to be strategically sewn into the glove in a manner to optimize the response to normal force. The signals from the resistive and fluidic pressure sensing are read synchronously, and the data are fed into a machine learning model for task-based inference. For comfort, electronics are packaged into a small form-factor and mounted on the user's wrist. This is another advantage of our fiber-based systemboth the conductive and fluidic sensors require only compact transducers and minimal signal condition electronics which can be unloaded from the hand. The electronics are connected to a laptop that reads raw input signals (resistance measurements and pressure signals) and feeds them forward through a (context-dependent) neural network. The choice of network is modular and depends on the user's task, be it hand pose reconstruction, grasped object stiffness estimation, etc. We train these neural networks with recorded sensor data and ground-truth labels captured for the task at hand. The system is shown in Figure 1, and a bill of materials is given in the Supporting Information.
In the remainder of this section, we describe and characterize the resistive and pressure sensor design, detailing their decoupled nature, before describing our machine learning pipeline that synthesizes sensor data into higher-level sensing.

Resistive Knit Strain Sensing
In this work, we explore a commercial off the shelf knitted glove (Aglove) which has been developed to allow gloved use of capacitive touch screens. Due to the inclusion of the high conductance silver threads within the knit, the glove has a resistance of %5 Ωcm À1 which varies when a resistance is applied. Due to the knit pattern, the material can undergo %70% strain and when relaxed, the knit returns to a similar state. The knit pattern is directional, and thus the sensitivity and sensing capabilities of the knit are highly directional. Previous work has demonstrated the capabilities of custom knit-based strain sensors, [44] identifying how fabric and knit parameters can change the properties of the sensor. [45,46] Our decision to add a finite number of electrode connections to the "off-the-shelf" glove enables rapid creation.
To characterize the sensing capabilities of this knit, a section of the material was placed in an Instron Machine and a strain of 0-50% applied for 80 cycles. This was repeated for directions along and across the knit to characterize the directionality of the sensing capabilities. Figure 2 shows that the knit shows high sensitivity to strain along and low sensitivity to strain across the knit. This is adequate for our purposes, as the majority of deformations of the hand (particularly in the distal interphalangeal (DIP) and proximal interphalangeal (PIP) joints of the fingers) bend in the direction along the knit. The response along the knit shows high repeatability but also a very high hysteresis. This is most likely due to the relaxation and nonlinear behaviors of the knitted fibers. The average response time was found to be 0.35 s, whereas the recovery time was longer, and for the 80 cycles performed was found to be %0.8 s. This does potentially limit the use of the knitted glove as a sensor for some application which require a fast response, such as slip detection, but was not a major inhibiting concern for any of the applications presented in this article.
The choice of a fully conductive knitted glove as a substrate for our sensors admits three further benefits. First, it is fiber-based, making it easy to attach additional sewable components, such as our fluidic tubes or conductive fibers for connecting wristmounted electronics. This latter connection helps keep electronics off the hand itself, avoiding the need for flexible electronics www.advancedsciencenews.com www.advintellsyst.com and promoting ergonomic use. Second, the knit structure of the glove has minimal strain-based response to normal force/pressure. This ensures that the (pressure) responses of our fluidic tubes are maximally decoupled from the response of the knit. Finally, because the glove is fully conductive, hand self-collision (including fingertip-fingertip or fingertip-palm contact, as seen in an "OK" or "thumbs-up" gesture, respectively) emits a noticeable signal, making it easy to computationally detect these types of gestures.

Fluidic Tube Sensing
The principle behind our fluidic tube sensing is based off the ideal gas law: PV ¼ nRT. Compression of the fluidic tube caused by contact results in an inversely correlated response in internal pressure. As we will later show, the choice of fluidic sensors has additional benefitsits correlation to temperature means we can also detect changes in held objects' temperatures (given constant holding force). The fluidic sensors complement the sensing of the knit matrix, providing out-of-plane deformation information typically caused by environmental interaction. Our fluidic sensors are constructed from soft flexible tubes, sealed at one end and connected to a pressure transducer with an airtight seal at the other end. Although any airtight tubes or bladders could be used, a thin soft tube allows for easy integration into the glove.
To characterize the fluidic tube sensors, a cyclic normal force profile from 0 to 50 N was applied using an Instron Machine, repeated 80 times. The results from this characterization of normal force are shown in Figure 2. The sensor is sensitive to normal force and shows a linear and repeatable response with minimal hysteresis.
The strain response of the sensor is also investigated; ideally, the sensor should show a low sensitivity to strain so that normal force is isolated. Strain was applied to the sensor incorporated in the knit and cycled between 0% and 50% strain 60 times. This response is also shown in Figure 2. Although there is some response, it is far lower than that arising from normal force.

Learning Architecture
Our goal is to learn pose reconstruction and task classification in a supervised manner. To do this, we implement neural network architectures that combine our six tube pressure readings and our 120 multiplexed resistance readings (from 16 electrodes) into single or multidimensional (in the case of hand pose reconstruction) predictions. Our goal is to choose a simple neural-network-based architecture that can be reused for all tasks. As MemGlove is carefully designed to provide robust signals regarding wearer interactions, we find that even simple learning architectures are capable of providing highly accurate task-based inference. As our data are not organized in a particular grid-based or sequential fashion, we use a fully connected multilayer perceptron (MLP), which is well-suited for general unstructured data. We further train a long short-term memory (LSTM) model, using an MLP for each cell, to improve inference on temporally correlated tasks (such as hand motion). For each task, we compare the MLP and LSTM architecture; typically, the LSTM outperforms the MLP when samples are highly temporally correlated and constantly changing, and vice versa.
For each task, we record combined pressure and resistance readings, along with the ground-truth target for prediction-this can either be labels (for classification tasks, such as object identification), or real numbers (for regression tasks, such as temperature prediction), and can be one or multiple output labels. While we learn different neural network weights for each task, we reuse the same network architectures for nearly every task, which we have found work with high efficacy. The fact that the same architectures can be reused for each task makes learning new tasks simple and straightforward, and requires little hyperparameter search.
Our network architectures use standard, state-of-the-art techniques to improve inference performance. We apply layer normalization at each layer of the neural network. [47] The activation at each layer of the neural network is a rectified linear unit (ReLU), except for the final layer, which uses the tanh or softmax function, for regression and classification respectively. Each network used two layers of 64 hidden units, except for the more complex multidimensional prediction of hand pose reconstruction. Inspired by the work of Glauser et al., [32] , hand pose reconstruction uses a 2048-2048-2048-2048-1024 architecture for the MLP and a 512-512 architecture for each cell of the LSTM. We use standard loss functions for training; for regression tasks, such as temperature classification or hand pose reconstruction, we minimize the mean-squared error of the target output prediction; for classification tasks such as object classification, we minimize the cross-entropy measure.
As described, our architectures are based on established, vetted components. This makes implementing highly optimized versions of our models easy and compact. In our case, all models were implemented in TensorFlow. [48] 5. Implementation

Fabrication
The wearable glove is fabricated in two processes, as shown in Figure 3. First, the electrode connection points are added to the knitted glove "substrate," after which the fluidic pressure sensors are integrated.

Process 1: Resistive Knit Sensor Design and Fabrication
Using a conductive knitted glove as a base structure for the wearable device, a number of fixed connection points are made. By sequentially measuring the resistance between all pairs of n connection points, there are nðnÀ1Þ 2 resistance readouts, providing quadratic scalability in sensing. This is a similar approach to impedance spectroscopy. In this work, we consider 16 readout points which provides 120 unique sensor readouts.
A key design consideration is the placement of the electrode connections to the glove. This is a finite number, limited by electronic and fabrication constraints. The locations chosen should maximize the information that can be gained. Electrode connections have been made at points between joints to optimize the capture of strain resulting from joint bending. The 16 connection www.advancedsciencenews.com www.advintellsyst.com points chosen provide one connection point per major facet (i.e., bone between joints) of the hand, consisting of one connection per finger joint and two on the palm, as shown in Figure 3b). While the knit sensing only provides sensing along the fingers, the change in bulk material or contact between fingers allows sensing of stretch and movement in the direction across the hand. Although more readout points would provide more information, this design provides a good trade off in the complexity of hardware and the information gained.
The electrode connections to the glove have been created using a thin, soft, flexible, insulated multicore wire. The ends have been stripped and tied to the glove in the appropriate location on the inner side of the glove. The insulated wire is then tacked to the back of the glove, using a zigzag pattern to provide slack such that the wire does not prevent or limit movement while being comfortable for the wearer. To achieve the sequential resistance reads from the glove, each of the wired connections to the glove connection points is connected to two 16 channel multiplexers. The analog outputs from the two multiplexers are connected to a balanced Wheatstone bridge. By cycling through all unique combinations of resistor pairs, this builds up a map of resistance network of the glove. An Arduino Due microcontroller is used to control the multiplexers, with the output from these connected to the 12-bit Arduino Analog to Digital Converter (ADC). The balancing resistors have been chosen to maximize the output to the ADC. The microcontroller reads from each of the unique combinations of the glove and then transmits these over serial to the control personal computer (PC).

Process 2: Integration of Fluidic Sensors
The pressure sensors use a 2 mm outer diameter soft tube, which is highly flexible and can be sewn or woven into the glove with few limitations in form. One end is sealed by knotting, and the other end is given an airtight connection to the pressure transducer using an epoxy resin. Our design uses six pressure www.advancedsciencenews.com www.advintellsyst.com sensors-one on each of the five fingers, and one on the palm. The tubes have been integrated such that they run along centrally along the flats of the fingers. They are then routed in a gentle curvature along the fingers to flex outward at the joints when bent. This is to maximize the response to normal force and minimize false-positive reads induced by finger bending, while also ensuring they do not limit or obstruct finger movement. This layout is shown in Figure 3b. The pressure is measured using a Freescale MPXV6115V ported pressure transducer which provides an analog response. Each sensor is connected to an analog input on an Arduino Due which provides 12-bit resolution. The changes in pressure that are measured are very small (ranging from around 3 to 100 Pa), and the transducer has a sensitivity of 38.26 mV kPa À1 ; thus we perform oversampling to increase the resolution. By capturing 256 samples and averaging, we increase the resolution to 16-bit corresponding to the detection of 0.05 mV changes; this allows us to capture the small changes in pressure which occur.

Electronics and Data Capture
The data from both microcontrollers are communicated via serial to a PC where it can be read live or stored locally. Data are read simultaneously from the two microcontrollers, with data captured at %16 Hz, which is sufficient for all the applications considered in this work. If needed, the hardware could be later optimized to increase the read-out frequency.
In many of the experiments, an additional sensing system was used to record ground-truth labels for learning. Of particular note, to obtain ground-truth hand-posture for hand pose reconstruction, a Leap Motion hand tracking system was used. This is a system that has been successfully used for other motion tracking research, [49] and has been demonstrated to have a positional accuracy of up to 200 μm. [50] To obtain synchronized motion tracking data, the raw data from the Leap Motion were also read over serial alongside the data from the glove. The data were converted to the hand model from the study by Tkach et al., [51] ignoring the (irrelevant) global and wrist degrees of freedom. This model is shown in Figure 4b. In other experiments, a temperature sensor or a force sensor was incorporated to provide a ground truth. Similarly, these were connected to a microcontroller, with the data simultaneously read over a serial connection.
To ensure classification and regression tasks transfer between different glove-wearing sessions, all datasets were captured in multiple sessions. Between each session, the glove was fully removed and put back on again after a time period of at least 30 min. This rules out wear on the glove as a source of potential overfitting. Data were recorded in at least five sessions for classification tasks, and at least three sessions overall for each regression task.

Training Procedure
Training data were randomly divided into three sets: 45% of the collected data was used as a training set, 45% of the data was used as a validation set, and 10% of the data was used as a test set. Data were normalized in the training sets; those normalization constants were then reused for normalizing validation and test sets. As much as possible, data from the same recording session were kept together in the same set to ensure generalization of learned models on one session's captured data to another's. For all tasks, training was stopped when the validation loss ceased to decrease any further.   For all tasks we report the loss, as defined earlier, as well as a more human interpretable measure of real-world performance. For classification tasks, we report the probability of correct label classification. For regression tasks, we compute the average absolute error of the measurement. In the case of pose estimation, which has multidimensional prediction, this is the average error over all output angles.

Results
In this section, we demonstrate the sensing and inference capabilities of the glove system on a wide array of tasks. First, we demonstrate the ability of MemGlove to measure the intrinsic state of the user, including state-of-the-art accurate proprioception (hand pose reconstruction), as well as heart-rate monitoring. Second, we demonstrate how MemGlove can be used to infer wearer interactions with the physical environment. Finally, we demonstrate how our sensing suite combines for complex task classification, specifically classification of grasped objects and written letters. All training experiments were conducted on a laptop with a 2.9 GHz Intel i7 processor and a NVIDIA GeForce GTX 1080 GPU. In the following section, we present smoothed training, test, and validation data which were calculated with a smoothing window of 200 such that the general trends can be better observed. The raw results can be found in the Supporting Information.

Wearer-Intrinsic Sensing
We begin by demonstrating MemGlove's capability to reconstruct wearer hand pose, traditionally a very difficult problem with major implications for applications in virtual and mixed reality. Second, we demonstrate a simple application in health monitoring, specifically the capability of MemGlove to accurately predict the user's heartbeat. For both of these applications, we use only the resistive information of the glove, as we are not interested in interactions with the environment.

Hand Pose Reconstruction
Our first application is hand pose reconstruction, in which our system must infer the pose of the wearer's hand. We aim to faithfully reconstruct all joint angles of the hand, as per the kinematic hand model in the study by Tkach et al. [51] In this model, each finger (including the thumb) is modeled as having a ball joint at the base with three degrees of freedom, and two revolute joints further along the finger. See Figure 4b for a reference of the hand model. Pose reconstruction is notable as this example has a multidimensional output (25 joints) to predict.
The resistance data from the glove and the ground truth of the joint configuration were captured, with the (approximate) ground truth data being captured from a leap motion device. About %30 min of training data was captured, where the user aimed to move the hand in varying poses. Our trained MLP and LSTM were able to achieve an average per-joint accuracy of 6.40 and 4.78 respectively. Training curves are presented in Figure 4; as this task involves a lot of motion, the LSTM is able to leverage this dynamic information for higher reconstruction rates. We note that our LSTM-based reconstruction dominates even the best results in the study by Glauser et al., [32] the previous state-of-the-art which boasted on average 5.3 per joint error for the most fine-tuned model. Training is also realistically fast: for even this, our largest and most computationally expensive task and model, wall-time for training is 4.2s per epoch for the MLP, and 19s for the LSTM (we note all other experiments ran faster). Table 1 shows a breakdown of how that error is distributed across each particular joint; we further include a summary of how that error is distributed among the different biological joint classes. As can be seen, the majority of the joint errors are exceptionally low, even much lower than 5 on average. The primary exception is the abduction of the fingers, which has 10 -15 average error. This is likely primarily because MemGlove has poor sensitivity to strain response caused by lateral positional motions, due to the structure of the knit. The large standard deviations (and a nontrivial contribution of the error) are caused from spurious frame captures from the leap motion which occasionally invert the entire hand-we obviously cannot model this, leading to a heavy tail in our reconstruction distribution. Still, even with these combined errors, MemGlove demonstrates superior overall average reconstruction.

Heart Rate
In this demonstration, we show MemGlove's ability to infer the wearer's heartbeat. For this one application, we did not use a neural network estimator, but rather used standard signal processing tools and techniques to capture the periodic response of the heartbeat.
Although the most significant changes to the measured resistance of the glove correspond to postural changes, there are small changes that correspond to slight expansion/contractions resulting from the heart rate. Due to the many data channels in www.advancedsciencenews.com www.advintellsyst.com the glove, it is possible to average the frequency component over all sensors, filter, and extract the heart rate, even though the effective induced strains are small. To obtain the heart rate, the raw data from the glove held in a static pose are windowed over a 5 s period. For each of the resistive channels on the glove, the signal is converted into the frequency domain using a fast Fourier transform (FFT) and applying a low-pass filter, over which the responses from all sensors combined. By computing the power density of FFT and performing peak detection, the heart rate can be estimated. Due to the 16 Hz sampling rate of our current glove, the precision of the heart rate is only to the nearest 1 or 2 beats per minute (bpm); still, this is sufficient for many medical and fitness heart rate monitoring applications and is competitive with consumergrade monitors, such as Fitbit.
Data were collected with the user holding their hand in a neutral, approximately still pose. Approximately 10 min of data was captured during which the user performed light exercise (walking in place) to vary their heart rate. A wearable Fitbit heart rate monitor which has a precision of %2 bpm was also worn to provide ground truth. To ensure that that the glove is responding to only the heart rate measurements and not to, say, small changes in hand kinematics, the glove was placed on a mannequin hand (which has no pulse) as a control, and the experiment was repeated, ruling out this possibility. Figure 4c shows an example of the power spectrum obtained from the FFT. There is a clear peak at around 1 Hz, corresponding to a heart rate of %60 bpm. A comparison between the heart rate determined by the glove, and that measured by the Fitbit for the 10 min period is also shown. Overall, there is a strong agreement between the two devices' estimates.

Extrinsic Sensing
In the following section, we demonstrate the ability of MemGlove to sense external stimuli as a wearer interacts with the environment.

Holding Force
As a first, straightforward demonstration of extrinsic sensing, we show how MemGlove can be used to estimate the force with which a wearer is grasping a target object. This is a straightforward task, as the data captured from the fluidic pressure sensors naturally correlates with force. Therefore, for this task, we only use the fluidic pressure data. Holding force is a useful statistic, as it can be an indicator of health and recovery from several medical conditions, including muscular atrophy. To capture ground-truth data, a calibrated force sensing resistor (FSR) was attached to a rigid plastic bar. The bar was held and repeatedly grasped with different forces with data recorded from the fluidic sensors. Approximately 2 min of test data was obtained. Figure 5b shows the experimental setup, the test and training losses, and the performance using the trained networks. Because the grasps were changed rapidly throughout data collection, training data varied rapidly, and so only an MLP was considered   www.advancedsciencenews.com www.advintellsyst.com

Training, Test & Validation Loss
here. The trained network shows an average error of 1.74 N, which is low in comparison with grasping forces which can range from 0 to 80 N.

Object Stiffness
In a slightly more complex but similar task, we seek to measure the stiffness of grasped elastic objects. As the pressure tubes are also compliant, different object stiffnesses induce different deformations on the tubes at similar grasp strength, which leads to different pressure readings. Similar to the holding force experiments, this task also only relies on the pressure data readings. To investigate the ability of MemGlove to infer grasped material stiffness, six geometrically identical cubes of side 25 mm were fabricated from materials of varying stiffness including 3D-printed materials, EcoFlex silicones, and foams (full details are given in the Supporting Information). Each object was then grasped with the same hand pose and similar grasp strength. Time series data from the two sensors were obtained for six objects, all with ten repetitions. Figure 6a shows the training, validation, and test losses as well as classification probabilities, alongside a visual demonstrate of the experiment. As expected, as this is mostly static task, the MLP outperformed the LSTM, but both networks approached 100% classification rates.

Object Temperature Sensing
Although not a mechanical task, both our pressure sensors and our resistive sensors have the ability to inform estimates of grasped object temperature. Changes in temperature have a direct correlation with the resistivity of the knit glove; further, the fluidic pressure sensor readings are directly correlated temperature through the ideal gas law. Both measurements are used to provide an estimate of the temperature of grasped objects. Although both sensor modalities show some hysteresis in raw signal, the combination provides increasingly unique signatures for temperature change offering the potential of improved temperature detection. Further, the LSTM model provides smoothing and increased accuracy by considering signal history, resulting in no noticeable hysteresis or lag in prediction.
To obtain temperature training data, a thin coffee cup was equipped with an LM35 temperature sensor and hot water added to the cup. As the water cooled, the cup was held with a constant grasp. Data were recorded from the glove and ground truth obtained from the temperature sensor. This process was repeated five times with the cooling process accelerated via the addition of small amounts of cool water. Figure 5b shows the experimental setup, the training and test loss of the trained networks, and the performance of the trained network in terms of estimated absolute temperature difference. Both the MLP and LSTM network   www.advancedsciencenews.com www.advintellsyst.com perform well, but as temperature readings vary smoothly over time, the LSTM network outperforms MLP, achieving an average temperature estimation error of just over 1.0 C. We further use this experiment to examine the benefit of using two sensing modalities instead of one. Herein, we examine the mean error and standard deviation of the minimum mean test error recorded during training, averaged over three training runs. Our experiments demonstrate that pure resistive knit sensing provides an error of 0.96°C AE 0.02°C, fluidic pressure sensing provides an error of 1.2°C5 AE 0.03°C, whereas both modalities combined provides an error of only 0.92°C AE 0.01°C.

Material Conductibility
The resistive nature of the glove allows for identification of the conductibility of grasped materials. When the glove comes in contact with a conductive material, the material will act as a conductive path, changing the resistance profile of the glove. Such capability could have applications, for example, in sorting in recycling units, for separating metallic and nonmetallic recyclables. We treat this as a binary detection (conductive or nonconductive) task and do not attempt to estimate the conductability of the grasped object. As only the resistive sensing is sensitive to conductibility, it was the only sensing modality used for this task.
Data were collected for five different objects (box, pen, small ball, tube and soft toy). In one training data set, they were covered in aluminum foil to make them conductive, and in the other, they were not, remaining nonconductive. Each object was grasped five times in both configurations. This is a particularly challenging task, as the changing grasp interferes with the readings caused by the conductance changes. Ultimately, our networks are able to achieve 84.7% accuracy in classification between nonconductive and conductive objectssignificantly better than random ( Figure 6). This demonstrates that conductive pathways can be detected by the glove despite changing postures.

Complex Task Classification Tasks
In this section, we demonstrate MemGlove on two more complex tasks, one static, and one dynamic, in the form of grasped object classification and handwriting classification. For both of these complex tasks, both resistive and pressure sensors were needed and used to achieve the accuracies reported.

Object Identification: Static Task
To demonstrate the combined capabilities of MemGlove's sensors on a complex classification task, we seek to identify classification of the natural grasps of 30 objects. This reflects a scenario in which the glove could be used to assist with task identification through identification of a task-relevant, held item. The 30 household items (Figure 7a) selected were chosen to test the capabilities of the glove. These objects are of variable stiffness, shape, size, density, and material. To challenge the ability of the glove to differentiate similar objects, we included a number of objects with the same shape, but with different material properties, e.g. metallic verses plastic, or soft verses hard.
The training data set was generated by grasping each object ten times and recording time series data from both sensors after the object was grasped. Figure 7b shows the training and test loss and the performance of the classifier. As expected in this static task, the MLP slightly outperforms the LSTM network and achieves close to 100% classification success.

Handwriting Identification: Dynamic Task
As a final, challenging demonstration of MemGlove's capabilities, we investigate its ability to disambiguate handwritten letters. This task not only highlights the glove's precision (as handwriting leads to only small changes in posture and pressure distribution within the hand), but also presents a practical use case, providing another mode as an input device (e.g., for giving regular pens the capabilities of smart pens, such as the LiveScribe). It is also is a task in which there is large noise in the dataset; no handwritten letter is ever the same twice.
For each of the first ten letters in the English language alphabet (a through j), data from the sensors when the user wrote each letter was recorded, with 15 samples collected for each letter. Figure 7 shows the experimental results, showing that each letter has a distinct enough signature to allow for accurate classification. The MLP and LSTM both providing classification rates in the mid 80% range, compared with the 10% accuracy of true random.

Conclusion and Discussion
In this article, we presented a novel wearable glove design that combines knitted resistive sensing and fluidic pressure sensing with machine learning-powered reasoning for advanced realworld, real-time inference applications. We have demonstrated that MemGlove not only provides state-of-the-art hand pose reconstruction, but also unlocks applications in heart rate, temperature, force, stiffness, and conductibility sensing, as well as static and dynamic task classification. The range of different stimuli which the glove can reason about is significant, and outperforms previous work. Further, our fiber-based glove has low material cost and is exceptionally simple to fabricate using COTS components and is amenable to semiautomated or fully automated machine textile manufacturing methods.
MemGlove currently possesses a few limitations that would be worth investigating in future work. First, MemGlove has currently only be tested in single domain inference tasks. This means we can infer, e.g., hand pose, temperature, or holding force independently, but not jointly. Since similar changes in signal are used to learn each output label, we would likely need to improve the design or augment the glove with sensors and a more sophisticated reasoning pipeline that can better disambiguate the cause of the signal. Second, the temporal and signal resolution of MemGlove is limited by our choice of electronic hardware. Better electronics with higher resolution could be incorporated at additional monetary cost, bulk (affecting comfort), and complexity. It would be interesting to investigate these trade offs. Third, although our sensor architecture is effective and was chosen strategically, there is no evidence that our sensor placement is optimal. Computational or data-driven methods for optimizing sensor placement could further improve performance. Fourth, it would be helpful to add additional sensors to improve accuracy and precision of the abduction estimation. Finally, as this glove is a fiber-based wearable, it would be useful if it its connections could be modified to be resilient to machine washing.
Given its accuracy, robustness, and versatility, we believe MemGlove can unlock exciting applications in many fields. Soft robotic hands can use a modified version of our sensor architecture and machine learning pipeline for better proprioception and tactile manipulation. Virtual and mixed reality games and other applications could benefit from more immersive gestural input. Further, from heart rate sensing to hand force and motion monitoring, we believe MemGlove has many exciting opportunities in application to the medical field. We look forward to further investigation and application of our work in these and other fields.

Supporting Information
Supporting Information is available from the Wiley Online Library or from the author.  Figure 7. a) Our object classification dataset, with associated grasps. b) Evolution of the loss function for training, test, and validation sets, as well as real-world performance (in terms of correct classification probability) for c) grasped object classification; and d) Handwriting recognition classification.