Multisensory Pseudo-Haptics for Rendering Manual Interactions with Virtual Objects

Recent advances in extended reality (XR) technologies make seeing and hearing virtual objects commonplace, yet strategies for synthesizing haptic interactions with virtual objects continue to be limited. Two design principles govern the rendering of believable and intuitive haptic feedback: movement through open space must feel “free” while contact with virtual objects must feel stiff. Herein, a novel multisensory approach that conveys proprioception and effort through illusory visual feedback and refers to the wrist, via a bracelet interface, discrete and continuous interaction forces that would otherwise occur at the hands and fingertips, is presented. Results demonstrate that users reliably discriminate the stiffness of virtual buttons when provided with multisensory pseudohaptic feedback, comprising tactile pseudohaptic feedback (discrete vibrotactile feedback and continuous squeeze cues in a bracelet interface) and visual pseudohaptic illusions of touch interactions. Compared to the use of tactile or visual pseudohaptic feedback alone, multisensory pseudohaptic feedback expands the range of physical stiffnesses that are intuitively associated with the rendered virtual interactions and reduces individual differences in physical‐to‐virtual stiffness mappings. This multisensory approach, which leaves users' hands unencumbered, provides a flexible framework for synthesizing a wide array of touch‐enabled interactions in XR, with great potential for enhancing user experiences.

been proposed is visual pseudo-haptics.In contrast to delivering mechanical stimulation, visual pseudo-haptics use visual, spatial, or temporal illusions to convey the sense of touch. [11]Using head mounted displays (HMDs), these methods often amount to creating an artificial discrepancy between the user's rendered and actual hand locations.Doing so has been shown to invoke sensations such as stiffness, mass, and friction and is hypothesized to function based on manipulating the user's perception of work or exerted effort. [12]While visual pseudo-haptic effects can deliver proprioceptive and kinesthetic cues of stiffness without grounding the user, they obviously lack true tactile and force-feedback cues that are necessary for believable haptic feedback.Naturally, some researchers have combined visual pseudo-haptic cues with tactile haptic interfaces, [13,14] but these approaches still display the issues of hand encumbrance and limited applicability to all-day XR.
In this article, we present a novel approach─multisensory pseudo-haptics─to realizing haptic interactions in a virtual environment that combines visual pseudo-haptic feedback with tactile pseudo-haptics, achieved by applying haptic feedback at the wrist for cues normally felt at the fingertips.We use Tasbi, [15] a wearable haptic bracelet that provides a continuous squeeze force radially around the wrist, coupled with distributed vibration cues, to convey sensations, forces, and transients that would otherwise be expected at the hands and fingertips.Our proposed multisensory pseudo-haptic approach addresses the two primary design objectives of grounded haptic devices introduced nearly three decades ago, [2] namely that free space must "feel free" (achieved via a wearable bracelet) and that solid virtual objects must "feel stiff" (achieved via a clever combination of discrete and continuous visual and tactile pseudo-haptics).Rendering haptic feedback in an indirect or referred manner at a site located away from the hand overcomes the encumbrance issues that handheld, glove-type, and fingertip devices have, but may result in the loss of fidelity of the haptic experience and reduction of haptic cue saliency since the density of mechanoreceptors at the wrist is much less than that at the fingertips and other areas of the hands. [16]The inclusion of continuous squeeze cues at the wrist has the potential to enhance the haptic experience of users compared to visual-only feedback, supplementing the proprioceptive nature of the interaction rendered via visual pseudo-haptics with continuous tactile haptic sensations that are coordinated with user actions.
We hypothesize that multisensory pseudo-haptics, combining tactile wrist-based pseudo-haptics and visual pseudo-haptics, may induce genuine perceptions of users' virtual interactions, beyond just metaphors for haptic feedback.Further, this approach offers a way to balance the encumbrance issues of current haptic wearables with the lack of touch sensations in visual pseudo-haptic renderings.To this end, we first describe our approach for achieving believable mid-air haptic experiences, all while keeping the hands free.Then, in two psychophysical studies, we show that users map this multisensory sensation to physical object stiffness, and that users can discriminate the stiffness of different virtual interactions that are modulated through the parameters of the pseudo-haptic and referred haptic feedback renderings.Our approach provides a framework for creating a wide array of touch-enabled interactions in XR, with great potential for enhancing user experiences.

Multisensory Mid-Air Interaction Paradigm
The primary contribution of this paper is a hands-free, haptic feedback paradigm for XR interaction that combines tactile pseudo-haptic feedback (squeeze and vibrations) from a bracelet with visual pseudo-haptic feedback delivered via an HMD. Figure 1 provides an explanation of how we combine multisensory pseudo-haptic cues to create a believable interaction of Figure 1.Multisensory pseudo-haptic paradigm applied to a mid-air button interaction.The top row represents the virtual interaction, while the bottom row depicts the respective motion of the user in physical space.A) The user approaches the button.The control hand (blue) and display hand (black) are initially collocated.B) The user makes initial contact with the button.Tasbi's linear resonant actuators (LRAs) render a vibration to simulate the contact event.C) The user begins to push the button downward.Tasbi squeeze force increases proportionally to the button displacement and spring force.The control hand continues to track the user's true hand position and orientation, while the display hand remains on the surface of the button.D) At the end of travel, squeeze reaches its maximum (desired) force level, and the C/D discrepancy has become more pronounced.Note that the blue control hand is shown in these figures only for illustrative purposes and is not displayed to users.
pressing a "stiff" virtual button.As the user's hand approaches the virtual button, the rendered hand (the display) and the actual hand (the control) are collocated (Figure 1A).Using terminology from the classic "god-object" rendering method, [17] the rendered (display) hand is the god-object, and the user's hand (control) is the haptic interface point.Note that the user cannot see the blue control hand in practice.When collision is detected between the button and fingertip, a vibration stimulus is rendered through the bracelet to simulate the contact event (Figure 1B).As the user depresses the button (Figure 1C), we manipulate the location of the rendered hand such that it lags the actual hand by a proportional amount parameterized by the control-to-display ratio (C/D), thus providing the sensation of impeded motion.Simultaneously, we deliver increasing squeeze forces through the wristband to convey the resistive spring force of the button.As the user reaches the end of button travel (Figure 1D), the C/D discrepancy and squeeze intensity reach their maximum (desired) values, and a vibration stimulus is rendered to depict the button hard stop or activation event.Specific details regarding our implementation of the visual pseudo-haptics (C/D manipulation) and tactile pseudo-haptics (squeeze and vibration) can be found in the Experimental section.
When combined, the visual and tactile pseudo-haptic cues deliver a believable sensation of pressing a stiff button, despite the interaction occurring in entirely free space with no feedback delivered to the hand or fingertip.Notably, we predict that no visual or tactile pseudo-haptic cue can lead to compelling feedback alone, but instead highly believable substitutive feedback for mid-air interactions with virtual objects will depend on their multisensory combination.To understand if this multisensory pseudo-haptic combination invokes believable perception of virtual objects, we devised two psychophysical experiments, presented in the following sections.

Experiment 1: Estimating Absolute Stiffness
Here, we sought to identify whether individuals map virtual button stiffness-conveyed through tactile or visual pseudo-haptics-to physical stiffness measured using a real button.We also sought to determine whether stiffness mappings differed when virtual stiffness was conveyed via multisensory pseudo-haptic cues compared to tactile pseudo-haptic or visual pseudo-haptic cues only.Our first experiment (Figure 2) tasked subjects (N ¼ 12) with adjusting the stiffness of a physical button until it was perceptually equivalent to the stiffness of a virtual midair button.The mid-air button could be in one of the three conditions: Tactile-only (T; i.e., tactile pseudo-haptics comprising squeeze and vibration), Visual-only (V; i.e., visual pseudo-haptics via C/D manipulation only), or Tactile-Visual (TV; i.e., multisensory pseudo-haptics).Each condition was tested in separate blocks, beginning with a series of repeated pretest trials, and followed by a series of test trials where repetitions of 4 virtual stiffness levels (mapped to squeeze forces or C/D ratios) were presented in random order.Displacement and force plots obtained from representative trials for each condition can be found in Figure S3, Supporting Information.
Subjects reliably matched the stiffness of a physical button to the stiffness of a mid-air button.As squeeze force or C/D ratio increased, participants tended to adjust the physical button to have greater stiffness (Figure 3A-C).A two-way repeated measures analysis of variance (ANOVA) on the adjusted stiffness values revealed significant main effects of condition (F(2,22) ¼ 3.6, p ¼ .043)and level (F(3,33) ¼ 211, p < .001) as well as a significant condition X level interaction (F(6,66) ¼ 6.2, p < .001).These results indicate that the matched stiffness values differed between the T, V, and TV conditions, and these differences varied as a function of level.
Given the significant interaction between condition and level on matched stiffness values, we fit a linear function to each subject's data to quantify behavior under the T, V, and TV conditions (Figure 3D-F).The slope parameter for the linear functions describes how physical stiffness varies as a function of virtual stiffness in each condition.The linear functions generally provided a good description of the stiffness reports (mean R 2 : .91AE .15)which increased monotonically as a function of level.Linear function fits were better for the TV condition (.97 AE .03)compared to the T condition (.81 AE .22;p ¼ .021),but not the V condition (.95 AE .06;p ¼ .15)(Figure 4B).These results Figure 2. Experiment 1 overview.A) Subjects were presented with two buttons─a virtual, mid-air button rendered through visual pseudo-haptics and/or tactile pseudo-haptics, and a physical button rendered through a custom apparatus collocated with the virtual environment.B) Subjects increased or decreased the stiffness of the physical button with the controller thumb stick until it was perceptually equivalent to the virtual button, which was held at constant stiffness for each trial.Subjects were allowed to transition freely between buttons within the 20 s time limit.    .A) Slopes obtained from the linear fits to subject level means.Slopes in the bimodal condition were significantly greater from those found in the unimodal conditions, indicating that bimodal feedback extends the range of stiffness that can be rendered.B) The quality of fit, R 2 , obtained from the same fits.C).The square of residuals from the group-level fit, collapsed across level.Significantly smaller residuals in the bimodal condition suggest that subjects were more consistent with each other in this condition compared to the unimodal conditions.All error bars represent a 95% confidence interval.
imply that the mapping between virtual and physical stiffness is linear when multisensory pseudo-haptics are used, but this linearity breaks down when only tactile pseudo-haptic cues are available.Estimated slopes also differed between conditions (T: 37.8 AE 19.9; V: 45.8 AE 15.4; TV: 57.1 AE 11.4) (Figure 4A).The significantly larger slopes in the TV condition compared to the T condition (p ¼ .013)and V condition (p ¼ .020)imply that the combined use of tactile and visual pseudo-haptic cues expanded the range of physical stiffnesses that could be associated with the rendered virtual cues.We next determined whether the unimodal and bimodal virtual stiffness cues influenced across-subject consistency in stiffness mappings.Visual inspection of estimated stiffness values under each condition (Figure 3) reveals substantially less individual variability in the TV condition compared to the unimodal conditions.To quantify this relationship, we considered how each subject's data deviated from the group-level linear fit in each condition (Figure 3 These collective results imply that the combined use of tactile and visual pseudo-haptic cues leads to greater uniformity in the mapping between virtual and physical stiffness.

Experiment 2: Discriminating Virtual Stiffness
The results of Experiment 1 indicated that subjects could map virtual stiffness cues─rendered through tactile pseudo-haptics and visual pseudo-haptics, individually or in combination─to the stiffness of a physical button.In Experiment 2 (Figure 5), we determined whether subjects could discriminate between the virtual stiffnesses of two mid-air buttons using the unimodal or bimodal virtual stiffness cues.In a 2-alternative forced choice (2AFC) paradigm, subjects pressed two mid-air buttons on each trial and reported the one that was perceived to be of a greater stiffness.On each trial, one button was always rendered at a standard stiffness level, and the other button was rendered at a comparison stiffness level that varied from low to high.Comparison stiffness values, chosen to equate the perceived stiffness of the T and V cues, were determined from the group-level fits from Experiment 1 (see Figure S1, Supporting Information).Displacement and force plots obtained from representative trials for each condition can be found in Figure S4, Supporting Information.
Subjects reliably discriminated the stiffness of mid-air buttons (Figure 6), and their performance was captured by a standard psychometric function (R 2 ¼ .96AE .05) that allowed us to quantify subjects' bias and discrimination thresholds under each condition.Discrimination thresholds (just noticeable difference, JND) differed significantly depending on whether the virtual stiffness was rendered using tactile pseudo-haptics, visual pseudo-haptics, or both modalities (Figure 7A) (F(2,20) ¼ 5.5, p ¼ .019).In general, discrimination thresholds were smaller with bimodal stiffness cues compared to the unimodal cues.Indeed, JND values in the TV condition were significantly lower compared to the JND values in the V condition (p ¼ .007).The difference between JND values in the TV and T conditions did not achieve statistical significance (p ¼ 0.07) despite the qualitative differences.JND values in the T condition did not differ significantly from JND values in the V condition (p ¼ 0.67).No significant differences were found comparing the point of subjective equality (PSE) values between conditions (F(2,20) ¼ 2.74, p ¼ .09).These results indicate that the combined use of tactile and visual pseudo-haptic cues results in finer sensitivity to stiffness variations with minimal impacts to bias.
Given the reduction of discrimination thresholds in the TV condition, we assessed how overall performance compared between conditions.Consistent with the JND effects, we found a significant effect of condition for the percentage of correct responses (F(2,20) ¼ 8.879, p ¼ 0.002) (Figure 7B).Post hoc tests Figure 5. Experiment 2 overview.A) Subjects were presented with two virtual buttons which were both rendered through either visual pseudo-haptics and/or tactile pseudo-haptics.One button, either left or right, was the standard stiffness, and the other button was the comparison stiffness.B) Subjects were allowed to press each button twice, in whichever order they preferred, before selecting the stiffer of the two buttons using the controller thumb stick.revealed that subjects achieved significantly higher performance in the TV condition compared to the V condition (p ¼ .002).Although performance in the TV condition was nominally higher compared to the T condition, this difference was not statistically significant (p ¼ .056).No performance differences were found comparing the T and V conditions (p ¼ 0.32).
In Experiment 1, multisensory pseudo-haptics yielded stiffness mappings that were more consistent across subjects compared to unimodal cues.Discrimination performance in Experiment 2 (Figure 6) followed a similar pattern with a greater uniformity of response profiles in the TV condition compared to the T and V conditions.We tested this by quantifying how each subject's performance deviated from the group-averaged psychometric function in each condition.Across subjects, the sum of squared residuals in each condition differed significantly (F(2,240) ¼ 6.3, p ¼ 0.005) (Figure 7C).Post hoc tests showed that residual errors in the TV condition were significantly smaller than residual errors in the T condition (p ¼ 0.002) and the V condition (p ¼ 0.002).Residual errors did not differ between the T and V conditions (p ¼ 0.69).These results imply that discrimination performance was more uniform across subjects when virtual stiffness was rendered using bimodal cues compared to tactile or visual pseudo-haptic cues alone.
In Experiment 2, a short survey was presented at the end of each block.The questions, listed in Table 1, highlighted subjects' disposition toward the buttons presented in that block.Subjects responded to these questions using a continuous slider on a scale of Strongly Disagree (0.0) to Strongly Agree (1.0).The same questions were presented after each block, with the order randomized.The survey results presented in Figure 8 show that subjects rated the TV button higher in terms of its believability, pleasantness, and naturalness of interaction.Subjects indicated that the TV and V buttons were roughly equivalent in terms of realism and higher than the T button.Intuitiveness of interaction was positive for all three buttons, with the TV button rated slightly higher.The questions regarding location of hand and body ownership seem to indicate that subjects were largely unfazed by the C/D manipulation and the discrepancy between their actual hand location and the rendered hand location.Finally, the TV button inspired more confidence in the selection process than either of the unimodal buttons.

Conclusions
Here, we described a multisensory hands-free strategy for rendering intuitive and believable haptic XR interactions.We demonstrated the feasibility of using multisensory pseudohaptics to render the stiffness of manual interactions with virtual buttons.Participants reliably perceived the stiffness of virtual button interactions, which could be equated to the stiffnesses sensed in interactions with physical buttons.Multisensory feedback expanded the range of physical stiffness that could be associated with virtual interactions compared to tactile or visual feedback alone.The efficacy of virtual stiffness rendering did not depend on referencing to physical objects as participants also systematically discriminated between the stiffnesses of two virtual buttons.Multisensory feedback increased sensitivity to differences in virtual button stiffness and discrimination performance.Lastly, multisensory pseudo-haptic feedback reduced individual differences in physical-to-virtual stiffness mappings and task performance compared to unimodal pseudo-haptic feedback.Collectively, our results demonstrate the potential for using multisensory pseudo-haptics to render interactions with virtual objects in a wide variety of applications.
Our multisensory pseudo-haptics strategy leverages the combined use of visual feedback via a head mounted display and tactile feedback via a wrist-worn bracelet to convey redundant information about manual interactions with virtual objects (Figure 1).In our experiments, dynamic visual and tactile cues─in the form of visual pseudo-haptics and tactile pseudo-haptics─provided participants redundant and  .Asterisks indicate significant difference from a neutral response of 0.5.See Table 1 for the full questionnaire.
continuous feedback regarding their interactions with a virtual button.Indeed, with little training, participants were able to interpret the visual and tactile cues separately as reflecting the stiffness of a virtual button press, as indexed by explicit comparisons to the stiffness of physical button presses (Figure 2-4).
Because modulation of the visual and tactile feedback correlated with the displacement of the virtual button (Methods), participants may have relied on these signals as kinesthetic cues.
Conceivably, the referral of button stiffness to wrist squeeze, analogous to the use of squeeze bands in teleoperation [18,19] and prosthetic [20] applications, may have invoked associations with muscle activation patterns normally involved in finger movements and grip control.Additionally, participants may have interpreted the feedback signals as a proxy of the effort exerted to displace the button, as described in other visual pseudo-haptics applications. [12]Redundancy in the visual and tactile feedback also frees users to direct their gaze flexibly as they maintain manual interactions with virtual objects, liberating the eyes as well as the hands in XR.Accordingly, our multisensory pseudo-haptics approach addresses the two design principles governing the rendering of realistic haptics, namely that the haptic device must make free space feel free and must render stiff virtual objects. [2]oncurrent presentation of redundant visual and tactile pseudo-haptic feedback conveyed several advantages over the use of either feedback modality alone.First, participants were better able to discriminate between the stiffness of two virtual buttons using multisensory pseudo-haptics (Figure 5-7).Enhanced performance accuracy may have resulted from a finer sensitivity to stiffness variations when redundant pseudo-haptic cues were available.In fact, the reduction in perceptual thresholds (i.e., JND values) with multisensory pseudo-haptic feedback is consistent with the notion that participants integrated the visual and tactile pseudo-haptic cues in a statistically optimal manner. [21]This possibility can be tested further by evaluating how users perceive incongruent tactile and visual pseudo-haptic feedback.The use of continuous multisensory pseudo-haptic feedback that was correlated with participants' dynamic interactions with the virtual button may have facilitated ultra-rapid learning for optimal integration, [22] similar to how a learningbased approach can lead to optimal integration of artificial sensory feedback with neuroprostheses. [23]Second, multisensory pseudo-haptics expanded the maximum range of physical stiffness values that were associated with virtual button interactions, from 161 N m À1 in the Tactile-only condition and 200 N m À1 in the Visual-only condition, to 236 N m À1 in the Tactile-Visual condition.Importantly, beyond merely increasing the slopes of the linear functions relating pseudo-haptic cues to stiffness, the use of multisensory cues also improved the overall linear fits.Whether multisensory pseudo-haptics also promotes linear mappings over larger stiffness ranges or in other virtual interaction domains remains to be tested.Lastly, multisensory feedback reduced individual differences in task performance.When participants compared the stiffness of a virtual button to its physical counterpart, the mappings between virtual and physical stiffness were more uniform across individuals when multisensory cues were available (Figure 3G-I, and 4C).Similarly, when participants discriminated the stiffnesses of two virtual buttons, multisensory pseudo-haptics yielded more consistent choice probability patterns compared to performance achieved with only visual or tactile cues (Figure 6 and 7C).These collective results reveal the potential benefits of multisensory pseudo-haptics to render virtual interactions that maximize the sensitivity and consistency across users.
A potential limitation of our study is that multisensory pseudo-haptics were tested only in a VR context where we had complete control over participants' visual experiences.As such, the utility and efficacy of our rendering method in augmented reality (AR) contexts remain to be verified.If an HMD leverages transparent displays with light projections or waveguided optics, any implementation of visual pseudo-haptic cues will necessarily require coordination with the unobstructed view of the user's hand in its veridical position.Alternatively, if the HMD comprises an opaque display with real-time video passthrough, the video feed can conceivably be intercepted and manipulated to produce the visual pseudo-haptic effects we achieved in the VR context.Notably, tactile pseudo-haptic cues remain available in all contexts.Another aspect of our approach that requires further consideration is the role of vibration in tactile pseudohaptics.For interactions with virtual buttons, vibrations were transiently delivered through the bracelet interface only to signal the moment that a participant's finger made initial contact with the button and when the button reached its end of travel.Mechanical vibrations serve established roles in texture perception [24] and sensing through handheld tools. [25]Accordingly, there is an obvious need and opportunity to leverage vibration cues in tactile pseudo-haptics for rendering manual interactions with virtual objects.Moreover, multisensory pseudo-haptics may further benefit from the addition of auditory feedback, which conveys rich information about object-based interactions [26] and systematically shapes tactile vibration perception. [27,28]inally, our experiments explored only how user performance with the combination of referred haptic feedback at the wrist and visual pseudo-haptics compares to each modality alone.Further research is needed to better elucidate the other end of the spectrum where high-fidelity haptic feedback via hand-held devices or gloves is compared to our approach, and the trade-offs of haptic fidelity and encumbrance are explored in more detail.
As an initial demonstration of the broad potential of multisensory pseudo-haptics, we have successfully incorporated our rendering approach into a variety of applications (Table 2) beyond button pressing.The paradigm can be easily extended to other motion primitives, such as pulling an object with linear stiffness (Pull Switch) or rotating an object with torsional stiffness (Rotary Dial).These interactions provide the same haptic cues as the buttontactile pseudo-haptic manipulation via continuous squeeze for forces and transient vibrations for tactile events, and visual pseudo-haptic manipulation of the control-to-display ratio.The believability of any interaction can be further enhanced by layering additional transient vibration effects on top of squeeze, such as in the case of detents on a knob (Rotary Dial).Similarly, surfaces can be rendered using squeeze for macroscale geometric features and surface normals, while microscale texture details are rendered with vibration effects (Ripples).Our paradigm is also applicable to in-hand manipulations or rendering forces between the fingers (Hand Gripper), conveying the sense of mass and inertia (Tennis), and providing a sense of locomotion (Ladder).In bimanual applications, two wristbands can be used to render reaction forces or tension that would be expected between the limbs (Longbow).Finally, even with relatively simple pseudo-haptic feedback, we can also deliver distinct sensations in multifaceted interactions when the haptics are effectively matched to the user's motion and the visuals rendered (Virtual Bop It).While the broad potential for multisensory pseudo-haptic applications is clear, it is important to note the requirement to fit and calibrate the bracelet to individual users (Experimental Section).We have widely demonstrated these midair interactions in public and private forums to both experienced hapticians and novices alike.Beyond our psychophysical Table 2. Applications of our multisensory pseudo-haptic paradigm.Ten examples of the proposed feedback paradigm are shown for unimanual and bimanual interactions.Each interaction attempts to address a unique rendering target using the combination of tactile (squeeze and vibration) and visual pseudo-haptics.Squeeze is represented by arrows and solid lines and vibration is represented with a buzz icon.The visual pseudo-haptic effect is not shown for visual clarity; refer to Figure 1 for an illustration instead.findings, positive feedback on the believability of interactions supports our overall hypothesis that no sensory modality alone can lead to compelling feedback, but through combining tactile and visual feedback in a multisensory fashion it is possible to render highly believable substitutive feedback for mid-air interactions with virtual objects.

Experimental Section
In the following section, we present detailed methods necessary to implement our multisensory pseudo-haptic paradigm and replicate our two experiments.First, we introduce the Tasbi haptic bracelet, which enabled us to convey mid-air haptic experiences by providing tactile pseudo-haptic feedback comprising continuous squeeze around the wrist and discrete vibrations.Then we derive the equations needed to control the visual pseudo-haptic C/D manipulation and squeeze forces that were used to render the virtual button throughout both experimental paradigms.Finally, we outline each experiment's protocol, including materials, methods, and statistical analyses.
Tasbi Haptic Bracelet: Tasbi is an advanced multisensory haptic bracelet that incorporates both localized vibration and uniform squeeze feedback. [15]Vibrations were delivered via six linear resonant actuators (LRA) radially spaced around the wrist.Each LRA was independently controllable through the Syntacts vibration rendering framework, [29] allowing for the development of expressive feedback patterns.Squeeze feedback was accomplished through a cord tensioning mechanism that had been shown to provide uniform and nominally normal (to the skin) forces of up to 15 N at each contact point around the wrist. [15]The amount of squeeze force, F squeeze , was rendered by means of a closed-loop force controller leveraging a capacitive force sensor located at the base of the tensioner housing.
To ensure that Tasbi delivered consistent squeeze forces across users, all experimental sessions began by fitting and calibrating the bracelet to the user.3 M Transpore and double-sided mounting tape were used to secure the bracelet to the subject's arm approximately 6 cm behind the ulnar styloid process.Once Tasbi was securely mounted, the subject placed their wrist and Tasbi under a controlled force applicator.The apparatus delivered a sinusoidal force profile sweeping from 0 to 15 N to the top side of Tasbi.Voltage measurements from Tasbi's internal capacitive force sensor were calibrated against the applied force.Following this, subjects removed their arm from the calibrator, and Tasbi was verified to render the full range of 0-15 N to the user's arm.
Virtual Button Formulation for Multisensory Feedback Paradigm: A detailed description of the virtual button presented in Figure 1 and subsequently used for Experiments 1 and 2 is shown in Figure 9. First, a vibration stimulus was rendered through Tasbi's LRAs when the user first contacted the surface of the button.We used an exponentially decaying sinusoidal model [30] of the form Ae ÀBt sinð2πωtÞ with parameters of A ¼ 0.125, B ¼ 20, and ω ¼ 175Hz to generate the contact event waveform.As the user began to press the button, the visual C/D manipulation and squeeze stimuli began to take effect.While C/D ratios are often arbitrarily chosen for simple UI applications, we used a physically based C/D approach that was loosely described in our previous paper [31] and subsequently formalized by our co-authors in ref. [32] This approach differed from traditional methods in that C/D was not simply a visual scaling from the control to the display, but rather the manifestation of changing the parameters of the button impedance.As such, the button was a simulated second-order system parameterized by a mass m, a damping coefficient b, a stiffness k, and a displacement x mx Interaction forces applied to the button were computed from the displacement of a virtual spring of stiffness k p connecting the control and display fingertips.Thus, greater penetration depths of x p resulted in larger forces being applied to the button We achieved a desired steady-state C/D by changing the ratio of object stiffness k to proxy hand stiffness k p .The C/D ratio λ was defined as Choosing to ignore the dynamic contributions of m and b, the force balance equation was simplified to Finally, combining (3) and (4), we defined the C/D ratio λ in terms of the two stiffnesses Thus, to achieve a desired C/D ratio λ the implementation could compute either k or k p while holding the other constant.Although either approach was valid, we chose to hold the proxy hand stiffness k p constant and let the desired C/D ratio drive the calculation of button stiffness k. Figure 9. Formulation of multisensory pseudo-haptic button.A) The user approaches the virtual button simulated by a mass m, stiffness k, and damping b.The control finger (blue) and display finger (gray) are coupled via a virtual spring of stiffness k p and are initially collocated.When contact is made, Tasbi's six LRAs C) render a vibration to simulate the event.B) The user begins to push the button downward.Tasbi squeeze force increases proportionally to the button displacement x.The control hand continues to track the user's true hand position and orientation, while the display hand remains on the surface of the button.The control-to-display (C/D) ratio is given by the ratio of x control and x display .At the end of travel, squeeze reaches its maximum force level, and the C/D discrepancy is most pronounced.
It is important to note that because we chose to ignore the object mass and damping in the force balance Equation (1) the actual C/D ratio fluctuated slightly during dynamic interactions.The amount of C/D error in the dynamic case was minimized by carefully tuning the free variables k p , m, and b to the anticipated speed of pressing.A more robust method may have chosen to include the object's mass and damping terms in the force balance equation, and/or add a damping term to the proxy finger impedance.
Finally, the amount of squeeze to be applied to the user's wrist was proportional to the simulated button's displacement Squeeze rates, k s , were computed from the maximum desired squeeze stimuli force for a particular trial, F squeeze,max , and the button's full displacement length of 35 mm For example, the squeeze stimuli levels F squeeze, max of 3, 7, 11, and 15 N in Experiment 1 gave squeeze rates k s of 85.7, 200, 314, and 249 N m À1 , respectively.Experiment 1 Overview: Subjects performed the first experiment in a VR environment created in Unity Engine.An Oculus Rift CV1 served as the VR HMD, and Oculus Touch controllers were used to track subjects' hand position and gestures.Within the environment, subjects were presented with two visually identical buttons placed side by side (Figure 2 and S2, Supporting Information).The button on the left represented the physical stiffness and was visuo-spatially aligned with the physical button apparatus (Figure 2) so that when subjects pressed the button in VR, they felt the physical button.The button on the right represented the mid-air stiffness and displayed one of the three haptic rendering methods as detailed in the following sections.Both buttons were 50 mm in diameter and 35 mm tall.Their color was nominally a shade of green but changed to pink if displaced beyond the table surface, indicating to the subject to stop pushing.
Variable Stiffness Button Apparatus: A physical, variable stiffness button (VSB) was constructed and served as the physical stiffness comparison (Figure 2).The VSB was driven by a Maxon RE-25 motor and capstan cable mechanism, like those found in many desktop haptic displays, with a transmission ratio of 0.105 mm deg À1 .Closed loop current control was accomplished via an Advanced Motion Controls AB15A100 servo drive and a Quanser QPIDe DAQ interface sampled at 2 kHz on the host PC.After gravity (0.59 N) and kinetic friction (0.18 N) of the button were appropriately compensated for with feed-forward control, a proportionalderivative (PD) position controller allowed for setting the desired stiffness k vsb and damping b vsb of the button.The physical button was able to simulate stiffnesses ranging from 5 to 400 N m À1 before over-drawing current from the power supply.Throughout the experiment, the button damping was computed such that the button was always critically damped: where the button mass m vsb was determined to be 0.06 kg.
Experiment 1 Methods: Using the method of adjustments, [33] Experiment 1 tasked subjects with adjusting the stiffness of the physical button on the left until it was perceptually equivalent to stiffness depicted by the mid-air button on the right.The experiment was divided into 3 experimental blocks defined by the pseudo-haptic rendering condition (unimodal or bimodal) of the mid-air button.Each conditional block tested four pre-defined virtual stiffness levels.
Visual Condition (V): The stiffness of the mid-air button was conveyed only through a unimodal C/D stimulus.The 4 virtual stiffness levels tested were C/D ¼ 3, 4, 5, and 6.The proxy hand stiffness k v was set to a constant 50 N m À1 , and the mass of the button m b was set to 0.06 kg, equal to the mass of the physical button.Like the physical button, the mid-air button was always critically damped given the desired C/D ratio and resulting computation of the button stiffness k.The choice to critically damp both buttons was made so that subjects would not inadvertently use oscillatory motion to assess stiffness similarities.No tactile cues (squeeze or vibrations) were presented in this condition.
Tactile Condition (T): The stiffness of the mid-air button was conveyed only through a unimodal wrist squeeze stimulus, and the C/D simulation was disabled.In this condition only, both the mid-air button and the physical button were made visually transparent and static (Figure S2E,F, Supporting Information), so that subjects could see their fingers move through the button volumes but could not see the buttons displace.The effect minimized subjects' usage of visual information and forced them to make the comparison based only on what they felt.The squeeze force was proportional to the amount of finger penetration into the button volume, reaching a maximum force at x control ¼ 35 mm.The 4 virtual stiffness levels were defined by this maximum squeeze force, F squeeze,max ¼ 3, 7, 11, and 15 N.In addition to the squeeze stimulus, vibration cues signaled initial contact with the button and when the button had reached its end of travel.In the context of our experiments, we consider the combination of squeeze and vibration cues collectively as tactile pseudo-haptics.
Tactile-Visual Condition (TV): Virtual stiffness in this condition was conveyed by multisensory pseudo-haptic cues comprising the same tactile and visual pseudo-haptic cues used in the unimodal conditions.The 4 virtual stiffness levels from the unimodal conditions were used and presented congruently (e.g., C/D ¼ 4 with F squeeze,max ¼ 7 N, etc.).In this condition, F squeeze was proportional to the C/D button displacement x instead of x control as was necessary in the Tactile condition.
Each conditional block consisted of 64 trials.In each trial, subjects were presented with a mid-air button displaying a fixed virtual stiffness level.The starting stiffness of the physical button was randomly set either near the low end of its rendering range at 25 N m À1 or the high end at 375 N m À1 .Subjects assessed both buttons and then used the thumb stick of the Oculus Touch controller to change the stiffness of the physical button until it was perceptually equal to the mid-air button.Subjects were allowed to transition between buttons freely but were required to complete the adjustment in 25 s.Subjects were given the option to advance to the next trial once they were confident both buttons were equivalent.
The first 16 trials of each block were practice trials and presented the same virtual stiffness level, which was taken from the center of the tested stiffness ranges (e.g., C/D ¼ 4.5 and/or F squeeze,max ¼ 9 N m À1 ).The remaining 48 trials randomly presented one of the 4 test virtual stiffness levels, each repeated 12 times.Subjects were given a short break in between each conditional block.Importantly, the presentation order of the three blocks was randomized between subjects so that each of the 6 possible orders was equally represented.Overall, the experiment lasted approximately 2 h for each subject.
A total of 12 subjects (age: {M ¼ 22, SD ¼ 2.9}, 8 female) completed the experiment.Subjects were recruited from the Rice University undergraduate and graduate student bodies under Rice University IRB protocol #IRB-FY2020-43 and provided written informed consent.They were compensated 20 USD upon completion of the experiment.Except for a single subject, none had any experience with squeezing haptic displays, and all reported no or very limited experience with VR systems.
Experiment 2 Overview: Subjects performed the second experiment in the same VR environment as Experiment 1; however, for this experiment, the physical variable stiffness button was removed, and both the left and right buttons were rendered through tactile pseudo-haptics (vibration and squeeze) and/or visual pseudo-haptic manipulation (C/D ratio).The size and visual variations of the buttons remained the same as described in Experiment 1 Overview.
Experiment 2 Methods: Using the method of constant stimuli [33] and a two-alternative forced-choice (2AFC) procedure, the experiment tasked subjects with selecting the stiffer of two mid-air buttons.Each of the three mid-air button conditions (V, T, and TV) was presented in a separate experimental block.Each block consisted of 8 practice trials and 220 test trials.In each trial, two visually identical buttons were presented on the left and right, and subjects were allowed either five seconds or two presses of each button, whichever came first, to decide on which button was the hardest or stiffest to press.Subjects were instructed to alternate between the buttons and to make their selection as soon as they were confident.Subjects made their selection by moving the thumb stick on the Oculus controller to the left or right.One button, the standard, presented the same stimulus in every trial and was randomized to appear on either the left or right side an equal number of times.The other button, the comparison, displayed one of 11 virtual stiffness levels: 5 below the standard level, 5 above the standard level, and the standard level itself.Each comparison level was repeated 20 times, and the presentation order was randomized for each subject.
The virtual stiffness levels for the T and V conditions were derived from the results of Experiment 1 by using the group mean estimated stiffness as a proxy to perceptually match the squeeze and C/D levels to each other.Noting the similarities between the first 3 levels of the T and V conditions in Experiment 1, we chose to test a proxy stiffness range of 50-150 N m À1 .Thus, the chosen comparison max squeeze levels for the T condition in this experiment were 3.0, 3.  S1, Supporting Information).The TV condition presented the unimodal levels congruently.Subjects were given a 5 min break in between condition blocks, and the presentation order of blocks was randomized between subjects in a counterbalanced manner.The experiment lasted 2 h for each subject.
Twelve new subjects were recruited for Experiment 2 (age: {M ¼ 25, SD ¼ 4.8}, 3 female).One subject was excluded from all analyses for failing to follow the provided instructions.Subjects were pooled from the Rice University undergraduate and graduate student bodies under Rice University IRB protocol #IRB-FY2020-43 and provided written informed consent.They were compensated 20 USD upon completion of the experiment.Only one subject reported any experience with squeezing haptic displays, and none had significant prior experience with virtual reality devices.
Statistical Analysis: MATLAB (version R2020A) statistical packages were used to analyze the data and perform all statistical tests.For Experiment 1, a two-way repeated measures ANOVA was used to compare differences in subject mean stiffness estimates across conditions and levels.Two-sided p values lower than .05indicated significant differences.Any sphericity violations found though Mauchly's test were treated with a Huynh-Feldt correction.Post hoc analysis compared the TV-T and TV-V conditions, separately using similar two-way repeated measures ANOVAs.The same full procedure was carried out on the measure of subject residual errors.The measures of subject slope, fit, and squared residuals were compared between the TV-T and TV-V conditions using pair-sample t-tests with a Bonferroni correction (i.e., significance indicated by p values lower than .025for two comparisons).A subject was considered an outlier in a condition if their mean stiffness response in two or more levels was more than 1.5 interquartile ranges above or below the upper or lower quartile range.One subject was found to be an outlier in the V condition, and another in the TV condition, as shown in Figure 3.The outlier data for slope, fit, and residual error summarized in Figure 3 were replaced with the group mean of the respective condition.
In Experiment 2, separate one-way repeated measures ANOVAs (with Huynh-Feldt corrections, if needed) were used to compare differences in JND, PSE, percent correct, and residual errors across condition.Pairwise comparisons between all three conditions were made using Tukey's honestly significant difference procedure.There were no outliers for Experiment 2.

Figure 3 .
Figure 3. Experiment 1 main results (N ¼ 12).A-C) Box plots of subject mean responses in each level for each condition (outliers are beyond 1.5 IQR).We observe an increase in stiffness responses given an increase in wrist squeeze and/or visual C/D stimuli.D-F) Mean responses and fits for individual subjects as well as the group level fit for each condition (error bars are a 95% confidence interval).G-I) Subject residuals from the group level fit, where we find smaller residual values in the TV condition.This suggests that individual differences are reduced when congruent visual and tactile pseudo-haptic cues are provided.

Figure 4 .
Figure 4. Experiment 1 metrics (N ¼ 12).A) Slopes obtained from the linear fits to subject level means.Slopes in the bimodal condition were significantly greater from those found in the unimodal conditions, indicating that bimodal feedback extends the range of stiffness that can be rendered.B) The quality of fit, R 2 , obtained from the same fits.C).The square of residuals from the group-level fit, collapsed across level.Significantly smaller residuals in the bimodal condition suggest that subjects were more consistent with each other in this condition compared to the unimodal conditions.All error bars represent a 95% confidence interval.

MaxFigure 6 .Figure 7 .
Figure 6.Psychometric curves obtained from Experiment 2 (N ¼ 11).(left) Psychometric curves fit to individual subject data in each of the conditions.The central black curves represent the aggregate psychometric curves, i.e., the curve when fitted to all subjects' data.Note the tighter grouping of fits in the TV condition.(right) The aggregate psychometric curves with the mean of all subjects' responses at each stimuli level shown as markers.We can visually observe a greater slope in the TV condition, indicative of a smaller difference threshold.

Table 1 .
Experiment 2 survey questions.Questions were presented at the end of each of the three condition blocks.Subjects responded on a continuous scale from strongly disagree (0.0) to strongly agree (1.0).Results are shown in Figure8.