Pseudo-Hologram with Aerohaptic Feedback for Interactive Volumetric Displays

The ability to display virtual objects midair that can be seen without special headwear is an attractive proposition for the virtual reality systems of the near future. Volumetric display technologies have demonstrated this concept, allowing users to view virtual objects with real dimensions in 3D space while enabling some forms of interaction. Developments have been witnessed in haptic feedback technologies which are used to deliver a sensation of touch in the digital world. As virtual environments find more applications in today's world, haptic feedback devices have potential to integrate with virtual reality systems and contribute to a more immersive and interactive user experience. Herein, an air‐based haptic feedback device, named Aerohaptics, is presented, which delivers mid‐air tactile feedback while the user is manipulating virtual objects within a pseudo‐holographic display. The developed system constitutes a cost‐effective approach with relatively low complexity and various potential applications as no wearable or handheld peripherals are required. The delivered haptic feedback can be accurately directed at specific locations on the user's hand and its intensity can be controlled to suit various interaction scenarios. Through experiments, localized feedback on the user's fingertips as well as varying feedback force according to the user's hand movement is demonstrated.


DOI: 10.1002/aisy.202100090
The ability to display virtual objects midair that can be seen without special headwear is an attractive proposition for the virtual reality systems of the near future.Volumetric display technologies have demonstrated this concept, allowing users to view virtual objects with real dimensions in 3D space while enabling some forms of interaction.Developments have been witnessed in haptic feedback technologies which are used to deliver a sensation of touch in the digital world.As virtual environments find more applications in today's world, haptic feedback devices have potential to integrate with virtual reality systems and contribute to a more immersive and interactive user experience.Herein, an airbased haptic feedback device, named Aerohaptics, is presented, which delivers mid-air tactile feedback while the user is manipulating virtual objects within a pseudo-holographic display.The developed system constitutes a cost-effective approach with relatively low complexity and various potential applications as no wearable or handheld peripherals are required.The delivered haptic feedback can be accurately directed at specific locations on the user's hand and its intensity can be controlled to suit various interaction scenarios.Through experiments, localized feedback on the user's fingertips as well as varying feedback force according to the user's hand movement is demonstrated.
feedback leading to interactive volumetric displays.The schematic representation in Figure 1 shows the developed system comprises three main components, the pseudoholographic display, the gesture recognition module, and the haptic feedback device.It uses jets of pressurized air to replicate touch sensations while providing location and intensity control.The integration of the "aerohaptic" device with an interactive pseudoholographic volumetric display demonstrates its potential application in feedback-capable virtual environments.The comparison with other interactive holographic displays (Table 1) highlights the potential the presented system has by bringing together features that were only implemented separately in other systems, such as 360 viewing angles, "at-location" interaction, and haptic feedback without the requirement for user-attached devices.A qualitative comparison of presented air-based haptic feedback against other technologies for delivering haptic feedback in 3D virtual environments is given in Table 2.The presented aerohaptic systems offer advantages such as feedback location and intensity control while maintaining low complexity and lower cost in comparison to other approaches.Table 1.Comparison between interactive holographic display systems. [1,5,10,11]

System Overview
The three main components of the presented interactive pseudoholographic display (Figure 1) are connected to a master computer and are controlled via an in-house-built program based on the Unity platform.To facilitate the operation of the system, a remote-control device in the form of a tablet running a custombuilt application is also included.Figure 1a shows a block diagram of the system's main components and operations and a 3D model is shown in Figure 1b.

Pseudoholographic Display
The pseudoholographic display used in the presented system is based on Pepper's Ghost projection scheme. [31]The system used here is the upscaled version of the pseudoholographic display presented in our previous work. [1,32]2D views of the final virtual 3D object are projected onto transparent screens placed at a 45 angle from the projection source.This arrangement creates the illusion of an object "floating" midair. [1]For the presented system, the screens are arranged in a square-based pyramid structure, hence allowing the 2D images to be created from orthogonal views of the virtual 3D object.One of the sloping sides of the pyramid structure is modified to create openings at the two bottom corners.As the projected object appears to be floating inside the pyramid structure, the user can reach through the openings and interact with the volumetric image as if it were a real object.

Gesture Recognition
The user can interact with the virtual object by using life-like gestures.A commercial sensor (Leap Motion) is used to track the user's hand movements inside the pyramid structure and the accompanied software is able to recognize various gestures. [33,34]he sensor uses a pair of IR cameras and is able to detect the motion of both hands in a field of view of around 150 in a region up to 600 mm from the sensing device.As gestures are detected, the virtual object is transformed accordingly and projected on the pseudoholographic display in real time.

Concept
To provide a more realistic interaction experience with the virtual objects, the presented system incorporates haptic (pressure) feedback in addition to visual feedback.[37] The proposed "aerohaptic" feedback system uses jets of air directed on the user's hand to replicate the sense of touch when manipulating a virtual object (Figure 2a).In this case, the small noise due to the air flow from the jet may also complement the pressure and visual feedback.

Design
The aerohaptic feedback system, shown in Figure 2b, is based on a pan-tilt mechanism located just below the pyramidal display structure.A single air nozzle is attached on the said mechanism and can direct the air jets toward the user's hands.The articulation of the mechanism, combined with the accuracy of the handtracking device, allows for the nozzle to be pointed toward specific locations of the user's hand, i.e., a finger or a palm.Pressurized air is supplied via a conventional air compressor.An air-flow-controlling device is used to trigger the air supply and vary the delivered air pressure to allow for different expressions of haptic feedback.The nozzle positioning motors and flow-control device are interfaced with the master computer and an Arduino microcontroller.

Kinematics of Aerohaptic System
At the core of the developed aerohaptic system lies an algorithm to extract the required information from the detected hand position to correctly position the air nozzle. [38]With reference to Figure 2c, the offsets from the stereoscopic camera to the end effector of the tracking system are the following: X offset ¼ 70 mm (the offset from Leap Motion origin to center of servo 1 on the X-axis); Y offset ¼ 125 mm (the offset from Leap Motion origin to center of servo 2 on the Y-axis); Z offset ¼ À30 mm (the offset from Leap Motion origin to center of servo 2 on Z-axis); b ¼ 25 mm (the distance from the center of servo 1 to the center of servo 2 on the Z-axis); c ¼ 70 mm (the distance from the end point of b to the center of servo 1).These variables can be examined in Figure 2c.Three other variables are defined in this context: x, y, and z are the Leap Motion camera position parameters in millimeters.
From Cartesian to spherical coordinates, the standard transformations are as follows (1) In this context, the variable r represents the radius, or the Euclidean distance from the origin to a detected point, φ is the azimuthal angle from the origin to the detected point, and θ defines the polar angle between the reference zenith and the line connecting the detected point to the coordinate system origin point.However, the φ angle can become ambiguous in real-time tracking and provide unreliable values due to the arctangent function definition.For this reason, the algorithm was modified to provide a single, correct value using the two-argument arctangent function, thus keeping the reference frame fixed.
The tracking algorithm was defined as follows To avoid extreme values being passed to the pan-tilt tracking system, which would result in unnecessary strain on the servos, the spherical angles were checked and normalized using the following function

Software Implementation
The holographic display and the stereospatial camera are controlled through Unity Game Engine.The C# algorithm responsible for the hologram-user interaction simply transforms the detected hand's coordinates and assigns a discrete value for the aerohaptic pressure.All these data are then packed into a string and sent through a serial connection (USB) to an Arduino board.
The firmware on the Arduino is responsible for controlling the aerohaptic interaction with the user and managing the handtracking system.On the firmware end, the data transmitted by the Unity Game Engine are parsed and converted numerically and then used to control the hand-tracking system (pan-tilt servos) and the air pressure outputted by the nozzle.However, the firmware is also capable of discerning between multiple modes of operation, dictated by the nature of interaction between the hologram and the user (see the Supporting information).The modes are the following: hand landmark detection/selection, constant pressure output, and dynamic aerohaptic topology.In the constant pressure output mode of the system, the user will be provided with a constant aerohaptic feedback, regardless of the spatial positioning of their hand.This is achieved through a proportional controller, based on the hand's coordinates.Essentially, the pressure output at the nozzle increases linearly with the distance between the Leap Motion sensor and the user's hand.In the hand landmark detection mode, the algorithm can select which part of the user's hand is targeted by the tracking system.There are six landmarks: one at each fingertip and one in the palm.By selecting the area to target the airstream toward, the aerohaptic system can potentially artificially increase the feedback resolution, providing not only a more realistic interaction, but also a more precise one.Finally, in the dynamic aerohaptic topology mode, the nature of the interaction between the user and the hologram system undergoes a paradigm shift.In this mode, the pressure is modulated mainly based on the topology of the 3D object and the kinematic interaction with the user's hand.This allows the user to feel the holographic object responding to real-life stimuli.As an example, if the user attempts to bounce a holographic basketball, they will be met with an aerohaptic feedback proportional to the amount of force they would "perceive" when bouncing a ball, and dependent on the total travel of the holographic ball.For this reason, the user will be able to feel the topology of the ball (exact positioning in the holographic space), the nature of interaction (hard bounce, soft bounce) and feel the ball bouncing back through aerohaptic feedback.
In this last case, the Arduino controller unit would receive both the hand spatial coordinates and the absolute value of the velocity seen by the fingers, as the resultant velocity, on the X-, Y-, and Z-axes.This enables the system to compute whether the user is interacting with the object (if the hand and the 3D body intersect) and to calculate how the pressure should be modulated based on the finger speed.The finger speed value is directly mapped to the pressure modulation unit and the interaction is created.Essentially, the response between the ball bouncing and the air pressure outputted is directly proportional to the finger speed.

Results
In this section we present experimental studies that demonstrate the operation of the developed aerohaptic system as part of the interactive pseudoholographic display.Specifically, the experiments discussed subsequently showcase two capabilities of the system, the accurate location control of haptic feedback delivery and the adjustable haptic feedback intensity.

Controlling the Location of Haptic Feedback Delivery
The articulate mechanism holding the air nozzle can move about two axes of freedom allowing for the nozzle to be directed at different locations with significant precision.Concurrently, the commercial hand-tracking device included in the system can detect different locations on the user's hands, i.e., fingers, phalanges, palms, and wrists (Figure 3e).When combined, the tracking system and articulated nozzle enable accurate control of the location where the haptic feedback is delivered.To demonstrate this, the aerohaptics system is programmed to sequentially target individual fingertips on the user's hand.Strips of paper are attached to the user's fingertips to provide visual feedback for the delivered air burst.Figure 3a-d shows haptic feedback being delivered on the middle and index fingers at separate occasions.From the images, it is apparent that the air-based feedback is concentrated at the targeted finger as the paper strips attached to adjacent fingers are not disturbed.

Haptic Feedback Through Varying Air-Pressure Delivery
To deliver haptic feedback of varying intensity, the developed system incorporates an air-flow-control mechanism to regulate the amount of air being projected onto the user's hand.This can be useful in a variety of scenarios; for example, the intensity of the haptic feedback can be kept constant while the user's hand moves closer or further away from the nozzle.In the experiment described here, we explore a different scenario where the intensity of the delivered haptic feedback is adjusted based on the speed with which the user's hand moves.As the user attempts to bounce a virtual basketball, large and quick movements (Figure 3f ) produce a stronger haptic feedback, which corresponds to a harder hit on the ball.A more subtle movement (Figure 3g) results in a weaker air burst, replicating a softer touch to the ball.To this end, a force sensor (Flintec AP8) was placed on the user's palm and the intensity of the delivered haptic feedback was recorded.The obtained plot (Figure 3h) shows the change in the force delivered to the user's hand as a result of the varying air flow after three hard bounces, followed by three softer ones.These results demonstrate how the developed system can deliver haptic feedback of varying intensity using the described simple arrangement.

Evaluation of System and Comparison with Other Implementations of Haptic Feedback in Virtual Environments
41] This approach does provide a means for more accurate user input, when compared to hand tracking, as these devices use simple buttons and built-in electronics (accelerometers, gyroscopes) to detect hand motion.However, as the user needs to hold onto these external devices, interaction with the virtual objects happens "from a distance" and the recognizable hand gestures are limited by the form factor and freedom of movement of the handheld controller.
,43] In this case, the user can perform life-like hand gestures as the glove for the most part does not constrain the user's hand movement.This form factor also allows for accurate gesture detection by placing sensors along the fingers and wrist.Haptic devices can be placed in a similar arrangement to deliver localized haptic feedback.Although such smart gloves have the potential to provide various modes of haptic feedback, they often lead to bulky devices which also need to be tethered to larger control units.High cost and a steep learning curve are also usually associated with these devices.As a result, potential applications of these systems are limited, mainly oriented toward personal use and not suitable for wider public use.
Haptic feedback delivered via ultrasonic waves eliminates the need for additional equipment to be held or worn by the user. [19,34]These devices use an array of ultrasonic speakers to generate midair haptic feedback on the user's hand as it moves over them.By controlling individual speakers in the array, intricate haptic patterns can be recreated while the lack of cumbersome peripheral devices broadens the application spectrum.However, the complexity of both the hardware and software involved significantly raises the cost for such devices.The aerohaptic feedback system presented here shares some of the merits found in the ultrasonic haptic devices as it also delivers haptic feedback midair without the need for handhelds or wearables.However, the use of pressurized air as the feedback delivery medium reduces the cost of the system as the required components are widely available and less expensive.In addition, the incorporation of a moving nozzle instead of a static array minimizes the complexity of the hardware as well as the associated controlling software.The achieved spatial resolution is comparable to the ultrasonic haptic systems at around 3 cm. [19]The presented system can generate touch sensations of greater intensity (6 N, <1 mN for ultrasonic [19,44] ) due to the available air pressure, thus allowing the replication of additional haptic feedback scenarios.Apart from a touch sensation, air can be used to deliver other types of haptic feedback such as hot/cold sensation by controlling the temperature of the air supply.Olfactory feedback could also be delivered via this system, by introducing various scents into the air stream.Having said that, the presented version of the aerohaptic system does not provide the feedback precision of the more expensive systems (ultrasonic-based haptics, smart gloves).Such systems allow simultaneous multipoint haptic feedback, while implementations of smart gloves can apply feedback from other directions, i.e., on the back side of the hand.For the presented aerohaptic system, further optimization in both hardware and software is required to improve the user experience.As an example, increased articulation of the existing nozzle and the addition of a nozzle at the top side of the pyramidal display could enable haptic feedback delivery on multiple points and on either side of the user's hand.

Conclusion
We, present an air-based haptic feedback system that delivers midair tactile feedback without the use of wearable or handheld devices.The system relies on an articulate nozzle that directs bursts of air at specific locations on the user's hand while also controlling the intensity of the air jet.The developed aerohaptic system has the potential to deliver tactile feedback in virtual environments to enhance the user's experience.As such, it is presented integrated within an interactive pseudoholographic display system.The system's ability to control the location and intensity of the delivered haptic feedback has been demonstrated experimentally.In comparison to other approaches for delivering haptic feedback in virtual environments, the presented aerohaptic system constitutes a cost-effective solution with relatively low complexity.Through further development, the system has the capacity to deliver other modes of haptic feedback, such as varying temperature, while also presenting great potential to become a multimodal interaction platform for other sensory interactions such as olfactory feedback.

Figure 1 .
Figure 1.a) Block diagram showing the main components and operations of the aerohaptic feedback system as integrated with the interactive pseudoholographic display.b) 3D illustration of the developed system.

Figure 2 .
Figure 2. a) Air-based haptic feedback delivered midair via an articulate air nozzle when the user interacts with a virtual object (button).b) Components of the developed aerohaptic system.c) Conversion of the hand-tracking data to positional information for the air nozzle.

Figure 3 .
Figure 3. a-b) Demonstration of the location control for haptic feedback delivery by independently stimulating the middle fingertip (a,b) and index fingertip (c,d).e) Hand-tracking data obtained from the Leap Motion sensor allowing detection of landmarks on the user's hand.f,g) Demonstration of controllable haptic feedback intensity based quicker (f ) and subtle (g) user hand motion.h) Force delivered on the user's palm via the haptic feedback system showing the change in feedback strength corresponding to the different user hand motions.A quicker and more pronounced motion produces a stronger feedback (green pulses), whereas a more subtle motion results in weaker feedback (yellow pulses).

Table 2 .
Comparison between haptic feedback approaches for virtual environments.