3D Touch Surface for Interactive Pseudo-Holographic Displays

richer user interaction with midair 3D virtual objects in a touch-interactive pseudo-holographic display are presented. Frustrated total internal re ﬂ ection (FTIR)-based touch sensing is used in combination with a four-sided pyramidal pseudo-holographic projection. The developed system allows gesture-based control and smooth touch interaction through facile and inexpensive hardware and open-source software tools. A software application is also developed as the interface between the touch/gesture-sensing system and the optical display. By bringing the virtual and real world closer through touch-based interaction, the presented system will enrich user experience and enable advances in areas such as and where holograms are currently used.


Introduction
Touch interaction constitutes the most intuitive and natural way for humans to interface with objects in the real world. [1][2][3] Consequently, it has been adapted for use in electronic devices to provide a simple and effective method of interaction which also eliminates the need for peripheral equipment. Physical properties of objects such as material softness and surface roughness can be perceived through touch. [4,5] A variety of touch sensors have been developed to emulate these capabilities by integration within an electronic skin (eSkin). [6][7][8] Touch screen interfaces of smartphones and other displays widely in use today could be considered as the most basic version of such eSkins.
The technologies for touch screen interfaces (based on resistive, capacitive, surface acoustic waves and infrared (IR) mechanisms) are able to detect a user's touch by monitoring electrical or optical properties. Commonly, these touch interfaces are 2D, and they accompany 2D visual displays, as in smartphones. Due to limitations of current electronics manufacturing and packaging technologies such devices are usually rigid and planar. [7,9,10] Recent developments in the field of flexible electronics have also resulted in bendable touch sensing and display devices. But, with touch sensing and display layers being in-plane, such devices are still 2D as their touch sensing-and displaying-capabilities are limited to two dimensions. [11] User interaction in 3D space is also being explored; however, the field is dominated by non-touch-based approaches, with midair gesture sensing technologies being among the most widely used. [12] These noncontact approaches allow for new modes of interaction by enabling detection of more complex life-like gestures. However, these technologies pose greater integrationrelated challenges due to their increased complexity, and usually require large and expensive equipment. In addition, they lack accuracy and robustness in gesture recognition compared to conventional 2D contact-based technologies, due to the increased complexity of detecting finger and hand motion in three dimensions as well as the large variability in potential user inputs. [13,14] This shortcoming becomes more apparent with uninitiated users interacting with such systems for the first time.
Alongside 3D spatial interaction, visual 3D displays are also being explored. Technologies that can project visual content extending outside the conventional flat plane will be able to take full advantage of 3D interaction techniques and give rise to exciting novel applications. Currently, this has been explored through pseudo-holographic displays that provide the illusion of a 3D virtual object as well as volumetric displays which create a true representation of a 3D virtual object. [15][16][17][18] The pseudoholographic display technology used for this work relies on the Pepper's ghost projection scheme. [19] It is a nonvolumetric pseudo-3D display technology, which is much simpler and cost effective than the virtual reality and lenticular technologies available today. Other reported implementations include projection of 3D images for medical applications as well as non-touch based interactive pseudo holographic displays. [20][21][22] In this article, the Pepper's ghost display was combined with an IR touch sensing DOI: 10.1002/aisy.202000126 Herein, the design and implementation of a transparent 3D touch-enabled surface for richer user interaction with midair 3D virtual objects in a touchinteractive pseudo-holographic display are presented. Frustrated total internal reflection (FTIR)-based touch sensing is used in combination with a four-sided pyramidal pseudo-holographic projection. The developed system allows gesturebased control and smooth touch interaction through facile and inexpensive hardware and open-source software tools. A software application is also developed as the interface between the touch/gesture-sensing system and the optical display. By bringing the virtual and real world closer through touch-based interaction, the presented system will enrich user experience and enable advances in areas such as education, entertainment, gaming, retail, and museums where holograms are currently used. scheme to create a fully touch-interactive pseudo-hologram to allow people to have a 360-degree visual experience along with smart and smooth touch interaction.
In this article, we present the touch sensing surface developed to enable contact-based touch interaction in 3D space. The proposed approach relies on sensing through the frustrated total internal reflection (FTIR) principle and preserves the accuracy and cost benefits of the conventional 2D methods. The developed technique allows for multipoint touch sensing along a transparent pyramidal surface. The sensor is subsequently integrated with a pseudo-holographic display to demonstrate its potential application in 3D interactive displays. The advantages of this approach are presented in relation to other modes of interaction with 3D displays as well as other touch-sensing technologies.

FTIR Principle
The developed touch-sensing device exploits the FTIR phenomenon. Total internal reflection (TIR) occurs when light waves traveling through a medium reach an interface with an external medium having a lower refraction index. At a certain critical angle of incidence, no refraction occurs, and the entirety of the wave is reflected within the medium. When another medium with a higher refractive index is presented at the interface, it frustrates the internal reflection, resulting in the light wave escaping the waveguide and being reflected on the new external medium. [23,24]

Planar FTIR Touch Screen Interface
In the case of the planar touch screen interface, IR light-emitting diodes (LEDs) are placed at the periphery of an acrylic panel that is used as a waveguide. IR light is fully reflected at the acrylic-air interface due to the angle of incidence being greater than the critical angle, resulting in the emitted IR light remaining inside the panel. Acrylic glass (or plexiglass) is suitable for this application because of its transparency, low density, minimum IR absorption and low cost. When the user touches the panel, the refraction at the interface is altered because the refractive index of the finger is greater than that of air and therefore the IR light is reflected at the point of contact and is then captured by an IR camera (Figure 1a). The reflected light appears as a blob and through image processing, multiple points of contact can be detected. Touch sensors and surfaces are usually paired with a display to provide visual feedback to the touch input. In the case of FTIR touch sensors, they have conventionally been implemented on flat surfaces and paired with rear projection displays or modified liquid crystal display (LCD) back panels. [23,25]

3D FTIR Touch Screen Interfaces
Here, we present a novel implementation of an FTIR-based touch-sensing interface on a 3D surface. The setup arrangement of FTIR touch interfaces allows for facile transition from 2D to 3D surface touch sensing with minor alterations in both hardware and software. For the demonstrated sensor interface, the flat acrylic panel is replaced by an acrylic pyramid made of four flat panels ( Figure 1b). The IR LEDs are placed along the bottom edge of the pyramid and an IR camera is placed at the bottom side of the pyramid, facing toward its tip as shown in Figure 1b and Figure 2. The 2D data received by the IR camera are mapped to specific locations on the 3D surface of the pyramid to enable nonplanar touch sensing. Through software, singleand multiple-point touch inputs can be detected, and various gestures can be recognized. These can be on a single side of the pyramid or even span across all its sides. It may be noted that this approach does not provide touch sensing in the 3D space inside the pyramid. It is limited to the surface of the waveguide material, brings touch sensing out of the flat plane, and enables some exciting novel applications in 3D interactive systems.

Integration with Pseudo-Holographic Display
As stated previously, 2D touch sensing surfaces are usually paired with an in-plane 2D display as a means of providing visual feedback to the user's touch. Therefore, the developed 3D touch surface can be used along with a 3D display. In recent years, the projection of 3D virtual objects has been explored by various research groups in both academia and industry, resulting in many interesting approaches of how this coveted movie special www.advancedsciencenews.com www.advintellsyst.com effect can be achieved in real life. The developed technologies are commonly divided into two categories, pseudo-holographic displays, which provide an illusion of 3D virtual objects, and volumetric displays, which have the ability to address individual pixels in 3D space (voxels). The proposed 3D touch surface can be integrated with such 3D display technologies to allow interaction with virtual objects and provide an immersive experience to the user. To demonstrate the potential of the presented sensor interface, a pseudo-holographic display was developed to project virtual objects in 3D space. Subsequently, it was integrated with the touch surface and interaction with virtual objects was achieved.

Pseudo-Holographic Display Design and Working
The pseudo-holographic display is created by projecting 2D images onto the four faces of an acrylic pyramid to obtain a midair virtual 3D image of an object. An LCD screen placed above the existing touch-sensing pyramid implements the Pepper's ghost projection scheme to produce the illusion of floating 3D objects ( Figure 2). [19] The formation of midair virtual 3D object images could be explained with the FTIR principle. When visible light, originating from the displayed four-sided images of an object on the LCD screen, falls onto the four sides of the acrylic pyramid, it gets reflected toward the center of the pyramid and creates a psudo-3D optical illusion ( Figure 2). The distance between each pixel on the 2D screen and the point of deflection on the surface of the pyramid appears as the virtual distance of the pixel (of 2D display) from the pyramid's surface toward the inner side of the pyramid. The rendered graphics, or captured images, of the objects displayed from all four surfaces of the pyramid form the 3D virtual object, which appears to be floating inside the pyramid. Since the reflected light travels horizontally toward the human eyes, the sides of the pyramid were designed to tilt at 45 degrees with respect to the LCD screen on top. In this way, a clear 3D-like illusion could be seen horizontally.

Interaction with Touch Surface Using Dynamic Virtual Objects
The LCD screen is connected to a computer, which generates the images that are projected onto the pyramid. To display a pseudo-3D virtual object, the 2D images of a 3D model as viewed from its four sides are used. Using specialized software, 3D models of objects or scenes are being generated and the four side views are rendered and projected. By combining the touch sensing surface with the aforementioned software, the 3D models can be dynamically altered based on the user's touch input and subsequently rendered in real time, thus enabling an interactive experience ( Figure 2).

Gesture-Driven Interaction
Through purposefully developed software, gesture recognition was introduced, allowing the user to interact with the virtual objects through intuitive hand motions. Four gestures were implemented: a) single touch, b) swipe with one finger to rotate, c) swipe with two fingers to change images, and d) using two fingers to zoom in or out (Figure 3a). The nonplanar form factor of the developed touch surface in combination with the integrated pseudo-holographic display to enhance the interaction creates an illusion of the user actually touching the virtual object.
To achieve life-like interaction, the touch and gesture implementation is based on mapping the touch inputs onto the virtual object. With reference to Figure 3b, the plane CDEF represents the bottom plane of the pyramid, while the top plane of the pyramid is GHIJ. The pyramid is in a 3D space rectangular coordinate system and C is the origin of this system. Point M (0.5, 0.5, h/r) is the geometric centroid of the pyramid. Herein, h is the height of centroid M from the bottom plane and r is the length of the bottom side of the pyramid (in this case, it is 325 mm). h/r is the normalized value. M is horizontally projected toward the four surfaces of the pyramid, resulting in the geometric centers of each pyramid plane, namely, W, P, Q, R www.advancedsciencenews.com www.advintellsyst.com separately, becoming the reference point of each surface. The 2D imaging plane of the IR camera is represented as plane XC 1 Y (or plane C 1 D 1 E 1 F 1 ), in which C 1 (0,0) is defined as the origin of this plane. Every point in this imaging plane could have its own position composed of X and Y coordinates (the maximum value is 1 considering normalization). For example, the detected blob L 1 has XY coordinates of (x 1 , y 1 ), where 0 < x 1 < 1 and 0 < y 1 < 1.
The idea of mapping is to project the points back from the imaging plane of the touch sensor's camera to the pyramid and finally link those points with the virtual object inside the pyramid. As seen in Figure 3b, the edges of the pyramid C, D, E, F, G, H, I, and J as well as the centroid M get projected in the imaging plane to C 1 , D 1 , E 1 , F 1 , G 1 , H 1 , I 1 , J 1 , and M 1 . Then the touch point L is projected to point L 1 on the imaging plane. As an example, a virtual cubic object is considered in the figure. Theoretically, this object will have a geometric centroid, named N (N x , N y , N z ). The virtual touch point K can be found by projecting touch point L to centroid N. This will intersect at the boundary of the surface, forming the object at point K, which is the point where the 3D object is being touched. The mathematical description of the coordinates of the virtual touch point starts from the touch point L. As it is the projection of L 1 , the XY coordinates should be the same, these being x 1 , y 1 . The Z coordinate of point L 1 can be found through the 45 angle of the pyramid surface against the bottom plane. Thus, the coordinates of L can be represented as This L coordinate is only for a touch event happening in plane DHIE, where the blob is detected at plane D 1 H 1 I 1 E 1 . For the other three circumstances, which occur on pyramid surfaces CDHG, CGJF, and FJIE, the representation of the Z coordinate of L x,y,z will change, as shown in Table S2, Supporting Information. Thus, the representation of K coordinates is different in the four cases. The case DHIE is analyzed subsequently and the other three are summarized in Table S2, Supporting Information. For the X coordinate of point K in the case of DHIE, first, the vertical distance of the reference point S to the LCD screen is measured. This is because the reflection of the light, making the virtual image inside the pyramid, has the same distance to the reference point irrespective of the virtual depth of the object displayed in the screen. Thus, the distance of point S to the 2D object is also D screen . This distance is normalized as D screen /r. The X coordinate of K can be calculated by S x À D screen /r, and S x ¼ 1 À S z ¼ 1 À N z .
For the blob detected in plane D 1 H 1 I 1 E 1 , the coordinates of K are The Cþþ coding for this implementation was conducted as an interface on top of a Tangible User Interface Objects (TUIO) library. [26] The idea of programming was to make use of the TUIO library to get the right mapping information. The source code normalizes the value of the X coordinates and Y coordinates of these points to the range of 0-1 and each of the points has an individual identity (ID). Therefore, as long as the starting position and ending position of a point (with a given session ID) are found, the gesture can be detected, and the feedback of the gesture is programmed. The visual feedback is projected from the screen onto the pyramid system. The programming process www.advancedsciencenews.com www.advintellsyst.com of producing visual feedback with different gestures is shown in Figure S2, Supporting Information. Single-Touch Implementation: As shown in Figure S2, Supporting Information, the program is developed to continuously read the TUIO packages. When the finger is not touching the pyramid, there is no session ID, thus implying that no position information could be transmitted. If there is one finger touching the pyramid, the program goes to the left flow. Single touch is defined in the program as a movement in which the X or Y coordinate shifts no more than 0.05, whereas the maximum value is 1. The single-touch operation was used to change the content displayed in the pyramid.
Swipe Implementation: Swipe as a gesture normally allows one or more functionality, such as rotating images and changing images in mobile devices. Apart from the number of fingers used in the swipe gesture, the one-finger and two-finger swipe used here indicate that a change of X or Y coordinate greater than a threshold would alter the display. For example, "Swipe to rotate" can rotate the 3D illusion by certain degrees. Smoother rotation can be obtained by using a graphical rendering or implementing true voxel processing in the code.
Zoom Implementation: Two-finger zoom in/out is another gesture implemented on the touch-interactive pseudo-holographic display presented here. The idea is to measure the distance between two fingers before and after a scaling gesture. If the distance between two fingers is increased over a threshold value (for instance, 0.2), another image with four bigger characters would show up. On the contrary, the feedback with an image having four smaller characters appears if the distance decreases more than 0.2.

Comparison between Approaches for Interaction with 3D Displays
We have presented a potential use case for our developed 3D touch surface as a means for incorporating user interaction to holographic displays. As mentioned previously, development of pseudo-holographic and volumetric displays has been widely explored and inevitably, various interaction paradigms have also been used to take advantage of the emerging technology. Some of the approaches for interaction with 3D displays are discussed and compared subsequently.

Physical Controls
The most basic mode of interaction includes the use of physical controls such as joysticks, controllers, and dials. Physical controls have been widely used for interfacing with virtual environments well before any touch-based solutions were introduced. They are very easy and cheap to implement while providing a great level of precision, aided by mechanical feedback during operation. However, physical controls have a fixed form factor and, when used alongside virtual environments with increasing complexity, they cannot adapt to different emulated scenarios. The lack of correlation between the real and virtual worlds results in a less immersive experience for the user. [27] Although physical controls are usually easy to use in basic applications, they can be less intuitive for the user in more complex scenarios, thus requiring some learning beforehand.

2D Touch Screens
Another method of interacting with 3D displays is through 2D touch-sensitive devices such as tablets and smartphones. [28] Nowadays, such devices are so widespread that operating them has become second nature. The adaptability of touch control through implementation of different gestures allows for more intuitive interaction. However, as with physical controls, the user input occurs away from the virtual object projected by the 3D display, meaning that the user does not have a hands-on interaction with the virtual environment. A form of direct feedback is provided to the user through the device's display, which may require some training to interact with a 3D virtual object.

Midair 3D Space Gesture Sensing
When interacting with 3D virtual environments, the aforementioned 2D sensing technologies have an inherent disadvantagenot being able to directly address all points in 3D space-that limits the level of user immersion. [13] To overcome these issues, technologies that enable gesture recognition in 3D space have been developed. These use cameras with spatial sensing capabilities which, when combined with image recognition software, enable detection of hand motions. Another approach uses specialized wearable devices to track the user's motions and has been predominantly used with virtual reality (VR) headsets. [29] In general, sensing of midair gestures allows for realistic interaction with virtual objects since the user is able to act as if they are interacting with the real object. Various implementations of midair gesture sensing devices have emerged in recent years and are receiving a lot of attention. [30] However, the complexity of the task at hand means that gesture recognition is not as reliable as the planar contact-based counterparts. Identifying hand and finger position in 3D space as well as distinguishing between the various complex poses can be challenging. [14] As a result, uninitiated users struggle to operate the systems and require some training. In addition, the increased complexity requires greater effort for integration with the 3D displays and significantly increases the cost.

3D Touch Surface
The proposed approach using the developed 3D touch surface could be considered as being halfway between the simple 2D and fully 3D approaches. By using a contact-based touch sensor implemented along a surface, the proposed method inherits the advantages of the 2D touch sensors related to reduced complexity and cost. The system is easily scalable and provides reliable gesture recognition with a minimal learning curve. At the same time, the 3D touch-enabled surface around the virtual object provides a more direct interaction which improves the user experience. It is worth noting that the proposed 3D touch surface could be implemented in other shapes, e.g., dome, and paired with other types of 3D displays. When considering the potential application of such systems as exhibition units, simultaneous multiuser interaction could be achieved without increasing the complexity of the setup. As a final point, the contact-based touch sensing can enable facile integration of local haptic feedback devices to further improve user interaction without the need for external wearable devices.

Comparison between Touch-Sensing Technologies for 3D Touch Surfaces
The usefulness of the proposed 3D interactive interface is anticipated in a wide variety of sectors, including retail, education, and home entertainment. To achieve a significant uptake of the interface, consideration must be given to simplifying the fabrication and assembly stages, reducing manufacturing costs, sourcing more sustainable materials, and giving the option of scalability to the device. From the user experience perspective, a key functionality that should be implemented is the ability to use multitouch technology. Simultaneous multiple contacts are needed for performing a variety of different gestures to interact with the displayed projections. All these points govern the choices made of the tactile sensing technology used and the materials used to realize the interface. To this extent, we have evaluated different touch-based display technologies that are implemented in various tactile interfaces all around us, [5] in the context of the proposed 3D surface approach and holographic display applications. For a touch-interactive holographic display, the transparency of the touch screen interface is also an important factor. An overview of this comparison is provided in Table 1, while a detailed discussion can be found in the Supporting Information section. [35,36] Table 1. Comparison between touch-sensing technologies with potential to be used for 3D touch surfaces. [5,11,[31][32][33][34]

Conclusion
In summary, we present the development of a 3D touch surface based on the FTIR principle. The proposed touch sensor is suitable for use with interactive 3D displays, as demonstrated here with the development of a pseudo-holographic display. The user is able to interact with the display through various touch gestures while being presented with the illusion of directly interacting with 3D virtual objects. The viability of the proposed interaction approach is evaluated through comparison with other existing interactive 3D display arrangements. Simplicity in implementation and reliable sensing capabilities are among the advantages that the proposed approach poses over other techniques. FTIR-based touch-enabled surfaces have the potential to integrate with existing and emerging display technologies and enable novel and exciting ways for interacting with the virtual world. Devices similar to the interactive pseudo-holographic display presented here could find applications in numerous sectors, such as education, retail, museums and exhibitions, medicine, and 3D modeling.

Supporting Information
Supporting Information is available from the Wiley Online Library or from the author.