Gaze-Matched Pupil Steering Maxwellian-View Augmented Reality Display with Large Angle Diffractive Liquid Crystal Lenses

(Ocean Optics HR2000CG-UV-NIR


Introduction
[4] Most of the development on AR headsets focuses on improving image quality, [5,6] field of view (FOV), [7,8] ambient light transmittance, [9] vergence-accommodation conflict (VAC), [10] optical efficiency, [8,11] eyebox size, [12] compact form factor, and lightweight. [8,13]e concept of Maxwellian-view was first introduced as early as 1966, [14] and then applied to a virtual reality (VR) system in the early 1990s, because it could produce high brightness images with potentially wider FOV at that time. [15]In the late 1990s, Maxwellian-view was integrated into the AR system with the help of holographic optical elements (HOEs). [16,17]The conventional Maxwellian-view AR system consists of a laser projector and a lens coupler, whose focal point is located at the center of the observer's pupil.The major advantage of such a display is that it can achieve a very high optical efficiency and small form factor, while avoiding the VAC issue.However, there is a severe tradeoff between optical efficiency and eyebox size if the display system does not have an active beam modulator.To enlarge the eyebox of a Maxwellian-view display, two major approaches have been developed: pupil duplication (Figure 1a) [18] and pupil steering (Figure 1b). [5,12,19,20]s Figure 1a depicts, pupil duplication methods usually utilize a HOE, [18,21] beam splitter array, [22] or spatial light modulator (SLM) [23] to split the collimated beam into multiple directions, so that each direction corresponds to one viewing point.This approach is cost-effective because no additional eye-tracking system is needed, but it will introduce some problems.The major one is that the space gap between different viewing points could be too large (Figure 2a) or too small (Figure 2b).When this happens, the user either cannot see any image or will see two partial/ ghost images when their eyeballs rotate to the middle of two adjacent viewing points.Such a problem arises because all the viewing points appear simultaneously.Another problem, as Figure 2c depicts, is that the direction of the chief ray does not match the user's viewing direction (eye gaze) except for the central viewing point.This gaze mismatch will let the user see an unnatural image and get a terrible viewing experience at these viewing points. [24,25]The third drawback of pupil duplication is that the optical efficiency will drop to 1/N when N viewing points are presented (e.g., N ¼ 3 in Figure 2), because only one viewing point is used at a time.
On the other hand, the pupil steering method can eliminate the first and third shortcomings mentioned earlier because there is only one viewing point presented.However, to accommodate eye rotation, an eye tracking and a beam steering device are required for the pupil steering system.Moreover, a conventional lens coupler can achieve diffraction limit only at one incident angle.Once the input beam is modulated toward various directions, the imaging quality at some viewing points will drop significantly. [12]Meanwhile, the mismatch between chief ray and eye gaze still exists in the pupil steering system.One solution to correct such a mismatch is to shift the position of the lens coupler, but it requires a mechanical shifting part, which will increase the complexity and weight of the system. [5]Therefore, there is an urgent need to develop a Maxwellian-view AR system with an expanded eyebox, good imaging quality, natural viewing experience, high optical efficiency, and high ambient contrast ratio, while keeping the system simple, compact, and lightweight.
In this article, we demonstrate a gaze-matched pupil steering Maxwellian-view AR system with a large angle diffractive liquid crystal lens.The key component of the optical combiner is a polarization selective off-axis cholesteric liquid crystal (CLC) lens array.Each lens corresponds to one viewing point.The location, diffraction angle, and lens profile of each lens can be customized, so that the optimal imaging quality and viewing experience can be achieved for each viewing point.

System Configuration
The operation principle of the AR system we proposed is shown in Figure 3.A laser projector is used as the light engine.The projected beam passes through a circular polarizer (CP) and a collimation lens (CL) before reaching the optical combiner.Thus, the input image is either left-or right-handed circularly polarized (LCP/RCP) light.The optical combiner (OC) consists of two laminated off-axis CLC lens arrays.The chirality of these two CLC lenses is opposite.Let us assume the first layer (brown color) works for LCP, and the second layer (green color) works for RCP.The second lens has a slightly longer focus to accommodate its slightly longer optical path.The location of each off-axis lens is specially designed so that the chief ray matches the user's gaze.In Figure 3, D is the average diameter of an adult's eyeball, which is 2.4 cm; L is the eye relief; θ is the off-axis angle of the CLC lens  which is designed to be 60 , so that the input beam to the optical combiner will not be blocked by the eyeball; d is the lens diameter, and φ is the eyeball rotation angle when the user sees the other viewing point shown in Figure 3b.The input light can be selectively diffracted by a specific lens.Each lens corresponds to one viewing point, whose incident angle and optical path are fixed.The wavefront of each diffractive liquid crystal lens is recorded independently and can be designed to minimize the optical aberrations.
When the viewer sees the object at normal angle, there is a maximum eyeball rotation angle (φ max in Figure 3b), which means that once the object position is outlying the FOV of AEφ max , the viewer tends to rotate head instead of rotating eyeball to continue gazing at the object.When designing the system, we set φ max ¼ 30 , and there are five viewing points in one dimension.Then, we can plot the relationship between eye relief L and central FOV or lens diameter d, as shown in Figure 4. From the figure, the following tendency is found: the shorter the eye relief, the wider the FOV.To obtain a relatively large FOV and reasonable eye relief, we choose the eye relief to be 15 mm.Under such conditions, the corresponding central FOV is about 55 and the lens diameter is 15.6 mm.The f-number of the central off-axis CLC lens is 0.96.

Off-Axis CLC Lens Fabrication
The optical combiner in the proposed system is a two-layer polarization selective off-axis CLC lens.The first layer diffracts LCP light and the second layer diffracts RCP.The structure of this off-axis CLC lens is a lens-patterned reflective polarization volume grating (rPVG). [11,26]Photo-alignment method was adopted in the lens fabrication.A thin photo-alignment layer was spincoated on a clean glass substrate.Then the substrate was exposed in the setup shown in Figure 5.In the figure, a collimated laser beam with wavelength λ ¼ 457 nm is split into two beams by a polarizing beam splitter (PBS).After being reflected by the mirrors (M 1 and M 2 ), the two linearly polarized beams are converted to LCP and RCP, respectively, by the quarter-wave plate (QWP) in each arm.The LCP beam works as the reference beam and the RCP signal beam passes through a template lens (L), so that the lens profile is recorded on the substrate (S).On the other hand, due to the linear optical path difference between the two beams, a grating pattern is coexistent.The reference beam angle θ 1 is equal to the off-axis angle θ shown in Figure 3a, and its value is the same for all the lenses on the optical combiner, because the input laser beams are collimated.However, the signal beam angle θ 2 , which equals to the eye rotation angle φ in Figure 3b, varies for each of the different lens on the optical combiner, since each lens corresponds to a unique viewing direction.Due to limited facility in our lab, we can only build two sets of the optical path as depicted in Figure 5, and we are unable to fabricate more than two off-axis CLC lenses with different lens profiles in one   substrate.Therefore, we only retain the central lens in the first substrate working for LCP, and the second substrate has two offaxis CLC lenses working for RCP.There are three viewing points in total in our demonstration.Based on the designed eye relief and lens diameter, the viewing direction (φ in Figure 3b) corresponding to the two lenses on the second substrate is þ16 and À16 , respectively.That is to say, the value of θ 2 should be þ16 and À16 when fabricating these two lenses.On the other hand, it is easy to see that θ 2 should be equal to 0 for the central lens on the first substrate.
After pattern exposure, a reactive mesogen mixture (RMM) solution was spin-coated onto the substrate.The material recipe is listed in Table 1.The components and their ratios in the RMM solution for two substrates are the same, except for the chiral dopant.The chiral dopant for the first substrate is S5011, which is left-handed, while the second substrate is R5011, which is right-handed.It is worth mentioning that when the substrates were exposed to the same pattern, chiral dopants with opposite charity will lead to an opposite diffraction angle as Figure 6 depicts.If we want the two substrates to have the same diffraction angle for LCP and RCP, respectively, we can flip top to bottom and rotate 180 horizontally for one of them as shown in Figure 6.
Due to the large off-axis angle and small f-number, the grating period will be small.For example, when the signal beam is at þ16 , the angle between the signal and reference beams should be θ 1 þ θ 2 ¼ 60°þ 16°¼ 76°for the central point, which is shown in Figure 7a.However, for the marginal points on the right, the angle will be larger, and θ 3 ¼ 76°þ FOV=2 ¼ 101°( the FOV is around 50 when φ ¼ AE16 ).The corresponding grating period is 300 nm.Such a small grating period is challenging to fabricate.We measured the intensity distribution along horizontal and vertical directions, and results are plotted in Figure 7b,c (this intensity has taken Fresnel reflection at the glass-air interface into account).The data points (dots) are fitted by the Gaussian function (lines).In the x-direction, we shift the peak intensity a little to the left from the center of the template lens L, so that the intensity mismatch between the reference beam and signal beam in the small period area (the right side of the sample) is smaller, which is helpful for the pattern alignment.In the experiment, we also found that a smaller grating period requires a higher intensity during pattern exposure.Figure 7d-f show the photos of samples fabricated with 30 s exposure time but different intensities during the pattern exposure.From the photos, as the laser output power decreases, the unaligned area on the right side increases, but the left side is still well aligned, because the grating period gets larger from right to left.Moreover, in order to eliminate the influence of exposure dosage, we add another control group with 150 mW laser output power but an exposure time of 40 s, so that the samples in Figure 7d,g have the same exposure dosage.However, the performance of the sample in Figure 7g is similar to that in Figure 7f.Therefore, the exposure dosage is not a key factor in making this large off-axis low f-number CLC lens, but the exposure intensity is.
Figure 8a,b show the photos of our fabricated samples.The first sample works for LCP, which serves as the central viewing lens.The second sample diffracts RCP and contains two off-axis lenses, corresponding to þ16 and À16 viewing directions, respectively.Figure 8c shows the transmission spectra of these two samples with the specified circularly polarized beams, and the incident angle is 60 , which is the same as the condition when they are applied as the optical combiner in the proposed system.The spectrums were measured by a white light spectrometer (Ocean Optics HG2000CG) with a halogen lamp and normalized with a clean glass substrate.The diffraction efficiency of these two samples is 98% and 91%, respectively, at the target operation wavelength (457 nm), according to Figure 8c.

Results and Discussion
The system setup is shown in Figure 9a.A bandpass filter (BPF), THROLABS FB457.9-10, is placed after the laser projector (Sony MP-CL1A), which aims to control the laser output, so that the signal intensity at the exit pupil is not too strong to be captured by the camera.The spectra of the laser projector and BPF are measured, as Figure 9b shows.When the laser projector has only blue signal input, the emission spectrum has a peak wavelength at 445 nm.The central wavelength of the BPF is around 457 nm, which is consistent with the pattern exposure laser wavelength.After calculation, only 0.43% of the laser projector output can pass through the BPF.The optical combiner consists of two layers of off-axis CLC lenses shown in Figure 8a,b.The first layer working for LCP is flipped and rotated 180 horizontally, then laminated with the second layer, which is similar to the condition presented in Figure 6.A camera (iPhone 11 Pro Max) is set on a multi-axis translation stage to capture the imaging results.In the experiment, we fabricated a circular polarizer consisting of two segments: one for LCP and another for RCP, and manually rotated the circular polarizer to control the handedness of the input beam.In practical application, we can simply apply a fast-response LC active half-wave plate to switch the handedness. [12]Moreover, we prepared three PowerPoint slides that have the same image content (UCF) but different positions for the steering process.During steering, we switch between the three slides to obtain different viewing points to imitate the condition shown in Figure 3.In practice, we can add one more microelectromechanical system (MEMS) mirror to steer the output beam from the laser projector.Before starting the imaging experiment, we first checked the positions of the viewing points.After removing the camera and BPF, we placed a rod with a 2.4 cm diameter at the focal point of the central lens to represent the user's eyeball.According to Figure 10a-c, the focal points of the three off-axis lenses are located at the desired positions, corresponding to viewing points when the eyeball rotates at 16 , 0 , and À16 , respectively.Then, we put the camera and the BPF back in place to capture the imaging results of these three viewing points.Results are shown in Figure 10d-f.From the imaging result of the central viewing lens (Figure 10e), the imaging FOV is around 55 as expected.The whole FOV of the camera is around 67 according to our measurement.From the photo, we can see that the imaging content nearly takes the full space of the camera horizontally.When taking the photos of oblique incident images, we also rotated the camera by the same angle to simulate the eyeball rotation.According to Figure 10d,f, the center of imaging content is located at the desired viewing angle, which means the chief ray matches the user's eye gaze.In addition, since these three off-axis lenses are recorded with three different holographic lens patterns, each of them is optimized for a designed incident angle and viewing angle.Therefore, no significant aberration will be     introduced by the beam steering between these off-axis lenses.All the results show good image quality, without noticeable ghost image and scattering.
However, the ambient light looks yellowish due to the high diffraction efficiency of the optical combiner in the blue region.To solve this problem, we dilute the RMM solution, until the ratio of solute to solvent is around 1:6.Then, the fabricated sample is thinner, and the diffraction efficiency is reduced to around 50%.The imaging results with this lower diffraction efficiency optical combiner are shown in Figure 11a-c.After lowering the diffraction efficiency of the optical combiner, the yellowish ambient light problem is improved greatly.On the other hand, as an AR system, the ambient light transmittance is also a very important factor.When we took the imaging photos, we kept the camera settings the same for the same set of imaging results.However, the cellphone camera would automatically balance the intensity between ambient light and signal when the signal intensity changed.The actual ambient light would be stronger, and the signal would look weaker than what we present in Figure 10d-f and 11a-c.Actually, the signal intensity for the Maxwellian-view display will not be a problem, since there is no etendue waste in the system. [27]Although only 0.43% laser output passes through the BPF, the signal intensity is still strong enough for indoor imaging.If we match the wavelength of the laser projector with the BPF, then the signal intensity will be too bright to be captured by the camera or human eye, unless we dim the output power of the laser projector.To present the real ambient light transmittance, we turned off the signal, and placed the camera at the focal point of the central viewing lens, and took the photos of ambient light with and without the optical combiners.Results are shown in Figure 11d-f.Figure 11d is the photo of the environmental background through the high diffraction efficiency optical combiner.In the photo, we can still clearly observe some yellowish color on the left.The reason for this nonuniform yellowish background is that the off-axis CLC lenses are angular dependent. [11]Our off-axis lens is designed to have the highest diffraction efficiency when the input light is at 60 .As the incident angle gets far away from the designed value, the diffraction efficiency will decrease gradually.Next, let us focus on Figure 11e, which is the environmental background with the low diffraction efficiency optical combiner.After reducing the diffraction efficiency to nearly half, the yellowish color is suppressed noticeably.On the other hand, we used brilliant yellow as the alignment layer material, which will also appear yellowish.Using a transparent photo-alignment material will relieve the yellowish background.Moreover, to further enhance the ambient light transmittance, we can use a low birefringence RMM material to fabricate the off-axis CLC lenses, because it has a narrower Bragg reflection band. [28]In this experiment, we used a monochromatic optical combiner to prove the concept of the proposed system.For full-color imaging applications, we can stack multiple layers of off-axis CLC lenses together in the optical combiner shown in Figure 3, and each layer has a different lens profile designed for the corresponding wavelength, respectively.
In conclusion, we demonstrate a pupil steering Maxwellianview AR system with an expanded eyebox.Each viewing point is gaze matching, which provides a natural viewing experience.Moreover, each viewing point corresponds to one independent off-axis CLC lens, so that each lens can be customized to achieve an optimal performance and imaging quality.In the meantime, the system exhibits some desirable properties, such as high optical efficiency, good ambient light transmittance, relatively large FOV, compact size, and lightweight.Multiple polarization selective off-axis CLC lenses are fabricated by the holographic method, which serve as the optical combiner.The influence of exposure intensity on different grating periods was observed in the experiment.These off-axis CLC lenses have a large off-axis angle (60 ) and low f-number (0.96).The diffraction efficiency of the fabricated off-axis CLC lenses can be as high as 98%.Another set of low diffraction efficiency off-axis CLC lens array was also fabricated, which can relieve the yellowish background and enhance the ambient light transmittance significantly.The proposed pupil steering system overcomes a critical technical barrier, and its widespread applications for AR systems are foreseeable.
Off-Axis CLC Lenses Fabrication Process: To align the LC monomer, brilliant yellow (0.2 wt%) dissolved in dimethylformamide (DMF) was spincoated onto a clean 1-inch by 2-inch glass substrate at 500 rpm for 10 s and then 3,000 rpm for 30 s as a photo-alignment layer.Next, the substrate was subject to the 457 nm laser holography pattern exposure for 30 s.The optical setup is shown in Figure 5, and we have two sets of the setup with different signal beam angles (θ 2 ).For θ 2 ¼ 16 and À16 , these two off-axis lens patterns were exposed on the same substrate.After that, the precursors with R5011 were spin-coated with 2000 rpm on the glass substrates, and then cured with a UV lamp in nitrogen environment for 12 min.Next, another substrate with an alignment layer was pattern exposed with θ 2 ¼ 0 , then a precursor containing S5011 was spin-coated on the substrate with the same spin speed and UV lamp curing process.Finally, the two substrates were laminated together to form the optical combiner.
Optical Measurements: To characterize the transmission spectra of the off-axis CLC lens and the BPF, a white light source (Mikropack DH-2000), and an optical fiber spectrometer (Ocean Optics HR2000CG-UV-NIR) were used in the experiment.By replacing the white light source with a laser projector, the emission spectrum of the laser projector can be measured.

Figure 1 .
Figure 1.Illustrations of Maxwellian-view system based on: a) pupil duplication and b) pupil steering.

Figure 2 .
Figure 2. Problems exist in the pupil duplication: the distance between two viewpoints is: a) too large and b) too small; c) mismatch between the viewing direction (eye gaze) and chief ray.

Figure 3 .
Figure 3. System configuration of the proposed pupil steering AR system with: a) LCP input and b) RCP input.

Figure 4 .
Figure 4. Relationship between eye relief and central FOV or lens diameter.

Figure 5 .
Figure 5. Experimental setup for holographic off-axis lens pattern exposure.

Figure 6 .
Figure 6.Schematics of holography CLC optical elements with the same exposure pattern, but opposite charity.

Figure 7 .
Figure 7. a) Schematic of the angle between the reference beam and signal beam during pattern exposure.Intensity distribution along: b) horizontal and c) vertical direction.Photos of fabricated samples with 30 s exposure time, but different laser output powers: d) 200 mW, e) 175 mW, f ) 150 mW during pattern exposure.g) A photo of the fabricated sample with 40 s exposure time and 150 mW laser output power during pattern exposure.

Figure 8 .
Figure 8. Photographs of the fabricated samples for: a) LCP and b) RCP, and c) measured transmission spectrum of these two samples at 60 incident angle for the specified LCP and RCP beams.

Figure 9 .
Figure 9. a) Photo of the optical system setup.b) Measured emission spectrum (blue lines) of the employed laser projector with only blue signal input and transmission spectrum of the BPF.

Figure 10 .
Figure 10.a-c) Photos of the focal point positions of the three off-axis lenses on the optical combiner, and d-f ) corresponding imaging results.(From left to right, φ ¼ 16°, 0°, À 16°, respectively).

Figure 11 .
Figure 11.a-c) Photos of the imaging results with a low diffraction efficiency optical combiner.Photos of real environment background with a: d) high diffraction efficiency combiner, e) low diffraction efficiency combiner, and f ) no optical combiner.

Table 1 .
Recipes of the materials and spin-costing speed in the device fabrication.