In vivo multiphoton microscopy using a handheld scanner with lateral and axial motion compensation

This paper reports a handheld multiphoton fluorescence microscope designed for clinical imaging that incorporates axial motion compensation and lateral image stabilization. Spectral domain optical coherence tomography is employed to track the axial position of the skin surface, and lateral motion compensation is realised by imaging the speckle pattern arising from the optical coherence tomography beam illuminating the sample. Our system is able to correct lateral sample velocities of up to approximately 65 μm s−1. Combined with the use of negative curvature microstructured optical fibre to deliver tunable ultrafast radiation to the handheld multiphoton scanner without the need of a dispersion compensation unit, this instrument has potential for a range of clinical applications. The system is used to compensate for both lateral and axial motion of the sample when imaging human skin in vivo.


| INTRODUCTION
Multiphoton fluorescence microscopy has found many applications for in vivo imaging of biological structures and processes [1][2][3] and a range of systems have been developed to increase the ease with which such images can be obtained, for example [4][5][6]. However, live subjects have a range of sources of motion including breathing, heartbeat and peristalsis that can compromise images acquired in vivo. A variety of methods have been applied to address motion of the sample, including physical immobilisation, [7,8] acquiring a 3-dimensional image stack and using the image data to monitor and correct for sample motion [9,10], gated imaging approaches [11,12] and the use of optical sensors to monitor the axial sample position. The latter have included off-axis illumination [13] and spectral domain optical coherence tomography (OCT) [14][15][16]. Generally, previous work addressing lateral motion compensation has employed image-based correction methods, for example using pairwise rigid transformations [17] or using more advanced approaches that model the intra-frame motion using methods such as Hidden-Markov-Models [18], the Lucas-Kanade framework [19], or algorithms based on Lie groups [20]. However, these methods rely on the quality of the image data acquired being sufficiently high to allow image-based motion correction. When imaging weak signals, such as tissue autofluorescence, or sparse structures such approaches may not be successful.
We have previously demonstrated a handheld multiphoton scanner including axial motion compensation utilising OCT, which was demonstrated using ex vivo samples [14]. We have also previously demonstrated the use of a negative curvature fibre (NCF) for delivery of tunable ultrafast infrared light without the need for pulse pre-chirp [21]. Here, we extend that work to include lateral motion compensation implemented by imaging the back-scattered light from the OCT beam that is used to determine the sample axial position. This approach has the advantage that lateral motion compensation can be applied irrespective of the level of fluorescence signal returning from the sample and irrespective of the type of image structures present. It also has the advantage that the frame rate of the camera imaging the back-scattered light can be much faster than the frame rate used for multiphoton imaging, allowing higher sample velocities to be accommodated. We demonstrate this motion-compensated imaging system applied to human skin in vivo in the dorsal forearm and the upper chest. Figure 1A shows a schematic diagram of the handheld scanner, which includes both the active axial motion compensation system and a complementary metal-oxide semiconductor (CMOS) camera for lateral motion tracking. The scanner head had overall dimensions of 35 × 30 × 7 cm 3 . Multiphoton excitation light was provided by a mode-locked Ti:Sapphire laser (Mai Tai HP; Spectra Physics, Santa Clara, CA) tuned to a centre wavelength of 760 nm. A half-wave plate and polarizing beamsplitter cube were used to control the excitation power, which was limited to a maximum average power of 25 mW at the sample for in vivo imaging. A Faraday isolator was employed to prevent unwanted back reflections affecting the mode-locked operation of the oscillator. The dispersion-broadened pulse full-width at half maximum (FWHM) after the polarising beamsplitter and Faraday isolator was measured using an intensity autocorrelator (FR-103XL; Femtochrome, Berkley, CA) to be 210 fs. The beam was then coupled into 4 m of NCF described previously [21] with a loss at 760 nm of 0.09 dB m −1 . The pulse FWHM at the output of the NCF was measured to be approximately 150 fs due to the weak anomalous dispersion of the NCF at this wavelength [21].

| Handheld scanner
The absence of modal cut-offs in NCF mean that higher order modes are also supported but propagate along the fibre with much higher loss than the fundamental mode. We ensured that the coupling of light into the fundamental mode was maximized by matching the size of the focused spot to the modal field of the fundamental mode at the input end of the fibre. Single mode guidance of the 760 nm light was confirmed by imaging the output of the NCF onto a charge coupled device camera. Only the fundamental mode of the fibre was visible in these images. Improved NCF fibre designs with single mode operation have now been demonstrated [22].
A pair of x-y galvanometric scanning mirrors (6210H; Cambridge Technology, Bedford, MA) and scan and tube lenses (LSM03-BB and AC254-100; Thorlabs, Newton, NJ) were used to scan the beam angle at the overfilled back aperture of a water immersion microscope objective lens (UPLSAPO60XW; Tokyo, Japan). The fluorescence generated by the scanned focal spot was collected by the objective lens and separated from the excitation light using a dichroic beamsplitter (FF705-Di01; Semrock, Rochester, NY, DBS1 in Figure 1A). A pair of lenses (Thorlabs AC254-50 and   were used to relay the fluorescence onto the end face of a 2.86 mm diameter active area fibre bundle that guided the light to a hybrid photomultiplier (HPM-100-40; Becker & Hickl, Berlin, Germany). Multiphoton images were acquired with a line scan time of 3.9 ms, allowing a 64 × 64 pixel image to be acquired in 0.25 seconds and a 256 × 256 pixel image to be acquired in 1 second. The point spread function (PSF) of this imaging system was determined previously by recording fluorescence images of single sub-resolution fluorescent beads (FluoSpheres F-8781, 24 nm diameter; Life Technologies, Carlsbad, CA) mounted on a microscope slide. The FWHM of the lateral and axial PSF were measured in water as 0.34 and 2.18 μm, respectively [14].
The output from a superluminescent diode (930-B-I-10-PM; Superlum, Corrigtwohill, Ireland) operating at 930 AE 35 nm was split by a 50/50 polarisation-maintaining fibre coupler into the two arms of a low coherence interferometer. As shown in Figure 1A, the sample arm was incorporated into the handheld imaging head where a dichroic beamsplitter (zt1064spr-dc; Chroma, Foothill ranch, CA, DBS2 in Figure 1A) was used to direct light through the objective lens and onto the sample. The back-scattered light was coupled back through the illumination fibre where it was recombined with light from the reference arm. The length of the reference arm was adjusted by translating the reference arm mirror until the optical path length of the sample arm was 100 μm longer than the reference arm so that the spectral domain OCT signal from the sample is separated from the direct current and autocorrelation terms. A microscope objective (Olympus UPlanFL N 40×) placed in the reference arm (L9) was used to match approximately the dispersion in the sample arm introduced by the objective lens (O). The resulting optical interference pattern was recorded by a spectrometer (USB2000+; Ocean Optics, Largo, FL, USA). In order to avoid an unwanted back reflection from the polarisation maintaining fibre coupler, quarter-wave plates were placed in both the sample and reference arms so as to rotate the polarisation states of the reflected beams by 90 and a linear polarizer was placed between the fibre coupler and the front of the spectrometer to reject the unwanted back reflection. An optical chopper was placed in the interferometer reference arm to decrease the effective exposure time of the spectrometer to below its hardware limit of 1 to 0.2 ms. The analysis of the recorded spectrum and the predictive Kalman filter that was used to mitigate for the 15 ms of latency in the system have been described previously [14]. The output from this filter was then multiplied by a measured distance-to-voltage calibration factor and sent to the piezo actuator (P-725 PIFOC; PI, Karlsruhe, Germany) operated in closed loop that controlled the position of the microscope objective.
In order to record lateral motion of the sample, a 50:50 non-polarising beamsplitter ( Figure 1A) was used to pick off half of the superluminescent diode light that was back-scattered by the sample. Light from the focal plane of the objective (O) was then imaged onto a CMOS camera (MQ013CG-ON; Ximea, Munster, Germany) by lens L6. When the sample consisted of a rough and/or scattering material the image at the CMOS camera consisted of a speckle pattern due to the spatially coherent illumination. Lateral motion of the sample therefore resulted in a lateral translation of the speckle pattern. The CMOS camera was configured to acquire images at 80 Hz and the acquisition sequence was hardware triggered at the start of a multiphoton image acquisition. Therefore, 1 CMOS image was acquired for every 3.2 multiphoton lines scanned.
When imaging fluorescence from white cardboard, a 30 second acquisition was configured to consist of 120 sequential 64 × 64 pixel multiphoton frames acquired at a rate of 4 Hz and 2400 speckle frames on the CMOS camera. The axial motion compensation was not used for this experiment.
When imaging human skin in vivo a 30 second acquisition was configured to consist of 30 sequential 256 × 256 pixel multiphoton frames acquired at a rate of 1 Hz with 2400 speckle frames. The axial motion compensation was enabled for the first 15 multiphoton frames and then disabled for the second 15 frames. Figure 2A shows an example of a speckle image acquired from a sample consisting of a piece of white cardboard. In order to avoid any bias due to static features in the speckle images from unwanted reflections in the optical system, the image stack was temporally averaged and the resulting average frame was subtracted from every frame of the stack. Lateral motion of the sample was determined using the peak of the normalised cross-correlation [23] γ(u, v) of temporally adjacent CMOS camera speckle frames f(x, y) and g(x, y)acquired 12.5 ms apart-according to

| Calculation of lateral sample motion from speckle images
where f and g are the means of the two images, respectively. The cross-correlation was implemented using the MATLAB function normxcorr2. In order to increase the precision of the subsequent peak-finding step, the resulting cross-correlation map was resized to increase the number of pixels by a factor of 16 in both dimensions using MATLAB's interp2 function with cubic interpolation. The location of the largest element in the resized cross-correlation map was then recorded to provide a readout of the lateral translation of the sample between the two frames, see example shown in Figure 2B. This process was repeated for all adjacent image pairs in the acquired sequence. The translation between frames was cumulatively summed to determine the total lateral translation of the sample since the start of the acquisition.

| Correction for lateral sample motion
The calculated lateral sample motion was then resampled in time from the CMOS camera acquisition rate (80 Hz) to obtain the sample displacement for each multiphoton image line (acquired at 256 Hz). Lateral motion-compensated multiphoton images were reconstructed line by line, taking into account the measured displacement of each line during the image reconstruction. This process was repeated for all multiphoton images in the stack. A map of the effective beam dwell time for each pixel in the reconstructed image space was also calculated and used to normalise any variations in image brightness arising from variations in beam dwell time between pixels in the reconstructed image caused by lateral motion of the sample. The key image processing steps for measuring and then correcting for lateral sample motion are summarised in the flowchart shown in Figure 2C.  Figure 2A shows an example raw speckle frame captured by the CMOS camera. Temporally adjacent speckle images were crosscorrelated and the normalised cross-correlation coefficient as a function of time is shown in Figure 3A. The spatial displacement of the cross-correlation peak was used to determine the velocity of the sample ( Figure 3B) which was then integrated numerically to yield the total sample displacement since the start of the acquisition ( Figure 3C). It can be seen that the normalised cross-correlation coefficient is reduced when the sample is moving, which we attribute to blurring of the speckle pattern over the CMOS frame integration time, and to the speckle pattern decorrelating as the sample translates due to scattering of light from other planes within the sample. Movie S1, Supporting Information shows the background subtracted speckle image overlaid with a red cross indicating the results of the speckle tracking. The motion of the red dot has been wrapped so that it stays within the speckle image frame for easier visualisation. Figure 4A,B shows exemplar raw multiphoton image frames from a sequence of 120 frames where individual fluorescent fibres within the cardboard can be resolved. Figure 4C shows the time-average image over the 120 frame image stack which has low image contrast. Figure 4D,E shows the effect of using the measured lateral sample motion shown in Figure 3C to correct the final image. In Figure 4D, an individual corrected frame (red) is overlaid on top of the corrected image obtained from all 120 frames (green), with regions where the channels coincide therefore being rendered yellow. If the lateral motion compensation worked perfectly then the red and green images would coincide perfectly. Movie S2 shows a false-colour 2 channel image where green shows the final corrected image and red shows each individual multiphoton image frame with lateral motion-compensation applied.
To estimate the precision of the lateral motion tracking, we compared the lateral shift calculated directly from multiphoton images of white card acquired at the beginning and end of a sequence of motion, with the cumulative net displacement calculated from all the speckle frames acquired during the motion. This was achieved by cross-correlating multiphoton image frames from the beginning of the image stack when the sample was stationary (t = 0-1 seconds) with multiphoton images acquired after the stage had translated 200 μm and then returned to be stationary again near its original position (t = 20-21 seconds). The lateral shift between these images was found to be 0.7 μm. We then used this information, together with the position information obtained from the lateral sample motion tracking (see Figure 3C), to determine the absolute error in the lateral sample motion tracking. The result was a tracking error of 4.1 μm after a round-trip sample translation distance of 400 μm, which is equivalent to a distance of 1.4 pixels in the multiphoton image.

| In vivo handheld imaging of the dorsal forearm
A volunteer's dorsal forearm was imaged with their arm lying on a flat rigid surface and with the scanner handheld such that the microscope objective lens was approximately vertical. Some of the weight of the scanner was supported by the operator and some was supported by the scanner resting gently on the arm. To introduce axial motion, the volunteer continuously opened and closed their fist with a period of approximately 0.9 seconds during the 30 second acquisition, thereby introducing a change in pressure between the skin and the front surface of the scanner due to the change in the size of the muscle beneath. Figure 5A shows the average of 15 seconds (ie, 15 frames) of multiphoton image acquisition without lateral or axial motion compensation. Figure 5B shows the effect of including axial motion compensation only, and Figure 5C shows the effect of both axial and lateral motion compensation. The dark cell nuclei are most clearly seen when both lateral and axial motion compensation is applied. The bright circular feature to the centre-bottom is thought to be part of a hair follicle. Figure 5D shows that the range of the axial displacement of the sample was approximately 20 μm, and Figure 5E shows that the lateral displacement was mainly in the y direction with a periodic displacement of approximately 10 μm.
Movie S3 shows the raw multiphoton image data with the axial motion compensation enabled for the initial 15 seconds and disabled for the final 15 seconds. During the final 15 seconds acquired without axial motion compensation, the focal plane of the imaging system can be clearly seen to be moving up and down as the cells periodically appear and disappear. During the initial 15 seconds, the cell nuclei are moving laterally-mainly in the y direction-but Movie S4 shows a false-colour 2 channel movie where green shows the final corrected image and red shows each individual multiphoton image frame with lateral motion compensation applied.

| In vivo handheld imaging of the chest
In order to demonstrate the ability of the system to image regions of the body other than the forearm, Figure 6 presents data obtained from skin on the upper chest. During the image acquisition, the volunteer was seated and the operator was free-standing while holding the scanner. Figure 6D shows the measured axial motion of the skin; there is a periodic variation with an amplitude of approximately 5 μm and a period of approximately 3.3 seconds that we attribute to motion of the chest due to breathing. Movie S5 shows the raw multiphoton data and the axial motion of the sample can be clearly seen in the second half of the data (t = 15-30 seconds) when the axial motion compensation is not applied. This is particularly clear when focusing on the wrinkle in the bottom left corner of the field of view which periodically changes its width. This is consistent with the vshaped structure of wrinkles, that is, they become narrower with increasing depth. Applying the axial motion compensation (t = 0-15 seconds) maintains the cell nuclei in the field of view and the width of the wrinkle feature in the bottom left-hand corner varies less with time. Figure 6E shows that the lateral motion is smaller than the axial motion and does not exceed approximately 1 μm during (t = 0-15 seconds), which corresponds to only approximately 1.3 multiphoton image pixels. Applying lateral motion correction in addition to axial motion compensation does lead to some sharpening of the borders of some cell nuclei, of which an example is shown in the inset to Figures 5A-C, but the largest qualitative improvement in image contrast is obtained from the axial motion compensation. Figure 7 presents data acquired in the same way as for Figure 5, but where there was no opening and closing of the fist. During this acquisition, the front surface of the scanner was sliding over the surface of the dorsal forearm. Figures 7A,B show the measured axial and lateral motion, respectively, and Figure 7C shows the processed image from the first 15 seconds of multiphoton image acquisition where both axial and lateral motion compensation have been applied. The lateral motion compensation was able to reconstruct the final image despite more than 250 μm of translation in the y direction. Movie S6 shows the raw multiphoton data with no lateral motion compensation applied. Movie S7 shows the lateral correction applied to the data obtained while the axial motion compensation is enabled (t = 0-15 seconds) with green showing the final corrected image and red showing each individual image frame with lateral motion compensation applied. During this work, we found that the motion artefacts acquired with the handheld scanner were qualitatively less severe than the motion artefacts seen in previous work carried out using a system with a rigidly fixed microscope objective. While we have no quantitative evidence to back up this observation, we hypothesise that this could be due to the operator actively compensating for sample motion to some extent by maintaining an approximately constant force between the instrument and the skin. When imaging skin on the upper chest using the handheld scanner, we observed axial motion of approximately 5 μm and it was compensating this axial motion that made the biggest qualitative difference in the final image. A more extensive study on a larger number of volunteers would be needed to provide a quantitative evaluation of the impact of the capability of the system to correct for sample motion.

| In vivo handheld imaging of the dorsal forearm during lateral sample motion
Our axial motion compensation approach has been shown previously to compensate for axial sample velocities up to 700 μm s −1 [14]. In this paper, we demonstrated in Figures 3 and 4 that our method for measuring and then correcting for lateral sample motion worked for a sample velocity of up to approximately 65 μm s −1 and that this led to a reduction in the normalised cross-correlation coefficient between temporally adjacent speckle frames to approximately 0.8. When the sample velocity was doubled, the normalised cross-correlation coefficient dropped to approximately 0.4 and the measurement started to fail (data not shown). This reduction in cross-correlation coefficient is due to blurring of the speckle pattern over the integration time of the CMOS camera and to speckle decorrelation caused by scattering within deeper layers of the sample. Further investigation is required to establish the relative contributions of the two effects. The impact of blurring of the speckle pattern could be reduced in the future by decreasing the CMOS integration time and the impact of speckle decorrelation could be reduced by increasing the frame rate of the CMOS camera. Increasing the frame rate of both the CMOS camera and the multiphoton imaging would enable lateral motion correction for higher sample velocities. Blood vessels with pulsatile flow in subcutaneous tissue are a potential source of image motion, and may also contribute to blurring of the speckle pattern over the CMOS integration time. Although normal epidermis is avascular, if there were any dermal capillaries within the volume being imaged, we would expect erythrocytes to be flowing sufficiently rapidly that the capillary contribution to the speckle pattern would blur out. However, the tissue contribution to the speckle pattern would remain and could still be used for lateral motion compensation. As the speckle image derives from light scattered throughout the object, it would not be greatly affected provided that the vessel fraction of the volume being imaged remained low.
The signal to noise ratio of the final corrected in vivo images was limited by the number of photons detected, which was determined by the fluorophores present in the sample and the image acquisition settings used. The average laser excitation power for in vivo imaging was limited to a maximum of 25 mW and the frame acquisition time was 1 second, which gave an average signal level of 6 detected photons per pixel. We acquired 15 frames each of 1 second duration with both lateral and axial motion compensation applied, and so in the final corrected images the average number of photons per pixel was approximately 90. The limitation on excitation power meant that we were not able to reduce the effects of sample motion by decreasing the image acquisition time, as this would have decreased the signal to noise ratio in the final image. Our lateral motion compensation approach increases the integration time that can be employed when imaging weak fluorescence signals in the presence of sample motion and it therefore increases the signal to noise ratio that can be achieved. In the future, this could help enable more accurate image quantification or more complex detection schemes, such as spectrally resolved detection or fluorescence lifetime imaging. Applications most likely to benefit are those requiring clinical or in vivo preclinical fluorescence imaging of the chest where sample motion due to breathing is most severe.

| CONCLUSIONS
We have previously presented a handheld multiphoton scanner and demonstrated its performance using ex vivo specimens [14]. In this paper, we have adapted our previous design to include a CMOS camera providing lateral motion tracking of the surface of the sample at a frame rate of 80 Hz. Crucially, this lateral motion tracking is performed using a static infrared beam at a different wavelength from the scanning multiphoton excitation. Our approach is therefore able to operate irrespective of the multiphoton fluorescence signal level or the type of features present in the multiphoton fluorescence image. In Figure 7, we demonstrated that the lateral tracking approach enables the effective field of view to be increased. In the future, it may be possible to use it to enable image tiling in order to further increase the size of the field of view. We demonstrated the performance of the lateral motion compensation using a sample of white cardboard. We also demonstrated the use of the system to perform multiphoton microscopy of human skin in vivo for the first time with motion compensation in all three directions.  Additional Supporting Information may be found online in the supporting information tab for this article. Movie S1. Movie corresponding to Figure 2A showing the background subtracted speckle image from the sample of white cardboard. Overlaid red cross indicates the results of the speckle tracking algorithm.

ACKNOWLEDGMENTS
Movie S2. False-colour 2-channel movie corresponding to Figure 4D showing the multiphoton fluorescence intensity images acquired from a sample of white cardboard. Green shows the final corrected image and red shows each individual multiphoton image frame with lateral motion compensation applied. Movie S3. Raw multiphoton image data from skin on the dorsal forearm with the axial motion compensation enabled for the initial 15 seconds and disabled for the final 15 seconds. Corresponding averaged frames without and with axial motion compensation are shown in Figure 5A and B, respectively. Movie S4. False-colour 2-channel movie corresponding to Figure 5 showing the multiphoton fluorescence intensity images acquired from skin on the dorsal forearm. Green shows the final corrected image and red shows each individual multiphoton image frame with lateral motion compensation applied. Movie S5. Raw multiphoton image data from skin on the chest with the axial motion compensation enabled for the initial 15 seconds and disabled for the final 15 seconds. Corresponding averaged frames without and with axial motion compensation are shown in Figure 6A and B, respectively. Movie S6. Raw multiphoton image data from skin on the dorsal forearm during lateral sample motion with the axial motion compensation enabled for the initial 15 seconds and disabled for the final 15 seconds. Corresponding data is shown in Figure 7. Movie S7. False-colour 2-channel movie corresponding to Figure 7C showing the multiphoton fluorescence intensity images acquired from skin on the dorsal forearm during lateral motion of the sample. Green shows the final corrected image and red shows each individual multiphoton image frame with lateral motion compensation applied.