Localization microscopy relies on computationally efficient Gaussian approximations of the point spread function for the calculation of fluorophore positions. Theoretical predictions show that under specific experimental conditions, localization accuracy is significantly improved when the localization is performed using a more realistic model. Here, we show how this can be achieved by considering three-dimensional (3-D) point spread function models for the wide field microscope. We introduce a least-squares point spread function fitting framework that utilizes the Gibson and Lanni model and propose a computationally efficient way for evaluating its derivative functions. We demonstrate the usefulness of the proposed approach with algorithms for particle localization and defocus estimation, both implemented as plugins for ImageJ.
Localization-based fluorescence microscopy relies on sparse activation of individual fluorophores within a sample (Betzig et al., 2006; Hess et al., 2006; Rust et al., 2006). The activated fluorophores are spatially well separated and can be imaged individually. This activate-and-image process is then repeated over many frames, after which the coordinates of each detected fluorophore are determined computationally and combined to yield the final super-resolved image. The point spread function (PSF) model of the microscope plays a key role in these techniques. Every point-source fluorophore gives rise to a PSF pattern in the image domain, and a localization procedure is applied to the individual patterns. The PSF model that is being used for the localization task determines the accuracy that can be achieved in describing the examined biological structure (Manley et al., 2008; Hedde et al., 2009; Märki et al., 2010; Geissbühler et al., 2011).
The Gaussian function provides a reasonable approx-imation of the main lobe of the Airy pattern while introducing relatively low computational complexity. Such approximation, however, discards the side-lobes of the PSF, which are particularly important in 3-D PSF modelling (Zhang et al., 2007). The trade-off between choosing realistic and simplified PSF models is execution time, and we propose here to apply a two-stage approach: fast algorithms that rely on simplified PSF models can be used to obtain preliminary results as well as immediate feedback about the quality of the experiment whereas more realistic 3-D PSF models can be used for a more accurate analysis, performed at a later stage.
In this work we introduce a least-squares PSF fitting framework that utilizes realistic 3-D PSF models. In particular, the Gibson and Lanni model was shown to be very useful for restoration problems in microscopy (Markham & Conchello, 2001; Preza & Conchello, 2004), and we demonstrate its usefulness for particle localization and for defocus estimation, too. The least-squares localization approach is likely to yield less accurate results than the maximum-likelihood approach in the presence of non-Gaussian noise sources (Aguet, 2009), and a quantitative comparison of these two criteria was carried out in Abraham et al. (2009) for the Gaussian and for the Airy disc patterns. It was shown there that in terms of performance, the least-squares fitting method follows the maximum-likelihood method quite closely, introducing standard deviations that are larger by no more than 2 (nm) for the estimated lateral position of a particle. An exception to that is the case of relatively strong mismatch between the width values of the simulated and the fitted PSFs. This can, however, be taken into account by estimating this parameter from the data itself, or by optimizing for it, too.
These findings make the least-squares criterion an attrac-tive and nearly optimal method for PSF fitting tasks. It is a simple yet powerful tool that depends on the fitted model only. Its additional advantage is that it lends itself to a fast minimization using the Levenberg–Marquardt algorithm. The maximum-likelihood criterion, by contrast, requires additional knowledge on the noise sources and relies on optimization procedures that are in many cases more involved in terms of the cost function and in terms of the numerical implementation of the minimization procedure (Aguet et al., 2005; Abraham et al., 2009).
The paper is organized as follows: we describe the Gibson and Lanni model and compute its partial derivative functions while taking into account the stage displacement, the particle axial position and the defocus measure of the detector plane. We then introduce an efficient way of evaluating these functions. As an example application, we utilize the Gibson and Lanni 3-D PSF model for localizing particles in a z-stack. We fit the data with the 3-D position coordinates and with an amplitude value that accounts for the random nature of the photon emission rate. Our algorithm uses adaptive threshold values for local maxima identification, and an adaptive window size for the least-squares fit. Motivated by multiplane imaging (Prabhat et al., 2004; Ram et al., 2008), we also introduce an algorithm for estimating the defocus distance of the detector plane. All of our algorithms were implemented as ImageJ plugins1; they are briefly described in Appendix B.
Gibson and Lanni model
The Gibson and Lanni model generalizes the Born and Wolf model by accommodating for a refractive index mismatch between the three imaging layers (Appendix A). It assumes an optical path that includes a biological sample, a cover slip layer and an immersion layer (Fig. 1). It relies on the Li and Wolf approximation of the Kirchhoff diffraction integral
where is a set of parameters given in Table 1, J0 is the Bessel function of the first kind of order zero and is the wave number of the emitted light; a is the radius of the circular aperture at the back focal plane and it can be approximated by
describes the optical path difference
and is the lateral distance between the particle position and the detector in the image domain
The original expression of in Gibson & Lanni (1992) distinguishes between nominal and actual refractive indices values of the immersion layer. Here, we assume that they are both equal, and nominal conditions are also applied to the cover slip thickness value. The Gibson and Lanni model assumes a homogenous sample layer and this is one of its limitations. Variations in refractive index values can be measured by differential interference contrast techniques (Kam et al., 2001) or be modelled as a stochastic process (Schmitt & Kumar, 1996). This means that is no longer a deterministic function, so one can interpret ns as an effective refractive index value.
Table 1. Parameters of the Gibson and Lanni model.
Numerical aperture of the microscope
Refractive index of the specimen layer
Refractive index of the immersion layer
Emission wavelength in vacuum
Stage displacement relative to the nominal working distance of the objective lens, i.e. axial step size
average magnitude of the spherical wave that impinges on the back focal plane of the microscope
Lateral position of the point source relative to the optical axis
Axial location of the point-source fluorophore in the specimen layer relative to the cover slip
Lateral position of a pixel in the image domain relative to the optical axis
Axial distance of the detector plane from the back principle plane. The nominal value can be approximated by the tube length value of the microscope
The optical path difference (3) can be alternatively expressed by means of the defocus measure in the object space (Aguet et al., 2005). The advantage of expressing by means of stage displacement values, as done here, is the straightforward calculations for z-stack acquisitions. We also included a defocus measure in the image space, allowing one to compute the PSF pattern of the multiplane design (Prabhat et al., 2004).
One of the advantages of the the Gibson and Lanni model is its ability to predict nonsymmetric PSF patterns. Such patterns occur due to refractive index mismatch between the sample layer and the immersion layer (Fig. 2). Another advantage is the distinction between stage displacements, detector location and particle depth. Each parameter has a different effect in terms of defocusing. The defocus measure can be approximated by the value of the Taylor coefficient of in , giving rise to the following relation:
This criterion implies that focusing can be achieved by moving the stage, the detector or the particle itself.
Numerical evaluation and fitting
We consider the task of fitting the Gibson and Lanni PSF model to light patterns that originate from single point sources. We minimize the least-squares criterion by using the Levenberg–Marquardt method. To this aim we express (1) as follows:
This, in turn, allows one to write
where is one of the parameters of . The work of Ram et al. (2005) introduces generalized expressions for the Fisher information matrix with respect to the 3-D particle position. Starting from the Kirchhoff diffraction integral (1), the authors computed the partial derivative of the PSF as a function of . We apply these results to the Gibson and Lanni model and provide in Table 2 explicit expressions for .
Table 2. Derivative expressions for the Gibson and Lanni PSF model.
To evaluate such integrals accurately, one needs to adapt the integration step to the oscillatory nature of the integrands. For example, as the particle is located deeper into the sample, the integrands in Table 2 oscillate more rapidly, requiring more sampling points for the numerical approximation. A possible approach for the integral calculation was suggested in Aguet et al. (2005). There, the number of sampling points was determined by the highest possible oscillation rate of either the optical path difference or the Bessel function. Such a Nyquist-based approach assumes that the integrands are essentially band-limited, although they are better modelled as chirp functions. For this reason, the bandwidth values of the integrands are relatively large and so is the required number of sampling points. We therefore take a different point of view that relies on approximation theory.
We approximate the integrals in a progressive manner. Let the sampling interval at the nth iteration be (1/2)n, and let be the approximated value for . The integrands that appear in I1, I2 are smooth functions and we approximate them by a piecewise quadratic function using the Simpson method. This means that the approximation error has a certain rate of decay. In particular, there exists N>0 for which for all n>N
does not depend on n, nor does the decay rate L which reflects the approximation order of the Simpson method. We define
and observe that
This means that has a decay rate that is not larger than the decay rate of . As L and are not known in practice, we extract them from as demonstrated in Figure 3. In particular, we can find and such that
We approximate by and have .
In order to express the error in terms of percentage, we use the relative error measure as a stopping criterion. The number of iterations is determined by a threshold on the relative error, say 1%. To ensure that the approximation error is governed by the decay rate (11), we require it to meet this threshold for at least three consecutive iterations. The numerical evaluation of the Bessel functions imposes no limitation on ; the Bessel functions J0 is evaluated up to an accuracy of and the J1 function up to times its argument (Abramowitz & Stegun, 1972).
The advantage of such progressive evaluation resides in the fact that less computational time is spent on calculating integrals of low values while controlling the accuracy level of their numerical approximation (Fig. 4). Extensions to other 3-D PSF designs are possible, too. In particular, to the biplane setup (Prabhat et al., 2004; Juette et al., 2008; Kirshner et al., 2012), to the astigmatic PSF (Huang et al., 2008), and to the double helix pattern (Pavani & Piestun, 2008). The only modification is in the expression of the phase term . The Kirchhoff diffraction integral remains the same so our numerical implementation can still be used for these cases.
We consider two applications: 3-D particle localization, and misalignment estimation. We introduce a least-squares fitting algorithm and analyse its performance in the presence of Poisson and Gaussian noise sources. The description of the software that was developed for this work is described in Appendix B.
3-D particle localization
One possible way of determining the 3-D location of a particle is by acquiring several PSF images that correspond to different sets of PSF parameters . A z-stack, for example, provides PSF images with different thickness values of the immersion layer. Another option is to have several detector planes located at different positions along the optical path in the image domain. The former corresponds to different values of whereas the latter to different values of zd. Assuming a z-stack of fixed point sources, we introduce the following algorithm (Fig. 5).
Normalization: The z-stack is normalized to have a unit maximum value.
Local maxima identification: We use different threshold values for the different slices. This is because the amplitude of the focused pattern fades away as the particle moves deeper into the sample. The parameter of a given slice provides us with an initial estimation of the particle depth by means of (5). We then calculate a PSF value for , denoted , and a PSF value for the global maximum, denoted . The ratio between these two PSF values approximates the expected ratio between a local maximum at a particular slice and the global maximum. Deviations from this ratio may occur due to sub-pixel axial and lateral positions, random nature of the emission rate and noise. For these reasons we choose the threshold to be
For large stage displacement values, may be lower than the mean noise level, and for that reason the local maxima should be higher than the noise level, too. We estimate the mean m and variance of the noise from the z-stack itself and set this threshold value to be . In addition to the voxel value, the local maxima should be higher than its 26 3-D neighbours.
Least-squares fitting: The 3-D neighbourhood of each local maxima is fitted with the Gibson and Lanni model, where the size of the fitting window is . We set to be the distance to the second minimum of the Airy pattern. This distance captures 90% of the energy of the Airy pattern and we use the same criterion for determining the axial parameter d by numerical means. For the PSF parameters of Figure 2, the values of r and d are 8 (pixels) and 9 (slices), respectively. Initial values for (xp, yp) are the coordinates of the local maximum and the initial value for zp is given by (5). The initial value for A0 is the ratio between the pixel value of the local maximum and the value of the PSF model for (xd, yd) = (xp, yp). Derivative values of the PSF model are given in Table 2 and (10).
Goodness of fit: We use several criteria for accepting the fitted parameters: the lateral position should be close to the local maximum, the axial position should be close to the initial guess and the signal-to-noise ratio value of the fit should be above a certain threshold.
Simulated z-stack data are shown in Figure 6 and least-squares localization performance is demonstrated in Figures 7–9. The fitting window was pixels. Our performance analysis included several particle depth values, several lateral positions and several noise levels. Each setup was repeated 20 times. Following Ober et al. (2004), the noise sources are Poisson-distributed scatter noise and Gaussian-distributed readout noise. The emission rate of every point source is a Poisson random number. Our results suggest that the lateral localization accuracy is less sensitive to the particle depth value, compared with the axial localization accuracy. The relatively low localization accuracy for the depth value of 200 (nm) follows from the smaller number of data points that can be used in the fitting process. This is due to the fact that the z-stack consists of positive stage displacement values only, demonstrating the importance of the fitting window size for the axial localization. We also observe that the subpixel lateral positions have minor effect on the lateral and axial localization accuracies.
Motivated by the calibration stage of multiplane microscopic design (Prabhat et al., 2004; Juette et al., 2008), we suggest here a fitting algorithm that allows for the determination of the axial position, zd, of the defocused detector plane. The input data is a z-stack of fixed point-sources, all lying on the cover slip zp= 0. Stage displacement values are both positive and negative with respect to the working distance. The output of the algorithm are the estimated values of xp, yp, zd and A0. The algorithm follows the stages of Figure 5 with the modification that the initial parameter for zd is determined by sweeping over several possible values and choosing the one that maximizes the PSF value at the particular slice of the z-stack. The threshold value is set to accounting for the Poisson distribution of the emission rate and for the noise. If all of the particles lie at the same depth position, then local maxima should appear in the same frame. Due to noise, the local maxima may appear in neighbouring slices and this does not require re-calculating , as was done in the previous algorithm. Fitting results are depicted in Figures 10 and 11. They suggest that the least-squares criterion provides an unbiased estimation with a relatively low-variance value for the axial position of the detector plane.
In this work, we provided a general approach to 3-D PSF fitting applications, while introducing an efficient and accurate evaluation of the PSF function. We focused on the Gibson and Lani model and showed its usefulness in particle localization and in defocus estimation. Our fitting algorithm relies on the least-squares error measure, and we provided analytical expressions for the required partial derivative functions. We then introduced an efficient and accurate way for evaluating the Kirchhoff diffraction integral. Our fitting algorithm includes local maxima identification with adaptive threshold values and window size for the least-squares fit. The algorithm utilizes the Levenberg–Marquardt minimization method and the initial values we use for it rely on the PSF model. Simulation results with z-stack data indicate that the lateral localization accuracy is less sensitive to the particle depth value, compared with the axial localization accuracy. It was also shown that axial localization accuracy can be improved substantially by taking more data points in the axial direction. We also observed that subpixel lateral positions have minor effect on the lateral and axial localization accuracies when using the least-squares criterion. Our results also suggest that this criterion provides an unbiased estimation with a relatively low variance value for the axial position of the detector plane.
This work was funded (in part) by the Euro-BioImaging project.
Appendix A: The relation between the Gibson–Lanni and the Born–Wolf models
The Born and Wolf model is
We set the aperture, a, to be the back focal plane of the microscope and rely on the sine condition to have the following relation (Fig. 12)
The parameter zd denotes the distance of the focused image from the back principle plane, and denotes the distance of the detector from the same plane. In practice, zd≫f and which leads to
We express the defocusing measure in the image domain in terms of defocusing in the object domain by relying on the analysis of Gibson (Gibson, 1990, chapter 3); is the distance from the front focal plane for which a particle would be focused on the detector plane , that is, just beneath the cover slip; is the distance for which a particle will be focused at zd in the image domain. The two pairs of similar triangles in Figure 13 yield
which is the Born and Wolf model for the wide field microscope.
The Gibson and Lanni model generalizes the model of Born and Wolf in the following manner. We assume that all layers have the same refractive indices ns=ni=n. It then follows that where the defocus measure in the object domain is due to the particle’s depth and stage displacement . The Taylor series for the OPD is
and the coefficient of which amounts to defocusing coincides with the phase term of (A.6).
Appendix B: Software description
The Gibson and Lanni PSF model was implemented in Java, and it can be used with a variety of software packages. As an example application, we developed the ImageJ plugin PSFGenerator, which evaluates and visualizes the 3-D pattern of the Gibson and Lanni model.2 It is fast and easy to use, requiring only few input parameters that are readily available for microscopy practitioners. The output of the plugin is a z-stack of any chosen size, which can also be visualized with orthogonal views. The plugin also provides a table that calculates the maximum value, the proportional energy and the effective radius of each slice (Fig. 14). The open architecture of the plugin allows for easy incorporation of additional PSF models, and the current version includes the Born & Wolf and the Richards & Wolf models.
We also used our Gibson and Lanni Java implementation for simulating data sets of photo-activated fluorophores.3 To this aim, a user-dependent biological structure is described by a 3-D density map of fluorophores (Fig. 15). Every fluorophore is assigned a random position (based on the density map), a random excitation time instant and a random photon emission rate value. We then use the Gibson and Lanni model to determine the image of each fluorophore and add the image to the time frame it belongs to. Background scatter noise and readout noise are added, too. The output of our software is a sequence of frames, composed of images of photo-activated fluorophores. We also provide the (xp, yp, zp, frame number) indices of every fluorophore, which can be used as a validation tool for other localization algorithms.