Simultaneous degradation estimation and restoration of confocal images and performance evaluation by colocalization analysis

Authors

  • F. ROOMS,

    Corresponding author
    1. Department of Telecommunications and Information Processing, Ghent University, Sint-Pietersnieuwstraat 41, B-9000 Gent, Belgium
    Search for more papers by this author
  • W. PHILIPS,

    1. Department of Telecommunications and Information Processing, Ghent University, Sint-Pietersnieuwstraat 41, B-9000 Gent, Belgium
    Search for more papers by this author
  • D. S. LIDKE

    1. Department of Telecommunications and Information Processing, Ghent University, Sint-Pietersnieuwstraat 41, B-9000 Gent, Belgium
    2. Department of Molecular Biology, Max Planck Institute for Biophysical Chemistry, Am Fassberg 11, D-37077 Göttingen, Germany
    Search for more papers by this author

Mr Filip Rooms. Fax: +32 9 264 4295; e-mail: frooms@telin.ugent.be

Summary

A novel method for joint restoration and estimation of the degradation of confocal microscope images is presented. The observed images are degraded due to two sources: blurring due to the band-limited nature of the optical system [modelled by the point spread function (PSF)], and Poisson noise contaminates the observations due to the discrete nature of the photon detection process. The proposed method iterates noise reduction, blur estimation and deblurring, and applies these steps in two phases, i.e. a training phase and a restoration phase. In the first phase, these three steps are iterated until the blur estimation converges. Noise reduction and blur estimation are performed using steerable pyramids, and the deblurring is performed by the Richardson–Lucy algorithm. The second phase is the actual restoration. From then on, the blur estimation is used as a criterion to measure the image quality. The iterations are stopped when this measure converges, a result that is guaranteed. The integrated method is completely automatic, and no prior information on the image is required. The method has been given the name SPERRIL (Steerable Pyramid-based Estimation and Regularized Richardson–Lucy restoration). Compared with existing techniques by both objective measures and visual observation, in the SPERRIL-restored images noise is better suppressed.

Introduction

The useful information in an image is usually degraded by imperfections in the observation process, i.e. blurring due to the band-limited nature of the imaging process (like an optical system), as well as a noise process due to, for example, detector noise [e.g. photon noise in a photo multiplier tube (PMT)]. Image degradation is usually modelled as g(x,y) = N((h * f)(x,y)), with g(x,y) the blurred image, f(x,y) the unknown ideal image, h(x,y) the point spread function (PSF) and N( … ) the noise process. The symbol * represents the convolution operator and models the image blurring. The goal of image restoration is to recover f(x,y) as well as possible from a degraded observed image g(x,y). When the degradation parameters (in particular the PSF) are known, we are left with a classical image restoration problem (Katsaggelos, 1989; Lagendijk & Biemond, 1991; Bertero & Boccacci, 1998). However, in some cases the degradation parameters are unknown, and one has two choices: estimating the image of interest and the degradation parameters simultaneously (blind restoration, Kundur & Hatzinakos, 1996), or estimating the degradation parameters before starting the restoration process (Katsaggelos, 1989). This paper follows the latter approach.

We present a combined method for degradation estimation and image restoration that is based on steerable pyramids (Freeman & Adelson, 1991; Simoncelli et al., 1992; Portilla et al., 2003). To our knowledge, it is the first joint approach for degradation estimation and regularized deconvolution with steerable pyramids. The deblurring itself is the only step not performed with steerable pyramids (with deconvolution we refer to the whole restoration process, and with deblurring to that part of the process that sharpens the image). It uses the Richardson–Lucy (RL) algorithm, but with the regularization step in the steerable pyramid domain, it adds a new form of prior knowledge to the restoration problem.

The paper is organized as follows: in Section 2, the outline of our algorithm is discussed; it is divided into the following subsections: in 2.1 we give some background and details on the variant of the wavelet transform, namely the steerable pyramid; in 2.2, the noise reduction is discussed; in 2.3, the estimation of the image blur is explained; in 2.4 the deblurring step is explained; and in 2.5, a stopping criterion is formulated. In Section 3, some experimental results are shown and discussed. Finally, a conclusion is given in Section 4.

To illustrate how our research evolved, we refer to preliminary results about the PSF estimation in Rooms et al. (2001, 2002) and to results on the estimation/restoration technique in Rooms et al. (2003a,b). Here, the full restoration and estimation technique is described, and evaluated on synthetic and real confocal images, using colocalization analysis as an objective criterion to evaluate the restoration performance of the microscopic images.

2. Outline of the algorithm

The outline of the algorithm is shown in a block diagram in Fig. 1. First, the image is prefiltered to reduce Poisson noise. This prefilter is equivalent to the ‘denoise’ steps further in the algorithm, except that the ‘denoise’ is applied to the image transformed by the Anscombe transform to convert Poisson noise to Gaussian noise (Starck et al., 1998) instead of to the normal image. The ‘denoise’ steps in the algorithm are performed by computing a steerable pyramid decomposition of the image and by applying noise reduction to the subbands of the decomposed image. The next step is the estimation of the image blur from the noise-reduced subbands. Next, the filtered subbands of the steerable pyramid are recombined. Finally, RL deblurring is applied. The estimation of the blur is unstable at first (therefore the estimated restored image obtained in this stage is unreliable and discarded in the actual restoration phase), but converges typically after two iterations and is then used to generate the PSF for the deblurring steps. After this training process, the iterations are restarted with the original degraded image and the stable estimation of the PSF and the blur estimation is merely used as a quality measure to control the number of iterations: when it converges, the iterations are stopped. We will first focus on the steerable pyramid transform, followed by a more detailed explanation of the different steps of the algorithm itself.

Figure 1.

Block diagram of the algorithm. In the first block (the training phase), the PSF is estimated. This PSF is used as input for the restoration block. Both blocks consist of a denoise step (regularization) and a deconvolution step (deblurring). The PSF from the training phase is used unaltered, but a sharpness measure is calculated to determine when to stop the algorithm.

2.1. Introduction to multiresolution and steerable pyramids

In the last decade, multiresolution techniques (e.g. wavelets, steerable pyramids) have acquired increasing popularity in the field of image processing. We do not intend to describe the full mathematics behind multiresolution, only the key ideas and their applications in image processing. More details about the theory and applications can be found in Daubechies (1992), Strang & Nguyen (1996) and Mallat (1998).

The principle of multiresolution is to decompose an image into a lowpass band (which consists of the low spatial frequency features) and several detail subbands, which contain the high spatial frequency features under different orientations. The lowpass band is again decomposed into a lowpass and several detail images. An example of such decomposition applied to the so-called zoneplate image is shown in Fig. 2. To see how this decomposition can facilitate the separation of the noise and image, we give an illustrative example in Fig. 3. One can see that edges in the image result in large coefficients (black is large-magnitude negative, white is large-magnitude positive and grey is zero) in all scales for both images. These large coefficients (which are due to the useful image features) are correlated both spatially in connected clusters and across scales. Noise, by contrast, results in uncorrelated, small coefficients. All wavelet-based noise reduction algorithms exploit these two properties to separate the noise from the useful features in an image.

Figure 2.

Steerable pyramid decomposition of zoneplate image (image consists of a radial sine wave with a low frequency in the centre, increasing towards the edges). The top row shows the response of a bandpass filter for the highest frequencies at different orientations: large coefficients at the borders, where a high response to the highest spatial frequencies is found. The middle row shows the response of a bandpass filter for lower spatial frequencies, which have their largest responses further away from the borders. The bottom image shows the response of a filter for low spatial frequencies. Note that for clarity of the illustration, the subbands were not subsampled (i.e. every subband has the same size as the original image). In practice, however, subbands are subsampled for more efficient memory usage, keeping for each dimension only half of the coefficients for the next lower resolution scale. This example illustrates the fact that in the different subbands, features with different orientations are well separated (i.e. only produce response in one of the four orientated subbands).

Figure 3.

Comparison of decomposition of noise-free and noisy structure in the steerable pyramid domain. The first column is the image, and the other columns display the decomposition. We can see in both cases that useful signal propagates well across scales (i.e. produces large coefficients at the same positions) in contrast with noise, which is uncorrelated across scales.

Classical wavelet decompositions, however, have several disadvantages, such as limited orientation resolution (edges with different orientations in an image are not recognized as such, e.g. a classical separable wavelet decomposition does not distinguish between edges under π/4 or under 3π/4) and a lack of shift invariance. (The concept of shift invariance means that the same feature at shifted positions in an image produces the same wavelet coefficients at shifted positions. Classical critically sampled wavelet decompositions do not have this property: an edge slightly shifted in an image can produce an entirely different set of decomposition coefficients across the resolution scales.) Therefore, a variant on the classical wavelet transform was proposed, i.e. the steerable pyramid (Simoncelli et al., 1992). We will now briefly explain some basic properties of this steerable pyramid.

One often wants to analyse orientated image structures, such as edges at a certain angle. In order to do so, one can filter the image with a range of orientated kernels to cover all orientations present in the image. However, this demands a high computational cost. To avoid this, one can filter the image using a fixed set of base kernels and interpolate for the other directions from the images filtered with the base kernels (Freeman & Adelson, 1991; Simoncelli et al., 1992). Such a set of base kernels is called a steerable filter set. An illustrative example of such a set of filter kernels are the first partial derivatives in x and y of the Gaussian function:

image(1)

These two kernels Gx(x,y) and Gy(x,y) can be interpreted as filters for horizontal and vertical image features, respectively. Let I(x,y) be the original image and let the symbol * denote the convolution operator. Then Rx(x,y) = (I *Gx)(x,y) is the image filtered for vertical features and Ry(x,y) = (I *Gy)(x,y) for horizontal features. If we want a kernel Gθ(x,y) tuned for features in the image under an angle, θ, we can take a linear combination of the two base kernels: Gθ(x,y) = cos(θ)Gx(x,y) + sin(θ)Gy(x,y). Because convolution is a linear operation, Rθ(x,y) can be calculated by taking a linear combination of the horizontal and vertical image: Rθ(x,y) = cos(θ)Rx(x,y) + sin(θ)Ry(x,y). For a large number of orientations, this is computationally far less expensive (two multiplications and one addition per pixel instead of a more complicated convolution).

Steerable filters are not restricted to derivatives of Gaussians. In Freeman & Adelson (1991), the general conditions were given for a set of kernels to form a steerable base set. In Simoncelli et al. (1992), the steerable pyramid is described. This decomposition based on steerable filters is similar to the standard wavelet decomposition (Mallat, 1998), but has a much better orientation resolution and provides approximate shift-invariance.

To implement the steerable pyramid, we used the method described in (Portilla et al., 2003). We first explain this choice, followed by some implementation details. Our choice was based on the need for a multiscale decomposition that is approximately shift-invariant, with good orientation resolution. All these requirements are fulfilled by the steerable pyramid decomposition. In addition, the implementation of the pyramid based on the Fourier domain (as described here) provides almost perfect reconstruction from the decomposed subbands (within the range of floating point errors). An additional advantage for future research is that the properties of steerability (as described above) are maintained in this decomposition.

We now discuss some implementation details of the steerable pyramid. First, the image is transformed to the Fourier domain and multiplied with a set of transfer functions (here a transfer function is a mask designed to select certain frequencies from the Fourier transform of an image). This defines different image subbands (a subband is an image that contains only spatial frequencies in a certain band of the Fourier domain, see also Fig. 4). Each subband is then transformed back into the spatial domain. We start by defining an initial highpass transfer function H0(r) (r and θ are polar coordinates in the Fourier domain), and a lowpass transfer function L0(r). Next, L0(r) is subdivided into orientated transfer functions Bk(r,θ) (k = 0, …K − 1, with K the number of orientated subbands) and a residual lowpass transfer function L(r). Each of these transfer functions is multiplied with the Fourier transform of the image, and transformed back to the spatial domain, thus producing the different subbands of the image. In the example of the Gaussian derivatives explained above, only two orientated subbands were used (which results in a very limited orientation resolution). For this paper, we used four orientated subbands per resolution scale, which is sufficient to separate the useful features in the image from the noise. In Portilla et al. (2003), it is shown that the increase in denoising quality by increasing the number of orientations from four to eight is 0.3 dB and is still computationally not too heavy.

Figure 4.

The Fourier transfer functions corresponding with the filterbank in Fig. 5 are shown. In (1), the application of inline image and inline image are shown. In (2), the application of L* and H* on inline image are shown. (3) The subdivision of the annular transfer functions inline image and inline image in orientated transfer functions inline image and inline image. In (4), the recursion is illustrated: inline image is further subdivided into inline image and inline image.

For the next resolution scale, the subband resulting from transfer function L(r) is subsampled and again subdivided in orientated subbands and a lowpass band. In Fig. 5, the part in light grey is recursively applied in the dark rectangle of the bottom subband. A schematic illustration of the transfer functions is given in Fig. 4 (the origin is at the centre of each figure). The transfer functions are defined as:

Figure 5.

The filterbank scheme of the decomposition is given. The light grey rectangle is the recursive part (recursively applied in the dark grey rectangle at the bottom). H is highpass filtering, L is lowpass filtering. The Bk's are the orientated subbands for a resolution scale, the Ak's the orientated subbands for the finest resolution scale H0. ↓2 means subsampling with a factor 2 in every dimension, and ↑2 means upsampling.

image(2)

The transfer functions H0(r) and L0(r) in the initial step are given by L0(r) = L(r/2) and Ak(r,θ) = H(r/2)Gk(θ), where the Ak(r,θ) form orientated subbands of H0(r) (Portilla et al., 2003). The subdivision of the outer bandpass function H0(r) into Ak(r,θ) allows us to define a parent–child relation for denoising this last highpass as well.

These transfer functions were defined so that:

  • • The sum of the squares of the transfer functions equals 1, i.e. reconstruction is the exact inverse of the decomposition.
  • • The angular transfer functions inline image divide the frequency domain in bandpasses of 1 octave of frequencies, as in other multiscale decompositions.
  • • The subdivision of the angular transfer functions into orientated transfer functions allows a good orientation resolution of the features in the image, and allows perfect reconstruction of the previously mentioned angular transfer functions. In addition, the set of orientated transfer functions results in steerable subbands.

For the remainder of this article, when we use the term wavelet decomposition, we in fact use this special variant of the wavelet decomposition, i.e. the steerable pyramid.

2.2. Noise reduction

Noise reduction algorithms implemented in the wavelet domain (Donoho, 1995; Pižurica et al., 2002; Portilla et al., 2003; Şendur & Selesnick, 2002) have proven to be superior to classical noise reduction algorithms, because they can operate on the image locally. These algorithms follow the same strategy:

  • 1apply the forward wavelet transform,
  • 2shrink some of the wavelet coefficients according to some rule, and
  • 3apply the inverse wavelet transform.

In Donoho & Johnstone (1994) and Donoho (1995), a statistical method was developed (called SURE) to calculate an optimal global threshold for all the wavelet coefficients within a subband, and to set all coefficients below this threshold to zero. For the larger coefficients, the value of the threshold is subtracted (soft thresholding). This has already been applied in microscopy by Boutet de Monvel et al. (2001) and Stollberg et al. (2003). Computation of the SURE threshold involves sorting the coefficients in the different subbands, which takes in the order of N log N operations, with N the number of coefficients in a subband (Mallat, 1998). This noise reduction is suboptimal for two reasons. First, it assumes Gaussian noise. In confocal fluorescence imaging, however, the major source of errors is Poisson noise (Pawley, 1995; Van Kempen et al., 1997a). Unlike Gaussian noise, Poisson noise is intensity dependent, which makes separating image from noise very difficult (Nowak & Baraniuk, 1999). Second, it only exploits the fact that useful coefficients should be large, and does not take into account spatial correlation and correlation across scales (see again Fig. 3). We deal with these problems in the following way. To handle Poisson noise, we used the Anscombe transform (Starck et al., 1998), which transforms the Poisson data into data with a Gaussian distribution with unit standard deviation. The Anscombe transform is given by

image(3)

The method transforms Poisson data with mean intensity Ipoisson,1 and variance inline image into Gaussian data with mean Igaussian,1 and variance 1.0, while Poisson data with intensity Ipoisson,2 and variance inline image are transformed into Gaussian data with mean Igaussian,2 but again with variance 1.0, so that the noise has become independent of the signal. This is illustrated in Fig. 6, where a piecewise constant dataset is shown before and after the Anscombe transform. For each constant segment, mean and variance are given before and after the transform. This transformation therefore allows us to use well-studied methods for Gaussian noise on data with the much more complex Poisson noise. We have chosen to apply the following wavelet-based method developed by Şendur & Selesnick (2002), where the following bivariate wavelet shrinkage function is proposed for data contaminated with Gaussian noise:

Figure 6.

Poisson data and the Anscombe transform: we generated a piecewise constant synthetic dataset, and applied a Poisson noise process to it (top). For each of the pieces where the signal is constant, we calculated the average (µ) and the variance (σ2). We notice the well-known fact that for Poisson data, mean and variance are the same. At the bottom, the dataset was transformed using the Anscombe transform, and for the five constant segments, the mean and variance were calculated again. Now we note that the variance has become independent of the signal, and is equal to 1.

image(4)

with (x)+ = max{x,0}. In this expression w1(k) represents a denoised wavelet coefficient in some resolution scale and orientation. This denoised coefficient is calculated from the corresponding noisy coefficient y1(k) in the same resolution scale and orientation, and its parent y2(k), which is the wavelet coefficient at the same spatial position as y1(k) and the same orientation band, but in the next lower frequency resolution scale. inline image denotes the noise variance, and σ(k) denotes the local standard deviation of the wavelet coefficients in the neighbourhood of the kth coefficient (in practice, the standard deviation of the wavelet coefficients in a 7 × 7 window around the coefficient at position k is used). This algorithm is simple to implement, has low computational cost (involves only order N operations per subband, with N the number of coefficients per subband) and yet provides noise reduction that is comparable with other recent wavelet-based denoising techniques which also capture both inter- and intrascale dependencies of the wavelet coefficients with more complex models: Gaussian scale mixtures (Portilla et al., 2003); Markov random fields (Pižurica et al., 2002). In Portilla et al. (2003), a comparison is made with Pižurica et al. (2002) and Şendur & Selesnick (2002) in terms of PSNR, and Portilla et al. (2003) is considered to be the current state of the art, though the differences are usually less than 0.5 dB.

The reasons why the noise reduction of Şendur & Selesnick (2002) outperforms that of (Donoho, 1995) are

  • 1it adapts locally to the presence of edges due to the presence of σ(k), and
  • 2the factor inline image captures the correlation between coefficients across scales.

We adapted this algorithm for implementation with steerable pyramids. After denoising, the inverse Anscombe transform was applied. The denoising step is first combined with the Anscombe transform as a prefiltering step to reduce Poisson noise. Later, the denoising step is applied as a regularization step (without the Anscombe transform) after each deblurring step.

2.3. Blur estimation

Determining the PSF in confocal microscopy is usually performed by imaging a small fluorescent bead image with a size of the order of the resolution limit of the microscope. Methods using this bead image as a starting point to construct a PSF model have been proven to be very reliable and to allow better restoration results than with a theoretical PSF (Van der Voort & Strasters, 1995; McNally et al., 1998), because not all possible causes of image aberration can be taken into account into a theoretical model. However, this bead imaging is very time consuming (one has to re-image these beads regularly) and researchers do not always take the time to do this for various reasons (when imaging there is no intention to restore the image, limited time, etc.). Therefore, we developed our own method for blur estimation, which also operates in the steerable pyramid domain (so no extra transforms are necessary), and is based on estimating the sharpness of the sharpest edges present in the image. First, we assumed the PSF to be modelled by a two-dimensional Gaussian function. One can of course question how useful a Gaussian PSF model is, because this type of PSF is not band limited (the Fourier transform of a Gaussian is again a Gaussian, which means that in principle all frequencies can be recovered). This is indeed true when the image is degraded due to blur only. However, no imaging system is free of noise processes. The noise level in fact limits the band of useful frequencies in the degraded original (see Fig. 7).

Figure 7.

Radially averaged spectra of ideal test image (blue line), blurred with a Gaussian PSF (green line) and blurred, after which a Poisson noise process was simulated (red line). The vertical axis is logarithmically scaled. This figure illustrates that for an image blurred with a Gaussian PSF, in principle all frequencies could be recovered in the absence of noise. However, as soon as the image is contaminated by noise, an upper limit is imposed to which frequencies can be recovered (i.e. where the red line becomes flat).

Now we explain how this Gaussian PSF is estimated from the wavelet transform of an image. Mallat (1998) has shown that the evolution of the wavelet modulus maxima across scales depends on three factors:

  • 1the regularity of the original underlying signal (e.g. a sharp edge or a smooth transition);
  • 2the properties of the wavelet basis functions used in the transform;
  • 3the blur of the signal at position k.

This means that the blur can be estimated from the evolution of the wavelet coefficients through scales when the other factors are known. For a step edge blurred with a Gaussian PSF with different widths, the evolution of the wavelet coefficients across scales is displayed in Fig. 8 for different degrees of blurring. Practically, we locate the modulus maxima of the coefficients of the steerable pyramid in the highest resolution scale and follow the evolution of their magnitude through the different resolution scales. By fitting a curve of the form

Figure 8.

Wavelet modulus maxima (WMM) vs. scale for Gaussian PSFs with different widths (σblur). The more the image is blurred (larger σblur), the smaller the WMM at the finest scales (because in an image with more blur, less high-frequency content is present). The curves from top to bottom represent the evolution of the WMM for a step edge blurred with σblur = 1 (top curve), σblur = 2; σblur = 3; σblur = 4 (bottom curve). From these curves, the parameter b is estimated, which is correlated with the image blur.

image(5)

to each of these series of modulus maxima, we obtain a local measure for the blur in the image. Empirically, we found that the value b in Eq. (5) is directly related to the width of the Gaussian. The evaluation of this method was discussed in (Rooms et al., 2001, 2002). Here, we quickly summarize the results in Fig. 9, where a graph with the results of the estimation method is shown.

Figure 9.

Evaluation of the blur estimation for Gaussian blur (left) and out-of-focus blur (right). A set of eight test images was blurred with a parametric PSF, for which different values for the blur parameter were used, after which the blur was estimated from the blurred image. In the graph, the real applied blur parameter (σ in case of the Gaussian PSF in the left figure; the radius of the disc in the case of out-of-focus blur in the right figure) is given on the horizontal axis, and the estimated blur on the vertical axis. The mean for a certain amount of blur over the whole test set is plotted as a dot, together with the variance (vertical error bars).

Our current implementation assumes isotropic blur, but can be easily extended to anisotropic blur, as we explain now. Preliminary results in this respect are promising. This blur estimation procedure is performed independently for the four orientations in the pyramid. Assuming that the PSF is Gaussian and spatially invariant, we average out the blur measures for the different edge pixels for each of the four orientations to obtain a robust estimate for the width of the projection of the PSF in the direction perpendicular to the orientation of the subband. In this way, we obtain σblur,k for k ∈[0,K – 1], where σblur,k represents the width of the PSF projected under orientation kπ/4. We know that for a projection of the PSF under an angle θ, we can write inline image, with x and y the data vectors obtained from the 2D distribution projected according to the x- and the y-axis. When E(x) and E(y) are zero (e.g. for point-symmetric, zero-centred PSFs), this equation becomes

image(6)

where σblur,x is the width of the PSF projected on the x-axis; σblur,y is the width projected on the y-axis and σblur,xy is the covariance of the 2D distribution that corresponds to the PSF. We estimate these values for π/4 and for 3π/4, so we can calculate σblur,xy, which allows us to reconstruct the PSF with the following formula:

image(7)

with X = (x,y) the vector of the spatial coordinates, µ the vector (E(x),E(y)) (which is the zero vector) and Σ the covariance matrix:

image(8)

This method can easily be adapted for parametric PSFs other than Gaussian (e.g. out-of-focus blur, where the PSF is a uniform disc, and the radius has to be estimated). Of course, when a real measured PSF is available, it can easily be plugged into our algorithm (we do not have to generate a synthetic PSF), and our sharpness estimation can be used to monitor the sharpening of the image.

2.4. Deconvolution step

For the actual deblurring step, the RL algorithm was used. The origin of the related Expectation Maximization (EM) algorithm is found in Dempster et al. (1977). This method was first applied in image reconstruction in Shepp & Vardi (1982). The algorithm is identical to the algorithms independently obtained by Lucy (1974) and Richardson (1972). More background information and references can be found in Molina et al. (2001. This algorithm is already common practice in confocal image restoration (Van der Voort & Strasters, 1995; Holmes et al., 1995; Van Kempen et al., 1997a,b; Verveer et al., 1999; Schaeffer et al., 2001). Let g(x,y) be the degraded data and f(x,y) the ideal image, then the RL algorithm maximizes the likelihood P(g(x,y) | f(x,y)) in case of pure Poisson noise. It has an iterative scheme of the form:

image(9)

where k+1(x,y) indicates a new estimate of the image, g(x,y) represents the observed, degraded data, h(x,y) is the PSF and * is the convolution operator. Of course, in our algorithm h(x,y) is the Gaussian PSF estimated as explained in section 2.3, where in classical methods h(x,y) is obtained from a calibration measurement.

The multiplication and the division are point by point. The main properties of this algorithm are (Bertero & Boccacci, 1998):

  • 1each estimate k(x,y) is guaranteed to contain only non-negative pixels;
  • 2the sum of the intensities over the whole image is preserved during each iteration;
  • 3the log-likelihood of the solution is non-decreasing during the iterations, and converges to a maximum.

Because the standard RL algorithm is obtained by maximizing P(g(x,y) | f(x,y)), no explicit regularization is applied. However, numerous authors have suggested different regularization schemes for the algorithm (Conchello & McNally, 1996; Verveer, 1998; Verveer et al., 1999). One particularly simple regularization approach that provides remarkably excellent results is to apply a slight blurring after every RL iteration (e.g. with a Gaussian PSF with σ = 0.5). This regularization by post-blurring is replaced in our algorithm by the wavelet-based denoising approach described in section 2.2.

2.5. Stopping criterion

In this subsection, we briefly discuss when to stop the iterations. We noticed that the estimation of the blur decreases with the number of iterations (as expected), as the likelihood of the solution is increasing. As an empirical rule of thumb, we stop the iterations when the ratio of the blur estimated during the previous iteration σblur,k−1 and the current iteration σblur,k converging to 1.0 is less than a threshold ɛ, i.e. when | (σblur,kblur,k) − 1 | < ɛ. Here, we choose ɛ to be small enough (e.g. 0.001 as a rule of thumb, although the precise value is not that important).

This criterion is the last ingredient of our algorithm, resulting in a method that estimates the degradation from the original image (so no extra calibration is necessary) and automatically determines when the restoration is complete.

3. Experimental results

3.1. Synthetic images

A first experiment was the restoration of a synthetic image with simulated degradation. Here, Fig. 10(a) is the ideal image and Fig. 10(b) is a simulated degraded image, where first blurring was simulated by convolution with a synthetic PSF and then Poisson noise was applied (as is common in synthetic experiments: (Van Kempen, 1998; Verveer, 1998). In our case, a Gaussian PSF was used. In Fig. 10(c) the result with classical RL (with slight Gaussian blur applied after every iteration as regularization) is shown. Figure 10(d) is the result of restoration with RL-SURE (Boutet de Monvel et al., 2001), which is RL deconvolution combined with simple wavelet-based regularization (Donoho, 1995). This method does not perform optimally here because SURE thresholding was designed for Gaussian noise and applies the same threshold for all coefficients in one wavelet subband, which means that it does not take into account local image features such as edges. The SURE threshold is too large in background areas (as Poisson noise has a low variance in low-intensity regions) and too small in bright areas (as Poisson noise typically has a large variance in high-intensity areas). We can see this in the restoration results because the noise in the background has been removed, but is still prominent in the bright image structures. Figure 10(e) is the result with our own method, which explicitly takes into account the Poisson noise (through the Anscombe transform). Notice also here that noise is best suppressed (fewer artefacts), resulting in homogeneous regions where the original image was homogeneous.

Figure 10.

Comparison of different restoration algorithms on a synthetic test image. (a) The ideal image; (b) simulated degradation (blurring with a Gaussian PSF with σblur of 2 pixels and simulated Poisson noise); (c) RL was applied, with slight Gaussian reblurring after each iteration as regularization; (d) the result of the RL-SURE (Boutet de Monvel et al., 2001); (e) is the result of SPERRIL.

For this image, the PSF was available and was used in all algorithms used in the comparison. In SPERRIL, however, we used our own PSF estimation. In fact, we have compared our blind method with non-blind methods that used the correct PSF, which was a disadvantage for our technique. Nevertheless, SPERRIL passed the test. For this experiment, the ideal image was also available, so we could calculate the PSNR, defined by:

image(10)

where MSE stands for the mean square error between the pixel intensities of the two images (ideal and restored). The PSNR values for the different restoration algorithms are shown in Fig. 11.

Figure 11.

PSNR values for restoration of the synthetic test image for different amounts of blur. The image was blurred with a Gaussian PSF with different widths (sigma of the PSF), and for each, Poisson noise was simulated.

As a test of the resolution of the methods, we took a synthetic grid and applied synthetic degradation (see Fig. 12). The grid contains lines of different widths separated by different widths of background. The grid was degraded (Gaussian PSF, σ = 2, Poisson noise 120 photons per pixel) and restored with the different methods. In Fig. 13, we plotted the average line profile after restoration with the different methods. Around half maxima of the profile lines, all methods indicate similar profile widths, an observation which is to be expected, because the deblurring part is the same for all algorithms the same (i.e. RL). However, we noticed that our method better suppresses the ringing in the background between the line structures, at the price of slightly wider tails of the profiles (which have a comparable width as with RL-SURE). In addition, SPERRIL generally tends to underestimate intensities slightly, while the other methods generally tend to overestimate the highest intensities.

Figure 12.

Restoration of a synthetic test grid. (a) The original grid; (b) the degraded grid (blurred with a Gaussian with σ = 2, and poisson noise of 120 photons per pixel); (c) restoration with standard RL; (d) the result of RL-SURE; (e) the result of SPERRIL. This figure illustrates that in terms of resolution, all restoration methods are more or less comparable, as the deblurring part of the algorithm (RL) is the same for all the methods.

Figure 13.

Line profile of Fig. 12. The original grid is shown in blue. Note that classic RL (red) tends to overcompensate the blur, giving higher intensities than original. RL-SURE (black) sometimes overcorrects and sometimes undercorrects. SPERRIL (green) generally underestimates, but is closer on the peaks of the signal in most cases. Ringing is also less severe with SPERRIL.

3.2. Biological experiments

To test the biological validity of our algorithm, we have chosen to compare different image restoration algorithms as a preprocessing step before colocalization analysis. In Landmann (2002), the topic of deconvolution as a preprocessing step to colocalization is discussed, and significant improvement in the analysis results were obtained when deconvolution was applied. In Fig. 14, we illustrate the effect of image restoration on colocalization analysis. Two separate objects seem to have a certain overlap due to image blurring and the presence of noise in the image. When an appropriate restoration technique is applied, we eliminate this false overlap, thus improving the accuracy of the colocalization analysis, which provides better results when the restoration technique is more reliable.

Figure 14.

Degraded observation of separate objects. (a) Two separate objects are shown (blue) which are observed under non-ideal circumstances (green). (b) Standard RL with slight reblurring (blue) already provides an improvement by eliminating part of the false overlap. (c) The ideal case where the two blurred peaks were restored in the absence of noise (blue), thus completely separating the two peaks.

Here, we evaluate the performance of our algorithm on the colocalization analysis of visible fluorescent fusion-proteins expressed in A431 cells and a fluorescently labelled ligand. The erbB family of receptor tyrosine kinases includes the epidermal growth factor receptor (erbB1) and erbB3. These membrane proteins regulate cell growth and differentiation through binding of ligands to their extracellular domain, which in turn activates the protein and initiates signalling cascades. One way to test for protein–protein interactions is to label the proteins fluorescently and monitor colocalization, or overlap, of the two fluorescent signals in the image. We have labelled the ligand for erbB1 (epidermal growth factor, EGF) with fluorescent Quantum Dots (EGF-QD), and either erbB1 or erbB3 with a visible fluorescent protein (erbB1-eGFP or erbB3-mCitrine) (Lidke et al., 2004). Figure 15 shows the test set of images used in our analysis: the top row (images H1–H4) display single confocal sections of living cells expressing erbB1-eGFP to which EGF-QDs have been added. After activation by EGF-QDs, the erbB1 will internalize, i.e. it is transported from the membrane to the inside of the cell. Because the QD-EGF remains bound to the erbB1-eGFP after internalization, the colocalization should be high. In contrast, the cells in the bottom row of Fig. 15 show A431 cells expressing erbB3-mCitrine. In this case, EGF-QD does not bind directly to erbB3, but still binds to the native, unlabelled erbB1 present in the cell. Upon activation of erbB1 by EGF-QD, erbB1 does not form complexes with erbB3, and therefore in this image the colocalization should be low.

Figure 15.

Set of test images used in the colocalization analysis. The top row displays cells where erbB1-eGFP was visualized, thus resulting in high colocalization; the bottom row shows cells where erbB3-mCitrine was visualized, thus resulting in low colocalization.

Following Landmann (2002), we applied and compared the result of colocalization analysis on the raw image to the result of colocalization analysis applied after three different image restoration methods: classical RL, RL-SURE and SPERRIL. Each restoration algorithm was combined with standard background correction (with ImageJ, available at http://rsb.info.nih.gov/ij/). The different colour channels were each time handled independently for the sake of simplicity, and the PSF was estimated separately for the three colour channels. In Tables 1 and 2, we present the results of our overlap analysis based on Manders et al. (1993). For the different methods, we calculate the overlap coefficient Roverlap, which is defined by:

Table 1.  The RPearson parameter obtained from analysis of the raw image, the image restored with classical Richardson–Lucy (classic RL), with RL-SURE and with SPERRIL. The values of RPearson for the images with high colocalization (H1–4) are shown at the top. Then, the values for the four images with low correlation are given (L1–4). The values in bold are the average of RPearson over a certain class. Δ is the difference between the average for the group of images with high and with low colocalization. Δ is largest for SPERRIL, thus indicating that this method is best capable of separating low and high colocalization.
Rpearson
 RawClassic RLRL-SURESPERRIL
H10.8770.9170.86740.8659
H20.6850.7710.64660.6622
H30.7260.7710.68280.6919
H40.7580.8720.7520.7382
average0.7610.8330.7370.740
L10.5320.6040.50460.4525
L20.6150.6510.50570.4824
L30.4340.5440.31750.3344
L40.4820.5350.55190.3711
average0.5160.5830.4700.410
Δ0.2460.2490.2670.329
Table 2.  The Roverlap parameter obtained from analysis of the raw image, the image restored with classical Richardson–Lucy (classic RL), with RL-SURE and with SPERRIL. The values of RPearson for the high colocalization images (H1–4) are shown at the top. Then, the values for the low correlation images (L1–4) are given. Values in bold type are the average of Roverlap over a certain class. Δ is the difference between the average for the group of images with high colocalization and the group with low colocalization. Again, Δ is largest for SPERRIL, thus indicating that SPERRIL is best capable of separating low and high colocalization.
Roverlap
 RawClassic RLRL-SURESPERRIL
H10.9280.9270.87850.8725
H20.8680.8070.8050.6793
H30.8310.7980.70880.7183
H40.8880.8950.75430.7418
average0.8790.8570.7870.753
L10.8320.6700.55550.5076
L20.7850.6960.55190.5407
L30.6850.6260.39520.4285
L40.7150.6330.61020.4546
average0.7540.6560.52820.483
Δ0.1240.2000.2580.270
image(11)

where S1(x,y) and S2(x,y) are the intensities in the first and second colour channels, respectively. RPearson is calculated in a similar way, with S1(x,y) and S2(x,y) replaced by (S1(x,y) −S1,aver) and (S2(x,y) − S2,aver). Here S1,aver and S2,aver are the average intensities of the first and second colour channels, respectively.

The Roverlap and RPearson values for the raw data (see Tables 1 and 2) show only a small difference between the two cell types. Restoration with the classical RL already improves the result of the colocalization analysis, in that the difference is clearer. However, when applying SPERRIL restoration prior to analysis the largest difference between cell types is seen; this is consistent with what is expected from the underlying biochemical process in the cells (Lidke et al., 2004).

In Fig. 16, a detail of cell H3 of Fig. 15 is shown. Again, we can conclude that for classical RL, the regularization is rather poor. The results of RL-SURE also do not remove all the noise in the bright areas, while the result of SPERRIL provides the better suppression of the noise.

Figure 16.

Detailed views of the different restoration results for cell H3. (a) The raw image; (b) the result after classical RL; (c) the result after RL-SURE and (d) is the result of SPERRIL. Note that (b) and (c) still contain noise, while (d) maintains the same sharpness and the noise is better suppressed.

In Fig. 17, we show a comparison of restoration of the same image as in Fig. 16, but now with SPERRIL and different PSF models. Fig. 17(a) shows the raw image, and Fig. 17(b) the result of restoration with the theoretical PSF. Due to greater complexity as compared with wide-field microscopes, confocal microscopes are susceptible to alignment errors. These may introduce severe shape distortions in the PSF (Van der Voort, no year). Because most confocal microscopes suffer from these effects, it is necessary to rescale the real theoretical PSF for accurate restoration. However, here we used a simplified PSF model in (Fig. 17c,d), which provides satisfying results.

Figure 17.

Detailed views of restoration results with different PSFs for cell H3. (a) The raw image; (b) the results after SPERRIL with a theoretical PSF; (c) the result after SPERRIL with a Gaussian PSF; (d) the result of SPERRIL with an Airy PSF. Note that (b) does not take into account optical misalignments etc., and should be rescaled (otherwise it performs suboptimally, as here). However, the difference between (c) and (d) is invisible.

4. Conclusions and future work

In this paper, we present an integrated, stable and automatic algorithm to restore degraded photon-limited images. The degradation parameters are estimated from the image itself without requiring a calibration image. This is an important advantage, without having to sacrifice restoration quality. The estimation as well as the regularization of the deconvolution are performed in the steerable pyramid domain. The sharpness estimation is used to formulate a stop criterion for the iterations, thus making the restoration fully automatic.

The algorithm was evaluated both on synthetic images and on real confocal images. Our algorithm performs best, both in terms of visual quality as well as in terms of the PSNR. The algorithm was also evaluated on a set of biological images, where colocalization analysis was chosen as an objective way to evaluate the restoration result. In this case, we also compared our algorithm with other algorithms, and it performs best in terms of visual quality as well as in terms of the result of the colocalization analysis. In comparison with other deconvolution techniques, SPERRIL restoration obtains the best results both on synthetic and on real confocal images.

Future work involves extending the SPERRIL restoration method to three dimensions. The RL algorithm has already been extended to three dimensions. In addition, the extension of the regularization in three dimensions is possible using 3D multiresolution decompositions, as 3D extensions of steerable pyramids are already available (Delle Luche et al., 2003). The biggest challenge would be the estimation of the PSF in three dimensions. The PSF in two dimensions can be approximated by a Gaussian. In the z-direction, this is obviously not the case. In that case, we would think of another, more realistic confocal PSF model in which the parameters of the confocal PSF model can be adjusted to compensate any observed aberrations from the blur estimation (e.g. a certain blurring in the z-direction is observed by our method, but the theoretical model predicts a smaller deviation. Then the parameters of the confocal PSF model can be adjusted to take this deviation into account).

Acknowledgements

This research was financed with specialization scholarship of the Flemish Institute for Stimulation of Scientific-Technical Research in Industry (IWT). We also would like to thank Professor P. Van Oostveldt and his staff of the Laboratory for Biochemistry and Molecular Cytology at the Faculty of Agricultural and Applied Biological Sciences of Ghent University for interesting discussions, feedback and previous test images.

Ancillary