SEARCH

SEARCH BY CITATION

Keywords:

  • Super-resolution imaging;
  • real-time dSTORM;
  • photoswitching microscopy

Summary

  1. Top of page
  2. Summary
  3. Introduction
  4. Materials and methods
  5. Results
  6. Discussion and outlook
  7. Conclusion
  8. Acknowledgements
  9. References
  10. Supporting Information

In the recent past, single-molecule based localization or photoswitching microscopy methods such as stochastic optical reconstruction microscopy (STORM) or photoactivated localization microscopy (PALM) have been successfully implemented for subdiffraction-resolution fluorescence imaging. However, the computational effort needed to localize numerous fluorophores is tremendous, causing long data processing times and thereby limiting the applicability of the technique. Here we present a new computational scheme for data processing consisting of noise reduction, detection of likely fluorophore positions, high-precision fluorophore localization and subsequent visualization of found fluorophore positions in a super-resolution image. We present and benchmark different algorithms for noise reduction and demonstrate the use of non-maximum suppression to quickly find likely fluorophore positions in high depth and very noisy images. The algorithm is evaluated and compared in terms of speed, accuracy and robustness by means of simulated data. On real biological samples, we find that real-time data processing is possible and that super-resolution imaging with organic fluorophores of cellular structures with ∼20 nm optical resolution can be completed in less than 10 s.


Introduction

  1. Top of page
  2. Summary
  3. Introduction
  4. Materials and methods
  5. Results
  6. Discussion and outlook
  7. Conclusion
  8. Acknowledgements
  9. References
  10. Supporting Information

In the recent past, a variety of methods have been developed that circumvent the diffraction barrier of light that restricts optical resolution to about 200 nm in the imaging plane (Born & Wolf, 1975) and enable far-field subdiffraction-resolution fluorescence imaging with resolution of ∼20 nm. To achieve subdiffraction-resolution imaging fluorescence emission of fluorophores has to be confined spatially and/or temporally. This can be realized either using phase masks or interference patterns that prevent fluorescence emission in specific parts of the laser focus (Hell & Stelzer, 1992; Gustafsson, 2005; Hell, 2007; Dedecker et al., 2007) or single-molecule based stochastic photoswitching of fluorophores to isolate fluorescence emission of individual fluorophores in time (Betzig et al., 2006; Hess et al., 2006; Bates et al., 2007; Clifford et al., 2007; Heilemann et al., 2008; Huang et al., 2008; Juette et al., 2008). Due to their simple and easy application wide-field single-molecule based approaches such as photoactivated localization microscopy (PALM) (Betzig et al., 2006), stochastic optical reconstruction microscopy (STORM) and direct STORM (dSTORM) (Heilemann et al., 2008) are currently widely used for super-resolution imaging. Because all these localization methods critically rely on the availability of efficient photoswitches or photoactivatable fluorophores they are unified in the following under the denotation ‘photoswitching microscopy’ (van de Linde et al., 2008a; Heilemann et al., 2009). Further potential applications of photoswitching microscopy comprise dynamic studies in live cells and quantitative high-resolution fluorescence imaging, e.g. to derive the number and structural organization of proteins in small subcellular structures or membranes, with a typical resolution of ∼20 nm (Shroff et al., 2008; van de Linde et al., 2008b).

The underlying principle of photoswitching microscopy is to separate fluorophores in time that are inseparable in space. This is achieved by light-induced switching of fluorophores between a bright fluorescent and a dark nonfluorescent state. After a subset of fluorescent probes is stochastically activated1, the fluorescence signal is read out and subsequently used for high-precision localization with nanometre accuracy. Thereafter, the fluorescent probes are deactivated. This procedure is repeated until a subdiffraction-resolution image can be reconstructed from the set of individual fluorophore localizations.

The major drawback of this technique is that the computational effort of localizing the fluorophores with nanometre accuracy is tremendous. First, the number and locations of spots indicating activated fluorophores are to be found; second, a model of the point spread function (PSF) is to be fitted to each such spot; third, by examining the parameters of the fits the true fluorophore localizations must be found. When viewing an isolated fluorophore, the emission signal can be mathematically described by its PSF. Since the exact, Besselian form of the PSF is hard to treat computationally (Karatsuba, 1993), its substitution with a Gaussian function has been fruitfully evaluated (Zhang et al., 2007) and offers the computational advantage of trivially computable partial derivatives. Central to these tasks is the knowledge that the reversal of this function yields the point source's position with an error substantially smaller than the optical resolution limit and proportional to σ/inline image, where σ is the standard deviation of the PSF and n is the number of photons detected (Thompson et al., 2002).

Researchers also used other methods than PSF fitting for tracking and localizing fluorophore positions, such as centroid computation, correlation, sum-absolute difference and Gaussian mask. However, these methods have been found (Cheezum et al., 2001; Thompson et al., 2002) to be less precise than a fit of a Gaussian PSF model.

Generally, subdiffraction-resolution imaging requires the acquisition of several thousand images with hundreds of thousands of localizations. Therefore, reconstruction of a subdiffraction-resolution image from experimental data measured typically within minutes can take up to several hours. This drawback has so far limited the broad application of photoswitching microscopy. In addition, with the ongoing quest for dynamic super-resolution imaging in live cells faster imaging speeds are currently developed (Steinhauer et al., 2008), widening the computational gap. However, faster data analysis is challenging and as a result reconstruction of super-resolution images substantially lags data acquisition. Here, we present a fast, parameter-free and robust image processing algorithm and system for nanometre-accurate localization of fluorophores. We developed a scheme for fast and accurate PSF reversal using fixed-width, fixed-orientation Gaussian functions for images with low signal-to-noise ratio. After initial tests of the algorithm we perform benchmarking tests with sample data demonstrating that the algorithm is ideally suited for real-time image reconstruction on a present-day personal computer.

Materials and methods

  1. Top of page
  2. Summary
  3. Introduction
  4. Materials and methods
  5. Results
  6. Discussion and outlook
  7. Conclusion
  8. Acknowledgements
  9. References
  10. Supporting Information

This section introduces the methods used for acquiring and processing image data, both simulated and real. The simulated data are used to benchmark algorithm characteristics against a ground truth. Their fabrication and processing is described in Sections ‘Simulated data’ and ‘Data processing’. The real data show the applicability and speed of the algorithm; their acquisition and processing is described in Sections ‘Experimental setup and sample preparation’ and ‘Simulated data’.

Experimental setup and sample preparation

dSTORM with carbocyanine dyes in the absence of an activator was performed on an Olympus IX-71 inverted microscope applying an objective-type total internal reflection fluorescence (TIRF) configuration using an oil-immersion objective (PlanApo 60×, NA 1.45, Olympus), as essentially described earlier by Heilemann et al. (2008). Two continuous-wave laser beams (647 nm and 514 nm) of an argon-krypton laser (Innova 70C, Coherent) were selected by an acousto-optic tunable filter (AOTF) and used simultaneously for readout and activation. Using appropriate filter sets the fluorescence was imaged on an EMCCD camera (Andor Ixon DU860ECS0BV). Photoswitching of Alexa 647 was performed with inline image at 514 nm and 35–inline image at 647 nm to increase photoswitching speed and thus enable a frame rate of 885 Hz using 64 by 64 pixels (inline image). A series of dSTORM images was acquired as previously described by Heilemann et al. (2008).

Indirect immunocytochemistry was applied to stain the microtubule network of COS-7 cells. The cells were fixed using 3.7% paraformaldehyde in phosphate-buffered saline (PBS, Sigma (St. Louis, MO, U.S.A.)) for 10 min. Afterwards, cells were washed and permeabilized in blocking buffer (PBS containing 5% w/v normal goat serum (Sigma) and 0.5% v/v Triton X-100) for 10 min. Microtubules were stained with mouse monoclonal anti-β-tubulin antibodies and with Alexa 647-labelled goat anti-mouse F(ab')2 fragments (Invitrogen, Carlsbad, CA, U.S.A.) serving as secondary antibody, both for 30 min. Three washing steps using PBS containing 0.1% v/v Tween 20 (Sigma) were performed after each staining step. Before dSTORM imaging, the PBS buffer was replaced by switching buffer, that is PBS (pH 7.4), containing oxygen scavenger (inline image glucose oxidase (Sigma), inline image catalase (Roche Applied Science, Basel, Switzerland), 10% w/v glucose) and 50 mM β-mercaptoethylamine (Sigma).

QDot565 surfaces were prepared via spin coating 50–100 μ l Qdot565 (Invitrogen) dissolved in water at a concentration of 1 nM onto a glass cover slide. Cover slides were treated with 3% hydrofluoric acid (Sigma) prior to use. Fluorescence of single QDot565 was acquired at 10–100 Hz under continuous irradiation with inline image at 514 nm using an appropriate bandpass filter.

Data processing

Fluorescent spots were found as maxima in a noise-reduced image and each fitted with a two-dimensional Gaussian function of the form

  • image(1)

where inline image is the point where the Gaussian is evaluated and V symbolizes the covariance matrix, which is a 2 by 2 matrix defining the width and form of the Gaussian. It can be expressed in terms of the standard deviation in X and Y directions (σx and σy) and the correlation ρ between the X and Y axes as

  • image(2)

An initial rough estimate for this covariance matrix was supplied (diagonal elements 4, off-diagonal elements 0) and improved iteratively as described in Section ‘Determining algorithm parameters’. To facilitate understanding, data processing operations described in the next paragraphs are visualised in Fig. 1.

image

Figure 1. Visualization of the rapid data processing scheme for photoswitching microscopy. From top to bottom, the data are processed from raw image form to a set of likely candidates for emission positions (spots), fitted by a Gaussian point spread function model and, given that this fit yields acceptable values, considered localizations. These localizations can be visualized in a result image or analysed quantitatively.

Download figure to PowerPoint

Noise reduction was performed by smoothing each image either with an average mask, a median mask, a Gaussian kernel mask as used by Thomann (2003), a morphological erosion or a morphological fillhole transformation2 followed by erosion (Soille, 2004). The first three operators were used with rectangular masks of size inline image for dimension k (i.e. 5 by 5 for the specimen shown in Fig. 2). The Gaussian kernel was built from two separated integer vectors with the standard deviations σx and σy equal to those of the two-dimensional Gaussian function. The erosions were performed with a rectangular structuring element of size sk= 0.5 mk (i.e. 3 by 3 for the specimen shown in Fig. 2) and implemented with the method described by Gil & Werman (1993).

image

Figure 2. Time-resolved view of the dSTORM image reconstruction on a real sample. The microtubule network of a mammalian cell was labelled via immunocytochemistry using a secondary antibody carrying the photoswitchable fluorophore Alexa 647 and processed with dSTORM. Laser powers of inline image (514 nm) and inline image (647 nm) were applied for fast acquisition with a frame rate of ∼1 kHz. Over 730 000 fluorophore localizations were found in the 70 000 images of 64 by 64 pixels acquired. A movie showing the complete acquisition process is supplied as supporting information.

Download figure to PowerPoint

Local maxima in the noise-reduced image were located using the non-maximum suppression (NMS) algorithm by Neubeck & Van Gool (2006) with a rectangular mask equal in size to the smoothing mask. This NMS algorithm extracts the positions of all pixels from an image that are local maximums. Local maximums are pixels that are strictly greater than all their neighbour pixels.3 We relaxed this condition to greater or equal in our implementation. The located maxima were sorted descendingly by their intensity and considered spot candidates. The candidates were fitted sequentially by means of motivational thresholding; that is, the candidates in the sorted list were fitted successively, beginning with the most intense, until a number M0 of fit attempts to successive candidates failed to meet the localization criteria stated in the next paragraph. M0 is a small constant that we have consistently chosen as 3 after evaluating its effects on a wide range of acquisitions. Higher values of M0 have little effect on the result, but increase computational cost.

Using the Levenberg–Marquardt least-squares algorithm, the exponential model seen in Eq. (3)

  • image(3)

was fitted to the pixels of the original, not smoothed image in a square mask of size inline image around each candidate's position. Herein, the amplitude A models the emission strength, inline image the subpixel-accurate fluorophore position and B the local background signal. The covariance matrix was either considered a fit parameter (free-form fit) or not (fixed-form). Fitting was terminated when a negligible step (both components of the step in inline image are smaller than 10−2) resulted from a position which had a smaller sum of squared residues than the previous position, as was suggested by Press et al. (1992). Fit results were discarded when they started within mk of the image border, diverged further than 0.5 mk from the spot candidate location or did not surpass an amplitude (i.e. fitted emission strength) threshold, whose choice will be discussed in the Section ‘Determining algorithm parameters’. Additionally, if the entries of the covariance matrix were fitted, we discarded fits if any matrix element differed from the starting values by more than a factor of 2. All other fit results were considered to represent true single fluorophore localizations.

For the sake of simplicity, we have neither treated overlapping fluorophore emissions (which happen with very low probability) nor linked multiple localizations of the same fluorophore in multiple frames.

The computational process described so far yields a set of fluorophore localizations. However, when a super-resolution image is to be displayed to a human operator, a high-contrast digital image is needed. Therefore, to visualize the localizations, we introduced a pixel raster with a 10-fold increase in resolution from the CCD pixels and added a weight of 1 for each found localization to the pixels in this raster, distributed bilinearly on the surrounding raster points. The resulting image depicted the localization density and was contrast-optimized with a weighted histogram equalization as described by Yaroslavsky (1985) with a weight factor of 2.

Simulated data

To simulate input data, all pixels not closer than 10 pixels to a localization were filtered from several different dSTORM series and histogrammed. The resulting histogram was used as a probability distribution for the background noise of the simulation. Each simulation background pixel value was drawn independently from a discrete probability distribution that consisted of all brightnesses present in the histogram, with probabilities proportional to the histogram value for this brightness. Fluorophore behaviour was modelled as a time-continuous Markov process between an on- and an off-state and a Besselian PSF. The average numbers of photons n captured in the pixel (xp, yp) from a fluorophore emitting at the subpixel-precise position (xf, yf) with intensity I was calculated from the Besselian of first kind and order J1 as

  • image(4)

with the integrals solved numerically using 87-point Gauss-Kronrod (Ehrich, 1994) integration and varied with a Poisson distribution. The camera response to incoming photons was set equal to the response given by a real camera's response curve as given by the manufacturer. The average lifetime of the on-state was chosen equal to the simulated acquisition time for a single image, but not synchronized with the simulated acquisition time intervals, to produce simulated images with a broad spectrum of spots with different photon counts typically ranging from 300 to 1000. The positions for simulated fluorophores were scattered randomly over the simulated acquisition area, excluding a small border at the image corners, but without further constraints.

Error rates in the detection process were measured by pairing, for each image, the active fluorophores [fluorophores were counted as active when emitting at least 30 photons, the transition point for photon-counting noise found by Thompson et al. (2002)] and the fluorophore localizations. The active fluorophore and the localization that matched best (i.e. those that had the lowest distance of all possible fluorophore/localization pairs) were paired and removed from their respective sets; this process was repeated until no pairs remained that were closer than 0.5 mk together.

We have simulated with two different datasets labelled Specimen 1 and Specimen 2, respectively. Specimen 1 used the PSF parameters and the noise distribution measured from the acquisition that is shown super-resolved in Fig. 2 and whose acquisition process was described above. Specimen 2 used the PSF parameters and the noise distribution of an earlier acquisition acquired with 50 Hz and Cy5 fluorophores. Specimen 2 was selected due to its high amount of background noise.

Results

  1. Top of page
  2. Summary
  3. Introduction
  4. Materials and methods
  5. Results
  6. Discussion and outlook
  7. Conclusion
  8. Acknowledgements
  9. References
  10. Supporting Information

In this section, we examine four topics: first, we will show that the algorithmic choices made in noise reduction can be compared with other proposed methods, such as those employed by Thomann (2003) and Zhuang et al. (2008). Second, we will do likewise for the algorithmic choices in PSF fitting. Third, we will show how the parameters for our method are automatically deducible from input data. Fourth, we will show that the resulting method is fully capable of real-time computation, resolving real biological specimens in less than 10 s.

Noise reduction

The first algorithmic choice for temporally confining photoswitching microscopy is the choice of a spot candidate detection method, i.e. a method that decides which locations in a source image should be fitted with a model of the PSF. This need has already been stressed by Cheezum et al. (2001), who named it ‘segmentation step’ in the context of fluorophore tracking. Most other authors have used some kind of spot candidate detection method. Basically, any spot candidate detection method performs detection of large, high-intensity regions in the image and therefore is, when viewed in the Fourier or frequency domain, a noise reduction algorithm. This suggests the usual noise reduction methods, such as morphological image cleaning (Peters II, 1995), averaging masks, median (Weiss, 2006) filters or alternating sequential filters (Goutsias & Schonfeld, 1991), can be applied to the spot candidate detection problem. To extract spot candidates from such a noise-reduced image, thresholding (Thompson et al., 2002) or non-maximum suppression (Thomann, 2003) are common methods; since a small non-maximum suppression only thins regions found by thresholding but does not eliminate them, and since a thinned region is sufficient for starting a Gaussian fit, non-maximum suppression is clearly the preferable choice here. Thus, the algorithmic choice in spot detection is the noise reduction approach.

The main goal in a spot detection algorithm is to find many spot candidates representing emission sites (true positives) while introducing few candidates representing noise (false positives). The requirement for few false positive candidates is necessary for two reasons: first, false positive candidates can result in false positive localizations, and second, computation time is wasted on false positive candidates. The first reason, the propagation of false positive localizations, is linked to some extent to the performance and the specific weaknesses of the candidate fitting and judging algorithm; therefore, we expect the whole performance of the spot candidate detection to vary somewhat with the candidate fitting stage. This dependency will be examined further in Section ‘Discussion and outlook’.

With this in mind, we compared the different noise reduction approaches introduced in the methods section and found little difference in the detected number of spot candidates, both for true and false positives and for false negatives. However, the precision and, to a lesser extent, the recall4 for localizations varied for different noise reduction methods as seen in Table 1. This indicates that the classification reliability of the fit amplitude threshold depends on the choice of the noise reduction method. Table 1 demonstrates how the resistance of a smoothing method to outlying pixels influences its performance. Median smoothing, known for its resistance to outliers in both directions, provides excellent precision and good recall, but is very slow.5 The Gaussian mask is, due to its large central weight, prone to outliers, explaining its very bad precision. The erosion filter is very resistant to high outliers, but susceptible to low outliers, and therefore performs badly under the high noise conditions and with the broader peaks of Specimen 2. If these low outliers are filled with a fillhole transformation prior to erosion, performance improves significantly. The average mask's outlier resistance is dependant on and grows with the mask size, as the difference between the relative performance of average and median smoothing for the two specimens shows.

Table 1.  Noise reduction algorithms compared at different amplitude thresholds (given in photons).
SpecimenAmpl. thresh.AverageGaussianErosionFillholeMedian
Rc.Pr.Rc.Pr.Rc.Pr.Rc.Pr.Rc.Pr.
  1. These numbers show the localization counts on a generated specimen with noise and PSF estimated from the specimen shown in Fig. 2. Because the data are simulated, the known positions of fluorophore emissions provide a ground truth. The Rc. columns give the recall, that is, the percentage of present fluorophore emissions our software correctly identified with localizations; the Pr. columns give the precision, that is, the percentage of the overall localizations that belong to actually present fluorophores, with the rest being false positives. Other specimens, both simulated and real, showed similar behaviour.

Spot finding time for a single image in ms0.050.080.220.360.51
Specimen 1 (small spots, low noise)12571%93%74%20%73%98%73%99%73%98%
15669%96%70%38%70%99%70%99%70%99%
18763%96%65%52%64%99%64%99%64%99%
25058%98%58%76%59%99%59%99%59%99%
Specimen 2 (large spots, high noise)12562%71%69%12%57%61%56%77%59%79%
15661%83%67%36%57%79%56%89%58%90%
18759%91%65%35%55%93%54%95%57%95%
25057%98%58%93%54%99%54%99%55%99%

Overall, we found that the average mask performs comparably to other methods for candidate detection and is the fastest operator considered, making swift spot candidate detection possible. Median smoothing or a fillhole transformation followed by an erosion are viable alternatives for small spot sizes, i.e. large pixel sizes, and high false positive resistance.

Fitting the covariance matrix

The second algorithmic choice is whether the entries of the covariance matrix V should be fitted to the data together with the position and the strength of the suspected fluorophore emission. We have measured the performance of the fitting algorithm in both variants on simulated data and on a real acquisition of isolated quantum dots. For the simulations, we compared important observables for fit accuracy in Table 2. It is important to note that the fit judging for free fitting presents a multidimensional optimization problem even if simple thresholding is applied (an amplitude threshold and up to four different thresholds for the covariance matrix). To make results comparable with the fixed-fitting approach, the statistics were normalized to common recall values. We randomly generated 15 different subpixel positions for fluorophores and simulated 100 000 images for each fluorophore, generating in total 75 000 images in which a fluorophore was active and 1.4 million images with pure noise or minimal fluorophore activity. Then, we chose optimal a posteriori thresholds for each fluorophore for both the amplitude threshold and, for the free-form fitting, the covariance matrix. These optimal thresholds were found by optimizing for a low false positive rate at a fixed recall rate in the parameter space spanned by the thresholds. We performed this optimization with the Simplex method (Nelder & Mead, 1965). These thresholds were searched independently for each fluorophore to gain information about their variability.

Table 2.  Comparison of the spot fitting and judging stage with different approaches.
QuantityRecallFixedFixed correlationFree
  1. Covariance matrix was either entries kept constant (Fixed), the correlation between X and Y axes kept constant (Fixed correlation) and fitting all entries (Free). The statistics given here were measured on Specimen 1, i.e. with PSF parameters and noise profile taken from the acquisition shown in Fig. 2. The fluorophore was fitted in 1.5 million images, averaged in blocks of 106. The amplitude prediction is the correlation between the measured amplitude and the number of photons emitted, and the resolution enhancement is the reduction factor between the full widths at half maximum of the point spread function and the distribution of localizations. Both amplitude prediction and resolution enhancement were calculated from true positive fits only to avoid interdependence with the false positive probability. Errors were not stated when below 0.005.

False positives per 104 fits to noise80%23.34 ± 0.36143.90 ± 2.15116.12 ± 9.27
60%1.62 ± 0.0911.54 ± 0.227.29 ± 0.44
40%0.32 ± 0.031.02 ± 0.050.70 ± 0.06
Amplitude prediction80%0.980.800.78
60%0.960.750.77
40%0.870.54 ± 0.020.61 ± 0.02
Resolution enhancement80%8.9 ± 0.16.0 ± 0.15.5 ± 0.2
60%13.0 ± 0.18.5 ± 0.38.8 ± 0.3
40%16.3 ± 0.110.3 ± 0.411.5 ± 0.3

In Table 2, we have measured three important characteristics of the spot fitting process: first, the false positive rate, which gives the probability that the spot fitting and judging code will falsely recognize a spot in random noise; second, the amplitude prediction, that is, the correlation between the fit amplitude and the number of photons in the spot; third, the resolution enhancement, that is, the ratio between the full width at half maximum of the PSF and of the spatial distribution of the localizations.

The false positive rate, i.e. the probability that a fit to random noise will be judged as a localization, is important because it determines how generously the fit results may be judged without introducing too many false positives. Since a generous fit result judgement will produce fewer false negatives and therefore more true positives in each image, the acquisition time to obtain sufficient structural information can be shortened considerably if a low false positive rate can be achieved. We see in Table 2 that the fixed covariance matrix approach gives the best false positive rates at all recall rates measured.

A high amplitude prediction is important if an additional amplitude threshold is to be applied,6 since it affects filter effectiveness. Surprisingly, the amplitude prediction gets worse for lower required recall rates (and, thereby, for higher amplitude thresholds and higher photon counts), an effect we did not investigate. However, fitting with a fixed correlation matrix performs best here as well.

The resolution enhancement gives an estimation of the localization precision attainable with dSTORM microscopy under the given conditions. It is measured by determining the statistical standard deviation of the localization centres around the simulated fluorophore position and comparing it with the standard deviation of the Gaussian approximation to the PSF. As expected, the resolution enhancement gets better with lower recall rates because the average number of photons per localization rises with the higher amplitude threshold. Again, fitting with a fixed correlation matrix proves superior.

It becomes obvious that fitting with fixed covariance matrix parameters performs comparably or better to fitting with free parameters in most respects. At first sight, this finding is surprising, since a reduction in the degrees of freedom available for fitting should make the fitting process more difficult, especially when the fixed parameters are not given precisely. However, this effect is mitigated by a higher noise tolerance: If the elements of V are determined from a large number of emissions and then fixed, the noise distortions in the covariance matrix cancel out and the PSF can be estimated more precisely from a large number of fits, enhancing the precision for a single fit. A method for this estimation will be given in the next section.

To measure the values for free-form fitting in Table 2, two thresholds (amplitude and covariance matrix) had to be optimized concurrently, whereas in fixed-form fitting one threshold suffices. Furthermore, the threshold optimization shows different covariance matrix thresholds for each desired recall rate, indicating that both parameters must indeed be adapted if free-form fitting is to be successful.

To check these simulation observations experimentally, we have analysed fluorescence images of real semiconductor quantum dots (QDs) immobilized on a dry glass surface. We fitted the same acquisition with both free-form and fixed-form fitting. Since there are multiple errors contributing to the uncertainty in localization, which was typically 8–15 nm, we were not able to quantify the contribution of the error in fitting. Hence we observed the changes in the localization precision (measured as described by Heilemann et al., 2008) and in the average distance the QD seemed to move between frames. Both observables were compared qualitatively for each QD. The fixed-form approach increased localization precision on 45 of the 66 measured standard deviations (∼500 single localizations each) and decreased the average distance between successive localizations on 26 of 33 quantum dots. This indicates a higher precision for fixed-deviation fitting with over 98% significance.

Determining algorithm parameters

Next, we examined the possibility of computationally determining the algorithm's parameters from input data. Apart from the smoothing and fitting mask sizes, there are two parameter choices to be made: first, the covariance matrix V for fitting the data, and second, the amplitude threshold that is used to judge the results of these fits. The decision for fixed-form fitting makes thresholds for the covariance matrix entries unnecessary, and the parameter M0 can in our experience be safely chosen as 3. Third, the resulting localizations must be visualized with sufficient contrast.

To estimate the covariance matrix, we employed an iterative statistical estimation procedure. Using arbitrary starting covariances, the pixels around each localization found were fitted with a Gaussian with centre coordinates fixed to the position of the localization and with amplitude, shift and covariance matrix entries as free parameters. The element-wise average of the found matrices, when differing significantly from the starting covariances, was then used as the starting covariances of a new iteration step. Using the simulation, we verified that this estimation process estimates a value within 4% of the correct covariances for normal signal-to-noise ratios and within 8% for very low choices. In all cases, 2000 localizations sufficed to estimate the covariances.

To obtain an amplitude threshold, we related the standard deviation of the background noise to the amplitude threshold necessary to achieve a false positive rate of 10−3. Our data show that, over a range of different noise spectra, an amplitude threshold of 25–35 times the noise standard deviation achieves the desired false positive rate. This indicates that a good amplitude threshold can be guessed when an allowable false positive rate is supplied.

Although these estimators automate the computation path from an input image to a set of localizations, the localizations must be visualized to be useful. Betzig et al. (2006) and van de Linde et al. (2009) have already described how the localizations are reduced to a spatial localization density map; however, the contrast in this image is often suboptimal and needs manual postprocessing. We have found that discretizing this high density map into a 16-bit deep intermediate image and mapping the intensities of this image to an 8-bit deep image by means of a weighted histogram equalization (Yaroslavsky, 1985) reliably produces images with good contrast. This is supported by the images shown in Fig. 2, which were produced with default values by our software and needed neither special parameter choices nor manual postprocessing. Although this technique reliably produces high-contrast visualization that are easy to process for human operators, the reader should be aware that quantitative methods should preferably work with the raw set of localizations. The reasons for this will be discussed in Section ‘Use of histogram equalization’.

Computational effort

The third topic we examined was the processing speed attainable with our algorithm. By optimizing the fit process, separating the average mask and utilizing advanced processor instruction sets through the Eigen linear algebra library, we achieved real-time speed even for the very fastest dSTORM acquisitions taken with a frame rate of ∼1 kHz. To process the 79-s-long acquisition that resulted in Fig. 2 and display a concurrent, live view on the intermediate result image (such as the subimages 2–5 shown in Fig. 2), our implementation needed 17 s of wall-clock time on an Intel E6550 processor7 and 70 Megabytes of memory. With the live view disabled, 14 s sufficed, with multithreaded CPU utilization over 98%. In this time, some 730 000 single fluorophore emissions were localized in 70 000 images, each of which was 64 pixels wide and high. During execution, the program spent 4 s with smoothing, non-maximum suppression and candidate sorting and 7 s with fitting, resulting in an average fitting time of 10 μ s per localization, including failing fits. The remaining 3 s were mainly used to read and convert the input data.

This computational speed proves that real-time computation of dSTORM images is possible and the data acquisition rather than the computational postprocessing is the speed bottleneck. By reducing this bottleneck with the fast dSTORM method described above, real biological structures can be super-resolved in 10 s, as seen in Fig. 2.

Discussion and outlook

  1. Top of page
  2. Summary
  3. Introduction
  4. Materials and methods
  5. Results
  6. Discussion and outlook
  7. Conclusion
  8. Acknowledgements
  9. References
  10. Supporting Information

In the previous sections, we have shown an accurate and real-time capable algorithm for processing photoswitching microscopy data. However, both the simulations used and the results obtained mitigate further discussion. First, we will discuss the limits of the simulated PSF and the background noise estimation and point out ways to overcome these limits; second, we will elaborate the caveats necessary for using histogram equalizations; third, we will discuss the entanglement in performance that occurred between the spot candidate search and the fitting stages and fourth, we will evaluate the results of fixed-form and free-form fitting.

Limits of the simulated PSF and the background noise estimation

Although the simulations we used have provided valuable ground truth to check and benchmark the different algorithms, they lack certain attributes of physical reality and the portability of these results to physical reality is therefore unknown. Regarding the PSF, we have used a coarse approximation to the PSF of a high-aperture microscope and have not modelled effects of Brownian motion, defocusing or dipole orientation. These effects might influence the quality of fitting and the preference for fixed-form fitting through introducing differences in the PSF between particles. To evaluate the numeric influence of all these factors, one would either modify the simulated PSF to account for all these effects or experiment with a physical system and try to isolate the effect.

Furthermore, our approach to background noise simulation did assume identically and independently distributed (IID) background noise over all pixels. Although we assume that this simplification did not disturb the extraction of background signal information from specimens, the lack of non-IID background phenomena in the simulated data could have influenced the results. Statistical tools such as co-occurrence matrices (Haralick et al., 1973) might be used to gain information about such phenomena in input data and to refine the simulations with this information.

Aside from these imprecisions, the scope of the simulations is also limited by their two-dimensional nature. In 3D, information about the Z coordinate must be extracted and is often extracted by way of properties of the PSF such as elongation or size. This causes the form of the PSF to vary from emission to emission depending on the Z coordinate; therefore, a fixed-form fitting approach becomes infeasible. However, this does not imply that a free-form fit must be used; the fixed-form versus free-form issue we have examined persists in the contrast between fitting with a minimal number of parameters or with the full set. For example, BP-STORM (Bewersdorf & Mlodzianoski, 2009) utilizes the PSF width in X and Y direction to determine the Z position of the particle, and both widths vary with the Z position. A fit routine for BP-STORM might only treat the Z coordinate as a free parameter and express the widths through it, thus fitting only one form parameter, or it might fit the PSF width, height and orientation and then extract the Z coordinate from the fit results in these parameters, thus fitting three additional parameters. Although we have only treated the 2D case, where no additional form parameter is needed, the arguments given in Section ‘Fitting the covariance matrix’ are also valid for methods with one or more required form parameters. Further research is needed to determine their truth for these cases.

Use of histogram equalization

For the reader with less image processing background, we want to re-stress the strong effect that histogram equalization, even in the weighted version, has on the quantization truth in an image. The most precise form of dSTORM results is the set of found localizations, with no image binning applied. The linear interpolation of this set onto a pixel raster (which produces the localization density image) and, even more so, the histogram equalization (which optimizes the contrast of this image) reduce the precision of results. Nonetheless, these steps tremendously ease interpretation of the results for humans, making them very useful for image-based microscopy and structural analysis, whereas the raw set of localizations is more useful for quantitative approaches.

In all image enhancing techniques, special consideration should be given to artefact creation, i.e. the introduction of new image features through the enhancement method. Since histogram equalization keeps the relative brightness relations between pixels, it is guaranteed that no new features will be introduced that were completely absent before, and that no feature will appear brighter than another feature when actually being darker. Only the relative differences will be amplified or reduced; however, in our experience this effect is small and mitigated by the enhanced contrast gained in the equalization. Readers concerned about such effects should keep in mind that neither standard-issue PC displays nor the human eyes are able to display contrasts accurately, thus limiting the usefulness of exact brightness differences.

Performance entanglement between candidate search and fitting

In Section ‘Noise reduction’, we have found that the choice of a spot candidate detection scheme strongly influences the precision of the fit judging stage. Although we were able to explain this result, the reader should note the implication of an interdependence between candidate search and fit judging. This result shows that the relative independence of the stages in high-precision localization, which authors such as Cheezum et al. (2001) have postulated, does not apply under the (from an image processing point of view) noisy conditions encountered in photoswitching microscopy.

For practical use, this result is not very restricting, since least-squares fitting with a Gaussian or similar-to-Gaussian function has become standard practice, and the arguments we used in Section ‘Noise reduction’ apply to all of these functions, making transferability of our results likely. However, researchers should be aware of the interdependency and check them when using a different particle localization method.

Fixed-form versus free-form fitting

We have seen in the results that fitting spots with a fixed covariance matrix provides advantages both in fit accuracy and speed. Additionally, it is easier to treat algorithmically because a single threshold suffices for fit result judging. Unfortunately, we were not able to isolate this effect numerically on real specimen. This might be due to two reasons: On the one hand, the difference between the simulated PSF and real fluorophore behaviour might have been too large with a bias towards fixed-form fitting, thus making the effect observed in simulations too large. On the other hand, other sources of noise, such as Brownian motion or the considerable size of quantum dots, might have masked the positive effect of fixed-form fitting.

Even though the results are not fully clear, we feel that a tentative conclusion can be reached. Fixed-form fitting showed a clearly better accuracy in simulations and a qualitative advantage on real acquisitions. Fixed-form fitting makes fit judging algorithmically easier, since it eliminates the need to judge matrix covariance entries. Last but not least, fixed-form fitting raises the computation speed. It does so by reducing the number of fit parameters and by making several numerical optimizations possible, which are based on the common occurrence of a negligible correlation between the X and the Y axes. For all these reasons, we deem fixed-form fitting to be the preferable approach at the current state of research.

Conclusion

  1. Top of page
  2. Summary
  3. Introduction
  4. Materials and methods
  5. Results
  6. Discussion and outlook
  7. Conclusion
  8. Acknowledgements
  9. References
  10. Supporting Information

We have shown a method for dSTORM image acquisition and for accurate and real-time super-resolution image processing within seconds: first, by finding candidates for spot fitting with a fast non-maximum suppression such as the Neubeck & Van Gool (2006) algorithm on an image smoothed by a fast outlier-resistant filter such as the average mask; second, by fitting only a small subset of these candidates consisting of those with the highest intensity in the smoothed image; and third, by fitting the point spread function with fixed covariance parameters to enhance fitting reliability and ease judgement of fitting.

With these principles, we implemented a software that can process photoswitching microscopy stacks on a timescale of seconds. We developed automated and robust estimators for all parameters and used a weighted histogram equalization technique for automated contrast balancing, making easy and fast use of dSTORM in the laboratory possible. We have additionally developed a graphical user interface and a direct link to the camera driver to ease practical use. Hence, neither knowledge of the computational process nor an informed choice for computational parameters is necessary to use the dSTORM method: the analysis and imaging process is reduced to choosing a region of interest and starting the process, making dSTORM a fast and easy method for widespread use.

With this software and fast dSTORM acquisition, we have shown how biological structures can be super-resolved in only 10 s. This core result adds greatly to the overall utility of photoswitching microscopy, making wide use of the technique less costly both in material and end-user knowledge.

Additionally, the fast acquisition method for dSTORM described in Section ‘Experimental setup and sample preparation’ opens up new applications for photoswitching microscopy. Resolving times of 10 s or less make high-resolved views of dynamic processes on a timescale of seconds, such as many processes in living cells, possible.

Footnotes
  • 1

    The mean distance between fluorophores residing in the fluorescent state has to be larger than the diffraction limit to ensure isolated point spread functions.

  • 2

    The fillhole transformation fills all holes, i.e. regions in the image with lower intensities than their border pixels, which are not connected to the border.

  • 3

    The neighbour pixels of a pixel p with respect to a mask are all pixels that are within the mask centred on p. To exemplify non-maximum suppression, consider a 64 by 64 image with a 3 by 3 mask. The NMS mask will be centred on each pixel, so excluding border cases 3 844 different masks will be considered. Each pixel that is strictly larger than the pixels at distance 1 horizontally, vertically or diagonally is output by a non-maximum suppression.

  • 4

    Recall and precision are error rate definitions stemming from the information retrieval sciences, where reliable true negative counts are unavailable. The recall is defined as the number of true positives divided by the total number of samples that should have been positive (here: number of localized fluorophores divided by total fluorophore count); the precision is defined as the number of true positives divided by the total number of positive classifications (Makhoul et al., 1999) (here: number of true localizations divided by total number of localizations).

  • 5

    Although there exist many fast implementations for median filters (Huang et al., 1979; Weiss, 2006; Perreault & Hebert, 2007), these are histogram based and therefore badly suited to the 16 bit deep images acquired in dSTORM. One of the few algorithms able to deal quickly with such deep images is the Ahmad & Sundararajan (1987) algorithm, which we used.

  • 6

    An amplitude threshold above the threshold necessary for a low false positive rate can filter out localizations with low photon counts and thereby raise resolution enhancement.

  • 7

    Dual-core, clocked with 2.33 GHz. This is a common and slightly outdated processor for desktop computers. We surmise that a similar multithreading efficiency can be achieved with a quad-core processor; if this assumption holds, performance would be 2.5 times higher with a current quad-core desktop CPU clocked at 3 GHz.

Acknowledgements

  1. Top of page
  2. Summary
  3. Introduction
  4. Materials and methods
  5. Results
  6. Discussion and outlook
  7. Conclusion
  8. Acknowledgements
  9. References
  10. Supporting Information

This work was supported by the Biophotonics program and the Systems Biology Initiative (FORSYS) of the German Ministry of Research and Education (BMBF; grants 13N9234 and 0315262).

References

  1. Top of page
  2. Summary
  3. Introduction
  4. Materials and methods
  5. Results
  6. Discussion and outlook
  7. Conclusion
  8. Acknowledgements
  9. References
  10. Supporting Information
  • Ahmad, M. & Sundararajan, D. (1987) A fast algorithm for two-dimensional median filtering. IEEE Trans. Acoust. Speech Signal Process. 34, 13641374.
  • Bates, M., Huang, B., Dempsey, G.T. & Zhuang, X. (2007) Multicolor super-resolution imaging with photo-switchable fluorescent probes. Science 317(5845), 17491753.
  • Betzig, E., Patterson, G.H., Sougrat, R., et al. (2006) Imaging intracellular fluorescent proteins at nanometer resolution. Science 313(5793), 16421645.
  • Born, M. & Wolf, E. (1975) Principles of Optics: Electromagnetic Theory of Propagation, Interference and Diffraction of Light. 5th edn. Pergamon Press, Cambridge .
  • Cheezum, M.K., Walker, W.F. & Guilford, W.H. (2001) Quantitative comparison of algorithms for tracking single fluorescent particles. Biophys. J. 81(4), 23782388.
  • Clifford, J.N., Bell, T. D.M., Tinnefeld, P., et al. (2007) Fluorescence of single molecules in polymer films: sensitivity of blinking to local environment. J. Phys. Chem. 111(25), 69876991.
  • Dedecker, P., Hotta, J.-i., Flors, C., et al. (2007) Subdiffraction imaging through the selective donut-mode depletion of thermally stable photoswitchable fluorophores: numerical analysis and application to the fluorescent protein dronpa. J. Am. Chem. Soc. 129(51), 1613216141.
  • Ehrich, S. (1994) Error bounds for Gauss–Kronrod quadrature formulae. Math. Comp. 62(205), 295304.
  • Gil, J. & Werman, M. (1993). Computing 2-d min, median, and max filters. IEEE Trans. Pattern Anal. Mach. Intell. 15(5), 504507.
  • Gustafsson, M.G.L. (2005) Nonlinear structured-illumination microscopy: wide-field fluorescence imaging with theoretically unlimited resolution. Proc. Nat. Acad. Sci. U.S.A. 102(37), 1308113086.
  • Haralick, R.M., Shanmugam, K. & Dinstein, I. (1973) Textural features for image classification. Syst. Man. Cybern. IEEE Trans. 3(6), 610621.
  • Heilemann, M., Dedecker, P., Hofkens, J. & Sauer, M. (2009) Photoswitches: key molecules for subdiffraction-resolution fluorescence imaging and molecular quantification. Laser Photonics Rev. 3(1–2), 180202.
  • Heilemann, M., Van De Linde, S., Schüttpelz, M., et al. (2008) Subdiffraction-resolution fluorescence imaging with conventional fluorescent probes. Angew. Chem. Int. Ed. 47(33), 61726176.
  • Hell, S. & Stelzer, E.H.K. (1992) Fundamental improvement of resolution with a 4Pi-confocal fluorescence microscope using two-photon excitation. J. Opt. Commun. 93, 277282. Provided by the SAO/NASA Astrophysics Data System.
  • Hell, S.W. (2007) Far-field optical nanoscopy. Science 316(5828), 11531158.
  • Hess, S.T., Girirajan, T.P. & Mason, M.D. (2006) Ultra-high resolution imaging by fluorescence photoactivation localization microscopy. Biophys. J . 91(11), 42584272.
  • Huang, B., Wang, W., Bates, M. & Zhuang, X. (2008) Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy. Science 319(5864), 810813.
  • Huang, T., Yang, G. & Tang, G. (1979) A fast two-dimensional median filtering algorithm. IEEE Trans. Acoust. Speech Signal Process. 27, 1318.
  • Juette, M.F., Gould, T.J., Lessard, M.D., et al. (2008) Three-dimensional sub-100 nm resolution fluorescence microscopy of thick samples. Nat. Methods 5(6), 527529.
  • Karatsuba, C.A. (1993) Fast evaluation of bessel functions. Integral Transforms and Special Functions 1(4), 269276.
  • Makhoul, J., Kubala, F., Schwartz, R. & Weischedel, R. (1999) Performance measures for information extraction. In DARPA Broadcast News Workshop, p. 249252. Morgan Kaufmann Publishers, San Francisco.
  • Mlodzianoski, M.J. & Bewersdorf, J. (2009) 3d-resolution in fpalm/palm/storm. Biophys. J. 96(3, Supplement 1), 636a637a.
  • Nelder, J.A. & Mead, R. (1965) A simplex method for function minimization. Comput. J. 7(4), 308313.
  • Neubeck, A. & Van Gool, L. (2006) Efficient non-maximum suppression. In ICPR '06: Proceedings of the 18th International Conference on Pattern Recognition, p. 850855. IEEE Computer Society, Washington , DC .
  • Perreault, S. & Hebert, P. (2007) Median filtering in constant time. IEEE Trans. Image Process. 16(9), 23892394.
  • Peters II, R.A. (1995) A new algorithm for image noise reduction using mathematical morphology. IEEE Transactions on Image Processing 4(5), 554568.
  • Press, W.H., Flannery, B.P., Teukolsky, S.A. & Vetterling, W.T. (1992) Numerical Recipes in C: The Art of Scientific Computing, 2nd edn. Cambridge University Press, Cambridge , U.K.
  • Schonfeld, D. & Goutsias, J. (1991) Optimal morphological pattern restoration from noisy binary images. IEEE Trans. Pattern Anal. Mach. Intell. 13(1), 1429.
  • Shroff, H., Galbraith, C.G., Galbraith, J.A. & Betzig, E. (2008) Live-cell photoactivated localization microscopy of nanoscale adhesion dynamics. Nat. Methods 5(5), 417423.
  • Soille, P. (2004) Morphological Image Analysis. Springer-Verlag, Berlin .
  • Steinhauer, C., Forthmann, C., Vogelsang, J. & Tinnefeld, P. (2008) Superresolution microscopy on the basis of engineered dark states. J. Am. Chem. Soc. 130(50), 1684016841.
  • Thomann, D.M. (2003) Algorithms for detection and tracking of objects with super-resolution in 3D fluorescence microscopy. PhD Thesis, ETH Zürich .
  • Thompson, R.E., Larson, D.R. & Webb, W.W. (2002) Precise nanometer localization analysis for individual fluorescent probes. Biophys. J. 82(5), 27752783.
  • Van De Linde, S., Endesfelder, U., Mukherjee, A., et al. (2009) Multicolor photoswitching microscopy for subdiffraction-resolution fluorescence imaging. Photochem. Photobiol. Sci. 8(4), 465469.
  • Van De Linde, S., Kasper, R., Heilemann, M. & Sauer, M. (2008a) Photoswitching microscopy with standard fluorophores. Appl. Phys. B 93(4), 725731.
  • Van De Linde, S., Sauer, M. & Heilemann, M. (2008b) Subdiffraction-resolution fluorescence imaging of proteins in the mitochondrial inner membrane with photoswitchable fluorophores. J. Struct. Biol. 164(3), 250254.
  • Weiss, B. (2006) Fast median and bilateral filtering. ACM Trans. Graph. 25(3), 519526.
  • Yaroslavsky, L. (1985) Digital picture processing. Springer, Berlin .
  • Zhang, B., Zerubia, J. & Olivo-Marin, J.-C. (2007) Gaussian approximations of fluorescence microscope point-spread function models. Appl. Opt. 46(10), 18191829.
  • Zhuang, X., Bates, W.M., Rust, M.J. & Huang, B. (2008) Sub-diffraction limit image resolution and other imaging techniques. United States Patent Application 20080182336.

Supporting Information

  1. Top of page
  2. Summary
  3. Introduction
  4. Materials and methods
  5. Results
  6. Discussion and outlook
  7. Conclusion
  8. Acknowledgements
  9. References
  10. Supporting Information

Movie S1. Real-time movie of the dSTORM image reconstruction on a real sample. The microtubule network of a mammalian cell was labeled via immunocytochemistry using a secondary antibody carrying the photoswitchable fluorophore Alexa 647 and processed with dSTORM. Over 730 000 fluorophore localizations were found in the 70 000 images of 64 by 64 pixels acquired.

Please note: Wiley-Blackwell are not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.

FilenameFormatSizeDescription
JMI_3287_sm_MovieS1.avi4087KSupporting info item

Please note: Wiley Blackwell is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.