In this section, we examine four topics: first, we will show that the algorithmic choices made in noise reduction can be compared with other proposed methods, such as those employed by Thomann (2003) and Zhuang et al. (2008). Second, we will do likewise for the algorithmic choices in PSF fitting. Third, we will show how the parameters for our method are automatically deducible from input data. Fourth, we will show that the resulting method is fully capable of real-time computation, resolving real biological specimens in less than 10 s.
The first algorithmic choice for temporally confining photoswitching microscopy is the choice of a spot candidate detection method, i.e. a method that decides which locations in a source image should be fitted with a model of the PSF. This need has already been stressed by Cheezum et al. (2001), who named it ‘segmentation step’ in the context of fluorophore tracking. Most other authors have used some kind of spot candidate detection method. Basically, any spot candidate detection method performs detection of large, high-intensity regions in the image and therefore is, when viewed in the Fourier or frequency domain, a noise reduction algorithm. This suggests the usual noise reduction methods, such as morphological image cleaning (Peters II, 1995), averaging masks, median (Weiss, 2006) filters or alternating sequential filters (Goutsias & Schonfeld, 1991), can be applied to the spot candidate detection problem. To extract spot candidates from such a noise-reduced image, thresholding (Thompson et al., 2002) or non-maximum suppression (Thomann, 2003) are common methods; since a small non-maximum suppression only thins regions found by thresholding but does not eliminate them, and since a thinned region is sufficient for starting a Gaussian fit, non-maximum suppression is clearly the preferable choice here. Thus, the algorithmic choice in spot detection is the noise reduction approach.
The main goal in a spot detection algorithm is to find many spot candidates representing emission sites (true positives) while introducing few candidates representing noise (false positives). The requirement for few false positive candidates is necessary for two reasons: first, false positive candidates can result in false positive localizations, and second, computation time is wasted on false positive candidates. The first reason, the propagation of false positive localizations, is linked to some extent to the performance and the specific weaknesses of the candidate fitting and judging algorithm; therefore, we expect the whole performance of the spot candidate detection to vary somewhat with the candidate fitting stage. This dependency will be examined further in Section ‘Discussion and outlook’.
With this in mind, we compared the different noise reduction approaches introduced in the methods section and found little difference in the detected number of spot candidates, both for true and false positives and for false negatives. However, the precision and, to a lesser extent, the recall4 for localizations varied for different noise reduction methods as seen in Table 1. This indicates that the classification reliability of the fit amplitude threshold depends on the choice of the noise reduction method. Table 1 demonstrates how the resistance of a smoothing method to outlying pixels influences its performance. Median smoothing, known for its resistance to outliers in both directions, provides excellent precision and good recall, but is very slow.5 The Gaussian mask is, due to its large central weight, prone to outliers, explaining its very bad precision. The erosion filter is very resistant to high outliers, but susceptible to low outliers, and therefore performs badly under the high noise conditions and with the broader peaks of Specimen 2. If these low outliers are filled with a fillhole transformation prior to erosion, performance improves significantly. The average mask's outlier resistance is dependant on and grows with the mask size, as the difference between the relative performance of average and median smoothing for the two specimens shows.
Table 1. Noise reduction algorithms compared at different amplitude thresholds (given in photons).
|Spot finding time for a single image in ms||0.05||0.08||0.22||0.36||0.51|
|Specimen 1 (small spots, low noise)||125||71%||93%||74%||20%||73%||98%||73%||99%||73%||98%|
|Specimen 2 (large spots, high noise)||125||62%||71%||69%||12%||57%||61%||56%||77%||59%||79%|
Overall, we found that the average mask performs comparably to other methods for candidate detection and is the fastest operator considered, making swift spot candidate detection possible. Median smoothing or a fillhole transformation followed by an erosion are viable alternatives for small spot sizes, i.e. large pixel sizes, and high false positive resistance.
Fitting the covariance matrix
The second algorithmic choice is whether the entries of the covariance matrix V should be fitted to the data together with the position and the strength of the suspected fluorophore emission. We have measured the performance of the fitting algorithm in both variants on simulated data and on a real acquisition of isolated quantum dots. For the simulations, we compared important observables for fit accuracy in Table 2. It is important to note that the fit judging for free fitting presents a multidimensional optimization problem even if simple thresholding is applied (an amplitude threshold and up to four different thresholds for the covariance matrix). To make results comparable with the fixed-fitting approach, the statistics were normalized to common recall values. We randomly generated 15 different subpixel positions for fluorophores and simulated 100 000 images for each fluorophore, generating in total 75 000 images in which a fluorophore was active and 1.4 million images with pure noise or minimal fluorophore activity. Then, we chose optimal a posteriori thresholds for each fluorophore for both the amplitude threshold and, for the free-form fitting, the covariance matrix. These optimal thresholds were found by optimizing for a low false positive rate at a fixed recall rate in the parameter space spanned by the thresholds. We performed this optimization with the Simplex method (Nelder & Mead, 1965). These thresholds were searched independently for each fluorophore to gain information about their variability.
Table 2. Comparison of the spot fitting and judging stage with different approaches.
|False positives per 104 fits to noise||80%||23.34 ± 0.36||143.90 ± 2.15||116.12 ± 9.27|
|60%||1.62 ± 0.09||11.54 ± 0.22||7.29 ± 0.44|
|40%||0.32 ± 0.03||1.02 ± 0.05||0.70 ± 0.06|
|40%||0.87||0.54 ± 0.02||0.61 ± 0.02|
|Resolution enhancement||80%||8.9 ± 0.1||6.0 ± 0.1||5.5 ± 0.2|
|60%||13.0 ± 0.1||8.5 ± 0.3||8.8 ± 0.3|
|40%||16.3 ± 0.1||10.3 ± 0.4||11.5 ± 0.3|
In Table 2, we have measured three important characteristics of the spot fitting process: first, the false positive rate, which gives the probability that the spot fitting and judging code will falsely recognize a spot in random noise; second, the amplitude prediction, that is, the correlation between the fit amplitude and the number of photons in the spot; third, the resolution enhancement, that is, the ratio between the full width at half maximum of the PSF and of the spatial distribution of the localizations.
The false positive rate, i.e. the probability that a fit to random noise will be judged as a localization, is important because it determines how generously the fit results may be judged without introducing too many false positives. Since a generous fit result judgement will produce fewer false negatives and therefore more true positives in each image, the acquisition time to obtain sufficient structural information can be shortened considerably if a low false positive rate can be achieved. We see in Table 2 that the fixed covariance matrix approach gives the best false positive rates at all recall rates measured.
A high amplitude prediction is important if an additional amplitude threshold is to be applied,6 since it affects filter effectiveness. Surprisingly, the amplitude prediction gets worse for lower required recall rates (and, thereby, for higher amplitude thresholds and higher photon counts), an effect we did not investigate. However, fitting with a fixed correlation matrix performs best here as well.
The resolution enhancement gives an estimation of the localization precision attainable with dSTORM microscopy under the given conditions. It is measured by determining the statistical standard deviation of the localization centres around the simulated fluorophore position and comparing it with the standard deviation of the Gaussian approximation to the PSF. As expected, the resolution enhancement gets better with lower recall rates because the average number of photons per localization rises with the higher amplitude threshold. Again, fitting with a fixed correlation matrix proves superior.
It becomes obvious that fitting with fixed covariance matrix parameters performs comparably or better to fitting with free parameters in most respects. At first sight, this finding is surprising, since a reduction in the degrees of freedom available for fitting should make the fitting process more difficult, especially when the fixed parameters are not given precisely. However, this effect is mitigated by a higher noise tolerance: If the elements of V are determined from a large number of emissions and then fixed, the noise distortions in the covariance matrix cancel out and the PSF can be estimated more precisely from a large number of fits, enhancing the precision for a single fit. A method for this estimation will be given in the next section.
To measure the values for free-form fitting in Table 2, two thresholds (amplitude and covariance matrix) had to be optimized concurrently, whereas in fixed-form fitting one threshold suffices. Furthermore, the threshold optimization shows different covariance matrix thresholds for each desired recall rate, indicating that both parameters must indeed be adapted if free-form fitting is to be successful.
To check these simulation observations experimentally, we have analysed fluorescence images of real semiconductor quantum dots (QDs) immobilized on a dry glass surface. We fitted the same acquisition with both free-form and fixed-form fitting. Since there are multiple errors contributing to the uncertainty in localization, which was typically 8–15 nm, we were not able to quantify the contribution of the error in fitting. Hence we observed the changes in the localization precision (measured as described by Heilemann et al., 2008) and in the average distance the QD seemed to move between frames. Both observables were compared qualitatively for each QD. The fixed-form approach increased localization precision on 45 of the 66 measured standard deviations (∼500 single localizations each) and decreased the average distance between successive localizations on 26 of 33 quantum dots. This indicates a higher precision for fixed-deviation fitting with over 98% significance.
Determining algorithm parameters
Next, we examined the possibility of computationally determining the algorithm's parameters from input data. Apart from the smoothing and fitting mask sizes, there are two parameter choices to be made: first, the covariance matrix V for fitting the data, and second, the amplitude threshold that is used to judge the results of these fits. The decision for fixed-form fitting makes thresholds for the covariance matrix entries unnecessary, and the parameter M0 can in our experience be safely chosen as 3. Third, the resulting localizations must be visualized with sufficient contrast.
To estimate the covariance matrix, we employed an iterative statistical estimation procedure. Using arbitrary starting covariances, the pixels around each localization found were fitted with a Gaussian with centre coordinates fixed to the position of the localization and with amplitude, shift and covariance matrix entries as free parameters. The element-wise average of the found matrices, when differing significantly from the starting covariances, was then used as the starting covariances of a new iteration step. Using the simulation, we verified that this estimation process estimates a value within 4% of the correct covariances for normal signal-to-noise ratios and within 8% for very low choices. In all cases, 2000 localizations sufficed to estimate the covariances.
To obtain an amplitude threshold, we related the standard deviation of the background noise to the amplitude threshold necessary to achieve a false positive rate of 10−3. Our data show that, over a range of different noise spectra, an amplitude threshold of 25–35 times the noise standard deviation achieves the desired false positive rate. This indicates that a good amplitude threshold can be guessed when an allowable false positive rate is supplied.
Although these estimators automate the computation path from an input image to a set of localizations, the localizations must be visualized to be useful. Betzig et al. (2006) and van de Linde et al. (2009) have already described how the localizations are reduced to a spatial localization density map; however, the contrast in this image is often suboptimal and needs manual postprocessing. We have found that discretizing this high density map into a 16-bit deep intermediate image and mapping the intensities of this image to an 8-bit deep image by means of a weighted histogram equalization (Yaroslavsky, 1985) reliably produces images with good contrast. This is supported by the images shown in Fig. 2, which were produced with default values by our software and needed neither special parameter choices nor manual postprocessing. Although this technique reliably produces high-contrast visualization that are easy to process for human operators, the reader should be aware that quantitative methods should preferably work with the raw set of localizations. The reasons for this will be discussed in Section ‘Use of histogram equalization’.
The third topic we examined was the processing speed attainable with our algorithm. By optimizing the fit process, separating the average mask and utilizing advanced processor instruction sets through the Eigen linear algebra library, we achieved real-time speed even for the very fastest dSTORM acquisitions taken with a frame rate of ∼1 kHz. To process the 79-s-long acquisition that resulted in Fig. 2 and display a concurrent, live view on the intermediate result image (such as the subimages 2–5 shown in Fig. 2), our implementation needed 17 s of wall-clock time on an Intel E6550 processor7 and 70 Megabytes of memory. With the live view disabled, 14 s sufficed, with multithreaded CPU utilization over 98%. In this time, some 730 000 single fluorophore emissions were localized in 70 000 images, each of which was 64 pixels wide and high. During execution, the program spent 4 s with smoothing, non-maximum suppression and candidate sorting and 7 s with fitting, resulting in an average fitting time of 10 μ s per localization, including failing fits. The remaining 3 s were mainly used to read and convert the input data.
This computational speed proves that real-time computation of dSTORM images is possible and the data acquisition rather than the computational postprocessing is the speed bottleneck. By reducing this bottleneck with the fast dSTORM method described above, real biological structures can be super-resolved in 10 s, as seen in Fig. 2.