Department of Physics, Washington University, St. Louis, Missouri, USA
Correspondence to: Zohar Nussinov, Department of Physics, Washington University, One Brookings Drive, Campus Box 1105, St. Louis, MO 63130, U.S.A. Tel: 314-935-6277; fax: 314-935-6219; e-mail: firstname.lastname@example.org
Inspired by a multiresolution community detection based network segmentation method, we suggest an automatic method for segmenting fluorescence lifetime (FLT) imaging microscopy (FLIM) images of cells in a first pilot investigation on two selected images. The image processing problem is framed as identifying segments with respective average FLTs against the background in FLIM images. The proposed method segments a FLIM image for a given resolution of the network defined using image pixels as the nodes and similarity between the FLTs of the pixels as the edges. In the resulting segmentation, low network resolution leads to larger segments, and high network resolution leads to smaller segments. Furthermore, using the proposed method, the mean-square error in estimating the FLT segments in a FLIM image was found to consistently decrease with increasing resolution of the corresponding network. The multiresolution community detection method appeared to perform better than a popular spectral clustering-based method in performing FLIM image segmentation. At high resolution, the spectral segmentation method introduced noisy segments in its output, and it was unable to achieve a consistent decrease in mean-square error with increasing resolution.
Fluorescence lifetime imaging microscopy (FLIM), a promising technique for imaging molecular processes, generates images using the characteristic fluorescence lifetimes (FLTs) from subcellular locations of biological samples (such as cells and thin tissue sections) that are treated with fluorescent contrast agents. The FLT is the average time a molecule resides in the excited state before returning to the ground state through fluorescence emission (Nothdurft et al., 2012). In this work, we propose a multiresolution community detection (MCD) method based on graph partitioning theory (Fortunato, 2010) to automatically segment FLIM images of cells. MCD (Girvan & Newman, 2002; Newman, 2004; Ronhovde & Nussinov, 2009, 2010; Fortunato, 2010; Hu et al., 2012a) seeks to divide groups of nodes with dense interconnections among them and with more sparse connections with the other nodes in the network. It thus partitions a large physically interacting system into optimally decoupled communities. To explore the performance of the proposed method, we segmented donor FLTs in FLIM images of cells transfected with Frster resonance energy transfer (FRET) protein pairs (Orthaus et al., 2009).
Image segmentation plays a crucial role in medical imaging applications by enhancing the detection of biological structures of interest. Existing medical image segmentation methods (Pham et al., 1998) are typically based on spectral clustering, normalized cuts (Scott & Longuet-Higgins, 1990; Perona & Freeman, 1998; Shi & Malik, 2000; Ng et al., 2002; Fortunato, 2010) and the mixture of Gaussian distributions (Dempster et al., 1977). FLIM is a relatively newer technique to the medical imaging community. Existing FLIM image analysis software packages (SPCImage, Becker-Hickl, Germany; SymPhoTime, PicoQuant, Germany; VistaVision, ISS Inc., Champaign, IL, U.S.A.) deliver FLIM images and corresponding FLT histograms for data acquired by FLIM systems. The VistaVision software package further provides phasor histograms of FLIM images (Stringaria et al., 2011). Users are able to manually segment pixels corresponding to distinct FLTs in FLIM images based on their locations in the FLT histograms or the phasor histograms. However, to date, there is no published method for automatically segmenting FLIM images.
The proposed MCD method performs automatic unsupervised segmentation of FLIM images for a given resolution of the network. The network is defined by image pixels as the nodes and similarity between the FLTs of the pixels as the edges. During this process, the input images are segmented starting from different initial states, and significant segments are determined in the final segmented images using information-theoretic correlations. Low network resolution leads to larger segments, and high network resolution leads to smaller segments. The outcome is a segmented image containing distinct average FLTs in each of its segments.
We compared the performance of the proposed method with a popular spectral clustering method developed by Ng et al. in segmenting two selected FLIM images. Using the MCD method, the mean-square error (MSE) in estimating the FLT segments in a FLIM image was found to consistently decrease with increasing resolution of the network. By contrast, the spectral clustering method was unable to achieve a decrease in MSE in segmenting FLIM images with increasing resolution, and this method introduced noisy segments in its output at high resolution.
The study is presented as follows. In Section ‘Fluorescence lifetime imaging microscopy’, FLIM imaging and its applications are discussed. In Section ‘Segmentation using MCD’, the proposed MCD method for FLIM image segmentation is described. In Section ‘Results’, the performance of the proposed method in segmenting FLIM images of cells transfected with FRET protein pairs is explored in a first pilot investigation on two selected images, and this performance is compared with that attained using the spectral clustering method developed by Ng et al. We conclude in Section ‘Conclusion’.
Fluorescence lifetime imaging microscopy
FLIM is typically performed in the frequency or time domains. In the frequency domain, a sinusoidally modulated (0.1–1 GHz) light source illuminates the sample, and FLTs are measured by detecting and analysing the amplitude and phase shift between the excitation light and fluorescence emission (Gadella et al., 1993). In the time domain, pulsed light illuminates the sample, and the time course of fluorescence emission is detected and analyzed for FLTs. Imaging systems use either time-gated wide field image intensifiers (Elson et al., 2002) or time-resolved laser scanning point detection (Morgan et al., 1995). FLIM images depict FLTs of fluorophore molecules in each pixel emanating from the sample microenvironment. Applications of FLIM include imaging molecular signalling (Webb et al., 2008) and trafficking (Verveer et al., 2000), imaging the spatial concentration of intracellular ions (Lahn et al., 2011), assessing the intracellular environment (Kneen et al., 1998), characterizing tissue slices in vivo (Ushakov et al., 2011) and determining molecular interactions using FRET (Keese et al., 2010). However, the literature does not describe any automatic method for segmenting FLIM images. This void leaves the users to manually select regions with distinct FLTs in the FLIM images based on FLT histograms or phasor histograms computed by existing FLIM image analysis software packages. Enabling automatic segmentation of FLIM images will beneficially eliminate the manual intervention in analysing FLIM data using these software packages.
Segmentation using MCD
Potts model Hamiltonian
To segment a FLIM image, we construct a network by using the image pixels as nodes and the absolute FLT difference between two pixels as the edge (weight) between the nodes formed by these pixels. The MCD method segments the nodes of the resulting network by minimizing a Potts model Hamiltonian,
The weight denotes the absolute FLT difference between a pixel pair formed by the ith and jth pixels in the input image with N pixels, and denotes the background of The Heaviside function ‘turns on’ or ‘turns off’ the edge designation,
The parameter γ controls the resolution of the estimated segments. With decreasing γ, the minima of Eq. (1) lead to solutions with progressively lower intracommunity edge densities, effectively ‘zooming out’ towards larger segments. The Kronecker delta is given by
In the above Hamiltonian, by virtue of the term, each spin interacts only with other spins in its own segment. The spin defines the segment identity for the ith pixel, and the algorithm optimizes it by minimizing the energy defined by Eq. (1). As such, the resulting model is local – a feature that enables high accuracy along with rapid convergence (Ronhovde & Nussinov, 2010). Thus, minimizing the Hamiltonian of Eq. (1) corresponds to identifying strongly connected segments of pixels.
Trials and replicas
We review the notions of ‘trials’ and ‘replicas’ used in our community detection algorithms. Both of these notions pertain to the use of multiple identical copies of the same system which differ from one another by a permutation of the initial site indices. Thus, whenever the time evolution depends on sequentially ordered searches for energy lowering moves (as it does in our greedy algorithm), these copies may generally reach different local solutions. By the use of an ensemble of such identical copies, accurate results are attained, information-theoretic correlations are determined (the Appendix) between the candidate solutions, and a detailed picture of the system is inferred from them.
In the definitions of ‘trials’ and ‘replicas’ given below, any given algorithm may be used to minimize the selected cost function. In our particular case, the Hamiltonian of Eq. (1) is minimized.
• Trials: We use ‘trials’ in our bare community detection algorithm. The algorithm is evaluated on the same problem T independent times. This may generally lead to different contend-ing states that minimize Eq. (1). Of these T trials, the lowest energy state is picked, and that state is used as the solution.
• Replicas: We use both ‘trials’ and ‘replicas’ in our MCD algorithm. Each sequence of the above-described T trials is termed a ‘replica’. The aforementioned T trials [and picking the solution that attains the lowest energy in the Hamiltonian of Eq. (1)] are replicated R independent times. By examining information-theoretic correlations between the R replicas, we infer which features of the contending solutions are well agreed on (and thus are likely to be correct), and on which features there is a large variance between the disparate contending solutions that may generally mark important physical boundaries. The information-theoretic correlations are computed within the ensemble of R replicas. Specifically, the information-theoretic extrema are a function of the resolution parameter, and generally correspond to more pertinent solutions that are locally stable to a continuous change of scale. In this way, the important physical scales in the system are detected.
The community detection algorithm minimizes Eq. (1) in four steps (Ronhovde & Nussinov, 2010).
The pixels are partitioned based on a symmetric or fixed K initialization.
Symmetric initialization is used for unsupervised image segmentation, where each pixel forms its own segment; i.e. initially, there are segments. Here, the algorithm does not know the number of segments, so the symmetric initialization provides the advantage of no bias towards a particular segment. The algorithm decides the number of segments K by means of the lowest energy solution. The method described here performs such unsupervised image segmentation.
Fixed K initialization is used in supervised image segmentation, where all pixels are divided into K segments using a random initial distribution. The community membership of an individual pixel is then changed to lower the solution energy, using the CD algorithm. Here the user decides the number of initial segments K based on the desired information. For instance, if only one target needs to be identified, is enough to describe the target and background.
Each pixel is then placed in the segment that best lowers the energy of Eq. (1) based on the current state of the system.
This process is repeated for all pixels. The iteration is continued until no energy lowering moves are found after one full cycle through all pixels.
The above three steps are repeated for T trials, and the lowest energy is selected as the best solution. Different trials differ solely by the permuted pixel order of the initial state.
Multiresolution community detection
We illustrate below how the MCD algorithm (Ronhovde & Nussinov, 2009) works.
To begin applying the MCD algorithm, users first specify the number of replicas R at each resolution γ, the number of trials T per replica and the starting and ending resolutions, γ0 and respectively. See Section ‘Trials and replicas’ for the definitions on ‘trial’ and ‘replica’. We typically use and . In the case of a symmetric initialized state of one pixel per community, the initial states of the replicas are generated by permuting the pixel labels. These permutations P simply reorder the pixel indices (with the state of i under a permutation), and thus lead to a different initial state.
The algorithm starts from the initialization of the system, as described in item (1) of Section ‘Community detection’.
Equation (1) is then minimized independently for all replicas at a resolution as described in Section ‘Community detection’.
For the entire range of the resolutions studied, the algorithm then calculates the average inter-replica information-theoretic measures, such as and V, at each value of resolution γ. These measures determine the most pertinent scales of the systems in the following manner. Extrema in the average inter-replica information-theoretic overlaps (Ronhovde & Nussinov, 2009) as a function of γ occur at a particular set of the values of γ (which we denote here by ). Community detection done at any one of these extremal values leads to an image segmentation (Hu et al., 2012b), which is robust and locally insensitive to a change of resolution (i.e. to small variations in γ about ). These locally stable segmentations generally highlight prominent features of the image on all natural pertinent scales. Different levels of detail and resolutions can be determined by allowing the resolution parameter γ to assume different values when the Hamiltonian of Eq. (1) is minimized following the CD method of Section ‘Community Detection’. Higher values of lead to finer segmentations on smaller scales; coarser larger range features appear in the corresponding segmentation when smaller values of are inserted.
This section describes selected examples of the proposed MCD method applied to FLIM images of cells transfected with FRET protein pairs (Orthaus et al., 2009) in a first pilot investigation on two images. A performance comparison between the MCD and a popular spectral clustering method developed by Ng et al. in segmenting FLIM images is also discussed.
To explore the performance of the proposed MCD method, we employed two FLIM images of live 12V HC Red cells expressing a protein fused to enhanced green fluorescence protein (EGFP) (donor) and red fluorescence protein (RFP) (acceptor) separated by a short linker. The images were acquired using an Olympus FV1000 laser scanning microscope equipped with the PicoQuant LSM Upgrade Kit for FLIM. Such a donor–acceptor fusion serves as a positive control for FRET. To capture images from cells transfected with EGFP–RFP fusion protein pairs, the samples received pulsed excitation at 470 nm with 40 MHz repetition. Photons were detected by a single-channel SPAD (PicoQuant, Germany) set-up. A fluorescence bandpass filter (500–540 nm) limited the detection to the donor (EGFP) fluorescence only.
The FLIM image in Figure 1(A) shows two cells with different average donor FLTs: a FRET cell and a cell where the acceptor RFP was irreversibly bleached, leading to an FLT shift from approximately 2.1–2.4 ns. In Figure 1(A), by carefully adjusting the colourmap of the image, this shift of the average FLT is clearly distinguished by yellow and red, respectively. In the FRET cell (shown in yellow), half of the EGFP–RFP fusion proteins could adopt a proper conformation because complete maturation allowed for FRET (Orthaus et al., 2009). For the acceptor-bleached cell (shown in red), the situation was different. Only 15% of the EGFP molecules were quenched by energy transfer to some remaining acceptor molecules, whereas the majority of donor molecules (85%) could not undergo FRET anymore because an appropriate acceptor molecule was missing. In the similar image in Figure 1(B), an average donor FLT of 2.2 ns (yellow) was obtained in the cell corresponding to the quenched EGFP, and an average donor FLT of 2.9 ns (red) was found in the other cell, where the acceptor molecules were irreversibly destroyed.
MCD for varying resolution
For the FLIM images shown in Figure 1, we define the edge weight between two pixels as the absolute FLT difference between them. The MCD was applied to segment the resulting networks formed by the image pixels as nodes. Figures 2(G) and 3(G) show plots of the respective information-theoretic overlaps between the replicas of the MCD for the FLIM images in Figures 1(A) and (B), such as their normalized mutual information and variation of information V, together with the respective number of estimated segments as a function of the resolution parameter γ. With decreasing γ, the minima of Eq. (1) lead to solutions with progressively lower intrasegment edge densities, effectively ‘zooming out’ towards larger segments. Natural network resolutions correspond to the values of γ for which the replicas exhibit extrema and plateau in the average of their information-theoretic overlaps when expressed as a function of γ (Ronhovde & Nussinov, 2009). Recall that the independent solutions of the MCD method attained from different starting points were defined as replicas in Section ‘Multiresolution community detection’.
Figures 2A–F and 3A–F show the results of automatic image segmentation using our MCD algorithm at different resolutions for the two FLIM images shown in Figure 1. The segments are depicted using false colours. As the resolution increases from Figures 2(A) to (F) and from Figure 3(A) to (F), the images show more detailed segments. In Figures 2(D)–(F) and 3(D)–(F), two major segments, one representing the respective FRET cell and the other representing the respective unquenched cell, are clearly visible for , in addition to the respective background. Thus, by using different resolutions γ, the MCD method was able to detect the segments at different scales. To generate the segmented images, we used eight replicas and one trial. Parallel implementation of the automatic segmentation in different resolutions will allow users to obtain segments without having any human intervention in the adjustment of the image colourmap.
Spectral clustering for varying resolution
To compare the performance of the proposed MCD method in automatically segmenting FLIM images, we used a popular spectral clustering method developed by Ng et al. In brief, we first constructed a network with image pixels as nodes and took the edge weight between two nodes as the squared distance between the FLTs of the corresponding pixels smoothed by a Gaussian kernel. Then we normalized the affinity matrix formed by the resulting edge weights, and performed eigendecomposition of the resulting matrix. Eigenvectors corresponding to the eigenvalues were chosen, these eigenvectors were normalized again, and then they were segmented along the rows to segment the network. As in the MCD method, α determines the resolution of the segmented images. Increasing α confines us to eigenvectors of smaller variation across the pixels, and decreasing α allows us to include eigenvectors of larger variation across the pixels. Consequently, increasing α effectively ‘zooms out’ towards larger segments.
Figures 4(A)–(F) and (G–L) show the performance of the spectral clustering method in segmenting the FLIM images shown in Figure 1 for decreasing α. For Figure 1(A), the segmented images shown in Figures 4(A)–(F) using false colours depict increasing noise at high resolution, and none of these segmented images clearly depicts the two major expected segments. For the FLIM image shown in Figure 1(B), using false colours, the segmented images show the two respective cells as one segment in Figures 4(G) and (H) in low resolution and as two expected segments in Figures 4(I) and (J) in a slightly higher resolution. At the resolution limit, similar to the case of Figure 1(A), the segmented images for Figure 1(B) exhibit a high amount of introduced noise. In comparison to the proposed MCD method, spectral clustering at high resolution was thus not able to provide the expected segments, and moreover introduced a high amount of noise in the segmented images.
Performance comparison between MCD and spectral clustering
To quantitatively compare the MCD and spectral clustering methods, we evaluated MSEs in segmenting the major segments of the FLIM images shown in Figure 1. To perform a fair comparison, the following procedure was used. The ground-truth FLTs of the cells in each FLIM image shown in Figures 1(A) and (B) were computed to be the average FLTs in the yellow and red coloured regions. The segmented images using the MCD method in Figures 2(A)–(C) and 3(A)–(C) and those using the spectral clustering method in Figures 4(A)–(D), (F), (G) and (H) depict the two respective expected major segments as one segment. Consequently, the average FLT of this single segment was used as the estimated FLT for both respective cells in each segmented image. The segmented images using the MCD method (in Figs. 2D–F and 3D–F) and those using the spectral clustering method (in Figs. 4E and I–L) depict the two respective expected major segments. Consequently, the average FLTs in these segments were used as the estimated FLTs for the two cells in each segmented image. The MSE in segmenting the FLIM images shown in Figure 1 at each resolution was the squared distance between the estimated FLTs and the ground-truth FLTs of the respective cells.
Using the MCD method, the MSE in the estimation of average FLTs of the correct segments consistently decreases with increasing resolution, as seen in Figure 5. As seen in Figures 5 and 6, the MCD method yielded a lower MSE than the spectral clustering method in all its network resolutions for the FLIM image shown in Figure 1(A), and in its high network resolution region () for the FLIM image shown in Figure 1(B). The MSE in estimating average FLTs of the correct segments using the spectral clustering method does not consistently decrease with increasing resolution for the FLIM image shown in Figure 1(A). For Figure 1(B), the MSE in estimating average FLTs of the correct segments using the spectral clustering method shows first a decrease with increasing resolution. Recall that the resolution of the images segmented by the spectral clustering method is increased by decreasing α. Here, for Figure 1(B), the decrease in MSE in estimating the correct segments is apparent for Decreasing α below 0.0625 introduces noisy segments in the output, and the MSE in estimating average FLTs of the correct segments becomes very high. Consequently, for clarity, such MSEs at limiting resolutions are not shown in Figure 6(B). In summary, in terms of MSE, the proposed MCD method appears to perform better than the spectral clustering method in automatically segmenting the FLIM images shown in Figure 1.
We suggest an MCD algorithm to automatically segment FLIM data. The proposed method is able to identify segments at different scales in the input FLIM images. In this study, it appears to perform better than a popular graph-based spectral clustering method developed by Ng et al. in segmenting two selected FLIM images. The MCD method was able to provide correct segments for FLIM images of cells transfected with EGFP–RFP fusion FRET protein pairs in our first pilot investigation on two selected images. The spectral clustering method was unable to provide such correct segments and introduced a high amount of noise in the segmented images, particularly, at high resolution. The MCD method offers lower MSEs in segmenting the FLIM images than that obtained using the spectral clustering method.
The MCD method for automatically segmenting FLIM images will avoid any manual selection of regions with distinct FLTs in the FLIM images based on FLT histograms or phasor histograms computed by existing FLIM image analysis software packages. Automatic segmentation of FLIM images will thus minimize error in analyzing FLIM data using such software.
Our study based on two images does not offer enough statistical power to conclude MCD method as a superior method in segmenting FLIM images than other image segmentation methods. Our study can however be considered as a first and promising pilot investigation in understanding how MCD performs in segmenting FLIM images in comparison with other graph-based image segmentation methods. Our future work will use a large population of FLIM images of diverse types to establish a statistical significance in this context. On another note, the success of our bare MCD graph theory–based method naturally suggests the possibility of yet more potent approaches which build on it. We will briefly propose and speculate on a possible extension involving the use of known prior information, and hope to explore this possibility in future work. We may apply expectation maximization (Dempster et al., 1977) to a given image via a library of finite number of images of known tissue types to infer probabilities that different parts of the image will be locally similar to any of the previously known types. We may then use the similarity of these local probabilities to define weights in the graph and employ MCD. In this approach, the MCD will not invoke bare weights resulting from only local intensity strengths in an image (as we have in the current work). Rather, the MCD will use edge weights given by these probabilities (inferred via expectation maximization).
This work is supported by the National Science Foundation (NSF) under grant numbers NSF DMR-1106293, NSF 1066293 (Aspen Center for Physics) and by the National Institute of Health (NIH) under grant numbers NIH R01 EB008111, NIH R01 EB007276, NIH R33 CA123537, NIH U54 CA136398 and NIH HHSN268207000046C. We wish to acknowledge a discussion on expectation maximization with Prof. Veit Elser from Cornell University. The images shown in Figure 1 were taken from the application note written by Orthaus et al., with kind permission from PicoQuant. We appreciate Prof. James Ballard's (Director, Engineering Communication Center, Washington University in St. Louis) close reading of the manuscript.
We use information-theoretic measures to calculate correlations between community detection solutions. The community detection method partitions N pixels for a replica r into segments, where segment k () consists of pixels. The ratio is the probability that a randomly selected pixel is found in the segment k ().
The Shannon entropy (Hu et al., 2012) is
The mutual information between the replicas r and s () is
where is the number of common pixels in the segment k1 () of replica r () and the segment k2 () of replica s ().
The variation of information between the two segments r and s is
which has a range of .
The normalized mutual information is
with the obvious range of .
Higher and lower values indicate better agreement between the compared segments.