- Top of page
- Materials and Methods
- Summary and Discussion
- Literature Cited
We describe a new supervised learning-based template matching approach for segmenting cell nuclei from microscopy images. The method uses examples selected by a user for building a statistical model that captures the texture and shape variations of the nuclear structures from a given dataset to be segmented. Segmentation of subsequent, unlabeled, images is then performed by finding the model instance that best matches (in the normalized cross correlation sense) local neighborhood in the input image. We demonstrate the application of our method to segmenting nuclei from a variety of imaging modalities, and quantitatively compare our results to several other methods. Quantitative results using both simulated and real image data show that, while certain methods may work well for certain imaging modalities, our software is able to obtain high accuracy across several imaging modalities studied. Results also demonstrate that, relative to several existing methods, the template-based method we propose presents increased robustness in the sense of better handling variations in illumination, variations in texture from different imaging modalities, providing more smooth and accurate segmentation borders, as well as handling better cluttered nuclei. © 2013 International Society for Advancement of Cytometry
Segmenting cell nuclei from microscopy images is an important image processing task necessary for many scientific and clinical applications due to the fundamentally important role of nuclei in cellular processes and diseases. Given a large variety of imaging modalities, staining procedures, experimental conditions, and so forth, many computational methods have been developed and applied to cell nuclei segmentation in 2D (1–9) and 3D images (10–14). Thresholding techniques (15, 16), followed by standard morphological operations, are among the simplest and most computationally efficient strategies. These techniques, however, are inadequate when the data contain strong intensity variations, noise, or when nuclei appear crowded in the field of view (7, 17) being imaged. The watershed method is able to segment touching or overlapping nuclei. Direct use of watershed algorithms, however, can often lead to over segmentation artifacts (6, 18). Seeded or marker controlled watershed methods (2, 3, 5–7, 19, 20) can be utilized to overcome such limitations. We note that seed extraction is a decisive factor in the performance of seeded watershed algorithms. Missing or artificially added seeds can cause under or over segmentation. Different algorithms for extracting seeds have been proposed. In Ref.20, for example, seeds are extracted using a gradient vector field followed by Gaussian filtering. Jung and Kim (8) proposed to find optimal seeds by minimizing the residual between the segmented region boundaries and the fitted model. In addition, various of postprocessing algorithms have been applied to improve the segmentation quality. For example, morphological algorithms (e.g., dilation and erosion) (7) can be used iteratively to overcome inaccuracies in segmentation. In Ref.21, learning-based algorithms were used for discarding segmented regions deemed to be erroneous. Similar ideas using neural networks can be seen in Ref.22.
When nuclei do not appear clearly in the images to be segmented (e.g., nuclear borders are not sharp enough or when a significant amount of noise is present), active contour-based methods (9, 11, 12, 23–27), especially those implicitly represented by level set (11, 12, 27), have been proposed to overcome some of these limitations successfully. As well known, the level set framework is well suited for accurate delineation of complicated borders and can be easily extended to higher-dimensional datasets. Ortiz De Solorzano et al. (11), for example, proposed an edge-based deformable model that utilizes gradient information to capture nuclear surfaces. Considering that strong gradients at object boundaries may be blurred and the noise and intracellular structures may also show strong gradients, Mukherjee et al. (28) proposed a level set model that also incorporates a region term using the likelihood information for segmentation of leukocytes with homogeneous regions. In segmenting cells in culture or in tissue sections, Dufour et al. (12) proposed a multilevel deformable model incorporating both a gradient term and a region term, adopted from Chan and Vese model (29), to segment cells with ill-defined edges. In Ref.30, Yan et al. also proposed a similar multilevel deformable model to segment RNAi fluorescence cellular images of Drosophila. In Ref.27, Cheng and Rajapakse utilized the Chan and Vese model (29) to obtain the outer contours of clustered nuclei, using a watershed-like algorithm to separate clustered nuclei. Similarly, Nielsen et al. (9) have described a method for segmenting Feulgen stained nuclei using a seeded watershed method, combined with a gradient vector flow-based deformable model method (31). Considering that some nuclei may appear to overlap in 2D images, Plissiti and Nikou (32) proposed a deformable model driven by physical principles, helping to delineate the borders of overlapping nuclei. In Ref.33, Dzyubachyk et al. proposed a modified region-based level set model, which addresses a number of shortcomings in Ref.12 as well as speeds up computation. To reduce the large computational costs of variational deformable models, Dufour et al. (34) proposed a novel implementation of the piece-wise constant Mumford–Shah functional using 3D active meshes for 3D cell segmentation.
Besides the methods mentioned earlier, several other approaches for segmenting nuclei based on filter design (35, 36), multiscale analysis (37), dynamic programming (38), Markov random fields (39), graph-based methods (40–42), and learning-based strategies (43–47) have been described. As new imaging modalities, staining techniques, and so forth are developed, however, many existing methods specifically designed for current imaging modalities may not work well. Later, we show that the application of some such methods can fail to detect adequate borders, or separate touching or overlapping nuclei, in several staining techniques. Therefore, considerable resources have to be spent to modify existing methods (or developing entirely new segmentation methods) to better suit the new applications.
Here, we describe a generic nuclear segmentation method based on the combination of template matching and supervised learning ideas. Our goal is to provide a method that can be used effectively for segmenting nuclei for many different types of cells imaged under a variety of staining or fluorescence techniques. We aim to guarantee robust performance by allowing the method to “calibrate” itself automatically using training data, so that it will adapt itself to segmenting nuclei with different appearances (due to the staining techniques for example) and shapes. The method is also “constrained” to produce smooth borders. Finally, given that the objective function used in the segmentation process is the normalized cross correlation (NCC), the method is also able to better handle variations in illumination within the same image, as well as across images. We note that template matching-based methods for image segmentation have long been used for segmenting biomedical images. One prominent example is the brain segmentation tool often used in the analysis of functional images (48). When segmenting nuclei from microscopy images, contour templates have also been used (43, 44). Here, we utilize similar ideas with some adaptations. First, our approach is semiautomated in that it first seeks to learn a template and statistical model from images delineated by the user. The model is built based on estimating a “mean” template, as well as the deformations from the template to all other nuclei provided in the training step. After this step, any image of the same modality can then be segmented via a template-based approach based on maximization of the NCC between the template estimated from the input images and the image to be segmented. We describe the method in detail in the next section and compare it to several other methods applied on different datasets in “Results” section. Finally, we note that our method is implemented in MATLAB computer language (49). The necessary files can be obtained through contact with the corresponding author (G.K.R.).
Materials and Methods
- Top of page
- Materials and Methods
- Summary and Discussion
- Literature Cited
Given the large variation in appearance of nuclei in microscopy images, a completely automated (unsupervised) approach for segmenting nuclei from arbitrary images may be difficult to obtain. We, therefore, focus on a semiautomated approach, depicted in Figure 1, where the idea is to first construct a statistical model for the mean texture and most likely variations of shape to be found in the dataset to be segmented from hand-delineated images. Segmentation of any image of similar type is then achieved by maximizing the NCC between the model and the local image region. Part A outlines the training procedure whereby the user utilizes a simple graphical user interface to isolate several nuclei samples, which are then used to build the statistical model. Part B outlines the actual segmentation procedure, which proceeds to first find an approximate segmentation (seed detection) of an input image by matching the statistical model with the given image and then produces a final segmentation result via nonrigid registration.
Figure 1. Overview of nuclear segmentation approach. Part A outlines the training procedure, which utilizes sample nuclei manually identified by the user to build a statistical model for the texture and shape variations that could be present in the set of nuclei to be segmented. The model is then sampled to form a detection filter-bank. Part B outlines the actual segmentation procedure which utilizes the detection filter-bank to produce a rough segmentation, and then refines it using nonrigid registration based on the NCC.
Download figure to PowerPoint
As outlined in part A of Figure 1, we utilize a simple graphical user interface to enable an operator to manually delineate rectangular subwindows each containing one nucleus sample from an image of the modality he or she wishes to segment. It is required by our system that each subwindow contains only one nucleus and recommended that the set of subwindows contains a variety of shapes and textures (small, large, bent, irregular shaped, hollow, etc.), because more variations present in the input images will translate into more variations being captured by the model. We note that it is not necessary for the user to provide the detailed outline for the nucleus present in each window. Rather, a rectangular bounding box suffices. In our implementation, given N such rectangular subwindows, which can be of different sizes, each subwindow containing one nucleus from the training set, we first pad each subwindow image by replicating the border elements so as to render each subwindow of the same size (in terms of number of pixels in each dimension). The amount of padding applied to each subwindow is the amount necessary for that subwindow to match the size of the largest rectangular subwindow in the set. The set of subwindows is then rigidly aligned to one subwindow image from the set (picked at random) via a procedure described in earlier work (50). As a result, the major axis of nuclei samples is aligned to the same orientation. In this case, we choose the NCC as the optimization criterion for measuring how well two nuclei align and include coordinate inversions (image flips) in the optimization procedure.
The set of N rigidly aligned subwindows, denoted as from now on, is then used to estimate a template that will represent an “average” shape as well as texture for this set. Several procedures can be used for this purpose. In this work, we choose the procedure outlined in Heitz et al. (51) where the idea is to iteratively deform all nuclear images (subwindows) toward a template image that is closest (in the sense of least deformation) to all other images in the set. Figure 2 contains a diagram depicting the procedure we use. The procedure depends on the computation of a nonrigid map that aligns two images I i, and I j, via , with x an input coordinate in the image grid Ω, and a nonrigid mapping function . In our approach, the nonrigid registration is computed via maximization of the NCC cost function, which is described in detail in the Appendix. Given the ability to nonrigidly align two nuclear images, the template estimation procedure consists of choosing a subwindow image from the set at random and denoting it . Then, starting with the iteration k = 1:
to each subwindow image
Calculate a temporary average shape template
the inverse of the transformation function f
(which we compute with Matlab's “griddata” function).
Compute the average texture on the same average shape template above by first registering each subwindow image in the set to
) and update the template via
(sum of squared errors). If
stop, otherwise set
and go to step 1.
In our approach, the statistical model is evaluated for in intervals of . The result of this operation is a set of images obtained by deforming the mean template and representing nuclear configurations likely to be encountered in data to be segmented. In addition, this set of images is augmented by including rotations (rotated every 30°, totaling seven orientations in our implementation) as well as variations in size (two in our implementation). Finally, we discard the top 1% and bottom 1% (in the sense of area) of the templates to avoid potentially segmenting structures that would be too small or too large to be considered as nuclei. A data-dependent way of choosing this threshold is also described in the “Discussion” section. The reason being that templates that are too small may cause over segmentation, whereas templates that are too large may merge nuclei that are close to each other. Figure 1 (top right) contains a few examples of template images generated in this way for a sample dataset. We denote the set of template images generated in this way as the “detection filterbank” to be used as a starting point for the segmentation method described in the next subsection.
Segmenting the mean template
Our procedure depends on the mean template, computed as estimated earlier, on being segmented in the sense that pixels in the foreground and background are known. Although many automated methods can be considered for this step, we choose to utilize a rough contour manually provided by an user. The contour is then refined utilizing a level set approach (53). The advantage is that such a process can be repeated until a satisfactory segmentation result has been made by the user. Figure 3 shows the outline of the procedure.
Figure 3. The mean template image must be segmented before the segmentation algorithm based on NCC maximization can be utilized. We utilize a semiautomated approach wherein an user draws an initial contour, and a level-set-based algorithm refines it to accurately match its borders.
Download figure to PowerPoint
The detection map is mined for potential locations of nuclei using the following two principles: (1) only pixels whose intensities in M are greater than a threshold μ are of interest. (2) The centers of detected nuclei must be at least a certain distance far away from each other. This helps to prevent, for example, two potential locations from being detected within one nucleus, causing over segmentation. These two principles can be implemented by first searching for the highest response in M. Subsequent detections must be at least a certain distance from the first. This is done by dilating the already detected nuclei (remember, the filtering step above not only defines regions where nuclei might be located but also the rough shape of each). This process is able to detect nuclei of different shapes due to simulated templates of various shapes and orientations generated in the previous step, and it is repeated until all pixels in the thresholded detection image M have been investigated. We note again that each detected pixel in M has its associated best matching template from the detection filterbank. Therefore, this part of the algorithm provides not only the location of a nucleus but also a rough guess for its shape (see bottom middle of Figure 1) and texture.
Once an initial estimate for each nucleus in an input image is found via the procedure described earlier, the algorithm produces a spatially accurate segmentation by nonrigidly registering each approximate guess to the input image. The nonrigid registration nonlinearly adapts the borders of the detected template so as to accurately segment the borders of each nuclei in the input image. In addition, the nonrigid registration approach we use also is constrained to produce smooth borders. Details related to the nonrigid registration are provided in the Appendix. Rather than optimizing all guesses at once, which could lead to difficulties such as a large number of iterations in our gradient ascent-type strategy, each nucleus is segmented separately.
Segmenting touching nuclei
An important feature of our template matching approach is that it is capable of segmenting touching nuclei without difficulties with a small modification of the procedure described earlier. In our method, if two (or more) nuclei are detected to be close to each other (e.g., the closest distance between their best matching templates' borders is smaller than 10 pixels), these nuclei are regarded as being potentially in close proximity to each other. If so, their best matching templates obtained from the filter bank procedure above are taken together under a subwindow and then nonrigidly registered to the same subwindow in the real image using the same optimization algorithm in the Appendix. An example showing the segmentation of two nuclei in close proximity to each other is shown in Figure 1 (bottom row). The left part of this portion of the figure shows the initial estimates from the filterbank-based estimation of candidate locations. The result of the nonrigid registration-based estimation of the contours for each nucleus is shown at the bottom right corner of the same figure. The black contours indicate the borders of the best matching templates (the initial guesses), and the white lines delineate the final segmentation result after nonrigid registration.
We demonstrate our system applied to several different cell nuclei datasets:1) a synthetic dataset BBBC004v1 generated with the SIMCEP simulating platform for fluorescent cell population images (54, 55);2) two real cell datasets (U2OS cells and NIH3T3 cells) acquired with fluorescence imaging (56);3) and a histopathology dataset obtained using thyroid tissue specimens with several different staining techniques. The primary goal for the simulated dataset is to obtain an accurate count for the number of nuclei in each field of view. Each simulated image contains 300 objects with different degrees of overlap probability (ranging from 0.00 to 0.60). The U2OS (48 images, each containing multiple nuclei) and NIH3T3 (49 images) cells were obtained with the Hoechst 33342 fluorescence signal, and the ground truth (including accurately delineated borders) is provided by experts (56). Of these, the U2OS dataset is more challenging, with nuclei tending to be more varied in shape and more clustered together. The intensity of the NIH3T3 images, however, is more nonuniform than the U2OS dataset. In addition, we apply our method to segmenting nuclei from histopathology images taken from tissue sections of thyroid specimens. Tissue blocks were obtained from the archives of the University of Pittsburgh Medical Center (Institutional Review Board approval #PRO09020278). Briefly, tissue sections were cut at 5- thickness from the paraffin-embedded blocks and stained using three techniques. These include the Feulgen stain that stains deoxyribonucleic acids only. If no counterstaining is performed, then only nuclei are visible demonstrating chromatin patterns as deep magenta hues shown in Figure 6a. The second is a silver-based technique that stains the intranuclear nucleolar organizing regions (NORs) (black intranuclear dots) and counterstained with nuclear fast red that uses kernechtrot that dyes nuclear chromatin red (Figure 6b). The third is the same silver-based staining for NORs without counterstaining (Figure 6c). All images used for analysis in this study were acquired using an Olympus BX51 microscope equipped with a 100X UIS2 objective (Olympus America, Central Valley, PA) and 2 mega pixel SPOT Insight camera (Diagnostic Instruments, Sterling Heights, MI). Image specifications were 24 bit RGB channels and 0.074 μm/pixel, field of view. More details pertaining to the image acquisition process for this dataset are available in Ref.57.
We note that our system is able to work with grayscale (single color) images as well as with color images. Equation (A1), in the Appendix, allows color images to be used, while the method can also be used to segment 3D data by defining the inter products and convolutions utilized in Eqs. (A1) and (A2) in three dimensions. In addition, we mention that for color images, each color channel (R, G, and B) is equally weighted in the approach we described earlier. This allows for segmentation even in the case when the optimal color transformation for detecting nuclei is not known precisely (as is the case in many of the images shown). In cases where this information is known precisely, the approach we proposed can be used with only the color channel that targets nuclei, or with the image after optimal color transformation. In each experiment, k sample nuclei (k is arbitrarily chosen as 20 in our experiments) were chosen by the authors for the training process. All but one of the parameters remained constant for all experiments. The percent of variance retained in PCA analysis was set to (95%), for the calculation of the average template, the step size in the gradient ascent procedure κ was set as 5×104, the scale number in the multi-scale strategy s was set as 2. For smoothing parameter σ in the gradient ascent procedure, a higher σ value helps to smooth the contour, while a lower σ value helps to better capture the real border of nuclei. In this paper, σ was experimentally set as 1.5 (pixels). The only parameter that varied from dataset to dataset was the detection threshold μ. Whereas a higher value of μ may miss some nuclei (e.g., out of focus), a lower value of μ may confuse noise and clutter for actual nuclear candidates. There are two ways to determine an appropriate value for detection threshold μ. When the ground truth (e.g., manual delineation of nuclei) for the training images is provided, μ value can be selected automatically by maximizing the dice metric (58) between the detections and provided ground truth. Here, counts the number of nuclei in different results, GT corresponds to the ground truth result, while corresponds to the nuclei detection result with respect to μ. When ground truth is not available, an appropriate μ value has to be empirically selected by the user to detect most nuclei in the training images for each application or dataset. In the experiments shown later, ground truth was not used for selecting μ. Rather, μ was empirically determined for each dataset based on empirical experimentation with a given field of view (containing multiple nuclei) from the corresponding dataset.
For comparison, we choose several different types of algorithms commonly used for cell nuclei segmentation. These include the level set method [Chan and Vese model (29)], an unsupervised learning method [color K-means (59)], and the direct seeded watershed method, which uses a shape-based method to separate clumped nuclei [implemented by CellProfiler (60)]. As the CellProfiler software (60) is only able to process 2D grayscale images, a typical choice is to convert the color histopathology image to grayscale image by forming a weighted sum of R, G, and B channels, which keeps the luminance channel ( ) (61). In addition, we prefer to take the general approach of normalizing all image data to fit the intensity range of [0, 1] by scaling the minimum and maximum of each image (discounting outliers set at 1% in our implementation). As the level set method and the K-means method may not be able to separate clumped nuclei very well, a common solution is to apply seeded watershed algorithm on the binary masks segmented from level set method and K-means method, in which seeds are defined as the local maxima in the distance transformed images of binary masks (62). Note that H-dome maxima (62) are calculated on the distance transformed images to prevent over-segmentation, and for different datasets, the H value is arbitrarily selected for the best performance. These techniques were chosen, as they are similar to several of the methods described in the literature for segmenting nuclei from microscopy images (12, 33). In the following sections, we will show both qualitative and quantitative comparisons of these methods.
Summary and Discussion
- Top of page
- Materials and Methods
- Summary and Discussion
- Literature Cited
We described a method for segmenting cell nuclei from several different modalities of images based on supervised learning and template matching. The method is suitable for a variety of imaging experiments given that it contains a training step that adapts the statistical model for the given type of data. In its simplest form, the method consists of building a statistical model for the texture and shape variations of the nuclei from the input of a user, and then segmenting arbitrary images by finding the instance in the model that best matches, in the sense of the NCC, local regions in the input images. We note that given an experimental setup, once the training operation is completed, the method is able to segment automatically any number of images from the same modality. We have demonstrated the application of the method to several types of images, and results showed that the method can achieve comparable, and often times better, performance compared with the existing specifically designed algorithms. Our main motivation was to design a method for segmenting nuclei from microscopy images of arbitrary types (scalar, color, fluorescence, different staining, etc.). To our knowledge, ours is the first method to apply a template matching approach that includes texture and shape variations to accurately delineating nuclei from microscopy images. In addition, to our knowledge, ours is the first method to utilize a supervised learning strategy to build such a statistical model, which includes texture and shape variations in multiple channels, for detecting nuclei from microscopy images.
In a practical sense, our method provides three main contributions. First, its overall performance is robust across different types of data with little tuning of parameters. We have demonstrated this here by applying the same exact software (with the only difference in each test being the value for μ) to a total of six different imaging modalities and showing the method performs as well or better than all other methods we were able to compare against. The performance was compared quantitatively and qualitatively, using both real and simulated data. Comparison results with a total of six alternative segmentation methods are shown here. Other, simpler, segmentation methods were also used for comparison, including several thresholding schemes followed by morphological operations. The results of these were not comparable to many of the methods shown here. Therefore, we have omitted them for brevity. Second, among the methods we have tested in this manuscript, we show that our method is the only method (besides manual segmentation) that is capable of handling significant intensity inhomogeneities. This is due to the fact that we utilize the NCC metric in the registration-based segmentation process. The NCC metric is independent of the overall intensity of the local region of the image being segmented. Finally, we also mention that, among all methods tried, the template matching method we described produced noticeably more smooth and accurate borders with fewer spurious contours. This can be seen, for example, by close observation of Figure 6. The smoothness in the contours obtained by our method is primarily due to the fact that the statistical modeling we use includes only the main modes of variation in nuclear shape. These tend to be, topically, size, elongation, as well as bending (in addition to rotation). High-order fluctuation in contours does occur in nuclei at times, but these do not occur as often as the ones already mentioned. We note that the method is still flexible enough to accurately segment nuclei that do not conform to these main modes of variation given the elastic matching procedure applied in the last step of the procedure.
We also note that our algorithm has several parameters including the percent of variance in PCA analysis, ε in the calculation of “average” template, in the nonrigid registration procedure, and μ in the approximate segmentation procedure. The algorithm is not unduly sensitive to these, as the same fixed parameters were utilized in all six experiments (datasets) used in this article. The only parameter that was selected differently for each dataset was the detection threshold μ. When ground truth is available, we described a method to automatically choose the optimal μ for the given dataset. In addition, in our current implementation, we discard the top and bottom 1% (in size) of the generated templates, in an effort to reduce outlier detections. This percentage too could be made dataset dependent through a cross validation procedure, when the precise ground truth is present.
Finally, it is important to describe the computational cost of our template matching approach, which is also important in evaluating the performance of an algorithm. Our approach consists of a training stage and a testing stage and is implemented in MATLAB 64 bit mode and tested on a PC laptop (CPU: Intel core i5 2.30 GHz, memory: 8 GB). The computational time for training a statistical model (560 simulated templates) from 20 nuclei samples (window size: ), for example, is about 1.6 h. Detecting and segmenting all cell nuclei (36 nuclei) from a fluorescence image () take about 20 min (about half a minute per nucleus). We note however, that the computational time can often be significantly reduced by implementing the algorithm in a compiled language such as C, for example. In addition, we note that the computational time should be considered in context to alternative segmentation methods capable of producing results (albeit not as accurate) on similar datasets. The level set algorithm by Chan and Vese, which is used in a variety of other nuclear segmentation methods, takes even longer to compute on the same image (23 min) in our implementation (also in MATLAB). Finally, we note that the computational time of our algorithm can be decreased by utilizing a multiscale framework. That is, instead of performing the filtering-based approach for detection in the original image space, we have also experimented with first reducing the size of the image (and templates) by two for the initial detection only (the remaining part of the method utilized the full resolution image). Thus, we were able to reduce the total computation time for the same field of view to roughly 10.6 min. The accuracy of the final segmentation was not severely affected (data not shown for brevity). Future work will include improving the computational efficiency of this method by further investigation of multiscale approaches, as well as faster optimization methods (e.g., conjugate gradient). Finally, we note again that the approach described above utilizes all color information contained in the training and test image. In cases where the nuclear stain color is known precisely, the approach can be easily modified to utilize only that color. In addition, many existing techniques for optimal color transformation (64) can also be combined with our proposed approach in the future for better performance.