Automated quantification of DNA demethylation effects in cells via 3D mapping of nuclear signatures and population homogeneity assessment


  • Arkadiusz Gertych,

    Corresponding author
    1. Translational Cytomics Group, Minimally Invasive Surgical Technologies Institute, Department of Surgery, Cedars-Sinai Medical Center, 8700 Beverly Boulevard, Los Angeles, California 90048
    • Translational Cytomics Group, Minimally Invasive Surgical Technologies Institute, Department of Surgery, Cedars-Sinai Medical Center, 8700 Beverly Boulevard, Los Angeles, CA 90048
    Search for more papers by this author
  • Kolja A. Wawrowsky,

    1. Translational Cytomics Group, Minimally Invasive Surgical Technologies Institute, Department of Surgery, Cedars-Sinai Medical Center, 8700 Beverly Boulevard, Los Angeles, California 90048
    Search for more papers by this author
  • Erik Lindsley,

    1. Translational Cytomics Group, Minimally Invasive Surgical Technologies Institute, Department of Surgery, Cedars-Sinai Medical Center, 8700 Beverly Boulevard, Los Angeles, California 90048
    Search for more papers by this author
  • Eugene Vishnevsky,

    1. Translational Cytomics Group, Minimally Invasive Surgical Technologies Institute, Department of Surgery, Cedars-Sinai Medical Center, 8700 Beverly Boulevard, Los Angeles, California 90048
    Search for more papers by this author
  • Daniel L. Farkas,

    1. Translational Cytomics Group, Minimally Invasive Surgical Technologies Institute, Department of Surgery, Cedars-Sinai Medical Center, 8700 Beverly Boulevard, Los Angeles, California 90048
    Search for more papers by this author
  • Jian Tajbakhsh

    Corresponding author
    1. Translational Cytomics Group, Minimally Invasive Surgical Technologies Institute, Department of Surgery, Cedars-Sinai Medical Center, 8700 Beverly Boulevard, Los Angeles, California 90048
    • Translational Cytomics Group, Minimally Invasive Surgical Technologies Institute, Department of Surgery, Cedars-Sinai Medical Center, 8700 Beverly Boulevard, Los Angeles, CA 90048
    Search for more papers by this author


Today's advanced microscopic imaging applies to the preclinical stages of drug discovery that employ high-throughput and high-content three-dimensional (3D) analysis of cells to more efficiently screen candidate compounds. Drug efficacy can be assessed by measuring response homogeneity to treatment within a cell population. In this study, topologically quantified nuclear patterns of methylated cytosine and global nuclear DNA are utilized as signatures of cellular response to the treatment of cultured cells with the demethylating anti-cancer agents: 5-azacytidine (5-AZA) and octreotide (OCT). Mouse pituitary folliculostellate TtT-GF cells treated with 5-AZA and OCT for 48 hours, and untreated populations, were studied by immunofluorescence with a specific antibody against 5-methylcytosine (MeC), and 4,6-diamidino-2-phenylindole (DAPI) for delineation of methylated sites and global DNA in nuclei (n = 163). Cell images were processed utilizing an automated 3D analysis software that we developed by combining seeded watershed segmentation to extract nuclear shells with measurements of Kullback-Leibler's (K-L) divergence to analyze cell population homogeneity in the relative nuclear distribution patterns of MeC versus DAPI stained sites. Each cell was assigned to one of the four classes: similar, likely similar, unlikely similar, and dissimilar. Evaluation of the different cell groups revealed a significantly higher number of cells with similar or likely similar MeC/DAPI patterns among untreated cells (approximately 100%), 5-AZA-treated cells (90%), and a lower degree of same type of cells (64%) in the OCT-treated population. The latter group contained (28%) of unlikely similar or dissimilar (7%) cells. Our approach was successful in the assessment of cellular behavior relevant to the biological impact of the applied drugs, i.e., the reorganization of MeC/DAPI distribution by demethylation. In a comparison with other metrics, K-L divergence has proven to be a more valuable and robust tool for categorization of individual cells within a population, with potential applications in epigenetic drug screening. © 2009 International Society for Advancement of Cytometry

Topological analysis of the distribution of proteinaceous and nucleic acid components of the cell, in particular mammalian cell nuclei, is helpful in understanding cellular functions in the state of health versus disease (1–10). Correlations between the distribution of cellular proteins and/or fractions of nuclear DNA and certain diseases has allowed mammalian cells to be utilized as useful models in the search for appropriate disease treatment, in the context of systems biology (11, 12). With the availability of today's more advanced imaging approaches (including confocal laser scanning microscopy, two-photon excitation microscopy, high content cell imaging, and automated tissue scanning), high resolution optical imaging has evolved into an essential tool for moving new chemical entities through the pharmaceutical discovery pipeline utilizing cell-based assays. Imaging advantages for drug discovery are realized through the ability of high-resolution microscopic imaging to measure the spatial and temporal distribution of molecules and cellular components, which is vital to understand the activity of drug targets at the cellular level. Thus, microscopic imaging applies to the preclinical stages of drug discovery for exploratory studies, target identification and validation, lead generation and optimization, and biomarker discovery (13). Drug efficiency can be measured by the uniformity of cellular response upon drug application, focusing on what percentage of cells in a population has reacted to the applied drug. More interestingly, compound effects can be evaluated by imaging changes in the relevant proteins' distribution patterns, and or nucleic acid loci which function as drug targets. This new, cytomic approach (1, 2) is gaining momentum by decreasing attrition in the very costly process of drug development.

Epigenetic changes, such as DNA methylation and histone modification, play a key role in cellular differentiation (14–16). Aberrant global methylation patterns are associated with several cancer types. Methylation pattern imbalances in cancer cells include genome-wide hypomethylation and localized aberrant hypermethylation of CpG dinucleotides (CpG islands) in promoter regions of tumor suppressor genes (17, 18). The reversible nature of epigenetic aberrations constitutes an attractive therapeutic target, and epigenetic cancer therapy with demethylating agents has already shown to be promising (19). Demethylating agents cause structural reorganization of the genome in cell nuclei, as they not only alter the DNA methylation load but also influence its spatial distribution (20, 21). Therefore, in a previous image-based cytometrical approach, we delineated MeC and overall DNA in AtT20 mouse pituitary tumor cells by means of immunofluorescence, and revealed significant differences in the patterns of MeC and (DAPI)-derived signals between untreated and a subpopulation of these cells treated with 5-AZA (22), a demethylating agent that has been reported to change methylation patterns on a genomic scale (23). Therefore, image-based assessment of DNA methylation patterns may provide a powerful technique for characterizing mammalian cells during differentiation and their status of health versus disease, as the underlying molecular processes involve large-scale chromatin reorganization, which is visible by light microscopy (24–29).

Today's advanced cellular imaging systems can produce multispectral two-dimensional (2D) and 3D data in quantities that often require machine vision support to assess and quantify the degree of individual cell similarity within an entire cell population based on cellular features. Topological analyses typically necessitate the segmentation of cellular regions of interest (ROI), including the entire cell and/or subcellular compartments such as the nuclei. This process involves the delineation of the ROI, recognition of residing patterns, and statistical quantification of these patterns with dedicated algorithms. So far, nuclear features have been analyzed in one of the following three ways: (i) comparing a known or unknown pattern with a reference pattern using statistical tests; (ii) classification of patterns through supervised learning, utilizing decision trees, support vector machines and neural networks; or (iii) clustering, in which the distance between points in feature space is used as a discriminating factor (30). The features are measurements reflecting complete cellular or just nuclear morphology, fluorescence intensity, and texture. For example Strovas et al. normalized the intensity of a variant of green fluorescent protein from methylotrophy promoter (PmxaF) of single cells to their size, in Methylobacterium extorquens AM1 culture. This served as a descriptor of cell-to-cell heterogeneity in growth rate and gene expression in response to antibiotics (31). Knowles et al. measured protein distribution through radial bright features within nuclei to identify changes in tissue phenotype (32). Lin et al. employed linear discriminant analysis with nuclear models that were constructed from userprovided training examples to distinguish different cell types (33). Markovian and fractal features (34), Zernike moments, co-occurrence matrices (35) and features generated by Gabor transformation have been commonly used in recognizing subcellular structures (36). Yet, the sensitivity of texture features depends strongly on the optical system setup, such as focusing, image magnification, and object positioning. In the description of cellular structures, the textural, morphological, and intensityfeatures are usually complementary.

The use of features in the quantitative description of 3D nuclear architecture is employed in many biological and medical applications, ranging from in situ studies of DNA, protein localization and migration in living cells, exploration of the structural aspects of cell division to investigations of the role of nuclear alterations in pathology (6–10, 37, 38). These approaches mostly consider the statistical distribution of one target, a protein or DNA fragment (single gene copy or genomic region) to be analyzed. In those cases, a reference pattern detected under specific conditions is usually defined and compared with protein/DNA distribution patterns that result from changes in culture conditions. However, image-based cytometry, which readily considers two or more parameters at the same time, would largely benefit from algorithms that can statistically assess patterns of multiple cellular targets. This is especially valuable in the discovery of pathways that can be targeted in drug discovery. Here, we report the development and application of a novel comparison-based approach that provides a statistical measurement on the two classes of DNAs; MeC and DAPI-positive global DNA, as nuclear targets. The algorithm compares the relative distribution of signals derived from these two targets (from two colorchannels), projects them onto scatter plots, and then measures the degree of similarities between the plotted signal distributions of cells within a population (22). This method offers a way to evaluate cellular response to external factors such as drugs and changes in culture conditions via dissimilarity assessment of relevant cellular structures.

Similarity between two data objects is perceived through measurement of the objects proximity in a multi-dimensional space, and is used to express the objects' relationships within a cluster or between clusters obtained through a partitioning process. Distance or similarity measurements between objects forming a cluster have been defined as equivalent notions (39); however, appropriate metrics are required to identify objects with similar or dissimilar profiles. Commonly applied similarity measures can be organized into three groups according to object representation: (1) point-based, including Euclidean and Minkowski distances, (2) set-based including Jaccard's, Tanimoto's, and Dice's (40) indices, and (3) probabilistic with Bhattacharyya (41), Kullback-Leibler's, and correlation-based Mahalanobis (42) distances, respectively. In many practical applications the objects are described by discrete features, by which the similarity is assessed (39). Furthermore, the sample homogeneity as cluster quality measure can be perceived as an averaged pairwise object similarity (36, 39).

We utilized the Kullback-Leibler's measure with its properties in our study. The background of this approach is introduced here. Let us consider a random discrete variable X with probability distribution p = {pi}, where pi is the probability for the system to be in i-th state. The measure log(1/pi) is called the unexpectedness or surprise (43). Two extreme states can occur: if pi = 1, then the event is certain to happen, and if pi ≈ 0 then the event is nearly impossible. Now, consider two discrete distributions p = {pi} and q = {qi}, where pi and qi are the probabilities of occurrence of the i-th state in a set of system states. The difference: log(1/qi) − log(1/pi) defines change of unexpectedness of the probability p with respect to probability q. Averaging the unexpectedness of the events over pi leads to:

equation image(1)

where: H(p) is the negative of Shannon's entropy (44) and K(p,q) is the measure of information referred to as inaccuracy (45). KL(pq) is nonnegative and delimited by the following constraints: equation image, and equation image.

Function KL(pq) is known as the Kullback-Leibler's divergence (46) of information linked to two probability distributions p and q. This is also a measure of how different two probability distributions (over the same system states space) are. Typically, pi represents data, observations, or a precisely calculated probability distribution, and qi represents an “arbitrary” distribution, a model, a description or an approximation of pi. Following (46) it is assumed that: (i) 0log(0/qi) = 0; and (ii) terms in Eq. (1) where the denominator is zero are treated as undefined and are neglected in order to provide absolute continuity of pi with qi.

The Kullback-Leibler's divergence can be used to measure the distance between various kinds of distributions (47). For instance, it has been employed in medical and systems biology applications including registration of image datasets (48), image segmentation (49), temporal analysis of gene expression (50), clustering of gene expression data (51), and similarity analysis of DNA sequences (52).

The objects' homogeneity assessment is then performed in two steps. First, distance-based similarity is measured between the combined 2D MeC/DAPI histograms of all nuclei and the histogram of each individual cell nucleus. Second, each nucleus (object) in the population is assigned into one of the predefined categories based on similarities.

Assessment of cell population homogeneity is not a trivial task as it is constrained by the imaging modalities and the cell type itself. In a typical setting, the evaluation of cellular response to external factors such as drugs can be achieved with a comparison of the treated population to an untreated (reference) population. However, in this work we present a method to assess each population by itself, in isolation. These populations were analyzed a posteriori, (i.e., without prior knowledge of relevant structural information). Regardless, our approach also allows for a global assessment of cellular patternsamong populations.

In 3D image analysis of nuclei, the segmentation of the nucleus and the quantification of residing features are the most vital components. A common scheme in existing approaches is the watershed algorithm followed by extraction of pertinent features (53–58). The aforementioned solutions require the extraction of tens of features for clustering or classifier training for the further application of a pattern recognition task. Hence, an algorithm utilized for feature extraction and pattern recognition, may be restricted by the morphology of a specimen, in which some features are redundant whereas others are irrelevant. Although some methods for cellular detection and segmentation have been proposed, a general-purpose system that can perform analysis and recognition tasks for a variety of confocal microscope images withoutnecessitating an approach modification or system training(related to the target-specific applications) is still not available.

The main aim of this work is to develop a software system that can be robustly applied to the topological analysis of nuclear targets, such as MeC and DAPI, which will provide useful parameters in the elucidation of epigenetic mechanisms as well as the evaluation of epigenetic drugs tested in cultured cell models. The algorithm developed combines the three major tasks: (1) automated segmentation of nuclei in a cell population, (2) subsequent nuclear pattern extraction, and (3) distance-based statistical measurement of cell dissimilarity using Kullback-Leibler (K-L) divergence. This method considers the strength of statistical evaluation of intra-nuclear MeC/DAPI patterns, especially valuable when cell population homogeneity is difficult to be assessed due to lack of standardized reference and sample size. In this study, we evaluate the potential of using an unsupervised 3D seeded watershed algorithm coupled with K-L divergence measurement to calculate the dissimilarity of mouse pituitary folliculostellate TtT-GF cell response to treatment with the demethylating agents, 5-AZA and OCT. This response was quantitatively measured and displayed as the differential co-distribution of MeC/global DNA signals in treated and untreated cells. A comparison of K-L divergence with other commonly used similarity metrics demonstrates the superior performance of our method.


Cell Culture

TtT-GF cells (ATCC) were grown in serum-containing low glucose Dulbecco's modified Eagle's medium (Invitrogen) supplemented with 10% fetal bovine serum, with addition of 2 mM glutamine and 1% antibiotic/antimycotic (100 units/ml penicillin G sodium, 100 μg/ml streptomycin sulfate) (Invitrogen), in 6% CO2, 37°C as described by Ben-Shlomo et al. (59). Cells were plated at 1 × 105 cells onto coverslips in multi-well plates, and allowed to attach for 24 hours. Then, cells were divided into two groups: (i) two control populations that were not treated for 48 hours (NT-TtT-1, NT-TtT-2), (ii) and two treated populations: AZA-TtT cells treated with 1 μM 5-azacytidine (Sigma-Aldrich) and OCT-TtT cells treated with 100 nM octreotide (Sigma-Aldrich), both for 48 hours.

Immunofluorescence and Imaging

To preserve the three-dimensional structure, cells cultured on coverslips in 12-well microplates were fixed with 4% paraformaldehyde/phosphate buffered saline (PBS) (Sigma-Aldrich) and permeabilized as previously described in Refs.60, 61. Subsequently, cellular RNA was removed with RNase A (Novagen), particularly because transfer RNA (tRNA) contains methylated cytosine as previously described (22). Cells were depurinated with 2N HCl and blocked with 2% BSA/PBS prior to application of antibodies: a monoclonal mouse 5-MeC antibody (EMD Biosciences) followed by a secondary Alexa 488-linked goat anti-mouse polyclonal IgG (Invitrogen). The specimens were counterstained with DAPI, and 3D imaging was performed using a confocal laser scanning microscope TCS SP2 (Leica Microsystems Inc.) equipped with a multi-line argon laser (458 nm, 488 nm, 514 nm) for Alexa 488 (MeC), and a 405 nm diode laser line for excitation of DAPI fluorescence: serial optical 2D sections were collected at increments of 200-300 nm with a Plan-Apo 63X 1.4 oil immersion lens; pinhole size was 1.0 airy unit. To avoid bleed-through, the imaging of each channel was acquired sequentially. The typical image size was 1024 × 1024, with a respective voxel size of 116 × 116 × 230.5 nm (x, y, and z axes), and resolution was eight bits per pixel in all channels. Example images of NT-TtT cells are presented in Figure 1. Fluorescence intensity of MeC and DAPI signals, IMeC and IDAPI, from optical sections were recorded into separate 3D channels.

Figure 1.

A maximum intensity projection of 3D confocal microscopy images of NT-TtT-1 cells: (A) cell nuclei with patterns of DAPI-staining (blue channel), (B) and MeC-staining (green channel), (C) merged projection, scale bar at the right left corner is 0.24 μm. The horizontal and vertical strips on right and bottom sides of the figure represent horizontal and vertical cross sections of the 3D image stacks along the lines in the image.

Image Analysis

Image analysis was performed in three main steps (see Figure 2): (1) 3D image segmentation resulting in the delineation of a 3D shell for each individual nucleus; (2) extraction of MeC and DAPI signal intensity distribution within each 3D shell; and (3) dissimilarity assessment of MeC and DAPI signal distribution patterns between each individual nucleus and a reference pattern derived from the entire cell population (Fig.2). This workflow was designed based on the images taken from the NT-TtT-1 and the following assumptions: the background in each image stack was considered to be quasi-uniform, meaning that there are very small to zero low frequency fluctuations or trends in the background through a single image plane or across the depth of the image stack. Moreover, all images in each stack are assumed to be acquired under nearly identical conditions and modality settings, and so the drift of the settings during acquisition can be considered minimal and thus neglected. In order to reduce computational complexity during the segmentation phase, the image resolution was decreased by a factor of four for this step only in the x and y directions. The developed methodology was subsequently applied to all image stacks.

Figure 2.

A three-step flowchart of the image analysis methodology.

STEP 1: 3D Segmentation of Nuclei

The IDAPI and IMeC image stacks were combined in the following way: equation image, thus intensity of the output image I is always a maximum of the intensities in corresponding channels at pixel position (x, y, z) (Fig. 3A). To separate the nuclei from the background a histogram of image I was constructed. We apply the technique described in (62) yielding the threshold value Tb that splits the histogram into two parts; a main peak representing the background, and a histogram tail reflecting intensities of the nuclear content. A binary image was obtained in which background pixels and nuclear content were converted to the values 0 and 1, respectively. This image was then subjected to enhancement by means of 3D morphological operations (closing and filling holes), yielding a refined binary image Ib (Fig. 3B). We note that in Ib the majority of nuclei were distinct. However, some nuclei touch (or nearly so) one another to form larger clusters. These two groups of objects were processed separately to better delineate all nuclei.

Figure 3.

3D image segmentation workflow, demonstrated with NT-TtT-1 cells: (A) combined MeC-DAPI channels, (B) binary image resulting from thresholding image in Figure 3A at the level of Tb; (C) distinguished groups of binary objects obtained by mean volume thresholding of objects in Figure 3B: clustered (violet) and distinct (cyan). (D) Image of modeled nuclei is created by assigning a uniform intensity Tb to pixels of image in Figure 3A covered by binary objects from Figure 3B. (E) Binary seeds are generated as a result of smoothing with different kernels followed by thresholding of image from Figure 3D, and superimposed onto the image in Figure 3A. The seed size depends on the degree of model nuclei smoothing. (F) Delineated 3D nuclear shells are reconstructed from a stack of 2D images, serve as ROIs. These shells are created by seeded watershed algorithm and overlaid onto the image in Figure 3A; nuclei previously appearing fused are now separated.

A reduction of the original resolution by factor of four of images Ib and I creates two down-sampled images Ib and I′, respectively. Labeling and counting of the binary objects in Ib was carried out according to Haralick et al. (34), and the volume of each object was found. A mean volume value Tvol, served as a criterion to split the image Ib into two binary masks, one with small components Ibs and one with large components Ibl (Fig. 3C). Then, all voxels of image I′ under the mask Ib were replaced by a constant value Tb, creating an image Im that models the nuclei (Fig. 3D). Such approach is useful for object segmentation, because it is comprised of image intensities equal to or lower than the automatically defined threshold Tb. This model, is used to create 3D seeds that define location of each nucleus, and serves also as the input for the seeded watershed segmentation technique.

Next, image Im was subjected to smoothing by two anisotropic Gaussian filters, Gs and Gl for small and large binary components, respectively. Infinite Gaussian kernel is approximated and its size is defined by Nx, Ny and Nz representing mask size in each direction. The smoothing effect in 3D is controlled by three parameters Gxyz). To assure that smoothing can produce a signal strong enough to detect a seed, the approximated filter kernels were adaptively adjusted to the relative volume of the binary objects in Ibl and Ibs respectively. The kernel size is adjusted first. We chose a spherical model for cell nuclei, and allocated three kernels for each x, y and z axis of a sphere. This approach provides a predefined number of filter kernels that fit the hypothetical nuclear size, in our case seven (n = 7). Since the image voxels in our data stacks are not isotropic, Nz can be almost twice as much compared to Nx = Ny, and the filter size can therefore be calculated from Tvoln · NxNyNz. Substituting Nz = 2Nx and Ny = Nx the filter size equation image can be derived as the largest odd number satisfying this inequality. Thus, mean volumes of binary objects in Ibs and Ibl can be used to calculate filter size Nx for kernels Gs and Gl. Second, the remaining filter coefficients σx, σx, and σx were empirically set to one half of the mask size in each direction. In general, sizes of Gs and Gl kernels are proportional to the mean volume of binary objects under respective masks, and so the corresponding filter coefficients. Also, the size of Gl is never smaller than Gs.

The image Im is separately smoothed once (by each kernel) to obtain the images Ims and Iml. The larger the kernel is, the smoother the created surface of the ROI (nucleus) will be. After filtering, the results were combined into one output binaryimage according to:

equation image(2)

where trh denotes a threshold function expressed as:

equation image(3)

and where Q(x,y,z) is an image, T is the threshold, If is the output image, Ims, Iml are the smoothed components, ⊗ denotes element-by-element multiplication and ∪ is the matrix logical union.

The smoothing procedure produces slowly varying intensity fields in Ims and Iml with maxima and local plateaus resembling blobs in 3D space which are located inside the nuclei, with intensities oscillating around Tb. The location and size of the maxima depend on the smoothing kernel and the nucleus size. The thresholding of the smoothed image at the level of Tb yields binary seeds in If, with one seed per nucleus (Fig. 3E). Small seeds were eliminated and converted to background.

The watershed algorithm (63) in its original form has several well-known limitations; it typically over-segments the image and does not take into account image-inherited cues such as intensity gradients, topology and content of segmented objects. Thus, the seeds serve as a priori knowledge about segmented structures and form numerous points for algorithm initialization. Such an approach has the potential to generate a number of unique regions that closely matches the number of seeds. In this study we extend the existing implementation of the 2D seeded watershed method (64, 65) to obtain 3D nuclear shells (Fig. 3F). During this segmentation each nucleus receives a label for further identification and visualization. Then, the segmented image Is was up-sampled by factor of four with the nearest neighbor interpolation technique, resulting in the image Is that contains the 3D nuclear shells. This image can also be superimposed onto IDAPI or IMaC and displayed, as shown later in Figure 4.

Figure 4.

Examples of cell populations with selected nuclear MeC/DAPI co-distribution patterns: NT-TtT-1 having an overall low number of dissimilar cells (A), OCT-TtT with a high number of dissimilar cells (G), and AZA-TtT with large majority of similar cells (M). The calculated K-L value is displayed within each nucleus. Selected nuclei NT1 (B) and NT2 (C) with different MeC/DAPI patterns and corresponding scatter plots (E and F) contribute to the plot for the entire cell population (D), and illustrate the intra-population diversity in MeC/DAPI patterns: The two selected nuclei display differential visual impressions of the distribution of the two types of DNA (B and C), confirmed by their respective scatter plots and their K-L values. NT2 is less similar to the reference (entire) population than NT1. Individual nuclei OCT1 (H) and OCT2 (I) with extreme differences in MeC/DAPI patterns (K and L, respectively) are selected and characterized by their strongly differing K-L values (4.76 and 2.81) and plot slopes (K and L) comparing to the entire population scatter plot (J). AZA1 (N) and AZA2 (O) with extreme differences in MeC/DAPI patterns (Q and R, respectively). (P) represents the combined plot of the entire population. Nuclei AZA1 and AZA2 are characterized by their highly different K-L values (0.12 and 1.71) and plot slopes (Q and R).

STEP 2: Extraction of MeC and DAPI Patterns

A powerful aspect of scatter plots is their ability to depict mixture models of simple relationships between variables. These relationships can reflect cellular patterns as specific signatures, in which the variables can be nuclear structures as shown in the case of DNA methylation patterns versus DAPI-stained DNA (22). These nuclear entities are not static and reorganize during cellular differentiation, as well as upon the application of demethylating agents. Earlier we showed that such reorganizations can be dynamically monitored by scatter plotting the two types of DNA, with their differential distribution becoming visible as changes in the plotted patterns. In this case, we first individually segmented nuclei to create three-dimensional ROIs (3D-shells). Then, we plotted the fluorescent MeC and DAPI signal distributions within these shells. Utilizing K-L divergence, the degree of similarity between two scatter plots can be easily measured, and reflects the similarity of target (MeC and DAPI signals) topology between two cell nuclei (in Kullback-Leibler sense).

STEP 3: Nuclear Pattern Analysis by Means of Kullback-Leibler's Divergence

In our approach, we applied the K-L divergence as a statistical measure of dissimilarity between two normalized scatter plots: the value of qi denotes a probability of occurrence of intensity i in an analyzed nucleus outlined by 3D shells and pi signifies a reference scatter plot component. The reference scatter plot is constructed from all individual plots. To the best of our knowledge, no such work on identification of nuclear patterns based on Kullback-Leibler's measure has been reported so far. Therefore, this is an innovative way to perform an intra-population assessment of cells with regard to their homogeneity in response to environmental changes in culture, and is especially suitable for high-throughput multi-parameter analyses.

The K-L divergences represent distinctive and relative measurements derived from a unique cell population. A comparison of K-L values between experiments, in principle, requires identical reference distributions to be applied. However, a lack of reproducibility in sample preparation, drift and instability of imaging modality settings is the primary constraint in determining such a universal reference. In order to reduce the influence of these constrains, and to make the K-L values more descriptive, we introduced four soft-qualifiers for defining the similarity degree of a cell versus the entire cell population. These degrees are associated with particular ranges of K-L divergences derived for two idealized Gaussian distributions. For the multivariate d-dimensional Gaussian densities given by equation imagethe Kullback-Leibler's divergence is expressed by:

equation image(4)

where: x is the random variable, μ is the vector of means, Σ is the covariance matrix, tr is the trace function, and |·| is the determinant of a matrix.

The K-L divergence in Eq. (4) between two one-dimensional univariate Gaussian distributions pG(x) = N(xpmath image) and qG(x) = N(xqq2) with x as the random variable comes down to (60):

equation image(5)

Furthermore, assuming that σp ≈ σq and that σ can be substituted instead of σp and σq in Eq. (5), we obtain equation image where the numerator reflects the distance between the peaks of the two Gaussian distributions. The KLG in the simplified formula can be also related to the fraction of the distributions' overlap area and used as a way of articulating dissimilarity. Also, when expressing μp−μq as a multiple of σ, the KLG value becomes solely dependent on the standard deviation in the evaluated distributions. Table 1 illustrates the four soft-qualifiers defining the similarity degree of KLG divergence linked to σ, obtained on the basis of the aforementioned assumption. The four soft-qualifiers are defined as: similarKLG ∈ [0,0.5), likely similarKLG ∈ [0.5,2), unlikely similarKLG ∈ [2,4.5), and dissimilar for KLG ∈ [4.5,∞). Thus this procedure can be perceived as a classification process. As a side note, the K-L divergence between two bivariate normal densities is a function of Pearson's correlation coefficient (66).

Table 1. Values of Kullback-Leibler's divergence and percent of overlap for two hypothetical univariate Gaussian distributions pG = Np, σ2) and qG = Nq, σ2)
μqpKLGPercent of Overlap Area Between pG and qG

Evaluation of Similarity Measures

Three commonly used similarity metrics including Mahalanobis, Bhattacharyya distances, and Dice's index were implemented into the image analysis workflow together with the proposed K-L divergence and then applied to NT-TtT-1, AZA-TtT and OCT-TtT cellular images. Since none of these metrics have been documented for assessing cell culture homogeneity through 2D methylation pattern histograms, we compared their performance to determine the most appropriate approach for measuring demethylating effects by nuclear topology. Unlike the method and system validation characteristics such as accuracy and reliability that are based on individual results, the characteristic of the uncertainty of results delivered by a classification method needs to be determined on a method-to-method based comparison (67). Therefore, using the uncertainty as a validation characteristic raises the objectivity of our comparative evaluation. In our case we used similarity values of nuclei within a cell population. Assuming that a similarity metric can label a nucleus in a way that it reflects its natural proximity to other nuclei in the feature (nuclearpattern) space, then such labeling should have a low uncertainty.

Our evaluation steps were as follows: (i) each of the tested metrics yielded a similarity value for all nuclei; (ii) the nuclei were grouped into classes based on assigned similarity value. For this a minimum distance criterion in class forming scheme was applied, and up to six classes were generated; (iii) clustering results were evaluated as described in (67). The entropies of the results were calculated as a measure of uncertainty, in which the lowest entropy indicates the least uncertainty of results produced by the evaluated method. (iv) Finally, a normalized certainty was used for method comparison (67):

equation image(6)

where: M is the number of classes used in the classification scheme, and EntropyM is calculated from the results of the similarity measure classification into M classes.


Untreated (NT-TtT-1, NT-TtT-2) and treated mouse pituitary tumor cells (OCT-TtT and AZA-TtT), (total number of cells n = 163) were imaged, and then analyzed by our in-house developed, MATLAB-based software. Following our algorithm, the three-dimensional nuclear shells were first delineated (Fig. 3), and then for each nucleus within an image field the fluorescent signals derived from MeC-specific staining and DAPI staining were mapped as respective scatter plots. The K-L divergences of the distribution of MeC and DAPI signals between individual plots (nuclei) and the reference plot (cumulative plot from all nuclei) were then calculated. The algorithm displays the K-L values and the digital ROI for each cell nucleus, as shown in Figure 4. Six nuclei (two from each of OCT-TtT, AZA-TtT and NT-TtT-1 cell group) illustrating different nuclear MeC and DAPI patterns were selected as examples for visualization purposes. The fields appearing in these figures are smaller than the complete microscopic field of view. Figure 3 shows the earlier intermediate steps of the algorithm described in the methods section, followed by the actual results in Figure 4.

The applicability of the K-L divergence was tested for the categorization of nuclear patterns with significantly different DAPI signal distributions. One-dimensional MeC and DAPI histograms were generated for each of the two 5-AZA-treated as well as the two OCT-treated nuclei, and plotted next to their respective 2D joint MeC/DAPI diagrams (Fig. 5). This separation shows that both signals, MeC and DAPI, differ in their intensities (indicated by the curves' shapes) between cells, which can be interpreted as the result of differences between cells in their response to the demethylating agents.

Figure 5.

Differential nuclear MeC and DAPI signal distributions; drug treated cells, 5-AZA-treated cells (A-F) and OCT-treated cells (G-L) displayed as 2D histograms (middle column) and individual 1D histograms (MeC, left column, and DAPI, right column). There are visible differences between respective histograms as follows: the range of MeC intensities in (D) is much greater than in (A), and the width of the MeC histogram peak in (G) is almost twice as much as in (J). Similarly, DAPI signal distributions in (C) and (F) differ in terms of histogram peak width and tail length. In (I) the DAPI signal is more compact compared to the wider spread of DAPI in (L). As a consequence, the K-L divergence applied to the 2D patterns in graphs (B), (E), (H), and (K) yields the following results (0.24, 1.03, 1.01, 1.58), whereas applied to 1D MeC histograms A, D, G, and J yields values of (0.23, 0.05, 0.76, 0.59), respectively.

Based on the definition of soft-qualifiers in Table 1, we have chosen four categories into which the processed nuclei fall: similar, likely similar, unlikely similar, and dissimilar.

This categorization helps to characterize a cell population in a quantitative and readable fashion (Table 2). The classification was performed twofold: (i) using solely the MeC histogram, and (ii) using joint MeC/DAPI histograms, of individual cells versus the entire population. In the first case a combined MeC histogram was used as the reference distribution. The outcome provides statistical information about the number of cells that fall into each category. Different cell populations can then be compared based on their category statistics.

Table 2. Results of soft qualification of nuclei in different cell populations.
MethodKL Divergence Applied to Joint MeC/DAPI Signal DistributionKL Divergence Applied to MeC Signal Distribution Only
CellsControl CellsTreated CellsControl CellsTreated Cells
  1. Application of the K-L divergence to 2D MeC/DAPI distributions revealed a significantly higher number of cells with similar or likelysimilar MeC/DAPI patterns among untreated cells (∼ 100%), 5-AZA-treated cells (90%), and a significantly lower degree of same type of cells (64%) in the OCT-treated cell population. The latter group was found to contain a subset of unlikely similar (28%) and dissimilar (7%) cells. Evaluation of 1D MeC distribution in the same cell groups resulted in a shift of the cells to the lowest category, i.e a reduced number of dissimilar cells in all populations compared to when 2D MeC/DAPI distributions are utilized for homogeneity analysis. In particular, OCT treated cells almost present an equal distribution of cells in the remaining categories, and the majority of 5-AZA treated cells appear as similar.

Similar13 (76%)38 (74.5%)1 (7.1%)27 (90%)15 (88%)45 (88.2%)4 (28.6%)29 (96.7%)
Likely similar4 (24%)12 (23.5%)8 (57.2%)3 (10%)2 (12%)5 (9.8%)6 (35.7%)1 (3.3%)
Unlikely similar01 (2%)4 (28.6%)001 (2%)6 (35.7%)0
Dissimilar001 (7.1%)00000

Utilizing the joint MeC/DAPI patterns in the categorization of the four groups of cell populations revealed that all NT-TtT-1 cells are classified as at least likely similar, with a majority of 76% being similar. This signifies a relatively high homogeneity of MeC versus DAPI distribution within the NT-TtT-1 cell population. Likewise, 74.5% similar, 23.5% likely similar and 2% unlikely similar cells were found in NT-TtTGF-2 population. Our assessment of untreated cells revealed that the distribution of the cell categories was quite consistent in populations with different numbers of cells. In comparison, OCT-TtT cells display a higher portion (64%) of likely similar cells and to a lesser degree (36%) also unlikely similar cells. The AZA-TtT cells represent very low ratio of dissimilarity, with 90% similar and 10% likely similar cells. However, one can note that their intracellular architecture is different comparing to NT-TtT and OCT-TtT cells in that, fewer loci is seen within AZA-TtT cell nuclei vs. nuclei in the remaining cultures.

Utilizing only MeC histograms to categorize cells yielded no dissimilar cells in all four tested populations. In NT-TtT-1 and NT-TtT-2 control cell lines there were identical fractions of approximately similar cells (88%). In NT-TtT-1 12% of cells and 10% in NT-TtT-2 were classified as likely similar, with 2% of cells found unlikely similar in NT-TtT-2 population. OCT-treated cells revealed almost equal (28–35%) allocation of cells among all three similar cells categories. The cell population treated with 5-AZA was characterized as highly represented by similar cells (97%) with only one cell (3.3%) classified as likely similar.

The cell categorization was implemented into the image visualization and analysis software we developed, as shown in Figure 6. Such visualization is a valuable feature of image-based cytometry, providing dual information of cell behavior/category and localization within the sample environment. Processed images of the three cell groups used in this study underwent a visual check by an expert (J.T.) and the dissimilarity evaluations between cells matched the automated analytical results.

Figure 6.

Visualization aid in the evaluation of cell population homogeneity: our software, developed in-house, is able to convert K-L values into pseudo colors, as illustrated for the NT-TtT-2 cell population with a large number of similar cells constituting a high population homogeneity. Here the pseudo-colors represent the K-L soft-qualifiers: green (similar), blue (likely similar), yellow (unlikely similar), and red (dissimilar, not present in this population).

In our definition of soft qualifiers, normality of the sampled population was assumed. To evaluate normality of the individual MeC/DAPI distributions, we estimated two Gaussian components by means of the expectation-maximization clustering algorithm (68) in each of the segmented nuclei of the NT-TtT-1 population (Fig. 7). The components estimated in this way constitute approximately 75% of data points of each nucleus. In addition, using Lilliefors' statistical tests (69) we tested a null hypothesis, which considers that the data derives from a multivariate family of normal distributions. This test was performed for each nucleus and separately for each dimension. The null hypothesis was not rejected at the 5% significance level. Therefore, we assume that the scatter plots obtained throughout our experiments can be approximated by multivariate Gaussian components.

Figure 7.

A scatter plot of a NT-TtT-1 nucleus with bivariate Gaussian components estimated by the expectation-maximization algorithm. The mean values are marked by the “+” sign and the ovals outline areas within one standard deviation from the means. These components constitute approximately 75% of all data points.

Selected similarity metrics, including the Mahalanobis and Bhattacharyya distances as well as the Dice's coefficient and the proposed K-L divergence, were calculated for each of the individual two-dimensional MeC and DAPI plots and the combined distribution. The normalized certainty (Eq. 6) of the results determined by the different metric methods is presented in Table 3.

Table 3. Normalized certainty of the results obtained with different metrics: similarity data was generated for the distinction of cells into two to six categories (classes). K-L divergence shows the highest certainty values (bolded) in the majority of tested cell populations.
No. of ClassesCell CultureSimilarity Measure

Our comparison of the different most applicable metrics indicates that in the majority of cases (73%) the normalized certainties reached their highest values when the classification was based on the K-L divergence. Moreover, if more than two classes (a more frequent scenario) are considered in the classification scheme, the proposed K-L similarity measure achieves the highest certainty scores in even more of the cases (91%).


The main goal of this study was to develop an automated image analysis tool that would be suitable for measuring the effects of demethylating agents through the differential analysis of relevant nuclear structures, as represented by methylated CpG-dinucleotides (MeCs) and global DNA, in cells. For this purpose, a dedicated tool was designed that performs the three sequential steps on individual cells within a population: (1) unsupervised segmentation of 3D imaged cell nuclei via seeded watershed algorithm, (2) multi-channel quantitative distribution analysis of nuclear entities, and (3) similarity testing of cells in regard of their distribution profiles by means of Kullback-Leibler's divergence measurement. Our experience with mouse pituitary tumor cells confirms that demethylating agents can exert the two known effects: (i) a decrease in the number of MeCs in global DNA (70), and (ii) the subsequent decondensation of highly compact heterochromatic regions of the genome, that lead to spatial reorganization in the nucleus and affect nuclear architecture (28). The image analysis we developed utilizes these coexisting phenomena to measure and display the relevant changes in intensity distribution of the two types of signals that reflect said phenomena: (a) MeC-signals created through immunofluorescence targeting of methylated cytosine and (b) DAPI-signals generated by subsequent counter-staining of the same cells, as DAPI intercalates into AT-rich DNA, the main component of highly repetitive and compact heterochromatic sequences. Our computational approach minimized the usual obstacles in automated cellular analysis such as intra-specimen variation in background and morphological properties of nuclei, including size, shape, and structural density. Furthermore, cellular clustering seen for some types of cells such as pituitary tumor cells in culture, can create a poor contrast between nuclear borders. The implementation of the seeded watershed algorithm in here allowed for a conservative separation of nuclei. In addition, the change of object resolution during image processing allowed for process acceleration through reduction of computational complexity. The segmentation masks can be overlaid onto the corresponding raw MeC/DAPI images for performing visual assessment of segmentation accuracy. It should be noted that the visual classification of the composite MeC/DAPI signals can be very time consuming and quite subjective, as compared with computer-aided classification in an automated fashion. This fact is especially true when large sets of image data with a highly non-geometrical distribution of nuclear targets need to be processed. In this way, both the delineation of the nuclei and the topological quantification of the complex patterns will be streamlined and results will be produced with higher confidence. The developed method is amenable to scale and suitable for high-content, high-throughput analysis of cells in both research and at the industrial volume.

In previous studies, we showed that the nuclear distribution of MeC versus DAPI signals, displayed as a 2D scatter plot, can serve as a signature by which cells differing in their state of differentiation or in treatment can be distinguished (22). We also observed that untreated and drug-treated cells of the same kind display different degrees of dissimilarity within their populations, as judged by the resulting scatter plots. This led to the development of the synthesized image analysis method described here, which utilizes the resulting scatter plots in a statistical fashion to assess structural and behavioral dissimilarity within a cell population. These features are generally studied in relevance to a variety of cell biological applications. Our notion was to develop and test an algorithm that can be meaningfully and robustly applied to the evaluation of demethylating agents such as 5-azacytidine and octreotide. However, the developed algorithm can be flexibly utilized for similar topological studies, in which nuclear entities and their distribution are targeted in a biological context. Especially, the modular integration of the K-L divergence measure is a valuable feature that allows for the statistical evaluation of cells, when the targets do not have a consistent location within the considered ROI, such as the nucleus. Furthermore, our analyses indicate that if only 1D histograms of MeC signal distribution were utilized for K-L divergence measurement, significantly different results were observed in homogeneity assessment when compared to those using the joint 2D MeC/DAPI histograms. The exclusion of DAPI causes a shift of the cells to the lower categories, suggesting that the DAPI signal is a meaningful dynamic parameter as it increases the differential resolution in the image-based analysis of nuclear methylation patterns. This can be reconciled with the aforementioned biological effects on nuclear DNA. In particular, heterochromatin decondensation, as a secondary effect of global demethylation, results in the relocation of heterochromatic sites within the nucleus (which is associated with genome destabilization). As a consequence of these conformational and organizational changes of the DAPI-positive nuclear sites, the same DAPI signal intensity is spread out over a higher number of voxels. Thus, both MeC and DAPI have dynamic patterns in the cell nucleus that become more discernable in a joint 2D plots than in a 1D MeC plot, or even when the two signals are separately displayed in one dimension (see Fig. 5). Notably, our snapshots of untreated cells also display dissimilarity in MeC/DAPI signal distribution, however, to a much lesser degree than treated cells (Fig. 4). We assume that this could be because of the fact that the cells were in different cell cycle phases, as this study did not apply any synchronizing agents for two reasons: (i) to minimize other induced effects that could interfere with demethylation, and (ii) to more closely model the in vivo situation in which synchronicity of cells within their native tissue environment is naturally not the case. Therefore untreated cells that display a lower MeC load signal may represent replicating cells in S-Phase that had not completed methylation of de novo synthesized DNA strands, as delay times between the two processes of replication and methylation have been reported for various types of cells (71, 72).

Our approach directly illustrates the distribution of voxel intensities. The changes of these distributions are derived from the underlying changes in the topology (spatial patterns) of global DNA in response to drug treatment. Consequently, we are able to demonstrate that when the topological nuclear distribution patterns of methylated cytosine and global DNA are converted into two-dimensional histograms, they can be utilized as differential biosignatures in the evaluation of cellular response to treatment with demethylating anti-cancer agents. This characteristic is in line with the larger purpose of our approach, namely to create a rapid image analysis method that is of low complexity and therefore computationally inexpensive with potential for high-throughput cell screening tasks.

Other statistical methods such as cluster or bimodal analyses (73, 74), commonly utilized in gene expression analysis (75), are important when targets (with respective intensities) have a definite location (coordinates). These methods are valuable in assessing ratio labeling of targets when hybridized to arrayed nucleic acid fragments that are immobilized and have defined coordinates on the supporting material (DNA microarrays) (76–78), or when hybridized to genomic loci with known chromosomal locations on metaphase chromosomes of normal cells (79). In contrast, nuclear targets such as genomic loci on largely decondensed DNA or proteinaceous entities may strongly vary in their localization between nuclei. In these cases, the K-L divergence becomes of value as it does not require dealing with absolute target coordinates for similarity testing. Moreover and unlike k-means clustering or bimodal analysis of gene expression, the K-L approach tolerates the occurrence of null categories that may not be filled by any object (in this case nucleus). Figure 6 shows an example in which the fourth category, namely dissimilar, is not represented by any of the nuclei in the tested population (no red-colored nucleus is present in Fig. 6B).

The Kullback-Leibler's divergence is a valuable method for quantitating dissimilarities within a cell population and this measure can be applied to any multi-color cellular assay that utilizes topological information of intracellular structures to assess cellular behavior. Our comparison of the metrics most frequently used for similarity measurements demonstrates that the K-L method produces the highest certainty (least uncertainty) for the nuclear MeC/DAPI pattern analysis within the imaged cell populations. Moreover, the Pearson's correlation coefficient between two distributions can be directly calculated from the K-L divergence if the distributions are normal, especially in cases when correlating samples do not have equal size. However, proving normality of multimodal distributions may increase computational complexity in practical cases. A way of identifying a distribution's normal components described and implemented in our study supports the suitability of K-L divergence to be used for our data, especially in determining the soft qualifiers, because in our study the majority of the acquired 2D signals had a normal distribution.

We observed the robustness of the K-L divergence against potential intra-experimental data variability introduced through the biochemical processing of specimen or the modality settings in between imaging sessions, which both may additively alter the intensity levels within the MeC and the DAPI channel. We did not detect any difference in K-L divergences, which was confirmed by the fact that the shape of the scatter plots remained unchanged. On the contrary, influences of multiplicative nature may skew the results of all types of metrics. Additionally, the K-L divergence measurement has the advantage of being independent from image rotation and the inherent anisotropy of confocal microscopy images.

As one would expect, statistical methods in the form of similarity measures gain more confidence when applied to large datasets, in this case large cell populations with thousands of nuclei. To our pleasant surprise, the K-L divergence outperformed the comparative metrics when utilized for smaller cell populations of only around 20 cells. This underlines not only the robustness of the method, but also its flexibility in dealing with a high dynamic range in sample size. This characteristic is quite valuable in connection with the current limited capabilities of our imaging systems that are restricted in the field of view size when acquiring highest-resolution 3D images. Thus, it is necessary to collect and tile multiple image stacks in order to obtain a complete picture of the entire sample. The robustness of the K-L measurement allows it to be applied across the entire tiled image. Such an approach could be helpful in the assessment of relationships between single cells and their macro- and micro-level neighborhoods for studying intra- and inter-population functional relationships through epigenetic effects such as DNA methylation via tissue diagnostics in disease pathology and cell-based assays for compound screening in drug development.


We thank Dr. Anat Ben-Shlomo (Cedars-Sinai Medical Center) for providing treated and control TtT-GF cells.