Departments of Medical Informatics and Radiology, Erasmus MC–University Medical Center Rotterdam, Rotterdam, The Netherlands

Biomedical Imaging Group Rotterdam, Erasmus MC University Medical Center Rotterdam, Departments of Medical Informatics and Radiology, P.O. Box 1738, 3000 DR Rotterdam, The Netherlands

For the investigation of the molecular mechanisms involved in neurite outgrowth and differentiation, accurate and reproducible segmentation and quantification of neuronal processes are a prerequisite. To facilitate this task, we developed a semiautomatic neurite tracing technique. This article describes the design and validation of the technique.

Methods

The technique was compared to fully manual delineation. Four observers repeatedly traced selected neurites in 20 fluorescence microscopy images of cells in culture, using both methods. Accuracy and reproducibility were determined by comparing the tracings to high-resolution reference tracings, using two error measures. Labor intensiveness was measured in numbers of mouse clicks required. The significance of the results was determined by a Student t-test and by analysis of variance.

Results

Both methods slightly underestimated the true neurite length, but the differences were not unanimously significant. The average deviation from the true neurite centerline was a factor 2.6 smaller with the developed technique compared to fully manual tracing. Intraobserver variability in the respective measures was reduced by a factor 6.0 and 23.2. Interobserver variability was reduced by a factor 2.4 and 8.8, respectively, and labor intensiveness by a factor 3.3.

The development of the nervous system is an intricate process involving many different factors (1). The molecular mechanisms regulating neurite outgrowth, differentiation, synapse formation, and plasticity are still being vigorously studied. In order to cope with the ever-increasing amount of image data generated for such studies, accurate and user-friendly image analysis tools are indispensable. Of particular importance is the ability, prior to morphometric analysis, to make accurate representations of the neuronal processes present in the images. Early attempts to automate this task were not very successful, and since then neurite tracing has been done primarily by the labor-intensive process of manual delineation (2). During the past decade, despite the considerable progress in computer technology, only a handful of renewed efforts toward automated neurite tracing have been published. Notable techniques developed recently are those based on vectorial tracking (3), reminiscent of earlier techniques for tracing chromosomes (4), or retinal vasculature (5), and multiscale feature analysis (6). Both techniques were developed for application to 3D confocal microscopy image stacks.

In this article, we are concerned with the problem of accurately detecting and tracing individual neurites in two-dimensional (2D) fluorescence microscopy images of cells in culture. Whereas the computational cost of digital image processing techniques for this task can be expected to be smaller in 2D than in 3D, from a morphological perspective the problem is, in fact, more complicated. The reason for this inherent to 2D imaging of 3D structures is that the images may contain ambiguities regarding the branching or crossing of neurites. In addition, the neurites may sometimes be out of focus, have very low contrast, or even contain large gaps. And when composed of several scans to increase the field of view, the images may show discontinuities in background intensity, as illustrated in Figure 1. It is clear that automated procedures based on intensity thresholding and skeletonization (7–10) would fail to yield correct results for such images. Several investigators have abandoned tracing altogether and have resorted to heuristic approaches to estimate the morphometric features of interest (11–13). An alternative, vectorial-tracking technique, published just before completion of our manuscript, was successfully applied to images of retinal explants (14); however, it does not address the just mentioned problems seen in our type of images.

Because of the complexity of the problem, it is not surprising that in practice nothing much has changed since the early days, as our recent literature survey revealed. A search in the PubMed database (National Library of Medicine, Bethesda, MD) for papers published from 2001 until the present containing, in the title, the keyword “outgrowth” combined with any of “neuron,” “neurite,” “axon,” “dendrite,” or their adjectives, yielded 190 responses. Of these 190 papers, we studied 102 that were electronically accessible through our library and in which it was stated explicitly that neurite length was measured. In 64 of these, no details whatsoever were given as to how the measurements were carried out or what image analysis tools were used, and it seems reasonable to suspect that neurites were traced fully manually. In 20 cases, use was made of commercial image analysis software tools, with a total of 10 different tools mentioned. In the remaining 18 cases, use was made of personally developed or public domain image analysis software, with NIH-Image (National Institutes of Health, Bethesda, MD) and its variants being by far most popular. To our knowledge, none of the reported software tools, whether commercial or not, provides an automated and scientifically validated neurite tracing facility.

Based on this observation and our own experiences with the burden and poor reproducibility of fully manual approaches, we conclude that there is still a great need for image analysis tools that facilitate neurite tracing and possibly leading to more accurate and more consistent measurements. It will be clear, from what has just been said, that the design of fully automated techniques for our purpose will remain a difficult problem, and it seems likely that some form of user interaction will always be required to resolve ambiguities. Accepting this fact and attempting to meet at least part of the needs, we have developed an interactive neurite tracing technique. It exploits the expertise of the user in resolving ambiguities but greatly simplifies the task of tracing long stretches of manifest neurites. In the present study, we describe the design of our technique and present the results of validation experiments revealing the potential improvement in accuracy and reproducibility and the reduction in user interaction of the technique compared to fully manual delineation.

MATERIALS AND METHODS

Tracing Technique

Neurite tracing using our technique consists of (1) a detection phase, in which every individual pixel in the image is assigned a value indicating its likelihood of belonging to a neurite; and (2) the actual tracing phase, in which consecutive pixels that are most likely to represent the centerlines of the neurites are linked together to constitute the tracings. The former is a preprocessing step and is carried out fully automatically; the latter requires user interaction. In the present study, we outline the respective algorithms developed for performing the two tasks. Details are given in the Appendix.

Our approach to detection is based on the observation that in fluorescence microscopy images, neurites are bright, elongated structures superimposed on a dark, noisy background. When representing images as gray-level landscapes, such structures manifest themselves as ridges. It is known (15–18) that ridge-like image structures are well detected by means of second-order differential operators. Specifically, the local principal ridge directions at any point in an image are given by the eigenvectors of the second-derivative matrix computed from the intensity values around that point. Because of the symmetry of this matrix, the eigenvectors are orthogonal, with the eigenvector corresponding to the smaller absolute eigenvalue pointing in the longitudinal direction of the ridge. Comparing the eigenvalue magnitudes, the algorithm computes for each pixel in the image a measure of “neuriteness.” In addition, it stores for each pixel the local ridge orientation as indicated by the mentioned eigenvector. The output of this detector, when applied to the image in Figure 1, is shown in Figure 2.

Our approach to linking of consecutive ridge pixels derives from the so-called live-wire segmentation paradigm (19–22). It consists of the selection of a starting point by the user followed by the application of a search algorithm to find the optimal paths from that point to all other points in the image, where “optimal” means having a globally minimal cumulative cost according to a predefined function. Once computed and stored, the paths can be displayed in real time as the user moves the cursor toward the end of the neurite of interest, until the presented path starts to deviate too much from what is considered the optimal tracing by the user. The tracing can then be fixed up to that point by a single mouse click, after which the algorithm proceeds by presenting optimal paths from that point. The process is iterated until the entire neurite has been traced and can be repeated to trace multiple neurites. In cases where the user is not completely satisfied with the paths presented, which may sometimes occur in regions with very low neurite contrast, it is possible to switch to manual delineation.

Cell Images

The images used for the validation of our neurite tracing technique were taken from a previously published study (23) into the involvement of the neuronal sec1, or murine unc18a (nSec1/Munc18a) protein in neurite formation. In brief, this study was carried out in culture by overexpression of nSec1/Munc18a and of Munc18b in PC12 cells followed by nerve growth factor differentiation, and in primary hippocampal neurons prepared from postnatal day-zero rats. For details regarding the experimental procedures, we refer to the mentioned paper. Fluorescence images of the cells after immunostaining for the transfected proteins (in the case of GFP-tagged proteins, their intrinsic fluorescence was used) were acquired by using a 10× objective on an Axioplan microscope (Carl Zeiss, Germany) in combination with a DC100 digital camera (Leica Microsystems, Germany). For many cells, to increase the field of view, multiple images were taken to capture all the cell's outgrowth. In these cases, composite images were created by matching the individual images visually, using Photoshop (Adobe Systems, San Jose, CA). From a total of about 170 cell images, we selected, prior to our validation experiments, 20 representative sample images, one of which is shown in Figure 1. Per image we selected a single neurite or neurite segment of interest. The neurites were chosen to include all potential sources of tracing errors, such as varying noise and contrast, background intensity discontinuities, and complexities in shape.

Validation Setup

The neurite tracing technique was implemented in the Java programming language (Sun Microsystems, Santa Clara, CA) in the form of a plug-in for ImageJ (National Institutes of Health, Bethesda, MD), the computer-platform independent public domain image analysis program inspired by NIH-Image. After initial, visual experimentation with the technique applied to a small number of cell images, not included in the sample set used for the validation, the different parameters of the detection and tracing algorithms were fixed to the values given in Table 1. As a representative instance of the fully manual tracing methods encountered in the literature we used the “segmented line selections” tool of ImageJ. Similar to our semiautomatic technique, this tracing tool requires indicating points on the neurite of interest. Contrary to our technique, however, the points are connected simply by straight lines.

Table 1. Parameters of the Neurite Detection and Tracing Algorithms and Their Values As Used in the Validation Experiments*

Parameter

Value

Description

*

The parameters are listed in the order of their first appearance in the Appendix.

σ

2.0

Gaussian smoothing scale

γ

0.7

Cost components weight factor

w

9

Snapping window size

p

5

Path coordinates postfilter size

s

5

Path coordinates subsampling factor

Four observers (two expert cell biologists and two expert computer scientists) participated in the validation experiments, which consisted of three sessions. The sessions were carried out separately and independently by the four observers. In the first session the observers traced the neurites of interest by means of the usual fully manual method. The start and end positions of the neurites were indicated by cross-hair pointers, the coordinates of which were loaded automatically along with the images. Upon completion, the tracings were stored automatically along with information on the amount of user interaction required. The session was carried out three times to allow for variability analyses. In the second session, also carried out in triple, the observers repeated the experiments, but now using the described semiautomatic neurite tracing technique. For both sessions, the observers were asked to minimize user interaction, as measured by the number of mouse clicks, while at the same time attaining a sufficient degree of tracing accuracy, according to their own visual judgment.

To obtain accurate reference tracings, required for the quantitative analyses described hereafter, high-resolution versions of the original images were generated by scaling up with a factor 10 using a high-quality interpolation method: cubic spline interpolation (24, 25). In the third session, the observers were asked to perform fully manual tracing once again to delineate the neurites of interest in these high-resolution images. The motto for this session was to be as accurate as possible, regardless of the amount of interaction required. After completion of this session, the resulting tracings were mapped back to the original images to yield reference tracings with a precision of 1/10th of a pixel. It will be clear that this approach to neurite tracing is infeasible in practice due to the 100-fold increase in the required storage capacity as well as a substantial increase in user interaction. For this study, however, the extra efforts allowed us to obtain very precise reference tracings, which we believe may reliably be taken as the “gold standard” in the quantitative analyses.

Quantitative Analyses

The accuracy of the tracings from the first two sessions was determined by comparison with the corresponding reference tracings from the third session, using two different error measures. As a first measure, we used the difference in length between the tracing and the actual neurite, as given by the reference tracing, divided by the length of the latter. A value larger or smaller than zero for this ratio indicates, respectively, an over-or underestimation of the true length. While relevant for studies measuring neurite length only, this measure does not, however, give an indication of how well the tracings really follow the neurite, since any two tracings may have the same length and yet be completely different. As a second, complementary measure, therefore, we used the average deviation of the tracing from the neurite, as given by the reference tracing, over the full length. Specifically, this measure was computed by adding the areas spanned by the segments of the tracing and the reference tracing between any two subsequent intersections of the tracings, and dividing by the length of the reference tracing.

For each of the 480 tracings (4 observers × 2 sessions × 3 rounds × 20 neurites), the values for the two error measures were computed by comparing the tracing to each of the observers' reference tracing for the corresponding neurite. For each of the two measures, the resulting four error values were averaged to yield a consensus error. The variability in the consensus errors was computed both per observer and between observers. Denoting by ϵ(o, s, r, n) the consensus error (either length difference ratio, ϵ_{L}, or average deviation, ϵ_{D}) for any observer (o), session (s), round (r), and neurite (n), we computed the intraobserver variability from the three rounds as

The interobserver variability was computed by comparing the consensus errors (again, either length difference ratios or average deviations) from the three rounds of all four observers as

The labor intensiveness of semiautomatic versus fully manual tracing was analyzed by comparing the number of mouse clicks required per tracing.

Finally, the statistical significance of the outcome of each comparison of the two techniques was determined by means of a two-sided paired Student's t-test (26). Specifically, for each of the measures separately, the test was applied to the corresponding 20 pairs of figures resulting from the 20 neurites (n) traced in each of the two sessions (s). Except for the interobserver variability, which combines the effects of all observers, the test was carried out separately for each observer (o). For the accuracy and labor intensiveness measures, the figures per neurite were first averaged over the three rounds (r). The null hypothesis for the test was that the two techniques would give similar results. The probability of this hypothesis being true was analyzed at levels P < 0.05, P < 0.01, and P < 0.001. To assess the overall significance of the different factors involved, we applied a full-factorial univariate analysis of variance (ANOVA), with “sessions” as fixed factor (corresponding to the two techniques being compared) and “observers,” “rounds,” and “neurites” as random factors. The analyses were carried out using the statistical software package SPSS (SPSS, Chicago, IL).

RESULTS

The results of the validation experiments are presented graphically in Figure 3. The graphs show, for each measure of comparison and for each of the observers, the full range and the mean of the observed values, as well as the statistical significance of the differences between the results of fully manual tracing and our semiautomatic tracing technique according to the paired t-test. From the results with the length difference ratio measure, shown in the top left graph, it follows that both techniques tend to underestimate the true neurite length—our semiautomatic technique even slightly more so than the fully manual method. Averaging over all observers, we found that the deviation from the true length was about −1.0% for the fully manual method and about −1.5% for our semiautomatic technique. The differences between the two techniques were not unanimously significant, however, suggesting that overall, our technique performs comparable to fully manual tracing when it comes to measuring lengths only. Indeed, the results of ANOVA indicated that the factor “sessions” was not significant (P > 0.05) for this measure. Nor were any of the other main factors significant. We did, however, find a significant interaction between “sessions” and “observers” (P < 0.01) and between “sessions” and “neurites” (P < 0.001), which suggests that the accuracy of length measurements with our technique is dependent on the user and also on the shape of the neurites being traced, as was to be expected.

All other measures of comparison yielded values larger than or equal to zero, in accordance with their definitions, with smaller values indicating better performance. From the results noted with these measures, presented in the remaining graphs of Figure 3, it follows that our semiautomatic technique outperforms the fully manual method. In virtually all cases, the differences between the two techniques were found to be very significant (P < 0.001) according to the paired t-test. This is in agreement with the results of the ANOVA, which indicated that with the average deviation and labor intensiveness measures, the factor “sessions” had a very significant effect (P < 0.001) on the outcome. Here too, we found significant interactions between “sessions” and “observers” (P < 0.05) and between “sessions” and “neurites” (P < 0.001). In addition, with the latter measure, we found a significant effect for the main factors “observers” (P < 0.05) and “neurites” (P < 0.001), as well as for the three-way interaction between “sessions,” “observers,” and “neurites” (P < 0.001), which confirms that with our technique, the amount of interaction required depends on the user and on the neurites being traced.

The improvement ratio per measure and per observer can be deduced from the mean values indicated in the graphs. Alternatively, computing the ratios on a per-neurite basis and then taking averages over all observers, we found that the reduction in errors with our semiautomatic tracing technique compared to fully manual tracing was a factor 2.6 for the average deviation measure, a factor 6.0 and 23.2 for the intraobserver variability of, respectively, the length difference and average deviation measure, and a factor 2.4 and 8.8 for the interobserver variability of the respective measures. User interaction was reduced by a factor 3.3. To compare, the interaction was a factor 15.5 less than what was required for creating the high-resolution reference tracings.

DISCUSSION

From the validation results, we conclude that the semiautomatic neurite tracing technique described in this article yields a significant improvement over fully manual tracing methods in terms of true neurite centerline representation, reproducibility, and user interaction. In terms of length measurement, however, it follows that, overall, the differences between the two are not significant. Analyzing the tracings afterward, we found that the underestimation of length may be ascribed to the fact that both techniques tend to shortcut strongly wiggling or sharply bending neurites. With fully manual tracing, shortcutting is inevitable when trying to limit the amount of user interaction. With our semiautomatic technique, shortcutting occurs mainly when the contrast is relatively low, which causes shorter “routes” to have a lower cumulative cost. In both cases, the influence on the results is user controllable. The results suggest that the observers were not biased toward either technique in performing the experiments: while showing significantly larger variability, the results with fully manual tracing were on average quite accurate. Note that, incidentally, this latter observation seems to justify the use of fully manual tracing methods in studies involving large numbers of neurite measurements, a result that, to our knowledge, has never been demonstrated before in the literature.

Because our technique employs a so-called global optimization algorithm and second-order image feature analysis, it is very robust against noise, varying or discontinuous background intensities, and varying or even locally diminishing neurite contrast, as illustrated by the sample tracings in Figure 4. As a result, it can be applied to a wide range of images without changing its parameters. The number of parameters to be set by the user is very small. More importantly, the parameters do not in any way represent hard thresholds—the algorithms are free of user-controllable binary decisions. A disadvantage of the technique is that it still requires user interaction, which limits its applicability to images containing manageable amounts of neurites only. Although, for reasons mentioned in the Introduction, it seems impossible to eliminate all user interaction, it remains a challenge to automate the technique further and to investigate its potential for other applications. One possibility to reduce the number of mouse clicks, for example, would be to automatically compute landmarks to serve as input for the optimization algorithm. Since the algorithm appears to have no problems with tracing neurite segments whose orientation changes only gradually from point to point, a logical approach would be to choose high-curvature and branch points for this purpose. This will be a topic for future research.

To facilitate the use of our technique, we have worked the algorithms into a full-fledged software tool. Because of our choice for the specific programming language and image processing environment described earlier, the tool is, in principle, computer-platform independent. By optimizing the implementation of the algorithms we have been able to achieve acceptable processing speeds, even when using a normal personal computer: for an image or region of size 800 × 800 pixels, the preprocessing time required for detecting the neurites and computing their orientations, and the time required for recomputing optimal paths after each mouse click, are both in the order of 1 second with a 2.4-GHz Intel Pentium IV processor. By analogy with its hosting application, we have named our tool NeuronJ. Apart from implementing the necessary ways and means to interact with the mentioned algorithms, the tool also provides the possibility to label and color tracings, to store them to and load them from disk, and to compute and display the dimensions of individual tracings as well as the statistics of measurements on selected or all tracings. At our institutes, NeuronJ has been received very positively and has replaced the labor intensive fully manual tracing tools. The software is freely available for noncommercial purposes from http://www.imagescience.org/meijering/software/neuronj/.

APPENDIX

DETAILS OF THE NEURITE DETECTION AND TRACING ALGORITHMS

In the implementation of our neurite detector, the required second-order image derivatives are computed by convolution with the second-order derivatives of the Gaussian kernel (27). Mathematically this means that if f denotes the image and G the normalized Gaussian, we compute

where * denotes spatial convolution, x = (x, y) denotes the pixel position, and the derivative directions i and j can be x or y. The eigenvectors and eigenvalues are computed in our algorithm, not from the standard second-derivative matrix, but from the slightly modified matrix given by

where α is a parameter whose optimal value will be given in the sequel. Denoting the normalized eigenvectors of this matrix by v′_{i}(x) and the eigenvalues by λ′_{i}(x), we compute them as

where the v_{i}(x) and λ_{i}(x) are, respectively, the normalized eigenvectors and the eigenvalues of the standard second-derivative matrix, or Hessian (28), at pixel position x.

The suitability of the matrix H′_{f} for neurite detection can be appreciated by analyzing the filter implicitly used in the computations. It is a fact from linear algebra (28) that λ_{i} = v · H_{f} · v_{i}, where the right-hand side is equal to f * (v_{i} · ∇)^{2}G, the second-order image derivative in the direction v_{i}. From the relations just given and the orthogonality of the eigenvectors, it follows that, implicitly, the eigenvalues are computed as λ′_{i} = f * h′_{i}, where the h′_{i} are instances of the filter h′ = {(r · ∇)^{2} + α(r^{⟂} · ∇)^{2}}G corresponding to r = v_{i}, which are the directions of maximum and minimum response. The parameter α is chosen in our algorithm such that h′, which happens to be a steerable filter (29), is maximally flat in its longitudinal direction. This translates into the criterion

Working out the details, we find that the left-hand side of this equation is equal to (1 + 3α)∥r∥^{4}/σ^{4}, with σ the standard deviation of the Gaussian kernel, from which it follows that the optimal value for the free parameter is α = −1/3. Visual impressions of the shape of the filter are given in Figure 5.

Based on the eigenvalue analysis described above, the algorithm assigns to each pixel in the image a measure of “neuriteness” according to the formula

where λ is the larger in magnitude of the two eigenvalues and λ_{min} denotes the smallest λ over all pixels, which in practice will always be smaller than zero. In addition, the algorithm stores for each pixel the local ridge orientation as indicated by v, the normalized eigenvector corresponding to the smaller absolute eigenvalue. Note that dark line-like structures, for which λ ≥ 0, are ignored by the detector. Also note that since we are using a second-order detector, responses to first-order structures, such as background intensity discontinuities, are suppressed. Moreover, since the Gaussian kernel involved in the computations is a scalable function, with scale parameter σ, the detector can be tuned to neurites of specific width, while at the same time suppressing noise.

In our implementation of the live-wire segmentation algorithm, the function used for computing the cost of moving from any pixel x to an eight-connected neighboring pixel y is computed as

where γ∈ [0, 1] is a user-defined parameter determining the relative weight of the two normalized cost components, C_{λ} and C_{v}. The former of these is computed from the eigenvalues at y as

The second component is computed from the eigenvectors at x and y as

where ϕ(x, y) = |v(x) · d(x, y)| with d(x, y) = (y − x)/∥y − x∥ the unit “link vector” from x to y. By taking the absolute values of the inner products, the algorithm ignores the actual directions of the eigenvectors and considers only their orientations. Optimal paths based on this cost function are computed by a version of Dijkstra's shortest-path algorithm (30) based on a discretization of the cost values and the use of a circular priority queue (19, 20, 22). The placing of starting points is facilitated by applying “local snapping,” which implies that the cursor is always moved to the locally lowest cost point in a small window, of size w × w pixels, around the actual cursor position. The smoothness of the paths computed by the algorithm is improved by application of a uniform postfilter, of size 2p+1, to the subsequent coordinates of the path pixels, after which they are subsampled with a factor s.