Multi-scale features for identifying individuals in large biological databases: an application of pattern recognition technology to the marbled salamander Ambystoma opacum

Authors

  • Lloyd Gamble,

    Corresponding author
    1. Department of Natural Resources Conservation, University of Massachusetts, Amherst, MA, 01003, USA; and
      Correspondence author. E-mail: lloydgamble2@gmail.com
    Search for more papers by this author
  • Sai Ravela,

    1. Physical Oceanography Group, Program in Oceans, Atmosphere and Climate, Earth Atmospheric and Planetary Sciences, 54–1624, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
    Search for more papers by this author
  • Kevin McGarigal

    1. Department of Natural Resources Conservation, University of Massachusetts, Amherst, MA, 01003, USA; and
    Search for more papers by this author

Correspondence author. E-mail: lloydgamble2@gmail.com

Summary

  • 1Capture–mark–recapture (CMR) studies provide essential information on demography, movement and other ecological characteristics of rare and endangered species. This information is required by managers to focus conservation strategies on the most relevant threats and life stages, identify critical habitat areas, and develop benchmarks for measuring success in recovery plans. However, CMR studies have been limited by individual identification methods that are not effective or practical for many types of organisms.
  • 2We develop a pattern recognition algorithm and photo-identification method that uses photographs taken in the field to identify individual marbled salamanders (Ambystoma opacum), using their dorsal patterns as ‘fingerprints.’ The algorithm ranks all images in a database against each other in order of visual similarity. We couple this technology with a graphic user interface to visually confirm or reject top-ranked algorithm results. Using this process, we analyse all adult salamander captures from one year of a long-term study.
  • 3In a database of 1008 images, the algorithm identified 95% of 101 known matches in the top 10 ranks (i.e. top 1% of all images). Time spent on manual elements of the matching process was estimated at one minute per image, permitting full indexing of all capture records.
  • 4Capture histories constructed from matched images identified 366 individuals that were captured between 2 and 5 times. Of these, less than 2% were captured at more than one of the 14 pond basins included in the study, suggesting that migrations were strongly directional to and from basins and that ‘pond-shopping’ among first-time breeders was infrequent. Females arrived at basins later, remained longer, and experienced more weight-loss than males during the breeding period.
  • 5Synthesis and applications. We develop, test, and apply a pattern recognition algorithm that enables efficient identification of individual marbled salamanders in a database exceeding 1000 images. We expect that this algorithm can be modified to facilitate individual identification in many other organisms because it does not rely on manual coding or discrete geometric pattern features. High performance results suggest that it can be scaled to larger databases, allowing biologists to address critical conservation-based questions regarding demography, reproduction and dispersal of rare and endangered species.

Introduction

The development of effective conservation strategies for rare, threatened, and endangered species requires unbiased and precise information on their life history requirements and population ecology. For example, a recovery plan for a declining amphibian species may require a population viability analysis, which in turn depends on robust estimates of survival, fecundity, dispersal and how these parameters vary over time and space. CMR studies enable researchers to identify specific groups (i.e. ‘cohorts’) or individual animals over time, and are essential to address these types of questions. Though some questions can be investigated using cohort marks (e.g. ‘what percentage of animals emigrates beyond a particular distance?’), many others depend on the ability to identify individual animals. These include questions related to individual growth and survival (including age- or size-dependencies), dispersal and movement ecology, and reproductive strategies.

CMR studies typically use techniques in which animals are physically marked or tagged (e.g. with ear tags, PIT tags, leg bands, shell notches or dyes; Nietfeld, Barrett & Silvy 1994; Williams, Nichols & Conroy 2002). These methods are intrusive to varying degrees and in some cases may affect the fate and/or behaviour of the animals being studied (e.g. Powell & Proulx 2003; McCarthy & Parris 2004). In addition, some taxonomic groups like amphibians may be difficult to tag efficiently in large numbers (e.g. due to small body size or constraints of field conditions) or may not retain marks or tags long enough to be useful (Davis & Ovaska 2001). Alternative identification techniques that overcome some of these limitations are needed.

Numerous efforts have been made to identify individual animals using photo-identification methods (e.g. Hammond, Mizroch & Donovan 1990; Beck & Osborn 1995; Arzoumanian, Holmberg & Norman 2005; Karlsson et al. 2005). Individuals are photographed and later identified through visual matching, sometimes aided by one of the following strategies to narrow the scope of manual searches: (1) coding of distinctive features (e.g. fluke patterns and scars in whales) into a searchable database; (2) manual or semi-automated extraction of morphometric information (e.g. dorsal fin ratio); or (3) analysis with pattern recognition algorithms. The first approach made searches feasible in catalogues with thousands of whale images; however, in performance tests only 57% of known matches were identified in the top 5% of database-ranked images (Mizroch, Beard & Lynde 1990). This approach has also been used to identify salamanders in relatively small databases (e.g. Stenhouse 1985). The second approach has been applied to dolphins and fish (e.g. Araabi et al. 2000), but is limited to cases where clear and variable morphometric information is available. Published accounts of the third approach using pattern algorithms have been mostly limited to large-bodied animals (e.g. whales, cheetahs) and small test data sets, though algorithms have also been developed for use in another salamander application (L. Hiby, unpublished data). These methods have used 3-D modelling of the animal's surface (Hiby & Lovell 2001; Kelly 2001; Karlsson et al. 2005) or the presence of discrete geometric features such as spots (Arzoumanian, Holmberg & Norman 2005) to extract useful matching information.

We develop a new algorithm that assesses attributes of a patterned surface (in this case, a salamander's dorsum) at multiple resolutions. A numerical representation of this multi-scaled pattern is then compared to that of other images, ranking all images in a database against each other by level of visual similarity. We believe that this approach offers many new opportunities in conservation-related research by: (1) improving recognition performance substantially, and thus increasing the spatial and temporal scales at which capture-recapture investigations can be conducted; and (2) extending the reach of pattern recognition algorithms to a greater variety of patterned organisms, in particular because it does not rely on discrete geometric pattern features such as spots or stripes.

We describe an application of this method, in which we identify individual adult marbled salamanders (Ambystoma opacum Gravenhorst 1807) from a database of 1008 images. Marbled salamanders are primarily terrestrial salamanders, characterized by a black and white ‘marbling’ pattern (Fig. 1) that appears to be individually unique and stable over time (Graham 1971). Marbled salamanders occur in upland and floodplain forests across much of the eastern United States, and migrate to seasonal pond basins in the late summer and early autumn, where they court and deposit eggs in terrestrial nests (Petranka 1998). The eggs hatch into aquatic larvae shortly after inundation by rising pond waters, and metamorphosis and emergence occur in the following summer. Though seasonal ponds provide essential breeding habitats for many amphibians and invertebrates, they receive limited regulatory protection and are considered to be a declining habitat resource (Semlitsch & Bodie 1998). As a result, there is concern that many pond-breeding amphibians, including marbled salamanders, may be declining due to habitat loss and fragmentation (Scott 2005).

Figure 1.

Illustration of three pre-processing steps for marbled salamander images. In the first program (a), the user clicks a series of points along the approximated dorsal mid-line, allowing a second program to straighten the depiction of the animal (b). In a third program, the user clicks four fiducial points corresponding to approximate limb insertions that are used to identify the approximate mid-section (i.e. body-window) of the animal. Five subsequent perturbations in each of four variables (length, width, x- and y-coordinates of centre point) depicted by double-arrows (c) result in the generation of 625 patches that are each rescaled to the size of the original body-window for use in the pattern algorithm.

Our objectives are: (1) to determine whether our new pattern recognition algorithm can be used to distinguish individual salamanders; (2) to describe the advantages and limitations of this approach relative to alternative identification methods; and (3) to demonstrate the utility of the algorithm in a simple ecological application with marbled salamanders. Toward this third objective, we use individual identification to evaluate whether individuals visit more than one breeding site in a particular season, how long they stay in basins during the breeding and nesting period, and how weight changes compare between males and females during this critical life stage.

Materials and methods

field component

As part of a long-term population study at a site in western Massachusetts, continuous drift fences were in place to monitor marbled salamander breeding migrations at each of 14 seasonal pond basins (see Jenkins, McGarigal & Gamble 2003 for detailed field protocols). Pitfall traps (#10 tin cans) were spaced every 10 m on both sides of each fence to capture immigrating (pre-breeding) and emigrating (post-breeding) adults. Previous analyses showed that drift fences were effective at capturing marbled salamanders with capture probabilities between 0·83 and 0·91 for immigrating adults and between 0·52 and 0·85 for emigrating adults (Gamble et al. 2006).

We checked traps once daily from 24 April to 22 November 2002, and continuously during large breeding migrations (n > 20 individuals per night). Upon capture, adult marbled salamanders were weighed to the nearest 0·25 g, sexed and photographed. To standardize image acquisition and optimize image quality, we designed a weatherproof, open-top light-box with a coloured grid background and a circuit of six white LED bulbs. The bulbs were powered by three rechargeable AA batteries held in an externally mounted casing. Individual salamanders were placed in the box and photographed at ‘medium’ resolution (640 × 480 pixels) with one of two hand-held Sony Mavica digital cameras (Model MVC-FD83). No anaesthetization was required. We recorded image numbers in field books and transferred image files to a central database weekly and after significant migration events. All captured animals were released on the opposite side of the drift fence (i.e. in their direction of movement) after data collection. During the off-season, we closed traps and created numerous openings along fences to allow passage of animals.

matching images

Our objective was to develop an algorithm that queries a database of images and ranks different images of the same individual (i.e. matches) above those of other individuals. To accomplish this objective, we must first identify the visual features that lend themselves to correlation. Second, we must factor out any ‘nuisance’ variables that may prevent a monotonic relationship between high-correlation and high-visual similarity. There are six nuisance variables, of which the first four relate to geometry and the last two relate to illumination. We marginalize the effects of these variables through a set of image pre-processing steps applied to all images.

pre-processing images

The first nuisance variable is pose. Salamanders have flexible bodies and their size and shape can vary due to growth, recent feeding and/or their degree of hydration. In addition, without use of an anaesthetic, it is difficult to get a salamander to maintain a straight pose. We address pose variability by digitally marking the dorsal midline of the salamander in database images with a series of points (identified with mouse-clicks; Fig. 1a) and interpolating between these points. The result is a smooth curve that outlines the shape of the animal's pose. We then re-map this curve to a straight line along the x-direction in a length-preserving manner. To do this, at each pixel along the interpolated curve, we extract a narrow image strip perpendicular to the local tangent direction. We then reassemble this strip on the straightened medial axis. The result of repeating this cut-and-paste operation along the entire length of the curved medial axis is a salamander body that is ‘straightened’, albeit artificially (Fig. 1b).

The second nuisance variable is rotation. We re-orient the ‘straightened’ salamander image so that the head is pointing to the right.

The third nuisance variable is location. Once we have straightened and rotated the salamander body, we need a local coordinate system by which we can refer to all pixels on the body. This coordinate system will allow us to match salamanders in a translation-independent manner. To accomplish this, we digitally mark each image with a set of fiducials, or reference points, where the legs emerge from the body (Fig. 1b). The mean of these fiducials becomes the body-centre of the new coordinate system with its x-axis positioned horizontally and the y-axis vertically.

The fourth nuisance variable is scale. Because we have already centred the animal, scale variation can be handled by normalizing the dimensions of the local coordinate system. We achieve this by using the four fiducial points to define a body-window (the approximate mid-section of the body from which we extract pattern information; Fig. 1b) and rescaling the salamander image using bicubic interpolation so that the body-window has a fixed length and width. This rescaling addresses dimensional changes that may occur in an individual over time (e.g. due to growth or weight change) as well as from other factors relating to the imaging geometry or the animal's pose.

With these steps in place, we have images of salamanders that are straight, with the head pointing right, normalized in length and width, and centred at a fixed location. However, because the manner in which we determine the body-window is based on digital marking, it is not precise. To account for this uncertainty, we perturb each of the four variables defining the body-window (centre coordinates in x-axis and y-axis, length and width) by a random value drawn from a normal distribution with 5 pixels of standard deviation (Fig. 1c). Each statistically perturbed rectangle is called a patch, and the values of its image pixels are resampled using bicubic interpolation. Because the perturbation of length and width of the body-window can produce different dimensions from the original body-window, we scale back the resampled patch to the original body-window dimensions, again using bicubic interpolation.

As a result of these steps, each image i, has a representation

image

of M patches, where inline image is the patch (in ith image corresponding to the jth perturbation) written as a vector. This vector is obtained by arranging the pixel brightnesses along a raster-scan of the patch. Because all patches are of the same size, each vector is of the same fixed length. With five perturbations each in length, width, x-position and y-position, there are M = 625 patch-vectors for each image.

The fifth nuisance variable is illumination, which can vary among images due to changes in ambient lighting and the surface conditions of the animal. To some degree we minimize this variability with the use of the customized light-box. We further compensate by contrast-normalizing the patch-vectors. In the case of marbled salamanders, we think that much of the useful information is in the brightness variability and not colour. Therefore, we convert the patches to grayscale and apply contrast normalization to the grayscale patches using the formula:

image( eqn 1)

where µi,j is the average brightness of the patch-vector inline image, and σi,j is the standard-deviation of brightness of the patch-vector. Note that both the average and standard-deviation are scalar quantities. Thus, the modified representation of an image is a set of normalized patch-vectors, inline image. The representation Ri can be interpreted as a matrix whose columns are vectors constructed from the brightness of patches extracted from the images.

The last nuisance variable is specularity. Images of a smooth, wet animal tend to contain numerous spots where light reflection may obstruct the underlying pattern of the animal. The distribution of these spots depends on the animal's position relative to the camera and the strength of the light source, thus motivating the design of the light-box for image acquisition. Our algorithm does not explicitly compensate for remaining specular effects, but deals with them implicitly as the subsequent discussion will show.

outline of the recognition algorithm

Having marginalized the effects of these nuisance variables, we compute the numerical distance between patches using Principal Component Analysis (PCA). This approach has been successful in a variety of visual recognition problems, most notably for face recognition (Turk & Pentland 1991). PCA can be viewed from the perspective of a Mahalanobis distance (Mahalanobis 1936) between two normalized patch-vectors

image

and inline image corresponding to images m and n, and written as:

image(  eqn 2)

where Lmrns is simply the distance between the rth patch of image m and sth patch of image n. The matrix C−1 is called the cost or information matrix. In the PCA approach, this is computed by (pseudo-)inverting C, which is the covariance of the population of patch-vectors. The analytical population covariance is unknown, and is empirically determined from population samples. A reduced rank approximation of the sample covariance is obtained by selecting few principal components that capture the most dominant modes of variability in the population (See Appendix S1 in Supplementary material for a detailed derivation).

The basic recognition algorithm then proceeds as follows:

  • 1Compute the representation R0 of the reference (or ‘query’) image, numbered 0.
  • 2For each database image: i = 1 ... N
    • (a) Load the representation Ri, computed in advance.
    • (b) Compute the distance L0ris of each of the s = 1 ... M normalized patch-vectors in Ri with every normalized patch-vector r = 1 ... M in R0 using equation 2. See Appendix S1 for details on how this computation is implemented efficiently.
    • (c) Assign the minimum distance between elements of Ri and R0 to the score O(i).
  • 3Sort the vector of scores O in descending order and return the top Q images to the user.

Simply stated, this algorithm finds the best match between the normalized patch-vectors computed for the reference and database images. Unfortunately, in this current form, its performance results are limited for the individual salamander identification problem (e.g. less than 40% of known matches are retrieved in the top 10 ranks). The primary reason for poor performance is that the image patch, when represented as discussed above, does not sufficiently discriminate between (a) variability arising between differences in patternation between individuals, and (b) variability in pose or illumination of the same individual across different captures. Numerical differences between patches can come from either source, and thus far, there is no explicit mechanism to factor one source from the other.

To improve performance, we use a multi-scale representation. This approach stems from the observation that visual information is contained in a patch at several scales. For example, human visual inspection of a salamander image quickly reveals the presence of both coarse and fine structures (i.e. visually discernible markings such as the edges, lines, shapes and their spatial distribution). We are able to easily match coarse structures in one image with those in another and separate their contributions from similarities between finer structures. The original algorithm has no separate representation for things coarse or fine. Thus, two images of different salamanders differing in their coarse structures may appear numerically more similar than two images of the same salamander with remnant nuisance artifacts at the finer scale. By decomposing the image along this scale dimension, we essentially inspect the image from different perspectives. Measuring numerical similarity or dissimilarity between all of these views allows the algorithm to discriminate more effectively.

To accomplish this, we adopt a formalism called the Gaussian scale-space (Witkin 1983; Lindeberg 1994; Ravela & Manmantha 1999). The idea behind this approach is that if we take a sharp image and blur it, fine-scale structures disappear leaving behind coarse-scale structures. Generating a family of blurred images from a single image produces a multi-scale representation. More formally, starting with an image I0, we generate a sequence of images It (t = 1 ... T), where

image(eqn 3)

The operator inline image denotes convolution and inline image is the two-dimensional normalized
isotropic Gaussian parameterized by inline image, the position coordinate, and σt, the standard deviation at the tth scale. The blurred images are generated at monotonically increasing scales (or standard deviations): σ1 < σ2 < ... < σT.

We produce a multi-scale representation by generating normalized patch-vectors for each blurred version of the input image. Denote inline image as the jth patch-vector for image i blurred with Gaussian of standard-deviation σt and inline image as the feature vector obtained by concatenating patch-vectors at each scale. The representation for image i can now be rewritten as inline image. The multi-scale version of the algorithm is essentially the same as the previously described version but substitutes the feature-vector based representation where patch-vectors were previously used. We generate image representations at eight scales in half-octave steps: inline image when applying the algorithm to evaluate performance on the database in this paper. The algorithm was implemented in Matlab (version 6·5) with the Image Processing Toolbox (Matlab 2002).

evaluating algorithm performance

To test the performance of the pattern algorithm, we manually identified 101 pairs of known matches from the full set of 1008 salamander images. The manual identification of matches for testing was facilitated by comparing small subsets of images (e.g. immigrating captures to emigrating captures at the same basin) where matches were most likely to be found. We included all manually identified matches regardless of apparent image quality. We then imbedded this test set into incrementally larger random sets from 200 to 1008 images, ran the pattern algorithm, and plotted the percentage of known matches identified in the top 5 and top 10 ranks.

Upon completion of testing, we designed a graphic user interface (GUI; Appendix S2) to read the algorithm output files and display each reference image with its top ranked ‘candidate’ matches. A visual review of the top 10 candidate images in the GUI allowed us to definitively confirm or reject potential matches and assign an individual identification number to each set of matched images. There were no cases in which visual confirmation was ambiguous. The individual identification number was then linked back to the original biological field data using the shared field value for the image file name. We estimated the time spent on pre- and post-processing steps as the total amount of active keyboard time devoted to each step (e.g. identifying fiduciary points with mouse-clicks) divided by the total number of images processed.

biological application

We used capture histories compiled from the pattern matching process to quantify the total number of captures per individual and their locations (i.e. recaptured at the same pond basin as original capture or at a different basin), as well as (1) the timing of arrival at pond basins, (2) the duration of stay at pond basins, and (3) weight-change during the breeding period, calculated as a percentage of ‘pre-breeding’ weight (wet mass measured upon first capture). We calculated these three variables only for individuals captured twice – once during immigration and once during emigration. We then used scatter plots to identify possible correlations between each pair of the three variables, both at the pond-level (for ponds with n > 10 individuals) and the full study area level (pooling individuals across ponds). Because pond-level observations were generally consistent with study area-wide observations, we reported only the pooled results for purposes of this analysis. Note, our data represented a nearly exhaustive sample of breeding salamanders at all pond basins in a limited geographical area. For this reason, our inference space in this application was technically limited to our study area and inferential analyses were not appropriate.

Results

image analysis

We collected 1008 images of marbled salamanders in the field, each corresponding to a single capture event. In the analysis of test matches imbedded in the full data set, the pattern algorithm ranked 95% of known matches among the top 10 images (i.e. in the top 1% of the data) and 70% were returned as the top-ranked image (see Fig. 2 for examples of algorithm retrievals). All of these known matches were positively identified in one iteration of viewing in the GUI. Four of the five known matches not retrieved in the top 10 (i.e. ‘missed matches’) were ranked in the top 5% of the data. The algorithm's absolute performance gradually decreased (e.g. from retrieving 98% to 95% of known matches in the top 10 ranks) as the size of test data sets increased from 200 to 1008 images (Fig. 3); however, relative return rates (e.g. 95% of known matches retrieved in the top 1% of the data set) were maintained in the larger queries.

Figure 2.

Top three ranked retrievals from the pattern algorithm for three reference images; (m) indicates visually confirmed matches to the reference image. Patterns were sufficiently varied that in no case was the visual confirmation or rejection of a match ambiguous.

Figure 3.

Performance of the pattern algorithm in identification of 101 known matches imbedded in incrementally larger test sets from 202 to 1008 images. When the test set included only reference images and their matches (total of 202), 99% of matches were identified in the top 10 ranks (solid line). When the test set was expanded to include the entire data set (n = 1008 images), 95% of known matches were retrieved in the top 10 ranks. The same performance criteria are displayed for the top 5 ranked images (dashed line).

biological application

We identified 447 matches from the 1008 total captures during the field season, representing at least 366 individuals that were captured more than once. The majority of these individuals (n = 345) were captured twice (once each during immigration and emigration), of which 159 were females and 186 were males. An additional 20 individuals were captured 3 or 4 times and one individual was captured 5 times. Repeat captures for all but 6 individuals occurred at the same pond basins as the original capture; the 6 exceptions occurred at pond basins separated by 100–400 m. There were 242 images for which no matches were identified, representing individuals that were captured only once (i.e. passed a fence without detection or died in a basin during the breeding period) or for which an image match was missed.

On average, females arrived at pond basins a week later (Julian date 247 ± 8 days [mean ± 1 STD]) than males (Julian date 240 ± 5 days), but the modal immigration date (Julian date 241) was the same for both sexes. Females remained in basins for an average of 13 more days than males (females, 40·3 ± 9·3 days; males, 26·6 ± 8·0 days), and lost twice as much body weight through this period (females, –25·9 ± 8·9%; males, –11·9 ± 12·9%). The duration of stay for females was not strongly correlated with the timing of arrival at the pond basin; however, males that arrived at pond basins later were more likely to stay more briefly than those who arrived earlier (Fig. 4). No strong correlations were evident between percentage weight-change and either immigration date or number of days in basin for either males or females.

Figure 4.

Duration of stay for male marbled salamanders plotted against date of arrival at pond. Males that arrived later stayed for a shorter period of time (least squares regression: no. of days in basin = 260·5 – 0·97*arrival date, R2 = 0·41).

Discussion

image analysis

Our results suggest that a semi-automated process using pattern algorithms can provide an effective alternative to more intrusive techniques for identifying individual animals. Using our recognition algorithm, we thoroughly cross-referenced a database containing 1008 images into capture histories for several hundred individual marbled salamanders while minimizing the risk of missing a match for any one individual to less than 5%. This probability was further reduced because (1) we evaluated the top 10 ranks of every image independently (e.g. a match between images A and D could be identified with either image as the reference image), and (2) the effective search intensity for any given individual increased significantly with each additional match identified (because each match is also queried). Our semi-automated approach was time-efficient as well, even though visual confirmation of matches was ultimately necessary. In this study, the manual element of pre-processing steps took less than 30 s per image and visual post-processing using the GUI required approximately 35 s per reference image (for 10 comparisons) for a total of roughly 18 h of work to process 1008 images. None of these steps required more than a few minutes of training for a new user. Computer processing time was more significant, requiring approximately 4–5 min per image (on a 3·2 GHz processor) for straightening, patch extraction and ranking; however, these steps could be run in large batches without supervision required. The equivalent analysis of these data through unaided manual matching would have required nearly half a million comparisons of paired images, or roughly 500 h of work at an estimated 3·5 s per comparison. For this reason, manual matching approaches have been feasible only in small-scale studies where sample sizes are low (e.g. Stenhouse 1985), or in cases where individual pattern markings are amenable to categorical coding (Gill 1978). Clearly, the use of pattern algorithms in cases such as this makes larger image databases analytically accessible.

From a performance perspective, the multi-scale algorithm compares favourably to other published algorithms used to identify individuals in wildlife applications. For example, Kelly (2001) applied a 3-D pattern mapping approach developed by Hiby & Lovell (1990) to images of cheetahs Acinonyx jubatus and reported that 93·5% of known matches had similarity coefficients above 0·45 (n = 1000 comparisons); however, the percentage of non-matches above this threshold was not reported and low-quality images were excluded from the analysis. More recently, Arzoumanian et al. (2005) modified algorithms designed for the recognition of star constellations to identify individual whale sharks Rhincodon typus and reported 92% success in identifying 27 known matches from sets of approximately 450 images. Estimates of time spent on pre-processing elements with each of these approaches were 1–3 min (Kelly 2001) and 10 min (Arzoumanian, Holmberg & Norman 2005), respectively, and both resulted in significant time savings as compared to manual methods.

We contend that our approach has at least two advantages over previous approaches used for individual identification. First, the method we have developed augments a fairly classical method, PCA, with multi-scale features, and doing so produces better recognition performance. Because nuisance variables such as specular reflections affect finer scales more than they do coarser scales, separating images along multiple scales provides a more robust estimate of the differences between individuals. Second, our approach does not depend on discrete geometric features such as spots or bifurcations in striped patterns, and therefore can probably be extended to a broad variety of organisms and patterns (e.g. leopard frogs, wild dogs, numerous moths and butterflies). Recently reported techniques for identifying individual whale sharks effectively use geometric information extracted from triads of points on each shark (Arzoumanian, Holmberg & Norman 2005); however, such distinctive pattern features are often unavailable.

It is important to note that, in the case of our study organism, the patterned dorsum provided a relatively planar surface, and it was both feasible and advantageous to control conditions of image acquisition (e.g. lighting, angle of camera) using a light-box designed for field use. For these reasons, straightening each animal's pose in a 2-D fashion (see Methods) provided adequate standardization to obtain the desired performance from the pattern algorithm. In many wildlife studies, however, obtaining images from multiple angles and in less controlled conditions may be unavoidable (e.g. field studies of large mammals). In these cases, the multi-scale approach may pair nicely with a 3-D rectification of field images to optimize recognition results. Our experience suggests that a trade-off exists where efforts are directed to standardize image data (i.e. whether during and/or after acquisition). The optimal solution in any case will depend on the study organism and constraints of data acquisition; however, careful consideration and testing of these issues in the study design phase is critical.

There are, of course, limitations to a visual recognition-based approach; most notably, the dependence on some permanent and ‘extractable’ pattern information (Ravela & Luo 2000). Marbled salamanders, for example, lack distinctive patternation as juveniles and thus require different identification methods in pre-adult stages (Ott & Scott 1999). Numerous other animals have been identified manually based on natural patterns, including other salamanders (Gill 1978), frogs (Kurashina et al. 2003), lizards (Perera & Perez-Mellado 2004), fish (Persat 1982), and larger-bodied animals such as elephants (Whitehouse & Hall-Martin 2000) and giraffes (Foster 1966). The investment in an algorithmic approach becomes essential in cases where sample sizes are large, making manual approaches labour-prohibitive. These cases are probably most common with smaller-bodied animals.

There are also analytical limitations presented in the interpretation of these data with regard to errors. Errors of omission (i.e. ‘missed matches’) can occur if a matching image is not ranked highly enough to be identified in the GUI, or if the image is highly ranked but is not recognized by the observer. Either case, if unresolved, would result in a fragmented capture history similar to one resulting from loss or misreading of a tag (see Rotella & Hines 2005). In our analysis, all known matches occurring in the top 10 ranks were positively identified in one iteration (but see Grant & Nanjappa 2006); however, in the analysis of a larger image database by several observers, occasional misses were noted (L. Gamble, unpublished data). A different type of error can occur when an observer incorrectly links two images of different individuals. This type of error is expected to be less frequent, but could be estimated by selecting random samples of identified matches and subjecting them to independent observer verification. As with other methods used for individual identification, investigators should be cognizant of error rates and their potential impacts on the specific analyses being undertaken (Bailey, Simons & Pollock 2004) so that results can be adjusted or qualified appropriately.

biological application

The most important element of the biological application presented here was to demonstrate the successful indexing of a large database using pattern recognition algorithms and to further demonstrate the utility of such an approach for addressing specific biological questions. In the application we describe, the capture histories obtained using the multi-scale algorithm allowed us to evaluate aspects of movement, timing and weight change in the breeding cycle of marbled salamanders. Briefly, of the animals captured and identified on more than one occasion, the majority (94%) were captured twice at the same pond basin, suggesting that migratory movements were strongly orientated to and from breeding sites in this species and that multiple entries and departures from basins were rare. This is in contrast to Trenham's finding that breeding California tiger salamanders Ambystoma californiense often entered and departed basins multiple times within the same season (Trenham et al. 2000), but may be explained by different breeding systems in these two species. It also seems noteworthy that even though our capture data are limited to one season, there were so few captures of individuals at more than one pond basin, particularly given the density of breeding sites and numerous opportunities for captures across our study area. At least 15% of breeding individuals in this season were first-time (i.e. inexperienced) breeders (Gamble, unpublished data); nonetheless, there was little evidence of pond-‘shopping,’ or active distinction among pond basins based on multiple basin visits. One possible exception was an individual female that spent a week in one basin before moving to an adjacent basin for the duration of the nesting period there. Based on consistent directionality and brevity of stays, the other 5 individuals with multipond captures appeared to represent incidental encounters with other pond basins en route to a known breeding site; however, a more extensive and multiyear analysis would be necessary to test these observations.

Little has been reported in the literature regarding the total period of time spent in basins (duration of stay) by males and females or the weight-change associated with this time period, though other studies have noted general nesting time frames (e.g. Paton & Crouch 2002) and early male arrival times relative to females (Noble & Brady 1933; Krenz & Scott 1994). Interestingly, the first significant exodus of males on September 19 (Julian date 262) coincided with the second largest immigration date of females, resulting in nonoverlapping basin visits for many individuals. There was also significant variability in the duration of individual stays, with the majority of males staying between 15 and 35 days and the majority of females staying between 30 and 50 days. Longer stays in females can be attributed to nest attendance (Noble & Brady 1933), which may improve egg survival by reducing predation or fungal infection (Croshaw & Scott 2005). Differences in weight change between males and females is likely explained by oviposition and lack of feeding by females during the breeding period (Kaplan & Crump 1978; D. Scott, unpublished data). Fairly broad variability among females may result from unmeasured variables such as individual differences in clutch size.

applied research and management implications

In this study, we develop a new, multi-scale pattern algorithm and GUI that enabled us to index a database of 1008 marbled salamander images into individual capture histories. This and other recent successes (e.g. Arzoumanian, Holmberg & Norman 2005; L. Hiby, unpublished data) suggest that refinements in pattern recognition technology and improved image acquisition protocols can make photo-identification strategies a more practical alternative to conventional ‘marking’ techniques for many organisms. This has several important research and management implications that we discuss below.

Facilitate individual-based research on a broader diversity of organisms

Because the multi-scale algorithm effectively addresses complex forms of patternation, we expect it can be extended to many other species. Amphibians and invertebrates provide numerous examples of organisms with limited alternatives for individual identification due to small body size, sensitivity to tagging, or need for expediency in handling. We are currently conducting trials with Fowler's toads Bufo fowleri Hinckley 1882 and spotted salamanders Ambystoma maculatum Shaw 1802 to evaluate the adaptability of our approach to these other amphibians.

Resolve limitations of scale in photo-identification studies

With improved performance, algorithm-based approaches can be effectively applied to larger databases; for example, spanning several seasons and/or multiple breeding sites. As a result, investigators can address ecological processes occurring at these larger spatial and temporal scales. As an example, we have exploited the larger database from which these images were drawn to rigorously test assumptions of breeding site fidelity in adult marbled salamanders (L. Gamble et al., in press), as well as to quantify year- and pond-specific survival and breeding probabilities (L. Gamble, unpublished data).

Provide low-impact identification alternatives

A major advantage of image-based identification techniques is the ability to establish samples of marked individuals for study while minimizing the potential impacts of the research on study animals and their behaviours. For example, there is recent evidence that toe-clipping, one of the most commonly applied techniques for marking amphibians, may significantly affect return rates and/or growth in marked individuals of some species (Davis & Ovaska 2001; McCarthy & Parris 2004; but also see Funk, Donnelly & Lips 2005) and can also be limiting due to regeneration (Donnelly et al. 1994). Though Passive Integrated Transponder (PIT) tags and visible implant elastomer (VIE) markers provide alternative methods, both require significant restraint of individuals or the use of anaesthesia, either of which may increase handling time and/or captivity-related stress (Beaupre et al. 2004). Though the image-based method we describe here for marbled salamanders does not avoid the need for capture, individuals can be photographed in seconds, allowing for quick release with minimum handling. This efficiency is also a major logistical advantage in the case of amphibian research, because hundreds or thousands of individuals may routinely be captured during one night of field work (Ott & Scott 1999). In these circumstances, any field technique requiring more than several seconds per individual may be logistically impossible unless applied to a small subset of total captures.

Inform conservation and management actions

Recovery and management plans for rare and endangered species rely on empirically derived parameter estimates for many life history attributes, including vital rates such as survival, factors affecting reproductive success, habitat use and movement ecology, and the significance of individual correlates (e.g. size, age) to each of these attributes. Addressing these and other ecological questions requires effective means for identifying individuals over time and space. For each of the reasons we list above, the identification process we develop provides another important tool to enable or facilitate conservation-based research in each of these areas.

There are several priority topics that, with additional research and development, can extend the usefulness of this approach to other ecological applications. First, we detected some degradation in performance in test databases approaching 1000 images, suggesting the need for additional work to see how the multi-scale algorithms perform in larger databases. Low error rates may not be problematic for many types of analysis (Miller, Joyce & Waits 2002); however, investigators need to be able to quantify error rates and their implications. We are also currently evaluating several options for improved algorithm performance, including online learning and relevance feedback (i.e. using confirmed matches to fine-tune subsequent searches) and alternate metrics of similarity. Pairing the multi-scale algorithm with 3-D models for pattern extraction (Hiby & Lovell 1990) may also improve performance in some cases, extending its applicability to organisms and study designs where standardized image acquisition is not possible.

Lastly, though the disciplines of computer vision and wildlife ecology have historically been disconnected, recent projects indicate potential for new collaborations. For example, Web-based libraries like EUROPHLUKES (Evans 2003) and ECOCEAN Whale Shark Photo-identification Library (Arzoumanian, Holmberg & Norman 2005) enable users to search and upload images of several marine species online, with several research and conservation benefits. Facilitating additional exchanges between scientists in these fields (e.g. with biologists providing image datasets and computer vision experts developing algorithms) can provide fertile opportunities for applied research in both fields.

Acknowledgements

The first two authors contributed equally to this work. We would like to thank B. Compton, A. Richmond, S. Jackson, C. Griffin, S. Melvin, C. Jenkins and B. Timm for valued advice through the duration of this project and for comments on earlier versions of this manuscript. Thanks also to Website and database consultant E. Haugsjaa for design of the GUI and to D. Pepin for assistance with light-box construction. Fieldwork was completed with the able assistance of K. Berger, N. Bovaird, D. Brennan, M. Grgurovic, M. Hey, K. McKenney, K. Montieth, O. Eziokoli, J. Roberts and E. Weymouth. This work was funded by the Natural Heritage & Endangered Species Program of the Massachusetts Division of Fisheries and Wildlife, Sweet Water Trust (Boston, MA), the US Geological Survey Amphibian Research & Monitoring Initiative (ARMI), the Silvio Conte National Wildlife Refuge (Challenge Grant #50181–1-J045A), the Massachusetts Environmental Trust (Boston, MA), and NSF grants CNS-0540248 and CNS-0540259. Handling of animals was conducted under the University of Massachusetts Amherst Institutional Animal Care and Use Protocol #22-02-02 and annual collection permits granted by Massachusetts Division of Fisheries and Wildlife.

Ancillary