MAPHIS—Measuring arthropod phenotypes using hierarchical image segmentations

Animal phenotypic traits are utilised in a variety of studies. Often the traits are measured from images. The processing of a large number of images can be challenging; nevertheless, image analytical applications, based on neural networks, can be an effective tool in automatic trait collection. Our aim was to develop a stand‐alone application to effectively segment an arthropod from an image and to recognise individual body parts: namely, head, thorax (or prosoma), abdomen and four pairs of appendages. It is based on convolutional neural network with U‐Net architecture trained on more than a thousand images showing dorsal views of arthropods (mainly of wingless insects and spiders). The segmentation model gave very good results, with the automatically generated segmentation masks usually requiring only slight manual adjustments. The application, named MAPHIS, can further (1) organise and preprocess the images; (2) adjust segmentation masks using a simple graphical editor; and (3) calculate various size, shape, colouration and pattern measures for each body part organised in a hierarchical manner. In addition, a special plug‐in function can align body profiles of selected individuals to match a median profile and enable comparison among groups. The usability of the application is shown in three practical examples. The application can be used in a variety of fields where measures of phenotypic diversity are required, such as taxonomy, ecology and evolution (e.g. mimetic similarity). Currently, the application is limited to arthropods, but it can be easily extended to other animal taxa.

cases of image analysis are based on image segmentation, that is, accurate separation of the object of interest from the background (Pennekamp & Schtickzelle, 2013), segmentation can be used in the automatic data collection of traits describing whole organisms.With the accumulation of images, we need feasible pipelines for large-scale analyses.Recent developments in machine learning and computer vision increase the promise of efficient and accurate automated processing, and have been used particularly for automated identification of various taxa, for example, butterflies (Wilson et al., 2023) or Foraminifera (Hsiang et al., 2019).
Convolutional neural networks (CNNs) are the front-line methods for dealing with autonomous image processing (Lamba et al., 2019).
Even the popular image analysis freeware ImageJ has recently been updated to include deep learning models (Gómez- de-Mariscal et al., 2021) for a similar purpose.
However, many traits of interest need to be measured on specific parts of a segmented image with a semantic meaning, that is, corresponding to biologically relevant parts.The accurate segmentation of regions, such as organs, and their classification is technically more challenging (Weaver et al., 2020).Although there are already applications capable of segmenting, for example, plant leaves (Ott & Lautenschlager, 2022), there are no applications that would be designed for animals.
Our original aim was to develop an application to extract a variety of traits of mimetic resemblance from detailed images of arthropods taken under a stereomicroscope.As mimicry is often multi-trait (e.g.Pekár, Martišová, et al., 2022;Pekár, Tsai, et al., 2020), the measurements include comparisons of body/ organ sizes, body shapes and colourations (colours and patterns) at different levels of detail (whole body or its parts).Phenotypic similarity in mimicry systems has so far been measured by means of different software and hardware tools: either general image analysis applications, such as ImageJ (e.g. Kelly et al., 2021), or specialised applications for shape and colouration (Libungan & Pálsson, 2015;Troscianko & Stevens, 2015).Our aim was to merge such methods in order to measure traits using a single application and provide more detailed measurements at different levels of detail in a hierarchical manner.
Hence, we developed an application called measuring arthropod phenotypes using hierarchical image segmentations (MAPHIS), which can be used not only to measure mimetic traits but also any morphological trait from arthropod images.Various traits are measured in a variety of ecological and evolutionary studies, as we document here using three examples from different fields.

| APPLIC ATION OVERVIE W
Our MAPHIS application, in its most common use case, allows the user to automatically segment a set of specimen images, extract various measurements from them and then export the results into a spreadsheet format.As a default, the application comes equipped with a U-Net-based (Ronneberger et al., 2015) segmentation plug-in trained to segment arthropods, mainly arachnids and insects.The plug-in consists of a U-Net segmentation network followed by two post-processing steps to further refine the network's predictions.In the contracting path of the segmentation network, a ResNet34 network (He et al., 2016), pretrained on the ImageNet database (Deng et al., 2009) and fine-tuned after training, is used.Shortly, we used about 1400 images of arthropods from seven orders belonging to 162 species.For each, we created a reference semantic mask of body parts (head, thorax, abdomen, four pairs of appendages).We used 1200 images for training and 100 images each for validation and testing.The segmentation plug-in was evaluated on a set of images representing various species of arthropods.
The performance of the developed neural network was evaluated using objective and subjective criteria (Table S1).See Appendix S1 for more details.
The segmentation is semantic and hierarchic-the whole image is divided into 'background' and 'specimen'.The latter is automatically divided into 'body' and 'appendages'.The 'body' is further divided into 'head' (or proximal part of prosoma), 'thorax' (or distal part of prosoma) and 'abdomen', and all 'appendages' are individually marked ('A1-left' through 'A4-right').Each of them is then automatically divided into three parts (proximal, medium and distal).The resulting segmentation can be manually adjusted, and it is also possible to add further body parts (such as wings) or divide existing parts into more segments.
The software stores projects in an open format and utilises a plug-in architecture, making it possible to add custom functionality.This can include, for example, different primary segmentation methods, mask post-processing, scale determination from objects other than the default scale bars, additional measurement types or result analyses.A plug-in can be written either in the software's native Python or a minimalistic Python wrapper can be utilised to use a pre-existing executable (such as a stand-alone segmentation utility) written in any language.
A typical processing pipeline is illustrated in Figure 1.The user first creates a project and imports the images.In this step, the images can be downsized to increase the processing speed.The application can also read the scale bars (if present) to convert pixels to mm.Following the image import, the user selects a segmentation method, adjusts its parameters (if any are available) and applies it to the selected images.This produces a hierarchical mask for each image, with each pixel belonging to a particular node in the region hierarchy.Individual levels of the hierarchy of the segmented mask can then be examined (Figure 2).
The segmentation mask can be manually adjusted using common drawing tools (brush, paint bucket, polygon, knife) available in the application.It is also possible to create a completely manual segmentation, including the definition of a custom hierarchy.A constraint system allows the user to limit any edits to individual regions or subregions of the specimen, reducing the need for any high-precision drawing (Figure 2a).
As a connected step, a reflection mask for each of the input images can be computed.This mask marks the areas in which the image colour information is unreliable because of glare or light reflections, and which should therefore be excluded from any colour or texture measurements.If needed, the reflection masks can be edited manually as well.
The images and their masks can then be used to compute the requested shape, size, pattern, and/or colour characteristics.For a complete list of available measurements, we refer the reader to Table S2.The measurements can be computed for regions at any hierarchy level.The calculated values are displayed in a table and can be exported to a spreadsheet format (.xlsx, or general .csv).
A quick ruler tool is also available for single or multiple ad hoc measurements in pixels or real units (if the scale information is available).

F I G U R E 1
The processing pipeline.
From the input image with a scale bar, the scale information, the reflection map (reflections marked in red) and the initial segmentation (with individual body parts marked in colour) are extracted.The segmentation can then be adjusted, and the selected measurements are computed using all of the information combined.Yellow arrows point to manual adjustments.
F I G U R E 2 (a) Image overlaid with a segmentation mask at the lowest hierarchy level, with an active constraint limiting any edits to the 'appendages' super-region.In this state, the splitting of the bottom right leg into femur, tibia and tarsus can be corrected with just two clicks with a wide brush, as indicated by the dashed circles.(b) Hierarchy tree of the segmentation (some nodes collapsed for brevity).(c) Specimen/background hierarchy level of the same segmentation.(d) Body/appendages level.(e) Individual appendages and body segments level.

| Size measurements
The developed application can be used to estimate a number of length, width and area measurements of the whole body and/or its parts, namely the head, thorax (or prosoma), abdomen and appendages, provided that the image carries the scale information.The precision of the measurements depends on the arthropod position, that is, whether all leg segments are visible and stretched.We show this on an image of Sphodromantis lineola mantis (Figure 3), in which the head, thorax, gaster, antennae and legs were recognised.

| Shape measurements
To estimate the body shape (head + thorax + abdomen, excluding appendages), the application offers two measures: (1) the index of circularity (a scalar quantity), and (2) a body profile (a vector).When a comparison of two groups of body profiles (e.g. two species each represented by several individuals) is required, instead of raw body profiles, it is useful to use aligned profiles in order to correct for different positions of body constrictions.The alignment procedure is detailed in Appendix S1.After determining the body profile, the vector of distances is scaled in both dimensions.This is shown (Figure 4) on an ant-mimicking spider whose profile is registered to the profile of its model.

| Colouration and pattern measurements
The application can estimate two colouration traits: (1) colour and (2) pattern.The colour, expressed in the RGB or the HSV systems, is calculated for each region as an average from its pixels.If the area includes reflections of light, these can be excluded from the calculation.The pattern is characterised by GLCM values (Haralick et al., 1973) calculated for each region (Figure 5).Here, we compare two true bug species, Graphosoma lineatum and Pyrrhocoris apterus.
We can differentiate between these two species automatically on the basis of the quantification of the striped (or non-striped) pattern.To do this, we first segment the specimens and then calculate the 'GLCM Contrast' measurement for the V (value) channel of the abdomen region for several (~10) distances from the (0, 0.25 mm) interval at angles of 0° and 90°.If we plot these values (Figure 5c), we can see clear extrema in the 0° series of the Graphosoma specimens (corresponding to the vertical stripes), but no such feature in the 90° Graphosoma series, or any of the Pyrrhocoris series.
To summarise these results into a single value for each specimen, we can calculate the correlation coefficient between its 0° and 90° series.For the Graphosoma specimens, the correlation values were Other features can also be extracted from these GLCM values, such as the width of the stripes corresponding to the distance of the local maxima in the 0° contrast series.

| DISCUSS ION
The MAPHIS application can be used to extract a number of traits from images-specifically, measurements of body shape, size, pattern and colour, which can be used in a wide variety of research fields, particularly those focused on arthropods.The developed segmentation plug-in works reliably for images of arthropods and generalises well also for unseen images.Thanks to the availability of manual annotation tools, the application can also be used when the segmentation produces bad results or fails.
The segmentation worked well even when the body was asymmetric or bent and when the appendages were curled, missing, short or long.It encountered problems in cases when appendages were detached from the body, or when there were more than four pairs of appendages-for example, when the arthropod had long palps or cerci.These were recognised as legs/antennae and the last pair of legs was then omitted.However, as the available correction tools are easy to use and can be constrained to the problematic regions only, potential problems can be quickly corrected.
Although we do not use segmentation mask corrections for model improvement, they could be used in multiple ways.The most straightforward one would be to extend the training data sets and to use such corrections for retraining the existing model.However, as the obtained IoU values are already high (Table S1), the expected benefit would be limited.A more promising direction would be to incorporate some shape or geometric constraints directly into the model to guide the segmentation process (e.g.Al Arif et al., 2018;Mesadi et al., 2018).Another possible improvement would be to use geometric features to segment higher level sections (e.g. to split leg parts at the bends instead of into regular geodesic length intervals).
The separation of the segmentation and measurement phases in the application is important as it makes the application generally usable even in entirely different domains (e.g. in material sciences).
The reason for this is that even without the automatic segmentation function, the application can easily be used for measuring objects of interest if the annotations are manually created.Such manually created annotations can also be used for training extension plug-ins.
The separation of the phases also helps with the interpretability of the measurements because it is possible to measure only selected subsections of the hierarchy, and the provided measurements are directly interpretable and linked to the regions of interest.Thanks to the reflection masks, it is also guaranteed that the colour and pattern measurements are not influenced by corrupted pixels.
The segmentation accuracy obtained for the test data sets (Table S1) was high enough for the application to be practically usable.Although it was necessary to manually correct some of the test images (33%), the required modifications were only subtle and the time needed for the corrections was negligible as compared to the time needed for sample preparation.The application is trained to segment arthropods pictured from a dorsal view.The angle of view should be close to 90° to minimise measurement errors.In spiders, the proximal part (labelled as head) includes the cephalic part of the prosoma, the palps and the chelicerae, whereas in insects, the head includes the palps and mandibles.
The thorax includes the petiole.In this version, wings are not recognised as separate body regions, but can be segmented manually.
To conclude, although the application in its first version is limited only to arthropods, in the future, we aim to include additional segmentation modules that could be used to process images of other animals, vertebrates and other invertebrates to reduce manual workload that might be necessary for these different images.

F
I G U R E 3 (a) An original image of the mantis, Sphodromantis lineola.(b) Mask of the mantis with differently coloured body parts.(c) Some of the extracted measurements (mm).
photographic systems, such as digital microscopes, can provide such high-quality images.

F
I G U R E 4 (a) The original image of an ant-mimicking spider, Myrmarachne helensmithae, and its model, Opisthopsis haddoni.(b) Median of raw and median of aligned body profiles of the spider (blue) and ant (red) extracted from the masks.F I G U R E 5 (a) Original image of two true bug species, Graphosoma lineatum and Pyrrhocoris apterus.(b) Mask of the two species with light reflections indicated (green).(c) Plots of GLCM contrast values for different distances in horizontal (0°) and vertical (90°) directions (V channel of the abdomen region).