A Visual Tool for the Analysis of Algorithms for Tomographic Fiber Reconstruction in Materials Science

We present visual analysis methods for the evaluation of tomographic fiber reconstruction algorithms by means of analysis, visual debugging and comparison of reconstructed fibers in materials science. The methods are integrated in a tool (FIAKER) that supports the entire workflow. It enables the analysis of various fiber reconstruction algorithms, of differently parameterized fiber reconstruction algorithms and of individual steps in iterative fiber reconstruction algorithms. Insight into the performance of fiber reconstruction algorithms is obtained by a list‐based ranking interface. A 3D view offers interactive visualization techniques to gain deeper insight, e.g., into the aggregated quality of the examined fiber reconstruction algorithms and parameterizations. The tool was designed in close collaboration with researchers who work with fiber‐reinforced polymers on a daily basis and develop algorithms for tomographic reconstruction and characterization of such materials. We evaluate the tool using synthetic datasets as well as tomograms of real materials. Five case studies certify the usefulness of the tool, showing that it significantly accelerates the analysis and provides valuable insights that make it possible to improve the fiber reconstruction algorithms. The main contribution of the paper is the well‐considered combination of methods and their seamless integration into a visual tool that supports the entire workflow. Further findings result from the analysis of (dis‐)similarity measures for fibers as well as from the discussion of design decisions. It is also shown that the generality of the analytical methods allows a wider range of applications, such as the application in pore space analysis.


Introduction
Fiber-reinforced polymers (FRPs) are in high demand for a number of manufacturing industries such as automobile or aeronautics, since they provide high strength, durability, and elasticity while being lightweight. To ascertain and optimize the targeted applicationspecific properties of FRPs, material scientists need to be able to quantify and model the characteristics especially of the reinforcement components, i.e., fibers. Important characteristics of a fiber are, for example, start and end point, center point, direction, length and diameter. A typical method to determine the fiber characteristics is acquiring images of an FRP specimen using X-ray Computed Tomography (CT) [KH18], and then applying customized data processing pipelines to extract the fibers and their characteristics. An overview of such methods as well as on existing analysis and visualization tools is given in Section 2.
In multiple discussions with our collaborators, both developers of fiber characterization algorithms and domain scientists using these algorithms, we found that there is a great need for a specialized tool for the analysis and comparison of algorithms for determining fiber characteristics. Our collaborators typically employ generic visualization tools, custom quantification scripts, written for example in MATLAB or Python, or resort to a manual analysis in specific cases. Each of these methods is tailored to the output format of the analyzed algorithm. Furthermore, these methods require prior domain knowledge, e.g., when setting up specific visualization pipelines in the generic tools, or when developing custom quantification scripts. Finally, each of these methods requires a considerable amount of time to set up, and does not provide inherent possibilities for comparing multiple datasets.
When analyzing the requirements of our collaborators, we found that they require support regarding the following analysis tasks: 1. to compare multiple results, specifically, a. to quantitatively compare average accuracy between results b. to qualitatively compare visual representations of results In an iterative design process we then started designing and developing methods to address these tasks. In each step of the iteration, we discussed prototypes of our methods with our collaborators and let them experiment with these prototypes on their own data. Based on their feedback, we refined our methods. The final prototype implementing these methods is the tool for FIber reconstruction Algorithm (K/)Comparison and ExploRation (FIAKER), described in this paper.
We see our contributions in the methods and tools for analyzing fiber characterization algorithms, which we describe in Section 4. A further contribution is the evaluation of these methods in Section 5, where we analyze some synthetic and real world datasets with our methods, and indicate how they speed up the analysis of fiber reconstruction algorithms. An additional contribution described there is the evaluation of a variety of fiber dissimilarity measures, as well as explanations how to apply our methods in the similar scenario of analyzing pore characterization algorithms. Section 6 provides a discussion of the design decisions and the lessons learned during the design of FIAKER. Figure 1 shows a schematic of pipelines for determining fiber characteristics from data acquired by CT. Conventional pipelines follow a sequential approach. In the CT device, X-rays are attenuated by a specimen (a) to generate projection images from multiple angles (b). From the set of acquired projections, a 3D image is reconstructed (c). Then, a variety of image processing steps are performed to extract and segment the individual fibers (d). In the last step, the characteristics of each segmented fiber are determined (e). Examples of such types of pipelines are presented in the works of Salaberger et al. [SKK * 11,SJKK15]. More recently, techniques for a direct extraction of fiber characteristics from CT projection data have been proposed. For example, Elberfeld et al. [Edd * 18] reconstruct volumes from a small numbers of projection angles and then estimate position, direction and length of the contained fibers using a priori knowledge of their shape, modeled as a geometric representation. This direct step is indicated by the red arrow from (b) to (e) in Figure 1. The estimates are then iteratively improved by minimizing the error that results when comparing with the original projection images. Konopczyńsky et al. [KRR * 18] show that applying deep neural networks also can deliver reasonable results. However, a significant amount of time has to be invested for training these networks on synthetic and real-world data.

Background and Related Work
We refer to the outcome of a single parameterization of a specific fiber characterization pipeline as a result. We use the term ensemble to refer to multiple results from either different parameterizations Figure 1: Schematic of fiber characterization pipelines: A specimen (a) is scanned via computed tomography, the scanned projection images (b) are reconstructed to a 3D volume (c), which is segmented (d). The characteristics of each fiber (e) are quantified either from the segmented data or from the projection images. of the same pipeline or from different pipelines. When developing a new fiber characterization algorithm, one typically starts with a synthetic dataset: a ground truth with a known characterization result, which we will refer to as reference. In the analysis, the reference is part of the ensemble.
The use of visualization techniques to optimize these algorithms can be considered as visual parameter space analysis (VPSA) as proposed by Sedlmair et al. [SHB * 14]. In the typical VPSA scenario, for each result a fixed number of global quality measures is computed. While average measures for a result can also be computed in our case, we require new approaches for analyzing and comparing the single fibers and their characteristics.
For the analysis of a single result, the methods by Fritz et al.
[FHG * 09] as well as by Weissenböck et al. [WAL * 14] can provide insights into the distribution of fiber characteristics. Yet their methods are not suited to analyze details of the algorithms, such as intermediate steps from iterative algorithms. Furthermore, these methods only deal with a single result and cannot help in comparing multiple results or in analyzing the differences between a result and the reference. Zhang et al. [ZFS * 19] provide similar capabilities for the analysis of the formation of bubbles in porous media. They analyze multiple similar pores, in volumetric data acquired via computed tomography. As their method enables the user to compare pores inside a dataset with each other, it is also targeted at analyzing a single result.
The algorithms we analyzed, model the fibers as straight cylinders. For 3D visualization, we therefore create triangle meshes in appropriate detail. Tools such as YMCA by Schmidt et al. [SPA * 14] or DreamLens by Matejka et al. [MGB * 18] are similar as they compare multiple 3D shapes. However, they are not required in our context because we only need the meshes for qualitative inspection and visual comparison. For the analysis of more general polytopes,  explore the concept of 1D slicing. Their methods target the comparison of polytopes with more than three dimensions, and do not provide means to compare additional characteristics associated with these polytopes.
To match the fibers of the different results, we compute their similarity. For this purpose, we employ fiber similarity measures. Such measures are also used in the imaging of white brain matter through diffusion magnetic resonance imaging (dMRI). For example O'Donnel and Westin [OW07] employ a distance based on the mean closest point distance between two fibers. Quan and Vo [QV15] propose a measure based on the center of mass as well as the start and end point of each fiber. Bhattacharya et al. [BWW * 17] employ distance measures for the clustering of fiber tracts of industrial fiber-reinforced materials into fiber bundles. While these measures are designed for clustering similar fibers, we are interested in determining matches of the same underlying fiber in different datasets. In our case, the ideal distance measure therefore must always rank the pair of best-matching fibers first, while for the use cases cited above, the absolute order is not that crucial. Analogously to Labra et al. [LGD * 17], we utilize multiple measures with differing accuracy and performance in order to speed up our computations. While in dMRI (for an overview see e.g., Schultz and Vilanova [SV18]) the input data are tensor fields or more general orientation fields from which the fibers have to be reconstructed with numerical methods, in our case the individual fibers are explicitly given.

Fiber Dissimilarity Measures
For analysis purposes, we need to determine which fiber in the predefined reference matches best a particular fiber in a result. This requires measuring the dissimilarity of fibers in the reference and the result, as shown in Figure 4(c). We considered and implemented a number of different fiber dissimilarity measures: 1. Measures based on differences in the characteristics 2. Measures based on fiber point distances 3. Measures based on fiber overlap The first category measures the dissimilarity through the difference in the fiber characteristics: the fiber center, the direction as azimuth (φ) and elevation (θ) angles in a spherical coordinate system, as well as the length of the fiber. The fiber dissimilarity can then be computed as the Euclidean distance in the 6-dimensional space spanned by these six fiber characteristics. Let us assume we are analyzing a result where for each fiber a we have the fiber center as a position vector ca, its orientation angles φa and θa as well as the fiber length la, as shown in Figure 2(a). With fa = (ca, φa, θa, la) being the 6-dimensional vector of all those feature characteristics, the dissimilarity between fiber a and b is computed aš where 2 denotes the Euclidean distance. This measure is used in the evaluation of the optimizing algorithm by Elberfeld et al.
[Edd * 18]. The problem with this measure is that the components of the vectors have different physical dimensions and are therefore not comparable. A methodically clean way would be the introduction of weight factors.
For this reason, we have also experimented with a normalized measure, in which we form a weighted sum of the Euclidean distance of the center points and the absolute differences of the other characteristics, the weights being used to normalize each term by its maximum possible value, so as not to over-emphasize a single characteristic. This dissimilarity measure is given as Here the coefficient w 1 is the inverse of the diagonal length of the bounding box containing the fibers from all results, w 2 and w 3 are the inverses of the maximum possible differences between two φ and θ angles, respectively, and w 4 is the inverse of the difference between the shortest and the longest fiber in all results.
We also evaluated three dissimilarity measures considering fiber point distances. The first is computed as the sum of the Euclidean distances between start, end and center points of two fibers: where s, c and e denote the start positions, centers and end positions of fibers a and b, respectively.
The measures introduced so far are not invariant to a rotation of one fiber by 180 degrees -meaning that even if fiber b in the result is exactly matching fiber a, but a is specified rotated by 180 degrees with respect to b, these measures will tell us that there is a high difference between them. As the result of structural analyses does not differ for these two cases, they should be considered equal in our tool as well. Therefore, we need to compute these distances twice, once as defined above, and a second time with the start and end position vectors for fiber b exchanged, or, for measures d c 1 and d c 2 , with direction angles φ and θ rotated by 180 degrees. The resulting fiber is denoted as b − . We then use the minimum of the two distances as final measure We always useď as an indicator that the metric d is afflicted by this orientation dependency issue.
In an attempt to overcome this limitation, we analyzed a second measure based on point distances, namely the sum of distances between all pairs of these three points: While this measure is not affected by the direction of the fibers, it does come with the disadvantage of being dependent on the fiber length. It cannot trivially be normalized to correct for this dependency on the fiber length.
We then sought a more mathematically grounded dissimilarity measure and came up with a measure based on the (square of) the average distance of corresponding points along the two fibers a and b. It leads to the measurě We can see that it depends only on the Euclidean distances between start and end points in R 3 , as well as the angle between the vectors sa − s b and ea − e b . This measure is also subject to the orientation dependency issue; therefore the final distance is again computed with the help of Equation 4.
We also considered dissimilarity measures based on volume overlap. To compute them, we follow the procedure schematically depicted in Figure 2(b): We first sample a collection pa of random points inside the cylinder of fiber a. We then count how many of those points are contained in the cylinder representing fiber b. The overlap is then defined as where c(pa, b) denotes the count of points in pa that are contained in the cylinder of fiber b, while #pa stands for the cardinality of pa, i.e., the number of sampled points.
In an attempt to create a symmetric measure, we always sample the fiber with less volume, and check inclusion in the fiber with higher volume, since a cylinder with larger volume can never be fully included in one with a smaller volume. This leads to our first definition of an overlap-based dissimilarity measure as where v(a) denotes the volume of fiber a. This measure might be sufficient for some cases, but it delivers a dissimilarity of 0 whenever one fiber is fully contained within the other one. The contained fiber could however be much smaller than the other. Therefore, we need to accommodate for their volume ratio, leading to a second overlap-based dissimilarity measure We also experimented with the non-symmetric version of this metric, where we always compute the overlap in a specific direction As noted above, these dissimilarity measures are used to compute the dissimilarity of fibers in the results to fibers in the reference result. A low value always indicates low dissimilarity, a value of 0 indicates a perfect match. Most measures are normalized to have a value in the range of [0, 1]. It is explicitly mentioned above if this is not the case. When the reference is set, we perform the dissimilarity measure computations. For each fiber in a non-reference result, we compute the dissimilarity to all fibers of the reference. We then chose a configurable number of "best" matches, that is, those reference fibers with the lowest dissimilarity, and store them for further analysis.

FIAKER
FIAKER provides methods for the analysis of data resulting from fiber characteristics pipelines. The workflow realized with FIAKER is depicted in Figure 4. Input is an ensemble of results, ideally along with a reference result, as described in detail in Section 4.1. The techniques employed to visualize this information are laid out in Section 4.2.

Input Data
The input to FIAKER are the computed characteristics of the fibers in each analyzed result. As the datasets analyzed so far contained only straight fibers, FIAKER visualizes fibers as cylinders.
The cylinder geometry for each fiber is specified by the start and end point as well as the diameter. The tool can also process supplementary characteristics per fiber. All characteristics are expected in table (CSV) format, as shown in Figure 4(a). In addition, FIAKER can handle the fiber characteristics from intermediate steps of an iterative algorithm. As a preprocessing step, the user selects one of the loaded results as reference, i.e., as base for comparisons, as shown in Figure 4(b). The tool then computes for each fiber in each result the dissimilarity to all fibers in the reference. In absence of a true reference, any one of the results can be chosen as a base for comparisons.

Visualization Design
The main interface of FIAKER utilizing multiple linked views can be seen in Figure 3. Its main views are the result list (a) and the spatial view (b), as well as detail views in the form of an iteration step chart (c) as well as a scatter plot matrix (d). Additionally, a selection view (e), an interaction protocol (f) as well as a settings view (g) are shown when required. We recommend viewing the supplemental video for a better impression of the interactions between these views. FIAKER is available as a module in the open_iA tool [FWS * 19].

Result List
The result list contains the name and a 3D preview for each result, as can be seen in Figure 3(a). All fibers in a preview are assigned a color which is specific to the single result. Alongside, aggregate weighted measure bar charts in an interface similar to LineUp [GLG * 13] enable ranking of the results for addressing Task 1a, the quantitative comparison of results, from Section 1. This task is supported in addition by a histogram, showing the distribution of a chosen characteristic. In addition this addresses task 3, the investigation of the influence of input data and parameters. To focus on a specific result, users can filter their analyses in the results view. Only the filtered results will then be displayed in the spatial view, and also the iteration step chart and the scatter plot matrix will be restricted to these results.

Spatial View
The spatial view shows a 3D visualization of all user-selected results, enabling the qualitative comparison of individual results, and thus addressing task 1b. Fibers can be color-coded by the color specific to the result which they belong to. Alternatively, the reference can be color-coded by the average dissimilarity measure value of the matches to that fiber across all results, as shown in Figure 4(d). Or, fibers can be colored by their individual match quality or all other characteristics. These color-codings enable users to identify fibers with specific properties, and thus addressing task 2b. The bars of the histogram in the result list are colored as the fibers and serve as color map. When fibers are selected, the unselected fibers are displayed with increased transparency to provide the context, as shown e.g. in Figure 5(a). For selected fibers, the closest matches with the reference can be shown. To clarify the link between result and reference fibers, connecting lines are shown, as can be seen in Figure 6(a). Task 4, providing visualizations for communicating the results, is addressed mainly through the spatial view as can be seen in the case studies.

Detail Views
Users start navigating in the results list and the spatial view. Additionally, several detail views are available: 1. Iteration step chart, see Figure 3(c): When an iterative algorithm is analyzed, where data from each of its iteration steps is available, this data can be tracked in the iteration step chart. It shows the evolution of selected characteristics on the y-axis over the iteration steps on the x-axis. The user can switch to any iteration step through a control in this view, and the spatial view will update accordingly. In addition, an animation can be enabled that automatically loops through all iteration steps. This view supports task 1a by enabling the comparison between intermediate and final results. 2. A scatter plot matrix, see Figure 3(d), shows the characteristics of all fibers in the currently selected results, revealing potential correlations between the characteristics. Each dot represents a single fiber, and is colored exactly like the corresponding fiber. It supports the analysis of the influence of fiber and dataset characteristics on the results in task 3. 3. Selection details, see Figure 3(e): Whenever a selection is made, its details are recorded in this two-part view. Its left part keeps a history of all selections as a list. The right part displays a tree of all fibers and the results they belong to for the current selection. Selections in general mainly address tasks 2 and 4, where users need to be able to select a subset of fibers for further analysis. In the spatial view, single fibers can be selected by dragging a selection rectangle, which selects all fibers where start or end point are in a view frustum spanned by the rectangle, or by clicking on a single fiber. Fibers can also be selected in the scatter plot matrix and the iteration step chart. 4. The interaction protocol, see Figure 3(f), logs all user interactions. This view, along with the selection details, makes it easy to keep track of the analysis process. Having a reproducible analysis process is crucial when providing visualizations for communicating the results, thus these views mainly address task 4. 5. A settings view, see Figure 3(g), provides access to visualization options such as modifying the opacity of selected and nonselected fibers, as well as specifying which fiber dissimilarity measure to use for finding the closest reference fiber.

Evaluation
FIAKER was developed in close collaboration with scientists working on fiber characterization algorithms. We evaluated the tool on results from two different algorithms. One aspect of our evaluation was the iterative refinement of our methods regarding usability. The most important aspects are found in the comparison of the analysis with our tool to the analysis using existing methods, as well as the kind of insights that can be gained with our tool. We further evaluate the fiber dissimilarity measures, and finally show how the methods can be extended to support the analysis of other datasets, demonstrated by pore space analysis. Where not noted otherwise, the case studies were conducted by the tool developer, and observed and commented on by a domain expert. The domain experts were also provided the tool to play around on their own, and several of our insights in Section 6 came from the discussion of analysis sessions performed by the domain experts on their own.

Evaluated Algorithms
We evaluate FIAKER using synthetic results as well as with results obtained from datasets acquired by computed tomography. These datasets show objects made of fiber-reinforced polymer with glass or carbon fibers. The projection images from the CT device, or reconstructed volumes, that serve as input to the fiber characterization algorithms, are typically several hundreds of megabytes to several gigabytes in size. That is, there are about a hundred to a few thousand projection images of size 2 300 × 2 300 pixels, yielding reconstructed volumes of up to 2 300 3 voxels. Since most of our visualizations don't deal with these image or volume datasets directly, we are not limited by their size. Instead, we almost exclusively use polygonal data extracted from the characteristics computed by the fiber characterization algorithms. Thus, our algorithms are limited only by the number of fibers found. So far, we have tested our visualizations with results containing up to 400 000 fibers.
The datasets that served as basis in the case studies below are the result of applying two different fiber characterization algorithms. The first one, described by Salaberger et al. [SJKK15], starts with a reconstructed volume and applies a sequential pipeline of image processing operations, including template matching. Through this process, the center lines of the fibers are extracted, from which labeled fiber images are created. We will refer to this algorithm as Fiber Characterization through Template Matching (FCTM).
The second algorithm we have been evaluating, Parametric Reconstruction (PARE), has recently been published by Elberfeld et al. [Edd * 18]. It directly inputs the CT projection images, performs an initial fiber characterization from a SIRT reconstruction using the ASTRA toolbox [vPC * 16] and then refines this characterization by projecting the model of the recognized fibers forward (again) into the space of the projection images. The result of this forward projection is used to compute the projection error, i.e., the difference to the original projection images. The characterization is then adjusted to reduce the projection error, using a gradient descent optimization method. This minimization of the projection error is repeated for each fiber separately until no further reduction in the error is noticed, or a maximum number of iterations is reached.

Usability Evaluation and User Centered Design
We developed FIAKER over the course of approximately six months. Designs and prototypes were regularly presented to our (d) Figure 5: Fibers of the synthetic dataset, color-coded by dissimilarity to reference (a); the color-mapping is given in the distribution chart (b). Fibers with the worst reference match are selected in the scatter plot (c). The long vertical fiber in the middle (green) and its development over the iterations is analyzed in the 3D view in comparison to closest reference fibers (grayish-violet) as well as in the iteration steps chart (d).
collaborators on average once per week, and we discussed the direction of further enhancements that we intended to implement. It was thus a thoroughly user-centered design process, following the best practices and avoiding the pitfalls outlined in the design study methodology by Sedlmair et al [SMM12].
We continuously improved the usability of our prototype based on user input. We also conducted an interview, focusing solely on improving usability. The result of this interview was a list of items where usability was lacking in the prototype version available at the time of the interview. One such issue, for example, was that coloring single fibers by characteristics such as their length was available only through the settings dialog of the scatter plot matrix. This restriction resulted from design decisions of previous tools built in the same framework as our prototype, where the scatter plot matrix played a more central role. In our context, that was pretty counter-intuitive. The list of items, together with the suggestions for improvement, was then discussed with the visualization experts among the co-authors, where the implementation details of each improvement where outlined and refined. This resulted in a workable, prioritized list of changes to the prototype that we were able to quickly implement in the next prototype release.
Overall, there were four collaborators involved in the development of the tool who are developing algorithms for fiber characterization or working with such algorithms on a daily basis. Furthermore, four experts on visualizations for material science lended their expertise in designing and improving the visualization techniques.

Case Study 1: Synthetic Data
This case study considers a simple synthetic ensemble consisting of only two results. One is the result of the PARE method, the other is the reference, a result artificially created to approximate the distribution of fibers in real datasets. The projection images, input to the PARE method, were generated directly from this reference. Therefore, the reference in this case is an actual ground truth.
The first goal in this case study is to test our methods on a simple dataset, and also to serve as an introduction into how to apply these methods. After defining the reference, we start by color-coding the fibers obtained by PARE by their dissimilarity to the best-matching fiber in the reference, as shown in Figure 5(a). Red indicates a bad match based on the color map depicted in (b). In (a), we already have selected the seven fibers with the worst match according to our overlap-based measure d o 3 in the scatter plot Figure 5(c). We chose these because they seem to separate slightly from the rest of the fibers. Interestingly, they come from both ends of the fiber length spectrum. It can be seen that two fibers extend beyond the bounding box of the reference, which is shown as a black wireframe box in Figure 5(a). Now let us focus on the long, near-vertical fiber in the middle, to see why the PARE algorithm performs badly for it. We selected this fiber by clicking on it in the 3D view. Figure 5(d) shows its length difference to the best matching reference fiber (Length Diff.) as well as its contribution to the projection error (Proj. Error), the optimization criterion of the iterative PARE algorithm, over the iteration steps. The picture on the left of the charts shows the fiber in the first iteration step. Note that we view it from a slightly different camera position in (a) that is better suited for our further analysis purposes. On the right of the charts we see the same fiber in the last iteration step. From the 3D visualizations and the charts we can see that the projection error decreases, while the difference in length to the closest reference fiber actually increases. In the 3D views, the fibers are colored by their result color, green for the PARE result and grayish-violet for the reference. We can see that there is another reference fiber that is initially only touched by the green fiber, but at the end of the optimization, the start of the reference fiber is completely covered by the green fiber. On its other side, in the course of the optimization the green fiber grows out of the bounding volume of the reference, which does not adversely affect the projection error, because it outgrows the field of view and the projection error can not provide negative feedback for this change. This is actionable information for the algorithm developer. On one  hand, care must be taken to avoid generating fibers which grow to cover more than one fiber in the reference. On the other hand, it shows that reducing the projection error as an optimization criterion is not sufficient and additional constraints are required to prevent fibers from being placed outside the analyzed regions. Previous methods for analyzing this dataset involved scripts to match result to reference fibers, as well as computation of errors for the best matches. In contrast to these, with FIAKER, the fibers going outside of the optimized volume were immediately apparent.

Case Study 2: Measure Validation
To analyze the fiber dissimilarity measures we utilized the synthetic data from the previous section. Figure 6 shows examples where the first developed dissimilarity measures performed badly. For the measures based on the fiber characteristics, d c , we quickly realized that using them would require a fine-tuned balance of weighting for all characteristic difference terms. In the example shown in (a), the length is obviously weighted too high. The reference fiber with the best match is the same size, but slightly different in orientation, and somewhat distant from the result fiber. The best measure considering only positions, d p 3 , for which the best match is shown in Figure 6(b), performs better and shows that there is a reference fiber that overlaps with the result fiber. The large difference in length is the most likely explanation for why the characteristics-based measure did not select this reference fiber as best match. An example where measure d p 3 also does not perform well is shown in Figure 6(c), where we see that start and endpoints of the matching reference fiber are pretty close, but there is no overlap. The best match shown in Figure 6(b), computed via the overlap measure d o 3 , is the true best match, according to our collaborators. We see that there is a lot of overlap, but the length varies consid- erably, which means that one of the endpoints of the reference is quite far from its counterpart on the result fiber.
When analyzing the best four matches according to measure d p 3 , which are shown in Figure 6(e), we see that it includes the best match of measure d o 3 , shown in Figure 6(d).
In the examples evaluated for the synthetic dataset, it is always the case that the best match according to d o 3 is contained in the first few matches of d p 3 . Even though d p 3 does not always give the correct order, it provides a good first estimate of match quality. We can use it to optimize the computation of overlap measures, which is too expensive to be computed for each possible pairing of fibers in reference and result, especially when analyzing a large ensemble or results with a large number of fibers. We therefore only compute the overlap measures for the best 25 matches according to the d p 3 measure.

Case Study 3: Influence of Noise in Synthetic Data
We also looked at the influence of noise on the outcome of the PARE algorithm. From a ground truth, several projection images were generated with different levels of additive Gaussian noise with a mean of 0 and a sigma value of 0, 0.5, 1, 2, 3, 4, and 5. A list of the results gathered for this analysis is shown in Figure 7. We show the difference in length measure as well as the overlap-based dissimilarity measure. Even though we have a separate ground truth available again, we could also consider the 0% noise level as a baseline, as we are only interested in how the noise affects the outcome. The difference in length remains approximately the same until noise with a sigma value of 4 is generated. At sigma value 5, the difference in length begins to increase steeply. The overlap-based dissimilarity measure yields slightly more differentiated results. From noise with sigma value between 0 and 2 it varies only slightly, then starts to grow more rapidly. Also here, we see a sharp increase at noise created with a sigma value of 5. We conclude that the algorithm is quite robust to noise with sigma values lower than 2.

Case Study 4: Influence of Resolution Differences in Real CT Scans
The data analyzed here are based on CT scans of a small cut-out of a larger glass fiber-reinforced polypropylene structure with 30 weight-percent fiber content. The specimen was scanned four times with different resolutions, the voxel sizes of these scans were 1 µm, 2 µm, 3 µm, and 5 µm. For each of these scans, the analyzed ensemble contains the result of the FCTM approach, plus an addi-tional manually labeled result for reference. The fibers in these datasets have an average diameter of 12 µm and a length varying widely between 30 µm and 460 µm. Before loading in our tool, this data was analyzed using some custom scripts [SJKK15] for a quantitative evaluation, in addition to a visual analysis with Fiber-Scout [WAL * 14]. As each result needed to be explored separately, and quantitative evaluation was done in scripts custom-written for this purpose, the time to conduct this previous analysis was in the range of at least a week. The goal of the previous analysis was to optimize the FCTM algorithm. For our case study, the goal was to analyze whether key findings were missed in the previous analysis.
In the list in Figure 8(a) we see the results in decreasing order of resolution, except for the last entry which is the reference. We observe that the fiber count decreases with the resolution, which clearly indicates that not all fibers are detected with lower resolution. The point-based dissimilarity measure shows similar values for the first two entries, while the overlap-based dissimilarity measure correlates better with the resolution.
The spatial overview in Figure 8(b) shows two fibers in the reference, marked in blue, without matches in any of the results, not even in the highest-resolution scan. The existence of two such unmatched fibers was new to the domain expert, and was missed the first time he analyzed the data. This is remarkable given that these important findings became clear within minutes after loading the data. These two fibers clearly deserve a closer look in the input data, for which currently an external tool is needed. Another interesting detail is revealed when looking at the fibers with high dissimilarity to their best reference match according to measure d o 3 . Figure 8(d) and (e) show the five fibers with the highest dissimilarity, for the characterization performed on the 1 µm and 2 µm scans, respectively. These fibers are all very short, leading to the preliminary hypothesis that the algorithm is working better for long fibers. However, as can be seen from Figure 8(c), where all fibers of lengths between approximately 30 µm to 60 µm are selected, there is no consistent trend for all short fibers. Two fibers in (d) and (e) are the same, indicating systematic problems of the algorithm with these two fibers, as they are not well recognized in the results of two completely different scans. In the study, we found a total of four fibers that require further investigation.

Case Study 5: Feasibility Study of Analyzing Other Datasets
The goal of this section is to lay out how FIAKER can be adapted to analyze algorithms other than fiber characterization pipelines. Our methods can be applied to the analysis of multiple results, when each of those results contains a number of parametrizable objects which can be visualized through a geometric primitive. To adapt our current methods to another object type, the following modifications or extensions are required: 1. Create a 3D visualization for the geometric primitive representing an object 2. Define at least one dissimilarity measure between two objects of the given type We have started experiments to analyze results from pore characterization algorithms, in our case generated by the FeatureAnalyzer  Pores often feature a circular or elliptical shape, and may therefore be represented as ellipses. As dissimilarity measure, the distance between the center points of the two pores, combined with the difference in their extents in x-, y-and zdirection may be considered. Figure 9 shows the result list and the spatial view with the results of three differently parameterized pore characterizations. The required adaptations to our tool were done within approximately half a day by a single software programmer.

Discussion of Design Decisions and Lessons Learned
In the result list, initially we only showed a list of result previews. Then we implemented a simplified version of LineUp [GLG * 13] for quantitative ranking. We considered implementing a design similar to WeightLifter by Pajer et al. [PSTW * 17], in order to explore the weight Space more comprehensively. But we decided against this, as it was not needed for the current analysis scenarios. For comparison of value distributions across results, we added histograms.
It turned out that scatter plot matrices were not used as much as we had expected. As they play a major role in FiberScout [WAL * 14], a previous tool for fiber analysis, we expected them to be a key element in the analysis again. In our case, it turned out that the main focus of the analysis is on the result list and the spatial view. We assume this is due to the fact that our tool focuses more on comparing multiple results. In FiberScout, the focus was more on correlations between different fiber characteristics.
Our users unanimously were asking for a 3D view and were thrilled to see its possibilities for visualizing the results when they first saw the prototype. This was surprising and contrary to the experience from previous work on the analysis of image processing tools in material sciences (e.g., for analyzing image segmentation results [FMH16]). The preference for 3D in this case comes, on the one hand, from the fact that we have polygonal data, not volumes, where slice images are easier to comprehend than direct volume rendering that is prone to occlusion. On the other hand, the many ways to select and thus concentrate on just a few fibers make it easy to focus in the 3D visualization on the currently interesting parts of the result. In this drill-down scenario, the majority of the fibers are rendered highly transparent so that one can focus on the few important fibers while maintaining their 3D context. Selecting fibers in the 3D view was highly demanded by our users. However, it turned out to be quite challenging. We first implemented selection through drawing a rectangle. This selects those fibers where any of their end points is contained in the view frustum spanned by the rectangle. We first implemented a selection mechanism by selecting those fibers whose endpoints are contained in the view frustum spanned by a user-drawn rectangle. It might have been more intuitive to select any fiber which passes through the rectangle. However, in practice, selecting only by end points already leads to selecting more fibers than wanted, as it is hard to pinpoint only the start or end point of the fibers without including others in the vicinity. In order to facilitate the selection of a particular fiber, we therefore also implemented a selection by a single click. The click selects the fiber hit first by a ray cast in viewing direction into the scene. Again, this can be problematic if the desired fiber is somewhere in the middle of the region of interest and thus occluded by other fibers. This motivated us to implement a selection view that allows the user to further refine a selection made.
An interesting lesson also was the design of the dissimilarity measures. Initial drafts had not put much emphasis on this topic. But during exploration of the datasets, we discovered problems with the initial measures. We could also verify with our tool that the measures we created as a consequence could overcome these limitations, as described in Section 5.4. This points towards the generalizability of our methods: The same methods that are helpful in understanding and debugging fiber characterization algorithms have also helped us in developing appropriate dissimilarity measures.
One potential future extension proposed by our collaborators is the ability to present the reconstructed CT volume as localized context for visualizing selected fibers for which the characterization did not work well.

Conclusion
We have developed FIAKER, a tool implementing methods for analyzing and comparing the results of fiber reconstruction algorithms. We refined the usability and utility of these methods by developing them in close collaboration with scientists working on fiber reconstruction algorithms or applying such algorithms on a daily basis. We evaluated the tool on the outcomes of two different fiber reconstruction algorithms, and were able to show that the tool quickly provides insight into where a fiber reconstruction algorithm still needs improvement. In addition, the methods can help to understand which algorithm performs better in comparison to a reference, both in general as well as for specific locations or specific fibers. We also showed how our methods may be generalized and applied to the analysis of any collection of objects representable by geometric primitives, and demonstrated this in our case study on pore characterization results.