Segmentation and Surface Reconstruction of the Detailed Ear Structures, Identified in Sectioned Images

Authors

  • Hae Gwon Jang,

    1. Multimedia and Computer Architecture Lab, Graduate School of Information and Communication, Ajou University, Worldcup-ro 206, Suwon 443-749, Republic of Korea
    Search for more papers by this author
  • Min Suk Chung,

    1. Department of Anatomy, Ajou University School of Medicine, Worldcup-ro 164, Suwon 443-749, Republic of Korea
    Search for more papers by this author
  • Dong Sun Shin,

    1. Department of Anatomy, Ajou University School of Medicine, Worldcup-ro 164, Suwon 443-749, Republic of Korea
    Search for more papers by this author
  • Seung Kyu Park,

    1. Multimedia and Computer Architecture Lab, Graduate School of Information and Communication, Ajou University, Worldcup-ro 206, Suwon 443-749, Republic of Korea
    Search for more papers by this author
  • Keun Soo Cheon,

    1. Department of Obstetrics and Gynecology, Dongguk University College of Medicine, 707 Seokjang-dong, Gyeongju 780-714, Republic of Korea
    Search for more papers by this author
  • Hyo Seok Park,

    1. Department of Anatomy, Dongguk University College of Medicine, 707 Seokjang-dong, Gyeongju 780-714, Republic of Korea
    Search for more papers by this author
  • Jin Seo Park

    Corresponding author
    1. Department of Anatomy, Dongguk University College of Medicine, 707 Seokjang-dong, Gyeongju 780-714, Republic of Korea
    • Department of Anatomy, Dongguk University College of Medicine, 707 Seokjang-dong, Gyeongju 780-714, Republic of Korea
    Search for more papers by this author
    • Fax: +82-54-770-2402


Abstract

The structure of the ear, which intervenes between gross anatomy and histology in size, can be best understood by means of three-dimensional (3D) surface models on a computer. Furthermore, surface models are the source of interactive simulation for clinical trials, such as tympanoplasty. The objective of this research was to elaborate the surface models of detailed ear structures, which contribute to learning anatomy or the practice of otology. We produced sectioned images of a cadaver head (pixel size, 0.1 mm; 48-bit color). In the sectioned images, the external, middle, and internal ear structures and other related components were delineated on Photoshop to acquire segmented images at 0.5-mm intervals. Segmented images of each structure were stacked, and the surface was reconstructed to generate a 3D-surface model on commercial software. Thirty surface models showed fine ear topographic anatomy (e.g., semicircular ducts), as expected. Herein, we present the corresponding sectioned images, segmented images, and surface models of ear structures that will be released together. It is hoped that these image data will stimulate the development of medical simulations. The efficient technique of segmentation and surface reconstruction enables the manufacture of surface models from other serial images (e.g., CTs and MRIs). Anat Rec, 2011. © 2011 Wiley-Liss, Inc.

Human body structures, which vary in size, are categorized as macroscopic and microscopic. The two kinds of structures are revealed during cadaver dissection or microscopy. The subject of this report is ear structures (e.g., tensor tympani and cochlea) intervening between the two categories to be inaccessible in both gross anatomy exercise and histology practice. Learning tools to experience the extremely irregular structures of the ear have been limited to plastic models for several decades. Alternative tools consist of three-dimensional (3D) models, any combinations of which are selected to display and rotate at arbitrary angles. Furthermore, the 3D surface models that can be modified in real time, even online, would be the source of an interactive simulation system contributing to clinical operations, such as mastoidectomy (Sorensen et al.,2009).

The objective surface models of the ear can be obtained by surface reconstruction after stacking serial outlines of the ear constituents. For serial images of the ear, microcomputerized tomographs (micro-CTs), even exhibiting bony trabeculae, seem suitable; however, in micro-CTs, only bone is shown (Lee et al.,2010). Sequential histologic slides have the advantage to show microscopic ear components, which become more prominent by staining. Because of the size limitation of specimens, however, the slides do not completely show the structures beyond the temporal bone, as needed for a macroscopic approach (Wang et al.,2006).

In contrast, the sectioned images of whole cadavers, such as the Visible Human Project, are to be considered. A problem with the Visible Human Project is that the pixel size and intervals are too large (>0.33 mm) to visualize fine ear structures (Spitzer et al.,1996). In the Visible Korean (Park et al.,2005b, 2008), Chinese Visible Human (Zhang et al.,2006), and Virtual Chinese Human (Tang et al.,2010), advanced sectioned images with 0.2-mm-sized pixels and 0.2-mm- or 0.1-mm-sized intervals were created. However, even 0.2-mm-sized pixels are not satisfactory for detection of minute ear architecture (e.g., bony labyrinth). It is for this reason that the visible ear, the subject of which was the unilateral temporal bone of a female cadaver (85-year-old Dane), was designed and executed to acquire sectioned images (0.05-mm-sized pixels and intervals, 24-bit color) (Sørensen et al.,2002). The fine ear structures were then identified to be outlined and surface-reconstructed to build the surface models (Wang et al.,2007).

We have prepared another dataset of bilateral ears, which are portions of entire head images. It was because the surface models of the ear might be connected to other head portions, such as the pharyngeal opening of the auditory tube or the auditory pathway through brain. Recently, in our laboratory for the Visible Korean, high-quality sectioned images of the cadaver head were generated with 0.1-mm-sized pixels and intervals (Park et al.,2009).

We also attempted to establish an effective technique of delineation, interpolation, and volume and surface reconstructions by the appropriate commands in commercial software packages and customized programs. Such a technique would contribute to other trials for making 3D models of the clinical images, like CTs (Park et al.,2005a,2007; Shin et al.,2009a,b).

The objective of this research was to prepare segmented images and sophisticated 3D-surface models of detailed ear structures, which can be used for teaching anatomy to medical students or in clinical practice by otologists. The data would be distributed with the source data, sectioned images of the head, which all correspond to one another. It was taken into account which types of the segmented images to be distributed are useful. Our goal was not to present the final product to end users but to provide other researchers with the suitable material for the various applications.

MATERIALS AND METHODS

The head of a male cadaver (67-year-old Korean) was serially sectioned to obtain 2,341 sectioned images (resolution, 4,368 × 2,912; pixel size, 0.1 mm; 48 bit color; tag image file format; Table 1; Park et al.,2009). Every five sectioned images was selected to increase intervals from 0.1 to 0.5 mm, then the sectioned images superior or inferior to the ear structures were excluded, so that 45 sectioned images remained. In the images, excessive margins beyond the bilateral temporal bones were cropped to achieve an image resolution of 1,622 × 420 with the pixel size (0.1 mm) retained.

Table 1. Features of sectioned images, segmented images, and three-dimensional (3D) models
CategoriesImagesUsed softwaresFile formatBit depthIntervals (mm)
  • Raw data and segmented images consist of 0.1-mm-sized pixels.

  • *

    *Self-developed software in this research is used.

Raw dataSectioned images TIFF24-bit color0.1
Segmented imagesOutlined imagesPhotoshopPSD24-bit color0.5
Color-filled imagesPhotoshopBMP8-bit color0.5
Interpolated color-filled imagesInterpolation*BMP8-bit color0.1
Black-filled imagesPhotoshopBMP2-bit color0.5
3D modelsVolume modelsReconstruction*TXT24-bit color0.5
Surface models3D-DOCTOR, MayaMB  

Thirty ear structures were chosen for segmentation. Our trial was focused on the anatomic components for auditory sensation and equilibrium (Table 2). Ear structures were segmented on both the right and left sides.

Table 2. Thirty ear structures segmented on the sectioned images and reconstructed to build the surface models
RegionStructures
  • a

    Structures are segmented semiautomatically.

  • b

    Structures are segmented manually.

  • c

    Structures are segmented automatically.

  • Segmentation intervals are 0.5 mm, except the oval window, stapes, and incus (intervals, 0.1 mm).

External earSkina, external acoustic meatusb, tympanic membraneb
Middle earTympanic cavityb, oval windowb, stapesb, incusb, malleusb, tensor tympani muscleb, stapedius muscleb, auditory tubeb
Internal earAnterior semicircular canalb, posterior semicircular canalb, lateral semicircular canalb, cochleab, internal acoustic meatusb, utricleb, sacculeb, anterior semicircular ductc, posterior semicircular ductc, lateral semicircular ductc, cochlear ductc
OtherTemporal bonea, internal carotid arteryb, superior petrosal sinusb, brainstemb, facial nerveb, chorda tympanib, vestibular nerveb, cochlear nerveb

Sectioned images were saved as Photoshop document (PSD) files on Photoshop CS3 version 10 (Adobe Systems, San Jose, CA). In each PSD file, 30 layers were created and named by the structures (Table 2; Park et al.,2005a). To make the boundary of the structure clear, the “brightness and contrast” of the sectioned images were adjusted or a “sharpen” filter was applied. Sometimes, a “median” filter was used to generate the color of the structure with less noise (Shin et al.,2009b).

The structures were segmented using the procedures we developed on Photoshop in a previous study (Park et al.,2005a). Most structures were delineated manually using the “lasso” tool. The preparatory process of filtering to highlight ear structures facilitated the manual segmentation (Fig. 1A). Apparent skin and temporal bone could be demarcated semiautomatically with the “quick selection” tool. After drawing outlines of the semicircular canals and cochlea, outlines of the semicircular ducts and cochlear duct were automatically created by a 97% decrease in the outline size (Tables 1 and 2).

Figure 1.

Various segmented images of the middle and internal ears. Outlined images (A; column), color-filled images (B; column), and black-filled images (C; column) around the tympanic cavity (D; row), cochlea (E; row), and semicircular canal (F; row). In the color-filled images, the background pink color signifies the temporal bone (B; column).

The manual segmentation of fine ear structures was time consuming, so assistance was required from a number of colleagues. Each colleague was responsible for segmentation of a set of adjacent structures (e.g., middle ear structures) through the entire set of sectioned images. The colleague put the outlines of a structure on the layers of the structure of the outlined images (PSD files). In every outlined image, we had to integrate the layers, including outlines drawn by different colleagues. Fortunately, the combing job was performed with enhanced automatization using JavaScript on Photoshop.

In the outlined images, the outlines of every structure were filled with a specific color to acquire 45 color-filled images of whole structures, which were saved as bitmap (BMP) files (Fig. 1B; Table 1); this was done automatically using “batch” on Photoshop (Park et al.,2005a).

By interpolation, intervals (0.5 mm) of color-filled images could be reduced to 0.1 mm. On the software manufactured in this study, two color-filled images (intervals, 0.5 mm) were loaded (Fig. 2A and F). The software is notified of the corresponding segmented structures in the two images by manual alternative clicking. By the computer graphics technique of interpolation, the software generated four new color-filled images, which were gradually changing in morphology between the original two images (Fig. 2B, C, D, and E; Table 1). In cases of very tiny structures (e.g., oval window and stapes), interpolation did not work properly. Thus, segmentation had to be performed on the original sectioned images at 0.1-mm intervals (Table 2).

Figure 2.

Color-filled images of the internal ear (intervals, 0.1 mm). The middle four images (BE) of which are created by interpolation between the original color-filled image (A) and its next image (F).

As the preprocess of volume reconstruction, color-filled images of the wanted ear structures (BMP files) were prepared from the outlined images (PSD files) on Photoshop (Fig. 1B). Then, on other self-developed software, the color-filled images were stacked and reconstructed by volume modeling. The rough volume models were rotated arbitrarily (Fig. 3A; Table 1). In case, the shape of volume models did not correspond to anatomic knowledge, incorrect segmentation of the structures, which led to the wrong volume models, was revised.

Figure 3.

Volume and surface models of ear structures. Volume models are built using color-filled images on the self-developed software (A; column) and surface models are built using black-filled images on Maya software (B; column) of the ear ossicles (C; row), the cochlea and its nerve (D; row), semicircular ducts, and utricle (E; row).

Black-filled images of each structure were made from the color-filled images on Photoshop. For instance, tympanic cavity in the color-filled images was filled with blue. We painted one top left pixel of the color-filled images with blue (the same color). The top left pixel was selected using the “magic wand” tool without the “contiguous” option. As a result, not only the top left pixel but also the tympanic cavity pixels were selected at the same time. After excluding the top left pixel, we filled the tympanic cavity with black. In succession, the “inverse selection” was ordered to fill outside of the tympanic cavity with white (Fig. 1D). The black-filled image of tympanic cavity was saved as BMP (2 bit color; Table 1). Taking all color-filled images together, the procedure was done using “batch” to produce the serial black-filled images of the tympanic cavity.

Surface models were produced from the black-filled images and refined. The black-filled images of a structure were sequentially accumulated using 3D-DOCTOR version 4 (Able Software, Lexington, MA). By using the “simple surface” command, the black-filled outlines were expanded to the next outlines to build combined volume models, defined as the volume reconstruction. Simultaneously, from this expansion, a surface 3D model was extracted (surface reconstruction). The surface model, consisting of stacked outlines and polygons between the outlines, was saved in the drawing exchange format, which could be opened in the next software, Maya version 7.0 (Autodesk, San Rafael, CA). To smooth the surfaces, stacked outlines were deleted using the “smooth” command on Maya, and the number of polygons was appropriately reduced using the “reduce” command. The refined surface model was saved as a Maya binary (MB) file. The surface reconstruction process was carried out repeatedly for all ear structures. In the new MB file, the individual surface models were assembled, which allowed them to be displayed selectively (Fig. 3B; Table 1; Shin et al.,2009a,b).

RESULTS

The authors regarded color-filled images of whole ear structures as the core result because the color-filled images were made from sectioned images by way of the outlined images (Fig. 1A and B), and the color-filled images were used to build volume models (Fig. 3A) directly as well as surface models (Fig. 3B) via the black-filled images (Fig. 1C; Table 1).

It took 30 days to acquire the 45 basic color-filled images of 30 structures; 29 days were required for segmentation; and 1 day was needed for color filling. Although computer-based work, such as color filling, could be performed with high automatization, knowledge-based work, such as delineation, required manual labor by medical experts. The duration of segmentation could be reduced only by the simultaneous work of several colleagues, followed by combining the work of outlined images.

In this study, the color-filled images were verified by the volume models. The volume models with low resolution (0.5-mm-sized voxels) did not show the real colors of the sectioned images. Nevertheless, the quality of volume models was sufficient to verify color-filled images as well as to compare the surface models (Fig. 3). The practical value of the volume models was enhanced by rapid manufacture from the color-filled images.

The color-filled images were a good source of 3D surface models. It was almost automatic to convert the color-filled images into the black-filled images of a selected structure (Fig. 1B and C). From the black-filled images, the surface model of the structure was obtained by a nearly automated method (Fig. 3B). Additionally, the color-filled images were capable of interpolation to create better 3D models (Fig. 2).

In the present study, essential procedures for outlining and surface reconstruction were performed on commercial software. Especially, the procedures were automatized based on the software tips devised in this research, indicating that any investigators could easily produce their own color-filled images and surface models independent of computer programmers. Two exceptional jobs (interpolation and volume reconstruction), requiring programmed software, were not included in the main stream of segmentation and surface reconstruction (Table 1).

Because of the best quality of original sectioned images (0.1-mm-sized pixels) encountered with a high technique of delineation and reconstruction, 3D surface models showed the following ear topographic anatomy. The tensor tympani muscle was shown to be connected with the tympanic membrane by way of the malleus (Fig. 4A). Between the malleus and incus, the chorda tympani crossed the medial surface of the neck of the malleus (Fig. 4B). The anterior, posterior, and lateral semicircular ducts were in contact with the utricle, then the saccule (Fig. 4C). The spiral cochlea began at the saccule and made two and one-half turns around the cone-shape modiolus for distribution of the cochlear nerve (Figs. 3B and 4C; Moore et al.,2010).

Figure 4.

Surface models of the left ear with annotation. Middle ear consists of the auditory ossicles, their muscle, and the chorda tympani (A, B), whereas the internal ear contains the membranous labyrinth and vestibulocochlear nerve (C).

DISCUSSION

The segmented images to be distributed will hopefully save outlining time and effort of other researchers. In the sectioned images of the head from the Visible Korean, numerous delineations of tiny ear structures >0.1 mm are possible (Park et al.,2009). Numerous delineations do not suggest that delineation is easy. Some graphic information in the sectioned images helps automatic segmentation, whereas other information prevents the automatization. Thus, the demarcation is inevitably carried out almost manually by medical experts. It is extremely ineffective for different investigators to outline the identical sectioned images, so we decided to share our segmented images (Fig. 1).

Among three sets of segmented images, the color-filled images are preferable for distribution. The reasons to present the color-filled images of ear structures with the original sectioned images follow.

First, the color-filled images, which have a smaller file size, simpler file format, and fewer file number than the outlined or black-filled images, are conveniently distributed and utilized. Outlined images have the merit to be simultaneously transformed into color-filled or black-filled images of the intended structures (Park et al.,2005a). However, the outlined images contain an enormous file size due to the number of layers and underlying sectioned images. Segmentation is expected to be continuously updated by the authors or others. It is inefficient to repetitively download heavy files of the outlined and sectioned images. In contrast to color-filled images (BMP files), the outlined images (PSD files) with many layers are not easy for every user to browse or handle. Meanwhile, black-filled images of each structure have the advantage of immediate surface reconstruction of the structure (Shin et al.,2009a). However, the entire files of 30 structures are too numerous to be readily managed. Therefore, color-filled images appear to be appropriate, even for online distribution.

Second, the color-filled images are suitably applied to build 3D volume and 3D surface models. In this research, volume models were built directly from the color-filled images (Fig. 3A). To make the genuine volume model comprising real human color, we have to prepare serial-sectioned images of each structure, excluding the outside. Each structure in the serial-sectioned images could be obtained using black-filled images of the structure. The black-filled images are required for the surface reconstruction as well (Fig. 3B; Shin et al.,2009a). Therefore, we developed an expedient technique to convert the color-filled images of all structures into the black-filled images of a chosen structure (Fig. 1B and C). For better 3D volume and 3D surface models, intervals (0.5 mm) of the color-filled images were reduced to the original intervals (0.1 mm) of the sectioned images (Table 1). Thus, we composed the interpolation software to produce intervening color-filled images at 0.1-mm intervals (Fig. 2). This research proves that the color-filled images have the potential to be used in many ways.

We also decided to distribute the elaborative 3D surface models of ear structures for the following reasons.

First, the surface models decrease the reconstruction labor of other researchers. Anyone is able to build the surface models from the color-filled images, as prepared in the present study. However, there is a time-consuming problem for converting color-filled images to black-filled images, for performing surface reconstruction, and for refining surface models, even if the procedures are optimized. We present the surface models to concentrate other investigators and technicians on the manufacture of applications from the surface models.

Second, the surface models generate additional models of adjacent structures with special needs. The surface models can be divided. For example, the cochlea, excluding the cochlear duct, is divided into scala vestibuli and scala tympani to establish the learning tool of anatomy on hearing. Moreover, the supplementary surface models can be pictured. For example, from the tympanic cavity, the mastoid antrum and mastoid air cells are stereoscopically drawn to create the orientation video of the tympanoplasty through the mastoid process (Moore et al.,2010).

Third, the surface models are the source of stand-alone interactive programs. The surface models (MB files), generated in this study, could be opened and manipulated on expensive software, such as 3D-DOCTOR or Maya (Table 1). Thus, it is hoped that browsing and simulation software of the ear models is composed to practically help users become aware of ear morphology (Wang et al.,2006,2007). The simulation system would be more valuable by being equipped with the haptic device to make users feel the physical property of each ear component (Sorensen et al.,2009).

Fourth, the surface models accompanied by the corresponding images will contribute to a realistic and comprehensive educational program. On the surface models, the original sectioned images (pixel size and intervals, 0.1 mm), as well as the color-filled images, can be superimposed to supply medical students and physicians with the concrete morphologic information of the ear (Fig. 5). In addition, the surface models can be replaced with the volume models, which enable users to see the real body color of free-angle sectional planes (Park et al.,2005b). Moreover, the sectioned images of the entire head would yield the segmented images and 3D models of other head structures (e.g., complete length of the facial nerve), which enrich the value of the surface models of ear.

Figure 5.

Sectioned image, superimposed on the surface model from the external ear to the internal ear with the vestibulocochlear nerve.

This is a report of state-of-the-art sectioned images, segmented images, and surface models of detailed ear structures. After obtaining permission to use the images from our group, the full data will be provided to the investigators at no charge. Before obtaining permission, the investigators are recommended to download and evaluate the reduced data from the menu, “Viewer of head” and “Images and movies of ear,” on the homepage (neuroanatomy.kr). The investigators are expected to select one among the Visible Korean and the Visible Ear data after considering their own aims. The supplementary data sets are likely to promote the further development of virtual simulators for medical students and otologists.

Ancillary