Automatic segmentation and supervised learning-based selection of nuclei in cancer tissue images

Authors

  • Kaustav Nandy,

    Corresponding author
    1. Optical Microscopy and Analysis Laboratory, Advanced Technology Program, SAIC-Frederick, Inc., Frederick National Laboratory for Cancer Research, Frederick, Maryland 21702
    • Optical Microscopy and Analysis Laboratory, SAIC-Frederick, Inc., Frederick National Laboratory for Cancer Research, Post Office Box B, Frederick, MD 21702
    Search for more papers by this author
  • Prabhakar R. Gudla,

    1. Optical Microscopy and Analysis Laboratory, Advanced Technology Program, SAIC-Frederick, Inc., Frederick National Laboratory for Cancer Research, Frederick, Maryland 21702
    Search for more papers by this author
  • Ryan Amundsen,

    1. Department of Assymetric Operations, Johns Hopkins University Applied Physics Laboratory, Laurel, Maryland 20723-6099
    Search for more papers by this author
  • Karen J. Meaburn,

    1. Cell Biology of Genomes Group, National Cancer Institute, National Institutes of Health, Bethesda, Maryland 20892
    Search for more papers by this author
  • Tom Misteli,

    1. Cell Biology of Genomes Group, National Cancer Institute, National Institutes of Health, Bethesda, Maryland 20892
    Search for more papers by this author
  • Stephen J. Lockett

    1. Optical Microscopy and Analysis Laboratory, Advanced Technology Program, SAIC-Frederick, Inc., Frederick National Laboratory for Cancer Research, Frederick, Maryland 21702
    Search for more papers by this author

  • This article is a US government work and, as such, is in the public domain in the United States of America.

Abstract

Analysis of preferential localization of certain genes within the cell nuclei is emerging as a new technique for the diagnosis of breast cancer. Quantitation requires accurate segmentation of 100–200 cell nuclei in each tissue section to draw a statistically significant result. Thus, for large-scale analysis, manual processing is too time consuming and subjective. Fortuitously, acquired images generally contain many more nuclei than are needed for analysis. Therefore, we developed an integrated workflow that selects, following automatic segmentation, a subpopulation of accurately delineated nuclei for positioning of fluorescence in situ hybridization-labeled genes of interest. Segmentation was performed by a multistage watershed-based algorithm and screening by an artificial neural network-based pattern recognition engine. The performance of the workflow was quantified in terms of the fraction of automatically selected nuclei that were visually confirmed as well segmented and by the boundary accuracy of the well-segmented nuclei relative to a 2D dynamic programming-based reference segmentation method. Application of the method was demonstrated for discriminating normal and cancerous breast tissue sections based on the differential positioning of the HES5 gene. Automatic results agreed with manual analysis in 11 out of 14 cancers, all four normal cases, and all five noncancerous breast disease cases, thus showing the accuracy and robustness of the proposed approach. © Published 2012 Wiley Periodicals, Inc.

Ancillary