Implementation of FocalPoint GS location-guided imaging system
Experience in a clinical setting
Recently, the Food and Drug Administration approved the use of the location-guided imaging system FocalPoint GS (FPGS), on SurePath Papanicolaou (Pap) tests for primary screening. The objective of the current study was to evaluate the impact of FPGS on the following: distribution of diagnostic categories; rate of high-risk human papillomavirus (HR-HPV)–positive ASC-US cases; and quality control (QC) data before and after FPGS implementation.
A search of the laboratory information system was performed to identify all SurePath Pap tests processed in our laboratory for the first 19 months after FPGS implementation. We also retrieved all SurePath specimens from a 16-month period prior to FPGS implementation to serve as the control. During the period from Janaury 2008 to April 2009, the FocalPoint Slide Profiler was used.
Implementation of FPGS resulted in a significantly higher percentage of LSIL and ASC-US interpretations, as well as a significant increase in the detection of candidiasis and bacterial vaginosis. The ASC-to-SIL ratio was 1.4 and 1.9 before and after FPGS implementation, respectively. There was a decrease in the HR-HPV positive rate in ASC-US cases, and a decrease in the estimated false-negative fraction after FPGS implementation.
In conclusion, our study seems to demonstrate a favorable performance of FPGS in the routine clinical setting. FPGS may have the potential to be a promising screening tool for gynecologic cytology in a low-risk patient population. Cancer (Cancer Cytopathol) 2012;. © 2011 American Cancer Society.
It is well known that the screening of Papanicolaou (Pap) tests by cytotechnologists is a subjective and tedious process that requires immense concentration and judgment. In addition, external factors such as the anticipated shortage of cytotechnologists, demand for increased productivity, and continuous threat of litigation for screening “mistakes” pose mounting challenges to the practice of gynecologic cytology. Hence, there is a recognized need to improve current diagnostic accuracy as well as the operative efficiency of gynecologic cytology. With the advent of computer technology and the introduction of liquid-based cytology during the past 2 decades, the use of computer-assisted screening has become a reality in the day-to-day operation of cytology laboratories.
Until recently, there were only 2 devices approved by the Food and Drug Administration (FDA) for computer-assisted primary screening of Pap tests. One was the ThinPrep Imaging System (TIS; Hologic Inc., Boxborough, MA), and the other was the FocalPoint Slide Profiler (BD Diagnostic Inc., Burlington, NC). TIS is a location-guided imaging system that is approved only for use with ThinPrep Slides (Hologic Inc., Boxborough, MA). The Slide Profiler scores, sorts, and ranks both conventional Pap smears and SurePath (BD Diagnostic Inc., Burlington, NC) Pap slides according to the probability and degree of abnormality present on the slide; up to 25% of the slides that scored below a certain threshold were eligible for archiving without any manual review. The Slide Profiler is also capable of designating areas on the slide that contain the most abnormal cells by displaying them on a printed map using PAPMAP technology.1 Although a number of studies have demonstrated improvements in accuracy and productivity with PAPMAP technology,2-6 it has not been widely applied in the clinical setting. About 7 years ago, improvement resulting in fully automated location-guided screening technology was achieved with the development of the FocalPoint GS imaging system (BD Diagnostics Inc., Burlington, NC).
The FocalPoint GS imaging system (FPGS) includes a computer-based slide review microscopy station and the GS review station (GSRS); the latter allows automatic localization of microscopic fields of view (FOVs) using information obtained from FPGS for review by cytotechnologists. Briefly, FPGS obtains digital images using a high-resolution scanner and a high-speed video microscope. These images are then analyzed by the instrument using proprietary algorithm-based interpretation software. The instrument selects 10 areas, each corresponding to 1 manual microscopic 10× objective FOV that is most likely to contain diagnostically relevant cells. Manual screening is performed at the GSRS. The monitor displays an electronic map of the slide indicating the locations of the 10 FOVs as well as a low-resolution 20× magnification black-and-white image of the FOVs. Other information such as ranking information and specimen adequacy is also available. Although FPGS, like its predecessor, has the ability to sort and rank the slides according to the likelihood of the slide containing an abnormality and to designate up to 25% of slides in each run as “no further review,” this functionality was not included in the FDA-approval application for FPGS in the United States. As a result, this functionality is not available in the US version of FPGS.
In December 2008, the FDA approved the use of FPGS for primary screening of SurePath Pap tests for its location-guided screening ability. Our laboratory upgraded our 2 FocalPoint Slide Profilers to FPGS in May 2009 and put these into routine clinical use 1 month later. The objective of the current study was to evaluate the impact of implementing FPGS in a low-risk screening population. Specifically, we assessed the impact on the distribution and frequency of various diagnostic categories, the rate of high-risk human papillomavirus (HR-HPV) detection in ASC-US cases, and the estimated false-negative fraction (EFNF) of the laboratory before and after implementation of FPGS.
MATERIALS AND METHODS
The Institutional Review Board of Yale University approved the current study. Our laboratory processes and evaluates about 80,000 Pap tests annually. The Pap tests are submitted predominantly by various obstetrics and gynecology practices in the surrounding community. SurePath-processed Pap tests account for 80% of all the Pap tests processed in our laboratory; the remaining include ThinPrep-processed Pap tests (19%) and conventional Pap test preparations (1%). The study population consisted of only SurePath preparations.
Before FPGS implementation, the Focal Point Slider Profilers were used for primary screening of SurePath specimens for 8 years. After the validation of FPGS and the completion of cytotechnologist training, the 2 FPGS imaging systems were put into clinical use in June 2009.
All SurePath slides were evaluated using FPGS according to the manufacturer's recommended protocol. For each imaged slide, a cytotechnologist reviewed all 10 FOVs selected by the instrument. If the cytotechnologist deemed there were no potential abnormalities in any of the 10 FOVs and the slide was satisfactory for interpretation, full-slide manual screening would not be required unless the slide was being selected for directed QC rescreening. The slide would be reported as “negative for intraepithelial lesion or malignancy” and “satisfactory for interpretation.” Full-slide manual screening would be required if 1 or more of the following conditions was met: (1) any potential abnormalities were identified in any of the FOVs, (2) no endocervical or transformation zone component was identified in any of the FOVs, (3) specimen adequacy could not be determined, or (4) a technical processing failure was identified by the instruments. Full-slide manual screening would be performed immediately following the FOV review by the same cytotechnologist who performed the FOV review. According to the protocol, FPGS selects 15% of the highest-scoring negative slides for full-slide manual screening as part of quality control (QC). The QC rescreen was performed by cytotechnologists who had not previously reviewed the slides. A systems change regarding QC was implemented post-FPGS. All Pap tests interpreted as “negative for intraepithelial lesion or malignancy” that were found to be positive for high-risk HPV DNA by HCII were manually rescreened. In addition, all cases that were reported to have no endocervical component by FPGS were manually rescreened.
Interpretations were rendered by a team of 15 cytotechnologists and 7 pathologists all board certified in anatomic pathology with subspecialty certification and fellowship training in cytopathology. The Bethesda System 2001 terminology and criteria were used for reporting. Cases interpreted as high-grade squamous intraepithelial lesion (HSIL) or above were reviewed in our daily consensus conference among the cytopathologists. HR-HPV testing was performed in house using the Digene hybrid capture 2 assay (Qiagen, Gaithersburg, MD) on residual fluid from liquid-based Pap tests. The probes used were directed against high-risk HPV types 16,18, 31, 33, 35, 39, 45, 51, 52, 56, 58, 59, and 68 using 1 relative light unit as the cutoff (RLU/CO). The clinical performance of HR-HPV testing was independently validated in our laboratory, as it is not currently FDA approved for use on SurePath specimens.
Using the laboratory information system, the reports of all SurePath Pap tests were queried for the periods from January 2008 to April 2009 (pre-FPGS implementation) and from June 2009 to December 2010 (post-FPGS implementation). Using Copath, the Pap test reports were searched (by subject text) for the following data for comparison: all diagnostic categories (ie, “NEG, ASC-US, LSIL, HSIL,” etc.), the lack of an endocervical component (“no ECC”), the presence of endometrial cells in a woman older than 40 years, and the presence of an organism (ie, “shift in bacteria flora, Candida sp., Trichomonas sp.,” etc.). May 2009 was not included in the analysis because it was the month when FPGS instruments were being validated and the cytotechnologists were being trained on FPGS. We also collected the results of HR-HPV testing in SurePath cases with an ASC-US interpretation for the 2 periods. The EFNFs were calculated before and after FPGS implementation. The EFNFs were based on cases initially interpreted as negative that were rescreened because they were either among the mandated 15% direct QC review or high-risk cases and were reclassified as squamous intraepithelial lesions (SILs) or above.
Statistical analyses were performed using t and z tests. The former was used for comparing frequency distributions of diagnostic categories, positive HR-HPV rates, proportion of cases for QC rescreen, and referral to pathologists, as well as ENFNs, whereas the z test was used for comparing the presence of endometrial cells in patients older than 40 years and the frequencies of reporting infectious organisms between the 2 periods. P ≤ .05 was considered statistically significant. The statistics were calculated using SPSS software, version 16.0. (SPSS Inc., Chicago, IL).
During the study period, a total of 188,588 SurePath Pap tests were processed and evaluated, 90,615 (5663/month) and 97,973 (5156/month) before and after FPGS implementation, respectively. Although there was a 9% decrease in the monthly test volume, both cohorts were derived from the same physician practices and same patient population in terms of demographics. During the study period, 2 cytotechnologists left and 2 pathologists joined the cytology laboratory practice. These laboratory staffing changes did not seem to affect cytologic interpretations or screening data during the study period, as there were no differences detected in the ASC-to-SIL ratio or HR-HPV positivity rate in ASC-US cases pre- versus post-FPGS implementation. In addition, post-FPGS, there was a change in the laboratory QC practice, accounting for the significant increase in the QC rate.
For the period prior to FPGS implementation, 3.66% of SurePath specimens could not be evaluated by the FocalPoint Slide Profiler, whereas 2.03% SurePath specimens could not be evaluated by FPGS post-FPGS implementation. This represented a 44.53% decrease in the rejection rate, which was statistically significant. Reasons for failure included problems with coverslipping or staining, insufficient cellularity, and obscuring factors such as blood, inflammation, or marked cytolysis. Instead of reimaging, all rejected cases were screened manually.
Table 1 summarizes the frequency and distribution of various diagnostic categories, as well as the frequency of detecting “no endocervical component” before and after FPGS implementation. Overall, there was an 8% increase in the detection of SIL or above; the majority of these cases were accounted for by the increased detection of LSIL (∼10%). There was no statistically significant difference in the detection of ASC-H, AGC, HSIL, and carcinoma. There was also a 50% increase in the reporting of ASC-US, resulting in a substantial increase in the ASC-to-SIL ratio. However, the ASC-to-SIL ratios for both periods were well within the acceptable range. There were also significant decreases in the rates of negative and unsatisfactory Pap tests. Another significant finding was the increase in the percentage of cases that lacked an endocervical component (ECC) post-FPGS. The percentage of cases that lacked an ECC rose from 12% pre-FPGS to 18% post-FPGS, representing approximately a 50% increase in the lack of an ECC in SurePath cases.
Table 1. Distribution of Diagnostic Categories Before and After Implementation of the FocalPoint GS Imaging System (FPGS)
|Unsatisfactory||0.61 ± 0.23||0.41 ± 0.23||−34.43||.015|
|NILM||91.58 ± 1.22||89.21 ± 2.23||−2.58||.001|
|ASC-US||4.86 ± 0.94||7.39 ± 1.05||+52.06||.000|
|ASC-H||0.23 ± 0.11||0.31 ± 0.12||+34.8||NS|
|AGC||0.14 ± 0.08||0.18 ± 0.10||+28.67||NS|
|LSIL||3.18 ± 0.37||3.49 ± 0.45||+9.71||.032|
|HSIL||0.24 ± 0.63||0.21 ± 0.69||−12.40||NS|
|Carcinoma||0.028 ± 0.05||0.023 ± 0.03||−17.86||NS|
|All SIL and above||3.45 ± 0.39||3.72 ± 0.45||+7.83||.066|
|No endocervical component||11.5 ± 0.86||17.59 ± 2.11||+52.70||.000|
There were significant increases in the frequencies of reporting a shift in bacterial flora and candidiasis after FPGS implementation, even though the screening was limited to 10 FOVs for about 80% of the negative SurePath cases with FPGS, whereas the frequency of reporting of trichomoniasis remained relatively unchanged (Table 2). The reporting of the presence of endometrial cells in women older than 40 years was significantly higher after FPGS implementation.
Table 2. Frequency of Reporting of Microorganisms and the Presence of Endometrial Cells Before and After Implementation of the FocalPoint GS Imaging System (FPGS)
|Shift in bacteria flora||5.41 (4906)||7.00 (6860)||+ 29.39||.000|
|Candida||5.80 (5257)||6.51 (6385)||+ 12.24||.000|
|Trichomonas||0.32 (288)||0.34 (337)||+ 6.25||NS|
|Endometrial cells > 40 years||1.82 (1645)||2.06 (2014)||+ 13.18||.000|
Table 3 summarizes the QC data. There was a significant increase in the number of cases referred to pathologists for review (16%), and selected for QC rescreen (34%) after FPGS implementation. We introduced a systems change in our QC practice postimplementation of FPGS. All Pap tests interpreted as “negative” that were found to be HR-HPV positive were manually rescreened, and all cases reported to have “no ECC” by FPGS were manually rescreened. As a result, there was a significant increase in our laboratory QC rate, which went from 17% pre-FPGS to 22% post-FPGS. This represented a 30% increase in QC. For the first 19 months after FPGS implementation, the average EFNF was 0.98% compared with 1.39% before FPGS implementation. This represented a 36% decrease in EFNF, which was statistically significant. Over 95% of cases that were interpreted as ASC-US were tested for HR-HPV. There was a 14% decrease in the positive HR-HPV rate after FPGS implementation, which was statistically significant.
Table 3. Quality Indicators Before and After Implementation of the FocalPoint GS Imaging System (FPGS)
|Rejection rate by imaging system||3.66||2.03||−44.53||.000|
|HPV positive rate for ASC-US||36.16 ± 5.65||31.13 ± 4.66||−13.91||.037|
|QC review||16.66 ± 1.35||22.41 ± 4.40||+34.45||.000|
|Pathologist referral||22.52 ± 3.22||26.13 ± 2.12||+16.03||.001|
|EFNF||1.39 ± 0.26||0.88 ± 0.37||−36.69||.000|
The FocalPoint GS location-guided imaging system was approved for the primary screening of SurePath Pap tests by the FDA in December 2008 partly based on the result of a multicenter clinical trial.7 The clinical trial, which was conducted in 4 diverse laboratories, compared manual screening and location-guided screening with FPGS. The clinical trial reported an increase in sensitivity of detecting HSIL+ and LSIL+ lesions by 20% and 10%, respectively, using FPGS when compared with manual screening. However, this study did note that these are potentially misleading increases because they resulted from differing classifications between the study arms. For example, if FPGS called a case HSIL and manual review called it LSIL, it was counted as a “miss” for manual screening. Therefore, the sensitivity for HSIL was higher. There were small decreases in the specificity of both HSIL+ and LSIL+ lesions with FPGS, but no difference in the sensitivity and specificity of detecting ASC+ lesions for both manual and computer-assisted screening.
There are also other studies in the literature that have evaluated FPGS for primary screening.8, 9 Because these studies were conducted in Europe, the researchers were able to evaluate the sorting and ranking, as well as the location-guided ability of FPGS. In 1 study that used conventional preparations, the authors reported that FPGS identified cellular abnormalities in the FOVs in 92% of the ranked abnormal slides, showing a sensitivity of more than 95% for SILs.8 On the other hand, a British study using SurePath specimens reported a 6% reduction in the sensitivity of detecting HSIL+ lesions with FPGS when compared with traditional manual screening.9
To the best of our knowledge, the current study is the first study in the English-language literature to evaluate the use of FPGS for primary screening of SurePath Pap tests in a routine clinical setting. Specifically, in the current study we examined the impact of upgrading from the FocalPoint Slide Profiler to FPGS for primary screening of SurePath Pap tests. With a historical HSIL+ lesion rate of 0.24%, our patient cohort represents a relatively low-risk screening population.
After FPGS implementation, our laboratory observed a significant increase in the reporting of squamous abnormalities at the level of ASC-US and LSIL, but not HSIL or carcinoma. There was also a significant decrease in the unsatisfactory rate. Our results were somewhat different from those observed in the clinical trial.7 As with the implementation of any new technology, the results in a clinical trial may not be readily reproducible in the routine clinical setting. In addition, results obtained in 1 laboratory may not be transferable to another laboratory because of differences in the laboratories' makeup, size, location, patient population, and so forth.
Our observations are supported by the finding that many other studies have reported inconsistent findings regarding the impact of implementation of TIS on their rates of ASC-US and SIL.10-21 Based on the data of the clinical trial, Biscotti et al reported an increase in the detection of ASC+ lesions but not LSIL+ or HSIL+ lesions using TIS.13 Other studies showed an increase in the detection of LSIL+ and HSIL+ lesions after implementing TIS.10-12, 15, 16, 18, 22 On the other hand, studies by Roberts et al19 and Schledermann et al20 reported either no change or a decrease in the detection of squamous abnormalities. With regard to ASC-US, several studies reported an increase,11, 12, 15, 20 whereas others reported a decrease or no change in reporting after implementing TIS.10, 16, 21, 23 Some authors attributed the increase in the ASC-US rate to the use of the proprietary ThinPrep Imager stain, which appeared to be somewhat darker than the previous stains used in their laboratories.10 A point to note is that there is no special staining protocol or requirement needed for FPGS. Another point worth noting is that the increased ASC-US rate observed in our study post-FPGS was not a result of early “overcall” in the first few months of operation, as the laboratory ASC-US rate has remained stable at approximately 7% over 1 year postimplementaion of FPGS.
Several authors have recommended the use of the rate of HR-HPV-positive ASC-US cases as a quality indicator in gynecologic cytology.24-26 According to a College of American Pathologists Q-Probe study, the mean percentage of HR-HPV positivity for ASC-US was 43.7% with a standard deviation of 17.7%.25 Our HR-HPV positive rates for ASC-US pre- and post-FPGS were both within 1 SD of the mean reported in the literature. However, there was a significant decrease (14%) in the positive HR-HPV rate for ASC-US after implementing FPGS. This finding is not unique to our study. A similar observation was noted with most of the studies using TIS.10, 11, 15, 16, 21, 23 One plausible explanation is the overinterpretation of reactive changes as ASC-US. However, with an increase in the reporting of ASC-US by 50%, a much greater drop in the HR-HPV-positive rate for ASC-US would be expected if overinterpretation of reactive changes as ASC-US was the sole reason. Another possible explanation is that FPGS identified more abnormal cells, especially those with “very mild and subtle” atypia that might have been missed by manual screening alone.
In an earlier study, Wilbur et al reported that the FocalPoint Slide Profiler (also known as the AutoPap imaging system) was less sensitive than manual screening in identifying infectious organisms based on the clinical trial data.27 In the current study, we observed an increase in the rates of reporting of candidiasis, trichomoniasis, and bacterial vaginosis after FPGS implementation. Therefore, the identification of organisms was not compromised even though the review of the majority of the negative slides was limited to 10 FOVs. Our results parallel the results reported by Papillo et al using TIS.10
In the current study, our EFNF decreased by 36% after FPGS implementation. Given the low prevalence of HSIL in our study population, LSIL diagnoses accounted for the majority of cases that led to the decreased false-negative rate. Other authors have also reported similar findings with the implementation of TIS.10, 12, 28 For example, Miller et al reported an EFNF of 0.04% and 0.02% before and after the implementation of TIS, respectively, representing a 50% decrease.
A puzzling finding was the 50% increase in cases without an adequate endocervical component after FPGS implementation. As mentioned earlier, all cases that were reported to have no endocervical component by FPGS received a full manual review. When we performed a retrospective review, only 10% of these cases were found to have endocervical or squamous metaplastic cells (data not shown). Therefore, screening errors cannot explain the dramatic increase in the number of cases without an adequate endocervical component. Prior to implementation of FPGS, the “no further review (NFR)” category from the FocalPoint Slide Profiler system signed out 25% of “normal” Pap tests without human review. It is plausible that within this subset of NFR Pap tests, a number of cases lacked an ECC and that this finding went undetected prior to sign-out.
One of the limitations of the current study was the inability to assess the performance of FPGS in detecting glandular abnormalities because of the small number of cases with glandular abnormalities. Others have found that TIS was effective in identifying atypical glandular cells within the FOVs.29, 30 However, the effectiveness of FPGS in identifying abnormal glandular cells remains to be elucidated. Interestingly, we observed an increase in the reporting of the presence of benign endometrial cells in women older than 40 years postimplementation of FPGS. However, the clinical significance of this observation is not known.
Although the impact of FPGS on productivity and operation was not the focus of this study, it should be noted that FPGS could also be used as a productivity tool in addition to improving the quality of gynecologic cytology. Because of a systems change in our quality control practice postimplementation, our QC rate increased by 34%. In addition, there were staffing changes that occurred during the study prior to FPGS implementation; 2 pathologists joined the practice and 2 cytotechnologists left just before implementation of FPGS. By applying Lean principles and implementing FPGS at the same time, we were able to shorten our turnaround time by more than 50% and increase staff productivity by 17%, despite the finding that our QC rescreening rate increased by one third, and our staffing changed.31
In conclusion, our study seems to demonstrate a favorable performance of FPGS in the routine clinical setting. We found a significant increase in the detection of ASC-US and LSIL with FPGS and a substantial decrease in the EFNF. However, a major limitation of our study is the lack of follow-up histologic data to support our findings. Although our results seem encouraging, they may be misleading, as our findings may be at the expense of an increased false-positive rate, especially because we found a significant decrease in the HR-HPV-positive rate in our ASC-US cases post-FPGS. Without follow-up data, we cannot validate our findings. In addition, we found a significant increase in the reporting of infectious organisms and in the reporting of the presence of endometrial cells in women older than age 40. With much needed additional long-term follow-up studies, FPGS may have the potential to be a promising screening tool for gynecologic cytology in a low-risk screening population.
Note Added in Proof
No specific funding was disclosed.
CONFLICT OF INTEREST DISCLOSURES
Dr. Chhieng is a paid speaker for BD Diagnostics. The other authors make no disclosures.