Validation of digital pathology imaging for primary histopathological diagnosis

Introduction

Digital pathology (DP) is the conversion of the light microscope image of a slide into a set of digitized files that allow the reproduction of the original slide on a computer workstation; it is also called virtual microscopy or whole slide imaging. Digitization also allows for the manipulation of the image and or its data to derive information of diagnostic, prognostic or therapeutic benefit, which would otherwise not normally be readily available.[1, 2]

Although it is widely used in research and teaching, this technology has only relatively recently become applicable to routine diagnostic work, through the development of large-capacity slide scanners capable of dealing with the workloads generated daily in modern diagnostic laboratories. These benefits include the ability to report cases remotely from the surgical laboratory generating the slides, thereby allowing for increased and more robust subspecialization, and more efficient use of the pathologist's time, the adoption of computerized analysis to aid and improve diagnosis, and the potential to automate some aspects of the pathologist's workload.

An important initial step in the wider adoption of this technology is the establishment of validation data assessing how effective pathologists are using digital workstations in comparison with conventional light microscopes and glass slide microscopy (GS) when examining cases for primary diagnosis. Previous studies using a range of differing technologies in different clinical settings have reported the comparison of DP with GS,[3-17] and criteria for such comparisons have been published.[18] However, although most of these studies have shown promise, no single study has been sufficiently powered to demonstrate a statistically significant equivalence (i.e. non-inferiority) of DP against GS.

Histopathology is an interpretive discipline, and pathologists’ performance in the analysis of diagnostic surgical specimens naturally varies both within and between observers, as has been measured in previous studies.[19, 20] To achieve a statistically valid outcome, any comparative study must include sufficient specimens to allow for this variation, which should ideally be measured among the group of pathologists taking part in the study. In addition to audit studies,[19] UK National Health Service (NHS) laboratories routinely review a proportion of their cases in multidisciplinary team (MDT) meetings, which are used in the planning of future care for the patient. Recording the number of variances detected when these cases are reviewed at MDT meetings gives a useful measure of a laboratory's baseline variation rate, i.e. GS to GS variance, which can inform the sample size calculation for DP validation studies.

Comparison studies of this type require the establishment of the correct diagnosis or ‘ground truth’, so that each method can be compared with the ground truth independently, thereby avoiding bias from either method. Finally, whichever method is used first, there must be a period of elapsed time before the same case is re-evaluated with the alternative method, to avoid the pathologist(s) remembering the case from previous viewings, the so-called ‘wash-out period’.

In this study, we retrospectively evaluated a series of cases reported on GS and then re-reported on DP workstations, to establish whether there was a significant difference between the diagnoses achieved beyond what would normally be expected through intraobserver and interobserver variation.

Materials and methods

Results

A total of 3103 cases were selected and scanned. The study closed when 3017 cases had been double-reported, leaving 86 cases that were incomplete at the conclusion of the study, comprising 24 that had been examined on the digital system, but required additional work (re-scanning of one or more slide, and cases incomplete at the time of scanning), and 62 that were awaiting re-reporting on the digital system.

The proportion of cases in each subspecialty group is shown in Table 1.

Ninety-seven (3.2%) cases required re-scanning before a DP report could be offered. No cases required additional stains or deeper sections before the DP report was issued.

The 3017 cases generated 10 138 slides, which, when scanned, resulted in a digital archive of 2.22 terabytes. Details of the mean scanning time and the size of the files generated are shown in Table 2.

The cases were re-reported by the same pathologist in 1009 (33.4%) instances and by a different pathologist in 2008 (66.6%) instances. A total of 17 pathologists took part in the study, comprising all substantive consultant histopathologists within the department, two visiting consultant pathologists, and one senior trainee. One pathologist experienced eye strain because of glare when using the digital workstation, owing to a longstanding eye condition, which was successfully alleviated by placing a filter (3M Privacy Filter; 24.0-inch Widescreen 16:10; PF24.0W) over the workstation screens. The other contributing pathologists reported no difficulties in using the system.

A comparison of diagnoses made with GS and DP is shown in Table 3. A total of 72 cases (2.3%) showed variance between GS and DP reports. Of these, 21 (0.7%) were deemed to be of clinical significance, and the details of these are shown in Table 4. Of these discrepancies, the ground truth lay with GS in 12 (57%) cases and with DP in nine (43%) cases. In the majority of cases, the variances were considered to probably represent normal intraobserver and interobserver variation. There were three occurrences for which the reporting pathologists felt that the digital platform was directly responsible for the variance. These were a gastric biopsy in which Helicobacter pylori was not visible on the DP images, and only became so when the slides were re-scanned at the ×60 setting, a bronchial biopsy showing squamous metaplasia in which moderate dysplasia was reported on DP, and a penile biopsy showing human papilloma virus changes in which penile intraepithelial neoplasia was reported on DP.

The primary analysis results are summarized in Figure 1. For concordance defined as complete concordance or variance of no clinical significance, the conclusions for analyses based on all samples, samples diagnosed by same pathologists and samples diagnosed by different pathologists were similar. The lower CI bounds were above the non-inferiority boundary (98%), and so DP was non-inferior to GS. The performances of the same pathologist and different pathologists were also similar with regard to complete concordance against all variances. The secondary analysis results are given in Figure 2. As expected from the primary analysis results, for the percentage of samples for which there was complete concordance, there was variance of no clinical significance, or DP provided the ground truth, DP was non-inferior to GS. For the percentage of samples for which there was complete concordance or DP provided the ground truth, for analysis based on all samples and analysis based on samples diagnosed by different pathologists, DP was non-inferior to GS. However, the results were indeterminate for analysis based on samples diagnosed by the same pathologists.

Discussion

This study is the largest to date of a pathologist's ability to assess histopathology samples on DP as opposed to GS. In line with previous guidelines, we instigated a small pilot study prior to this validation, with the aim of giving each of the pathologists experience in using the DP system.[18] Measurement of the departmental observer variability prior to the commencement of the study ensured that it was adequately powered to confirm that there is no difference in reports made on glass slides and those made on computer workstations. All of the pathologists were able to use the workstations for reporting, and, with the notable exception of oversized slides, the cases included in the study covered the full spectrum of histopathology specimens in the department, including frozen sections, immunocytochemistry, and special stains.

From 3017 cases, differences in diagnosis were identified in 72 (2.3%) cases, of which 21 (0.7%) would have, or were likely to have, resulted in a difference in patient management. These cases are all listed in Table 4, along with the number of slides that they contained; approximately half were single-slide cases, and the others were multi-slide cases. The majority of these variances mapped to a single slide and were very similar to our observed variability prior to the study commencing, as indicated by the ground truth being approximately equally distributed between GS and DP. Several variances were attributable to instances in which important morphological clues, such as Candida hyphae, had been missed on the GS, and it is reassuring that these were picked up on review with DP. The observer variation in our department was 1.22%. This is smaller than the figure used in some comparable studies,[13] and hence necessitated a larger number of cases to achieve the 95% power targeted. Nevertheless, the observed variation is in line with other studies.[21] Interestingly, the variation observed in the study was less than that seen in the MDT review audit. There are several reasons why this might be so. First, the MDT review audit looked at both macroscopic and microscopic sections of the report, whereas the validation study reported here only examined differences in the microscopic report. Second, many of the cases included in the validation study had already been through MDT meetings prior to recruitment, and had therefore been subjected to one review prior to the study review. Finally, the case population for this study was consecutively selected, whereas MDT review cases are selected clinically from patients suspected of having malignancy, where the incidence of clinical significant variances might be expected to be higher.

However, two situations were identified that gave cause for concern regarding the DP system. First, two cases were reported on DP as showing dysplasia that was not present. These two cases were reported by different pathologists, but with the same pathologist reporting both the GS and DP sections. In both instances, no satisfactory reason for the difference was apparent, apart from the fact that the nuclei appeared much darker on DP than on GS, giving an erroneous impression of dysplasia. This potential problem may benefit from further investigation, as we are aware of other groups who have encountered similar problems in Barrett's oesophagus (Dr D Treanor, personal communication) and cervical squamous dysplasia.[14] The other situation concerned the detection of microorganisms. A case of H. pylori-positive gastritis was missed because the organisms were only visible on the ×60 scans. This led to the adoption of scanning rules whereby all such cases are scanned at ×60 by default. Other studies examining this problem have shown that z-plane focusing is also an effective means of improving detection of these organisms.[22]

The intensity of the red colour in some special stains, namely diastase periodic acid schiff (DPAS) positivity for fungi and positive staining for Ziehl–Neelsen in mycobacteria, was noticed to be not as strong on DP as on GS. This problem is related to the colour balance of the DP image, and other studies have also identified this as a problem for identification of eosinophils.[13] Once this problem had been identified, all of the monitors in use were checked, and the colour balance was formally standardized. No further problems were encountered. As with all special stains, it is imperative to include appropriate positive controls to educate the observer about the intensity of the positive stain achieved in any particular run. These events support the findings of other groups that have highlighted specific areas in which DP may differ from conventional GS, and highlight the need for local validation before its implementation.[23]

In our series, we did not encounter any problems with z-plane (vertical) focusing, a weak point of several DP systems that do not allow focusing through the plane of the section. This has been cited as a cause for error in some of the older studies,[24] and, although this factor may still be relevant for cytology samples, it does not seem to be a common problem in histopathology samples, where sections are now routinely cut by most departments at 4 μm.

The DP system gives the operator some advantages, but also imposes some limitations. Although not formally assessed in this study, some aspects of routine reporting were improved by DP; for example, measurements of tumour depth, diameter or margin of clearance appeared to be much quicker and more accurate with the DP system than with GS. Likewise, DP keeps the entire case together, so there is no time lost in searching for individual slides of a case. There is clearly potential for rapid review of previous cases held on an individual patient as the digital archive grows, which will reduce the time lost in waiting for slides to be retrieved from the file. When the laboratory is working fully digitally, there is great flexibility in distributing the reporting workload across the consultant workforce and in sharing of difficult cases, which is likely to be particularly relevant in highly complex small-volume areas of pathology, such as transplant and renal pathology.[25] The impacts that these advantages may have on the diagnostic service are quite complex and outside the scope of this study. Naturally, the switch from GS to DP takes time to adapt to, but all of the pathologists in this study became sufficiently confident in using the system to be happy in ‘signing out’ their diagnoses. The examination of multi-block resection cases did not cause any problems, and nor did these cases appear to be more cumbersome or time-consuming to report on DP than GS.

However, some pathologists found the systematic screening of slides at medium power (×10 magnification) to be more difficult with the DP track ball used in this study than with a conventional microscope stage. The measurement of mitotic rates requires the operator to record mitoses per unit area, which is, in any case, more precise than high-power field areas, which should not, in general, be used. DP permits this to be calculated relatively easily by showing the size of the viewing area on the screen, and offers the potential for automating this function through algorithm development. It is not possible to examine slides with polarized filters with the current generation of DP systems, and immunofluorescent slides could not be scanned on the system used in this study. Of all of the samples studied, these limitations affected the assessments of renal biopsies most. The renal biopsies in this study were scanned at ×60, but, even with this resolution, the interpretation of subtle features of membranous change was difficult. However, immunohistochemistry, which is routinely used to increase the sensitivity for membranous nephropathy in any case, was clearly interpretable on the DP system. Congo red stains could not be examined under polarized light on the DP system, but areas of positive staining could nevertheless be detected, if not confirmed, on DP. It is possible that alternative strategies for identifying amyloid, such as immunocytochemistry for protein P, may assume dominance in the digital era.

Histopathology is a scientific discipline based on the visual observation of the variance of tissues from normal, aligned with knowledge of the pathological processes likely to cause the changes observed, and the astute use of additional tests on the tissue samples to provide additional information of use in the making of a diagnosis. DP enables pathologists to make the same judgements on a computer screen as they would with a microscope. DP image quality at high power is not as good as that obtained with a microscope, to the extent that identifying bacteria, small-cell carcinoma and granulocytic inflammatory cells was best performed by scanning the slides at ×60. Similar observations were made in the study by Bauer et al.,[13] suggesting that, in future, many specimens will have to be scanned at ×60 for diagnosis. However, DP images are clearly adequate for reporting the majority of cases. Retention of the facility to use GS when needed is clearly important at the moment. It remains to be seen whether this will continue to be the case as the ability of slide scanners progresses and alternative strategies to circumvent the existing problems are developed.

In conclusion, this study shows that DP is equivalent to GS for the vast majority of tissue specimens taken for diagnosis. Error rates are similar, and, although there may be some differences in the types of error encountered, some of these can be mitigated by acquiring experience with the system and the appropriate use of higher-resolution scans.

Acknowledgements

This study was supported by an educational grant from Omnyx LLC, 1251 Waterfront Place, Pittsburgh, PA 15222, USA.

Author contributions

D. Snead designed the study, took part in the GS and DP reporting, took part in the steering group meetings, assisted with the database management, and wrote the manuscript. Y. W. Tsang contributed to the study design and slide scanning, took part in the GS and DP reporting, assisted with the database, and contributed to writing the manuscript. A. Meskiri scanned the slides, reviewed GS and DP reports, managed the database and trial records, and chaired the steering group. P. Kimani and R. Crossman carried out the statistical analysis, and contributed to writing the manuscript. N. Rajpoot advised on the study design, and contributed to writing the manuscript. K. Chen, P. Matthews, N. Momtahan, S. Read-Jones, S. Sah, E. Simmons, B. Sinha, S. Suortamo, Y. Yeo and H. El Daly reported GS and DP, and took part in the steering group meetings. I. Cree advised on the study design, took part in the GS and DP reporting, took part in the steering group meetings, and contributed to writing the manuscript.