Relevance of the College of American Pathologists guideline for validating whole slide imaging for diagnostic purposes to cytopathology

Abstract Whole slide imaging (WSI) allows pathologists to view virtual versions of slides on computer monitors. With increasing adoption of digital pathology, laboratories have begun to validate their WSI systems for diagnostic purposes according to reference guidelines. Among these the College of American Pathologists (CAP) guideline includes three strong recommendations (SRs) and nine good practice statements (GPSs). To date, the application of WSI to cytopathology has been beyond the scope of the CAP guideline due to limited evidence. Herein we systematically reviewed the published literature on WSI validation studies in cytology. A systematic search was carried out in PubMed‐MEDLINE and Embase databases up to November 2021 to identify all publications regarding validation of WSI in cytology. Each article was reviewed to determine if SRs and/or GPSs recommended by the CAP guideline were adequately satisfied. Of 3963 retrieved articles, 25 were included. Only 4/25 studies (16%) satisfied all three SRs, with only one publication (1/25, 4%) fulfilling all three SRs and nine GPSs. Lack of a suitable validation dataset was the main missing SR (16/25, 64%) and less than a third of the studies reported intra‐observer variability data (7/25, 28%). Whilst the CAP guideline for WSI validation in clinical practice helped the widespread adoption of digital pathology, more evidence is required to routinely employ WSI for diagnostic purposes in cytopathology practice. More dedicated validation studies satisfying all SRs and/or GPSs recommended by the CAP are needed to help expedite the use of WSI for primary diagnosis in cytopathology.


| INTRODUC TI ON
Digital pathology consists of viewing, sharing, and/or analysing digitised pathology glass slides employing computer-based technology. 1 There are numerous clinical (e.g., primary diagnosis, telepathology, image analysis) and non-clinical (e.g., research, education) applications of digital pathology. Imaging technology related to digital pathology has evolved over time, from static images (microphotographs of a field of view on a slide) to dynamic images (transmission of images in real time), and more recently to whole slide imaging (WSI). WSI technology refers to scanning glass slides to generate digital slides that can be viewed on a computer monitor to recreate a virtual experience that is similar to examining the glass slides with a traditional light microscope. 2 To demonstrate that this technology works safely for diagnostic patient care and that it can accordingly be adopted for routine clinical work, WSI systems should ideally undergo validation before deployment in clinical service. The crux of such a validation study is to ensure that pathologists' diagnoses using WSI are as accurate as those rendered with glass slides and a light microscope. To assist pathology laboratories with this validation process, in 2013 the College of American Pathologists (CAP) published a specific guideline on how to validate WSI for diagnostic purposes. The CAP guideline incorporated 12 statements to guide pathology laboratories. 3 The CAP guideline was subsequently updated in 2021, 4 and differed from the previous publication because a Grading of Recommendation Assessment, Development, and Evaluation (GRADE) 5 framework was adopted to evaluate available evidence. Moreover, the concept of good practice statements (GPS) was introduced. GPSs differ from strong recommendations (SRs) because while they support important issues they lack the published evidence typically needed for a recommendation. The updated CAP guideline comprising three SRs and nine GPSs is summarised in Table 1.
Most validation guidelines related to WSI for diagnostic use, including the aforementioned published CAP recommendations, do not specifically include cytology. In fact, the authors of the CAP guideline underline that at the time of publication, due to lack of published evidence, validation of WSI in cytology was considered beyond the scope. Indeed, the adoption of digital cytology has lagged behind that of digital histopathology for several reasons, such as the difficulty of scanning cytology material on glass slides in different focal planes using Z-stacking. 6 Not surprisingly, published clinical validation studies in cytology are less numerous than those involving surgical pathology.
The aim of this study was accordingly to investigate the published literature concerning the validation of WSI systems specifically in cytology, with reference to the CAP guideline.

| Literature search and article screening
The review question was formulated according to a Population, Index, Comparator, Outcome (PICO) model. Population was represented by a series of cytology cases collected retrospectively or prospectively for the validation study; the Index was the WSI modality for pathology cases, while the Comparator was represented by conventional light microscopy. Outcome was represented by concordance between a diagnosis rendered with WSI and light microscopy, the latter being taken as the reference standard. The main aim of the study was to investigate the adherence of validation studies for WSI in cytology to the CAP guideline. Studies represented by abstract only with limited information were excluded.
A systematic review was conducted according to standard methods and reporting in accordance with the Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA). 7 The databases PubMed and Embase were systematically searched up to 20 November 2021 to identify any article regarding a validation study of WSI in cytology. The search strategy comprised combinations of the terms "digital pathology," "validation," and "cytology" with their conceptual aliases and variations, adequately adapted to the two databases' search engines. Four authors (AE, IG, NS, PA) independently reviewed all article titles and abstracts with the aid of the Rayyan reference manager web application. 8 Papers dealing with digital pathology other than human cytology (e.g., histopathology, frozen sections of surgical specimens, etc.), with static and dynamic images, or with animal or experimental models, were excluded, as well as papers in languages other than English.
Full texts of the articles fulfilling initial screening criteria were acquired and reviewed against the eligibility criteria. Any disagreement with respect to inclusion of a particular article was resolved by consensus.

| Data extraction
Two investigators (SN, PA) independently extracted data from the included studies with a standardised form. Data extracted included: author(s) and publication year, country of origin for the research, total number of cytological cases, site(s) of origin of the cytological material, and compliance with the CAP guideline criteria for SRs and GPSs.

| Overview of the papers
A flow diagram of the screening, selection, and exclusion of articles for this review is shown in Figure 1. Briefly, 3963 papers were found and screened with the aid of the Rayyan reference manager web application. 8 After title and abstract screening were undertaken, 69 papers were selected as potentially relevant to the review and after subsequent full text assessment 44 articles were then excluded.

| DISCUSS ION
Whole slide imaging technology involves the acquisition of digital images of entire pathology glass slides. 34 WSI has numerous benefits such as portability of pathologists, easy sharing of digital slides, side-by-side comparison of slides on a monitor, image analysis, and

SR 1
The validation process should include a sample set of at least 60 cases for one application, or use case (e.g., haematoxylin-eosin-stained sections of fixed tissue, frozen sections, haematology), that reflect the spectrum and complexity of specimen types and diagnoses likely to be encountered during routine practice. The validation should include another 20 cases to cover additional applications such as immunohistochemistry or other special stains if these applications are relevant to an intended use and were not included in the 60 cases mentioned above.

SR 2
The validation study should establish diagnostic concordance between digital and glass slides for the same observer (i.e., intra-observer variability). If concordance is less than 95%, laboratories should investigate and attempt to remedy the cause.

SR 3
A washout period of at least 2 weeks should occur between viewing digital and glass slides GPS 1 All pathology laboratories implementing WSI technology for clinical diagnostic purposes should carry out their own validation studies.

GPS 2
Validation should be appropriate for and applicable to the intended clinical use and clinical setting of the application in which WSI will be used. Validation of WSI systems should involve specimen preparation types relevant to intended use (e.g., formalin-fixed, paraffin-embedded tissue; frozen tissue; immunohistochemical stains). If a new application for WSI is contemplated, and it differs materially from the previously validated use, a separate validation for the new application should be performed.

GPS 3
The validation study should closely emulate the real-world clinical environment in which the technology will be used.

GPS 4
The validation study should encompass the entire WSI system. It is not necessary to separately validate each individual component (eg, computer hardware, monitor, network, scanner) of the system or the individual steps of the digital imaging process.

GPS 5
Laboratories should have procedures in place to address changes to the WSI system that could impact clinical results.

GPS 6
Pathologists adequately trained to use the WSI system must be involved in the validation process.

GPS 7
The validation process should confirm all of the material present on a glass slide to be scanned is included in the digital image.

GPS 8
Documentation should be maintained recording the method, measurements, and final approval of validation for the WSI system to be used in the anatomic pathology laboratory.

GPS 9
Pathologists should review cases/slides in a validation set in random order. This applies to both the review modality (ie, glass slides or digital) and the order in which slides/cases are reviewed within each modality. that included 600 total cases, 204 of which were cytology specimens, did not report on concordance rates. Only six of their cases revealed minor discordances, none of which involved the cytopathology cases. Hence, supposedly the intra-observer concordance for their cytology cases was 100%. Mukherjee et al 30 reported an intra-observer concordance for 12 thyroid cases that were scanned with three, five, and seven focal planes. Their intra-observer concordance ranged from 92% to 100%.
Finally, SR 3 states that a washout period of at least 2 weeks should occur between viewing digital and glass slides. This recommendation is intended to address the issue of recall bias when cases are reviewed using different modalities by the same observer. 43

| Good practice statements (GPSs)
In the CAP guideline, GRADE introduced the concept of GPS for several issues where published evidence was lacking to support specific recommendations. Overall, 50% of all the GPSs were met, compliance was not specified for 47% of GPSs, and 3% were not satisfied at all. Evaluation of included publications for fulfilment of GPSs was difficult given that extensive descriptions of the study setting were not always available. GPS 1, 2, and 4 were satisfied in all 25 studies.
GPS 3 states that the validation study should closely emulate the real-world clinical environment in which the technology will be used and laboratories are free to incorporate whatever they feel would be appropriate to achieve this goal. In our review, while six studies failed to report, only two studies provided data that did not comply with this parameter. Namely, Bongaerts et al, 16  Only two studies fulfilled this criterion.
GPS 6 states that pathologists adequately trained to use a WSI system must be involved in the validation process. As clearly reported by the CAP, this was not an evidence-based recommendation. Moreover, no metrics were suggested to determine technical competency of pathologists using WSI systems. Instead, adequate training is best defined at the discretion of the laboratory medical director. 4 The same applies for the number of pathologists participating in the validation process. In our review, for 64% of the studies pathologist training and competency was not specified.
Of interest, Hang et al 22

| CON CLUS ION
Increasing global experience and published data support the diagnostic use of WSI for cytopathology. However, extensive validation studies for such diagnostic use in routine cytology practice is still required. Most publications to date about validation studies using WSI for diagnostic use in cytology failed to satisfy many of the recommendations established in the CAP guideline. We accordingly recommend that future validation studies in this field be conducted with more a rigorous study design, in terms of better adherence to the guideline, which will help generate robust evidence to support the successful deployment of WSI for diagnostic use in cytology. Finally, WSI is the first step towards the implementing of artificial intelligence-aided diagnostics, which may be particularly useful in screening cytology, where most of the cases are negative. This will create the need for new validation criteria which will have to be included in future recommendations and guidelines.

AUTH O R CO NTR I B UTI O N S
All authors participated in the conception and design or analysis and interpretation of the data. All authors contributed to the drafting of the manuscript and approved the final version of the manuscript.

ACK N OWLED G EM ENT
Open Access Funding provided by Universita degli Studi di Verona within the CRUI-CARE Agreement.

CO N FLI C T O F I NTE R E S T
The authors declare no conflicts of interest.

DATA AVA I L A B I L I T Y S TAT E M E N T
Data sharing is not applicable to this article as no new data were created or analysed in this study.