A comparative evaluation of three consecutive artificial intelligence algorithms released by Techcyte for identification of blasts and white blood cells in abnormal peripheral blood films

Digital pathology artificial intelligence (AI) platforms have the capacity to improve over time through “deep machine learning.” We have previously reported on the accuracy of peripheral white blood cell (WBC) differential and blast identification by Techcyte (Techcyte, Inc., Orem, UT, USA), a digital scanner‐agnostic web‐based system for blood film reporting. The aim of the current study was to compare AI protocols released over time to assess improvement in cell identification.

Traditionally, morphological analysis of blood films are performed manually by light microscopy.However, digital pathology artificial intelligence (AI) systems are beginning to be used as diagnostic tools to replace manual white blood cell (WBC) differentials and blast identification.AI platforms have the capacity to improve over time through "machine learning" by using input data to refine their predictive models and increase accuracy.Therefore, depending on the volume of input data and manufacturer specific algorithms, initial investment in AI should result in improved cell differentials over time.
There are a number of commercially available automated digital microscopy systems that are currently marketed for use in diagnostic laboratories, with CellaVision (CellaVision AB, Lund, Sweden) proving to be arguably the most popular and having the most supporting published literature. 1 Similar competing products include Vision Hema (West Medica, Weiner Neudorf, Austria) 2 and EasyCell (Medica Corporation).However, these are all are closed systems that incorporate proprietary hardware for slide smear making, staining and scanning, as well as downloaded software for analysis to be performed on site.
There are few high-throughput systems that offer the flexibility of manually made slides, agnostic scanners and internet hosted software to enable easy online remote access by multiple users.Such web or cloud based systems may be more practical and economical for regional or multi-site areas.
Like all AI systems, Techcyte (Techcyte, Inc., Orem, UT, USA) uses deep machine learning and image analysis technology for WBC identification, and has the capacity to incorporate expert feedback to further refine its algorithms and improve image analysis over time. 3We have previously reported on the accuracy of peripheral WBC differential and blast identification in abnormal blood films by the Techcyte online AI system. 4 The aim of the current study was to perform a comparison of the three AI protocols released over time (2019 to 2022) to assess for improvement in cell identification.

| Study protocol
The slides for this study were preselected based on diagnosis and then reviewed by the AI configuration on Techcyte simulating closely a morphology bench workflow in a diagnostic setting.Manual microscopic WBC differentials were established on a selection of 124 abnormal blood films as per normal diagnostic laboratory practices.These were then digitized and uploaded to the Techcyte online morphology analysis platform and cells were classified using the provided AI algorithm in 2019 (AI1).This was a prototype developed in 2014 and the results of this analysis were extensively reported. 4en the online AI software was updated in 2020, the same images were processed using the new algorithm (AI2), and the process again repeated with the current version of the software released on Oct 21, 2022 (AI3).The AI differentials were individually correlated to those obtained by manual microscopy (gold standard) for each WBC class, and results compared to determine the superior AI version.

| Peripheral blood samples
Abnormal peripheral blood films were randomly obtained from our local haematology pathology service (New South Wales Health Pathology, New South Wales, Australia) and included 39 acute myeloid leukaemia (AML), and 22 acute promyelocytic leukaemia (APML) and 22 chronic lymphocytic leukaemia (CLL), as well as other pathologies as listed in Table 1.Patients were aged 18-94 (median = 63) years.Slides had been prepared within the last 12 months and were made based on routine laboratory criteria (quantitative abnormality, qualitative flags from Beckman Coulter DXH 800 and 900, Beckman Coulter Australia, Lane Cove West, Australia) and required a manual differential to be performed by trained morphology laboratory personnel, with review by a haemato-pathologist or their delegate before the differential could be validated.As part of normal workflow, the abnormal blood films were stained with Wright's stain, cover slipped, and the WBC differentials were performed at the time the slide was first made based on a 100-cell count by light microscopy, as per routine laboratory procedures.All slides were de-identified and a waiver of human ethics review for this evaluation study was granted by the Hunter New England Local Health District Human Research Ethics Committee.

| AI digital differential
The blood films were scanned using a commercial scanner, Motic Easy Scan (MOTIC, www.motic.com,British Columbia, Canada) at 40Â magnification.The images were stored in .svsformat on a local hard drive and then uploaded to the online morphology platform, Techcyte (www.techcyte.com).The Techcyte cloud-based software analysed 500 WBC (AI1) or 200 WBC (AI2 and AI3) per slide, as recommended by the manufacturer, to generate each AI informed digital differential.
The cell types analysed included basophils, eosinophils, blasts, promyelocytes, metamyelocytes, myelocytes, neutrophils, segmented or band forms, promonocytes, monocytes, lymphocytes and smudge cells.Techcyte automatically classifies neutrophils into segmented or band forms, however we chose to analyse these together as neutrophils.
Because of the limited numbers of promyelocytes, metamyelocytes and myelocytes, these cells were grouped together as immature granulocytes.There is significant inter-observer subjectivity in classifying precursor granulocytes, therefore for purpose of this study we chose to compare the total granulocytes (promyelocytes, metamyelocytes, myelocytes and neutrophils) and immature granulocytes separately. 5,6Blasts were counted separately.The AI differential count was then downloaded for comparison to the manual microscopy film examination results.The AI differential was performed in 2019 (AI1), 2020 (AI2), and again in 2022 (AI3) using the same images that had been retained in the online software.For the purposes of the present evaluation, there was no manual reassignment by a morphologist at any time point.

| AI2 and A3 categorized all cells
The original AI1 contained a classification category termed "other," where cells that were not able to be identified by the AI were listed.
WBC differentials of the 124 abnormal blood films analysed resulted in 81/124 slides with cells in this default group (Table 1), ranging from 0.2%-70% of cells (data not shown).Overall, there were 43/124 (35%) of slides containing more than 10% of cells that were classified as "other" by the AI1 analysis.These higher rates of unclassified cells were more common in non-leukaemic films and those with red blood cell abnormalities.Because there was no manual reassignment of cells, these unclassified cells were omitted from the present analysis.
Hence, the AI1 data is inherently biased towards the cells it was confident to classify.In contrast, AI2 and AI3 no longer contained this category, forcing all cells to be categorized into the differential results.
Of note, only 1 slide was correctly identified as having no blasts by all three AIs.There were 2 slides that were correctly identified as having no blasts by both AI1 and AI2, and 3 in common between AI2 and AI3.

| DISCUSSION
We analysed 124 abnormal adult peripheral blood films by manual microscopy and compared the results to digital differentials produced by the Techcyte web-based AI platform over time.Like most proprietors, Techcyte does not reveal the intricacies of its AI algorithms or training protocols.To our knowledge, this is the first independent report documenting the improvement of AI results after a 3 year period of machine learning.
We observed significant improvement from the original Techcyte 2019 algorithm to the latest 2022 release for identification of neutrophils, total granulocytes, immature granulocytes, and promyelocytes.Many studies focus on user defined reclassified cell results rather than the automated AI generated differential since this is how the technology is used in clinical practice.However, the accuracy of the automated pre-reclassification results is an important factor in determining the amount of time required to generate reliable results and produce a validated report.Hence for the purposes of the current study, there was no manual cell reassignment by a morphologist at any time point, and only the raw AI generated data was compared to gold standard of manual light microscopy WBC differentials.
The most notable difference in the later Techcyte protocols compared to the original 2019 AI1 release was the removal of the "other" cell category, forcing all cells to be identified by AI2 and AI3.In 2019, 43/124 (35%) slides had >10% of cells in this category, perhaps explaining why AI1 required analysis of 500 cells to produce a meaningful WBC differential.Although we have previously reported on the manual reclassification required for the AI1 results, we did not record the cell types that needed to be manually reclassified or those that were unable to be classified by AI1. 4 Nor have we performed manual reassignment on the AI2 and A3 results to determine if a particular cell type is more likely to be wrongly forced into classification by the newer algorithms.This is a limitation that will need to be addressed in future studies.
Most available AI analysis programs, including CellaVision, contain a grouping for "unidentified" cells; however, the number of cells preclassified into this category is rarely included in published reports.accuracy rates and perhaps limiting the use of AI in this setting. 10The performance of Techcyte AI on paediatric samples has yet to be evaluated.
Although elimination of the "unidentified" cell category may seem technically warranted in some instances, it is unlikely to be acceptable for the majority of laboratories where traditional workflow relies on this labelling to prompt additional review of slides with suspicious looking cells.
All three Techcyte AIs maintained high sensitivity for blast identification in malignant films, culminating with 100% sensitivity by AI3.This compared favourably with published results of CellaVision DM96 pre-classification sensitivity of 74%-83% for blast detection. 11,12However, Techcyte specificity for blast identification declined from 24% for AI1, to just 12% for AI3, performing much worse than published reports of 59%-66% for the DM96. 11,12though this high rate of false positives would require additional ble due to the ever increasing amount of generated data that needs to be effectively managed, stored and shared.In addition to providing such a platform, Techcyte is unique in its ability to process data from any stained slide uploaded from any scanner, allowing the technology to be easily incorporated into current workflows without the requirement for major infrastructure investment.This flexibility will undoubtedly enhance patient care, particularly in remote and regional areas where haematopathology expertise is limited.
Raw counts for individual cell types were converted to percentage of total cells counted and summary data is presented as mean ± standard deviation (SD).Statistical analysis was performed using MedCalc Statistical Software version 20.123 (MedCalc Software Ltd., Ostend, Belgium).For correlation analyses, Passing and Pablok regression slopes were plotted and Spearman rank order correlation coefficients (r) calculated.Lin's concordance correlation coefficients (CCC) were calculated to assess the level of agreement between each AI and manual microscopy, and a comparison of correlation coefficients was performed by converting them to z-scores to assess the statistical significance of the difference between the correlations.Clinical sensitivity and positive predictive value (PPV) as well as specificity and negative predictive value (NPV) of blast detection by the AI software were defined by the ability to obtain positive and negative results concordant with manual microscopy (true positives [TP] and true negatives[TN]), compared to the number of discordant results (false positives [FP] and negatives [FN]).Results were calculated as: sensitivity = TP/(TP + FN), PPV = TP/(TP + FP), specificity = TN/(FP + TN), and NPV = TN/(TN + FN).p-values <0.05 were considered statistically significant.

1
Correlations of A1, AI2 and AI3 to manual microscopy for indicated cell types.Passing and Pablok regression curves with 95% confidence intervals are illustrated for all cell types except for immature granulocytes, where the data was not suitable for such analysis.Traditional regression curves have instead been substituted for all AI analysis of immature granulocytes.Perfect 1:1 correlations are indicated by 45 black lines and calculated Spearman rank order correlations (r) are provided.a Total granulocytes = neutrophils + myelocytes + metamyelocytes + promyelocytes; b Immature granulocytes = myelocytes + metamyelocytes + promyelocytes.[Correction added on 6 October 2023, after first online publication: Figure 1 has been replaced in this version.]

F I G U R E 2
AI concordance with manual microscopy.Graphical comparison of Lin's concordance correlations (LCC) calculated for each AI compared to manual microscopy for individual cell types.Statistically significant improvements are indicated by *p < 0.05, **p < 0.01, ***p < 0.001.
review of Techcyte results by a pathologist, this is clinically more desirable than high specificity where false negative results could lead to a missed cancer diagnosis.With digital smart phones becoming an integral tool for clinical decision making, it is no surprise that cloud based AI pathology platforms such as Techcyte are steadily increasing in availability and use, with examples including the recently FDA approved Scopio Full-Field™ Peripheral Blood Smear Application (Scopio Labs, Tel Aviv, Isreal), 13 as well as Celly.AI (Celly.AI Corp, Covina, CA, USA), Morphogo (Hangzhou, China), EatherAI (Taipei, Taiwan), and Morphle (Bengaluru, Karnataka, India).Despite the current diagnostic laboratory propensity towards downloadable software in a secure local server environment, the future use of cloud based platforms is inevita- Distribution of pathologies and slides with unidentifiable cells by AI1.[Correction added on 6 October 2023, after first online publication: The data in the last row, last column, was corrected to 43 (35%) in this version.] Abbreviations: ALL, acute lymphoblastic leukaemia; AML, acute myeloid leukaemia; APML, acute promyelocytic leukaemia; B-NHL, B-cell non-Hodgkin's lymphoma; CLL, chronic lymphocytic leukaemia; MDS, myelodysplastic syndrome; MM, multiple myeloma; MPN, myeloproliferative neoplasm; RBC, red blood cell.
AI sensitivity and specificity for blast identification compared to manual microscopy.