Combined strategy of knowledge‐based rule selection and historical data percentile‐based range determination to improve an autoverification system for clinical chemistry test results

Abstract Background Current autoverification, which is only knowledge‐based, has low efficiency. Regular historical data analysis may improve autoverification range determination. We attempted to enhance autoverification by selecting autoverification rules by knowledge and ranges from historical data. This new system was compared with the original knowledge‐based system. Methods New types of rules, extreme values, and consistency checks were added and the autoverification workflow was rearranged to construct a framework. Criteria for creating rules for extreme value ranges, limit checks, consistency checks, and delta checks were determined by analyzing historical Zhongshan laboratory data. The new system's effectiveness was evaluated using pooled data from 20 centers. Efficiency improvement was assessed by a multicenter process. Results Effectiveness was evaluated by the true positive rate, true negative rate, and overall consistency rate, as compared to manual verification, which were 77.55%, 78.53%, and 78.3%, respectively for the new system. The original overall consistency rate was 56.2%. The new pass rates, indicating efficiency, were increased by 19%‒51% among hospitals. Further customization using individualized data increased this rate. Conclusions The improved system showed a comparable effectiveness and markedly increased efficiency. This transferable system could be further improved and popularized by utilizing historical data from each hospital.


| INTRODUC TI ON
Medical laboratory technologists usually have to make a quick judgment on a vast number of reports in a short period of time.
However, with the increasing number of test specimens, the traditional manual auditing mode has drawbacks under high pressure.
Technologists inevitably become fatigued, and mistakes may occur.
Therefore, an autoverification (automated result verification) system is needed to reduce the proportion of manual audits, improve turnaround time, and identify false reports. 1 Autoverification is the automated action of a computer system, related to the release of test results to patients' medical records, using rules and criteria established, documented, and tested by the medical staff of the laboratory. 2  Guideline (AUTO 10-A). 2,3 At that time, limited numbers of test specimens were processed. The knowledge-based autoverification ranges, set by experienced clinical pathologists and medical laboratory technologists, were strict, with a true negative rate of 69%.
Consequently, laboratory technologists expended much effort to review false positive reports. Currently, the increasing numbers of test specimens have imposed an added reviewing workload, bringing challenges to Zhongshan laboratory and other medical laboratories across China. [4][5][6][7] Thus, there is a need to replace the original knowledge-based-only system with a system with comparable effectiveness and higher efficiency.
The experience of laboratory technologists is based on their history of auditing reports. Due to the dietary changes, upgrading of laboratory technology, etc., data generated in a laboratory are dynamic. Rather than relying on human experience, a regular statistical analysis of periodic historical data may be a better way to determine the autoverification ranges. In this study, we explored the feasibility and practicality of using knowledge to select autoverification rules and historical data to determine autoverification ranges. The system established in this way was compared with the original knowledge-based-only system to assess whether it was an improvement.

| Original autoverification system
The original autoverification system of Zhongshan Hospital was designed and established according to Valdiguié's study and the CLSI AUTO 10-A document in 2012. 2,3 The system covered 60 items (tests), including liver function, kidney function, lipids, immunoglobulins, electrolytes, enzymes, and hemolysis, icterus, and lipemia indexes (HIL indexes). It included 359 rules (Table S1).

| Modification of autoverification system framework
Two new types of rules, ie, extreme value (60 rules) and consistency check (9 rules) rules were added in the improved system; thus, the new system comprised 428 rules in total (Table S1) The improved system also collected data for a report (patient information, HIL indexes, test results, sample information, and instrument flags) and worked following the modified framework ( Figure 1).
In this framework, if there was a previous result, the report would not undergo a limit check. If both the current and previous results were within the reference range, a delta check was not needed. The priorities of the rules in the system indicated their importance. This modified framework was defined as the Zhongshan framework.
F I G U R E 1 Framework of the improved clinical chemistry autoverification system. The workflow included 4 steps, the readiness of the test results (red text), the instrumental preset rules (yellow text), the laboratory established rules (blue text), and the comparison (with historical results) rules (green text). The rules in the filled boxes (light yellow) were newly added or rearranged. The autoverification would not be initiated until all the results of a report were ready. If any rule was violated, the report could not be autoverified and it would be transferred to manual verification. QC, quality control

| Determination of the range of each autoverification rule
In the original system, the ranges of limit check rules were deter- The finalized ranges were defined as the Zhongshan criteria.

| Validation of the improved autoverification system
The improved autoverification system contained the Zhongshan framework and the Zhongshan criteria. The system was validated according to the guidance documents of the International

Organization for Standardization (ISO), College of American
Pathologists, and CLSI. 2,8,9 The validation process involved 20 laboratories throughout China (Table S1). All these laboratories utilized the same instruments and reagents. They were all accredited by the governmental authority and qualified by the National External Quality Assessment of China. In addition, 90% of them had achieved ISO 15189 accreditation, which is not mandatory in China. In all these laboratories, IgG, IgA, IgM, immunoglobulin E (IgE), complement C3 (C3), complement C4 (C4), and homocysteine (HCY) were analyzed using a Hitachi modular P analyzer (Hitachi Ltd.) with DiaSys reagents (Shanghai, China), while iron and unsaturated iron binding capacity (UIBC) were analyzed with the same analyzer, using Wako reagents (Osaka, Japan). The other clinical chemistry tests were conducted on a Roche Cobas 8000 modular analyzer series using Roche reagents (Roche). The test results were transferred to LIS using Roche middleware IT3000.
The routine clinical reports generated for the Han Chinese population were collected from the participating hospitals, totaling 2,246,697 (Table S1). Of these, 20,996 reports (ca. 1% of the total reports) were randomly selected and manually veri-  Figure S1). To determine whether the improvements reduced the proportion of manual audits required and increased efficiency, all 2,246,697 reports were analyzed by both the original system and the improved system, and the efficiencies (pass rates) of the two systems were compared for the 20 laboratories.

| Further improvement of the autoverification system
For the laboratory with the lowest pass rate for the improved system, the autoverification system was further improved by determining the ranges of all 60 delta check rules, using its own historical data rather than the Zhongshan data, and the same percentiles as set in the Zhongshan criteria. The efficiency (pass rate) of this laboratory was then re-evaluated using the further improved system consisting of the Zhongshan framework and partially customized criteria.

| Logic and consistency check rules
The tests in a report reflect the condition of the same patient from different angles; hence, their results are internally related. Some minor problems, such as sample aspiration error, drug interference, substrate depletion, or a high-dose hook effect, may not cause extremely abnormal results but will have violated the internal relations. Thus, consistency check rules were newly added to identify these violations. The 9 consistency check rules included (IgG + IgM + IgA)/(KAP + LAM), (IgG + IgM + IgA)/(TP-ALB), CRE/ UREA, product of CA and P, whose frequency distributions of historical data are shown as representative data in Figure 2. The distributions of the four consistency check rules were convergent, and each curve had only one spike, which supported the existence of inter-test relations. The 2.5th-97.5th percentiles of each distribution were used as the autoverification ranges for the four consistency check rules (Figure 2).
In addition, the logic rules reflect the known logical relations among tests. The autoverification range of each logic or consistency check rule is shown in Table 1.

| Extreme value ranges, limit check ranges, and delta check ranges
Similar to the consistency check rules, the frequency distribution of each item was analyzed using the historical data. The 0.1th-99.9th percentile and the 2.5th-97.5th percentile of each distribution were used as the extreme value range and limit check range, respectively (Table S2).
Since two consecutive results for the same patient generally do not vary markedly, the delta check also helps to judge if the current result is plausible. The current results from the historical data were  Table 2.

| The effectiveness of the improved autoverification system
The effectiveness of the improved system, according to the rule ranges based on Zhongshan Hospital data, was evaluated with  (Table S2). These reports were judged by both the expert panel (manual verification) and the autoverification system (Table 3; Figure S2). The extreme value ranges (the 0.1th to 99.9th percentile) and limit check ranges (the 2.5th to 97.5th percentile) were determined using routine laboratory results. The determination of delta check ranges is described in Figure 3

| The pass rates of 20 laboratories
The purpose of the autoverification system is to emancipate laboratory technologists from needing to perform a manual audit. Its efficiency may be assessed by the pass rate. The reports from each of the 20 laboratories were analyzed by the original and improved autoverification systems (Table S2). The improved system increased the pass rate among the 20 laboratories by 19%-51%, and the increases were statistically significant, indicating that efficiency was markedly enhanced by the improved system (Figure 4; Figure S3).
For the laboratory (No. 20) with the lowest pass rate in the improved system (43%, Figure S3), we analyzed the intercepted false positive reports and found that some of these reports violated delta check rules. Considering that the Zhongshan criteria, which were possibly not suitable for laboratory No. 20, were used as the ranges for these rules, we customized the ranges of all 60 delta check rules using this laboratory's own historical data. The autoverification system using the Zhongshan framework and partially customized range criteria for this laboratory resulted in an increased pass rate of 49% (compared to 19% without customization), suggesting that, for each individual laboratory, an autoverification system consisting of the Zhongshan framework and individualized criteria using its own historical data may be a better option (Figure 4).

| DISCUSS ION
In this study, we attempted to enhance autoverification systems by selecting autoverification rules based on knowledge and ranges based on historical laboratory data. Compared with the original knowledge-based system, the improved system showed a comparable effectiveness and markedly increased efficiency. This system was transferable across laboratories and could be further improved and popularized by utilizing historical data from the individual laboratory.
In manual verification or knowledge-based autoverification, the report is typically considered to be abnormal (and intercepted), because a test result is too high or too low and beyond a   Since the follow-up actions for rule violations vary, the preset priority of autoverification rules helps laboratory technologists to determine which response is needed. Rules with a higher priority should be taken more seriously (Figure 1). Violation of logical rules is unacceptable, and the intercepted report cannot be ver- based on their experience. [12][13][14][15][16][17] The key cause of these challenges is the slow generation of the quantitative experience (autoverification ranges) from the vast amount of historical clinical report data.
More specifically, although the production of the historical clinical report data is rapid, the transformation rate by humans (from data to experience) is limited. This study showed that analysis of historical data by computer expedited this transformation. Moreover, efficiency was not affected by the complexity of the rules (Figures 2   and 3). The effectiveness of the improved autoverification system was competitive and the pass rate (efficiency) increased, which indicates that it is both feasible and practical to use a historical data percentile-based range-determining strategy ( Table 3).
The consistent increases in the pass rates among the 20 laboratories and the further increase with the customized system indicates that Zhongshan framework did not reverse the efficiency outside Zhongshan Hospital. The historical data percentile-based range-determining strategy made the knowledge-based Zhongshan framework widely applicable (Figure 4). To examine the improved system outside of Zhongshan Hospital, the clinical chemistry tests at other laboratories were all analyzed by similar instruments. In fact, the construction of the framework relied on medical knowledge and determination of the criteria depended on historical data, rather than being restricted to specific instruments. Therefore, even if transferability of the improved autoverification system with the Zhongshan criteria was limited, the Zhongshan framework and the combined strategy to generate an autoverification system was still transferrable.
This study had some limitations. For example, the frequency distributions of different tests/rules vary, and the percentile may need to be optimized for each of them. Optimization of the autoverification system should be explored in future studies.
Taken together, our study illustrates how to improve and popularize an autoverification system using a combined strategy of knowledge-based rule selection and historical data percentilebased range determination. The demand for autoverification is increasing, but the experience with autoverification is insufficient for many laboratories. Knowledge-based rule selection allows an experienced laboratory to construct a comprehensive autoverification system with a follow-up plan. Additionally, use of historical data percentile-base range determination makes it possible to transfer the system framework to another laboratory. Such an improved autoverification system may help to reduce the proportion of manual audits, improve the turnaround time, and identify false laboratory reports.

CO N FLI C T O F I NTE R E S T
The funding organization(s) played no role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the report for publication. All the authors declare that there is no conflict of interest.