Evaluation of the analytical performance of endocrine analytes using sigma metrics

Abstract Background (a) To evaluate the clinical performance of endocrine analytes using the sigma metrics (σ) model. (b) To redesign quality control strategies for performance improvement. Methods The sigma values of the analytes were initially evaluated based on the allowable total error (TEa), bias, and coefficient of variation (CV) at QC materials level 1 and 2 in March 2018. And then, the normalized QC performance decision charts, personalized QC rules, quality goal index (QGI) analysis, and root causes analysis (RCA) were performed based on the sigma values of the analytes. Finally, the sigma values were re‐evaluated in September 2018 after a series of targeted corrective actions. Results Based on the initial sigma values, two analytes (FT3 and TSH) with σ > 6, only needed one QC rule (13S) with N2 and R500 for QC management. On the other hand, seven analytes (FT4, TT4, CROT, E2, PRL, TESTO, and INS) with σ < 4 at one QC material level or both needed multiple rules (13S/22S/R4S/41S/10X) with N6 and R10‐500 depending on different sigma values for QC management. Subsequently, detailed and comprehensive RCA and timely corrective actions were performed on all the analytes base on the QGI analysis. Compared with the initial sigma values, the re‐evaluated sigma metrics of all the analytes increased significantly. Conclusions It was demonstrated that the combination of sigma metrics, QGI analysis, and RCA provided a useful evaluation system for the analytical performance of endocrine analytes.

The use of sigma (σ) metrics is a great success in the areas of customer satisfaction and global profitability, 4 It was introduced into clinical laboratories by David Nevalainen1 in 2000. 5 Currently, sigma metrics had been widely used in many aspects of laboratory quality management including pre-analytic, 6,7 analytic, [8][9][10] and post-analytic 11 phases of testing. The analytical performance of analytes is quantitatively estimated as a sigma value. The value is calculated based on three parameters: allowable total error (TEa), bias, and coefficient of variation (CV). 11 Though sigma metrics were applied for quality management of analytical biochemistry processes, 12 it is rarely used for the quantitative immunoassay testing processes. This is particularly the case in testing the analytical processes of endocrine analytes.
In this study, the analytical performance of thirteen endocrine immunoassay analytes was evaluated by calculating their sigma values based on their TEa%, Bias%, and CV%. The quality control (QC) strategies were then personalized and redesigned for each analyte based on their sigma value. Moreover, the quality goal index (QGI) ratios of the analytes with σ below 4 were calculated to determine whether its precision or accuracy that needs to be improved first.
Besides, the root cause analysis (RCA) and corrective actions were performed to reveal and eliminate the potential negative factors that affect analytical performance. Finally, the sigma metrics of the analytes were re-evaluated to verify the validity of the RCA and corrective actions.

| Study design
This study comprised three steps: the initial evaluation phase, RCA and corrective action step, and the re-evaluation step ( Figure 1). The study was conducted in the department of laboratory medicine of YueBei People's Hospital between October 2017 and September 2018. Sigma metrics values for the analytes were calculated using the following formula: σ=|TEa − Bias|/CV. 13 This was the initial σ values of thirteen endocrine analytes. The QGI analysis, RCA, and corrective actions were performed, respectively, to find and eliminate the potential causes of poor clinical performance of the analytes. The σ values of the analytes were then re-evaluated to verify the effectiveness of previous RCA and corrective activities.

| Instrument, reagents, and analytes used
The analyzer of automatic electrochemical luminescent immunoassay analyzer (E602, Roche, Switzerland) and specific reagents were all

| TEa
In this study, there were two sources of TEa as follows: one TEa was  (Table S1). According to the 2014 Milan strategic conference, 14 the analytes' TEa-NCCL and TEa-EFLM specifications, respectively, were constructed based on the effect of test performance on clinical outcomes and the components of biological variation of the measured.

| Bias
In  . The normalized performance decision diagram was drawn with CV/TEa as abscissa (imprecision) and Bias/TEa as ordinate (inaccuracy), and the chart is divided into six areas by five performance lines. Different colored circles represent different sigma grades factors on analytical performance. The calculation formula of bias was as follows, 15 (taking the bias of FT3 as an example):

| CV%
The daily internal quality control (QC) material Level 1 (

| QGI
QGI analysis helps laboratories to identify the main causes of low sigma value of analytes as well as excessive CV and bias or both. [16][17][18] In this present study, the QGI ratios of the analytes with the initial σ NCCL < 4 were calculated based on the formula QGI = Bias/ (1.5 × CV). 19 QGI < 0.8 indicates that the precision of the measurement procedure needs to be improved, QGI > 1.2 indicates that the accuracy of the measurement procedure needs to be improved, while 0.8 to 1.2 indicates that the precision and accuracy of the measurement procedure all need to be improved.

| Construction of the normalized QC performance decision chart
The normalized QC performance decision chart was constructed by registering an account in the CLInet (http://www.clinet.com.cn) with CV/TEa as abscissa and Bias/TEa as ordinate. [16][17][18]20,21 The chart is divided into six grades by five lines. 22 Based on the sigma level, the performance of the analytes was divided into six grades 23 : worldclass (σ > 6), excellent (5 ≤ σ < 6), good (4 ≤ σ < 5), marginal (3 ≤ σ < 4), poor (2 ≤ σ < 3), and unacceptable (σ < 2) ( Figure 2). The sigma value of the analyte was represented by colored circles marked in certain sigma grades of the chart when the parameters of the analyte's name, TEa, bias, and CV were inputted into the interface. This approach helped laboratory staff to obtain a visual synthesis view of the analytes' performance in a single chart at each QC measurement level.

| RCA and corrective activities
RCA was applied to determine the poor performance reasons for analytes with σ < 4. 15,24 It was performed based on five vital aspects: personal, equipment, material, method, and environment-related to poor performance. This was done to determine multiple sources of poor performance rather than simply classifying an error as precision and/or an accuracy problem. Based on RCA results, appropriate improvement strategies were framed through brainstorming sessions with clinical quality management. The framed strategies were implemented for 6 months (from April to September 2018) in our clinical laboratory.

| Initial sigma metrics evaluation of the analytes' performance
The sigma metrics of every analyte at the QC material Levels 1 and 2 were calculated based on two kinds of TEa and summarized in Table 1. Normalized QC sigma charts were also constructed to visually evaluate the performance of the analytes at each QC material level ( Figure 2). When we chose TEa-NCCL for the sigma metric evaluation, nine of the thirteen analytes exhibited a performance of at least 4σ (good) at the QC material Level 1, and three of these analytes (FT3, TSH, and PROG) had a world-class performance (Table 1 and Figure 2A). In the same line, seven of the thirteen analytes had a performance of at least 4σ (good) at NCCL Level 2. Two of these analytes (FT3 and TSH) had a world-class performance (Table 1 and  (Table 1 and Figure 2C). In the same line, eight of the thirteen analytes had a performance of at least 4σ (good) at EFLM Level 2. Five of these analytes (LH, TSH, PROG, CROT, and PRL) had a world-class performance (Table 1 and Figure 2D).

| QC procedure redesigned for the analytes based on sigma metrics
The redesigned QC procedures for the thirteen analytes at different QC material levels are shown in Table 2. For analytes FT3 and TSH that had a "world-class" analytical performance (σ ≥ 6) at both QC material levels, only one QC rule (1 3S ), one measurement at two QC material levels (N2) per QC event, and a run size of 500 clinical samples between adjacent QC events (R500) were adopted for QC management ( Table 2). For analytes CROT, LH, and PROG that had "excellent" analysis performance (5 ≤ σ < 6) at one or both QC material levels, three multi-rules (1 3S /2 2S /R 4S ) with N2 and R500 were adopted for QC management. For analytes TT3, TT4, E2, FSH, LH, PRL, and TESTO that had "good" analysis performance (4 ≤ σ < 5) at one or both Q C material levels, four multi-rules (1 3S /2 2S /R 4S /4 1S ) with N4 and R200-500 were adopted for QC management. For analytes FT4, TT4, CROT, E2, FSH, PRL, TESTO, and INS that had "marginal," "poor," or "unacceptable" performance (σ < 4) at one or both QC material levels, five multi-rules (1 3S /2 2S /R 4S /4 1S /10 X ) with N6 and R10-380 were adopted for QC management. Only the run size of TT4 was smaller than its average daily measurements at QC materials level 1.
This suggested that two or more QC events could be performed per day at QC materials level 1 ( Table 2). These results further suggested that the sigma metrics values could help in designing personalized QC procedures for the analytes at each QC material level.

| QGI analysis, RCA, and corrective actions
The QGI analysis was thus performed to explore reasons for the low sigma metrics values. Four analytes (TT4, CROT, E2, and FSH) had poor precision at one QC material level, two analytes (FT4 and PRL) had undesired accuracy and precision at one or both QC materials levels, while two other analytes (TESTO and INS) exhibited low accuracy at one or both QC materials levels (Table 3). Five root causal factors, personnel, equipment, material, method, and environment, were scrutinized to identify the root causes of poor precision, accuracy or both (Table S2). For instance, four analytes (two analytes with σ NCCL < 4 and two analytes with σ NCC L > 6) independently detected by two staff were evaluated using sigma metrics to explore the personnel factors ( Cognizant to this, corrective actions that included relearning of the standard operation processes, operational skills retraining, and basic knowledge reassessment of all staff were performed to improve the quality of analysis (Table S2). Also, two analytes (FT3 with σ NCCL > 6 and INS with σ NCCL < 4) were analyzed by the same staff from January

| Re-evaluated analysis performance of the analytes in September 2018
The sigma metrics of thirteen analytes were re-evaluated in Note: The run sizes, Ped, and Pfr of QC procedures were estimated value based on this novel study. 17,28 The average daily measurements of analytes were sourced from statistical analysis of the total measurements in 2017. N: the total number of QC measurements per run of Roche E602, N2 represents two measurements at a single QC material level or one measurement at two QC material levels, similar definitions apply to N4 and N6. R: the run size of patient samples between QC events, R500 represents a run size of 500 patient samples between QC events. (-): represent the Ped and Pfr of this QC procedure were not clear.
a When the two levels of QC procedures of the same analyte were different, the more strict QC procedure (more rules, larger N, and smaller R) was selected for daily QC management.
level (Table S3). Besides, three analytes (FT4, FSH, and PRL) exhibited a world-class analysis performance at one QC material level (Table S3). The remaining three analytes also exhibited significantly improved sigma metrics values (σ > 4.6) at both QC materials level compared to the initial assessment results (Table S3). Circles of all the analytes that had initial sigma metrics values of less than 4 (σ < 4) moved down to the bottom left of the normalized QC performance decision chart (Figure 4). This was an indication that the bias and CV of these analytes had decreased with the improvement in precision and accuracy. Ultimately, the second sigma metrics evaluation results proved that the RCA and corrective actions performed were effective in improving the analysis performance of the analytes. B whose research in the practical application of sigma metrics management in analytical biochemistry processes. 15 Zhou also reported that the factors of different detection systems, the sources of TEa, and the algorithms of CV and bias could be the common causes of this phenomenon. 15 Moreover, acquiring the appropriate TEa is an important challenge while using sigma metrics for performance as-   Though the normalized QC performance decision chart provided visual performance differences of the analytes, it could not present the reasons for quality errors such as those caused by imprecision, inaccuracy, or both. This phenomenon was also observed by Qiu HW et al 22 The

| D ISCUSS I ON
QGI analysis was to remedy this defect by providing easy insights into where sigma quality improvement was required. Further to this, the RCA analysis provided a structural and standardized framework to investigate five potential causal factors (Table S2). This analysis also helped the laboratory staff to identify and efficiently solve the problems. However, the possible root causes were only established in the conditions of our laboratory (Table S2). As such, other superficial and deep-seated problems could also exist. Based on the re-evaluated sigma metrics, it was ev- were not assessed and thus their influence in the results was not reflected. In future studies, these aspects should be prioritized to generate more conclusive results.

| CON CLUS ION
The combination of sigma metrics evaluation, QGI analysis, RCA, corrective actions, and sigma re-evaluation was adopted as a useful approach for performance improvement of analytes with σ < 4.
Indeed, the sigma metrics method provided a useful evaluation system for the analytical performance of endocrine analytes.

F I G U R E 5
The workflow for performance improvement of endocrine analytes based on sigma metrics. The sigma value of analytes was calculated using the TEa-NCCL. R, Run size of patient samples between QC events, R500 represents a run size of 500 patient samples between QC events; R200-500 represents the run size interval from 200 to 500 depending on the sigma value of analytes, and a similar definition applied to R10-500. When the two levels of QC procedures of the same analyte were different, the more strict QC procedure (more rules, larger N, and smaller R) was selected for analyte's performance improvement

ACK N OWLED G M ENTS
We appreciate Chen Chen, quality supervisor of QUALAB Company (Shanghai, China), who gave some important advice for this study.

CO N FLI C T O F I NTE R E S T
The authors stated that there are no conflicts of interest regarding the publication of this article.

AUTH O R S' CO NTR I B UTI O N S
YL and YC designed the study, searched the literature, performed the experimental procedure, analyzed and interpreted the data, and wrote the study; XL, LW, WC performed the experimental procedure and searched the literature. All authors read and approved the final study.

E TH I C S A PPROVA L A N D CO N S E NT TO PA RTI CI PATE
Not applicable.

CO N S E NT FO R PU B LI C ATI O N
All the authors agree on the publication of this article.