Validation of data quality in a nationwide gastroenterological surgical database: The National Clinical Database site‐visit and remote audits, 2016‐2018

Abstract Background and Aim In 2015, the Japanese Society of Gastroenterological Surgery (JSGS) initiated data verification in the gastroenterological section of the National Clinical Database (NCD) and reported high accuracy of data entry. Remote audits were introduced for data validation on a trial basis in 2016 and formally accepted into use in 2017‐2018. The aim of this study was to audit the data quality of the NCD gastroenterological section for 2016‐2018 and to confirm the high accuracy of data in remote audits. Methods Each year, 45‐46 hospitals were selected for audit. Twenty cases were randomly selected in each hospital, and the accuracy of patient demographic and surgical outcome data (46 items) was compared with the corresponding medical records obtained by visiting the hospital (site‐visit audit) or by mailing data from the hospital to the JSGS office (remote audit). Results A total of 136 hospitals were included, of which 88 (64.7%) had a remote audit, and 124 936 items were evaluated with an overall data accuracy of 98.1%. There was no significant difference in terms of data accuracy between site‐visit audit and remote audit. Accuracy, sensitivity, and specificity of mortality were 99.7%, 89.7%, 100% for site‐visit audits and 99.8%, 97.3%, 100% for remote audits, respectively. Mean time spent on data verification per case of remote audits was shorter than that of site‐visit audits (10.0 minutes vs 13.7 minutes, P < 0.001). Conclusion The audits showed that NCD data are reliable and characterized by high accuracy. Remote audits may substitute site‐visit audits.


| INTRODUC TI ON
The Japanese National Clinical Database (NCD) is a large-scale, nationwide, web-based data entry system linked to the surgical board certification system and covering >95% of the surgical operations carried out in Japan. 1 The goal of the NCD is to provide a basis for the best surgical treatment for the citizens and maintain the high standard of surgical quality in Japan. This is done through systematic gathering and analysis of clinical data serving to inform advances in quality improvement.
In the gastroenterological section of the NCD, the Japanese Society of Gastroenterological Surgery (JSGS) selected 115 gastroenterological operative procedures as important for surgical training in the board certification system. From 2011 to 2018, a total of 4 420 168 gastroenterological surgical cases were recorded. [2][3][4] The JSGS also selected eight main procedures (esophagectomy, distal gastrectomy, total gastrectomy, right hemicolectomy, low anterior resection, hepatectomy, pancreaticoduodenectomy, and surgery for acute diffuse peritonitis) as particularly important as medical standards for improving surgical quality. Risk models of mortality and morbidity for these eight main procedures have been developed, and a risk calculator has been implemented using these risk models, which enables predictions of patient morbidity and mortality upon entering of the preoperative data; this calculator is available on the NCD website for physicians in clinical practice.
The JSGS began its data verification activity in 2015, focusing on NCD data representing these eight main procedures and found a high accuracy of data entry. 5 In this initial audit, 17 hospitals (2% of the JSGS-certified hospitals) were selected, and data from all relevant medical records were obtained by visiting the hospitals (site-visit audit). Later, in 2017, the JSGS initiated a full-scale audit: Each year, 45-46 hospitals (5% of the JSGS-certified hospitals) were selected for audits. In addition to site-visit audits, the JSGS introduced the remote audit as an alternative to the sitevisit audit; in the remote audit, data from relevant medical records were obtained by mailing the data from the hospital to the JSGS office. In the present study, we document the NCD data verification activity for the gastroenterological section for the period of 2016-2018.

| Data sources
The NCD was established in 2010 as a general incorporated association in collaboration with several clinical societies. In the gastroenterological section of the NCD, the JSGS selected 115 gastroenterological operative procedures for the board certification system. These 115 procedures were stipulated by the Training Curriculum for Board Certified Surgeons in Gastroenterology. The JSGS selected eight main procedures (esophagectomy, distal gastrectomy, total gastrectomy, right hemicolectomy, low anterior resection, hepatectomy, pancreaticoduodenectomy, and surgery for acute diffuse peritonitis) as particularly important in terms of medical standards for surgical quality assessment. As previously reported, 5 the subjects were patients whose surgical data were recorded in the NCD and who underwent one or more of the eight main procedures.
JSGS-certified hospitals were requested to register their gastroenterological surgery cases in the NCD. Data were collected using specialized web-based data collection forms, which contained approximately 250 variables, including data on demographics, preoperative risk, operative procedure, postoperative complications, and outcomes. Registration was closed annually on a fixed date to block further entries. Each participating hospital was requested to appoint a data manager to be accountable for data traceability. The protocol for the NCD project, which included a site-visit audit, was approved by the Japan Surgical Society and the institutional ethics board of Tokyo University (registry number: 2976). In order to be able to enter patient data into the database, each participating hospital was requested to obtain ethical approval from its local institutional review board, the facility director's permission, or ethical approval from a proxy review board. Remote audits were approved by JSGS and the institutional ethics board of Kobe University (registry number: 170169). For remote audits, each participating hospital required permission from the facility director or ethical approval from the local institutional review board. Accuracy assessment involved variables, such as patient demographics, intraoperative information, and outcomes. We established a protocol to evaluate these variables in the test and initial audits.

| Methods of data verification
We defined the priority of checking the source documents and developed a flow chart to judge each variable. For example, surgical profile (gender, hospital admission data, preoperative diagnosis, postoperative diagnosis, and hospital discharge data) was checked and judged by the discharge summary. If the data was judged to be imprecise, the next step was to check the operation record, doctor's record, and nurse's record. Operative data (operation date, operator and assistant, surgical procedure, wound class, anesthesia technique, intraoperative blood loss, and operative time) was checked and judged by operation record and anesthesia record. Identification of the variables in the existing medical records was relatively straight-forward, and the definitions of the terms were unambiguous because they were standardized. During each remote audit, we referred to the medical record mailed by each hospital and evaluated the consistency of the variable values with those in the registered data. If the data submitted to the NCD matched those in the source documents, we judged the items as "consistent." Meanwhile, in cases of inconsistent values/information, we sought additional information to identify the cause of the discrepancy and requested additional material from the hospital.
Two or three staff members who were independent of the JGSG, not involved in any clinical practice, and who had general medical knowledge (nurse or health information manager) performed the established audit according to protocol under the supervision of a medical doctor belonging to the quality management subcommittee of the JGSG. All auditors were required to sign a written contract in which they consented to adhering strictly to the confidentiality obligations with regard to the hospital information and they were allowed access to the data for the purpose of verification only.
The feedback system followed an established process. All discordant results were provided to each hospital and were corrected.
All audit results were analyzed by members of the JSGS database committee and the NCD data quality management subcommittee. A general outline of audit results and solutions to improve data quality was provided to all members of JSGS.
We measured the total audit time required for each facility. The time spent for data verification per case was calculated by dividing the total audit time by the number of audit cases.

| Statistical analyses
The accuracy of data entry was expressed as the proportion of consistent items per verified case. We also calculated an item-wise proportion of data consistency between the source data and the NCD data. Some items, for which the original source could not be identified by the unified method, were considered indeterminable and excluded from analysis. We calculated the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for complications, transfusions, and mortality measures at discharge and 30 days after surgery. The time taken to verify the data for each case was compared by type of audit. All statistical analyses were carried out using the JMP v.14.0 software (SAS Institute Inc., Cary, NC, USA).

| RE SULTS
A total of 136 hospitals were selected for data validation (Table 1)

| Accuracy of data entry
The total number of cases selected for the 2016-2018 audits was 2716, and 124 936 items were assessed. Table 2 provides an overview of the concordance rate for site-visit audits and remote audits.
The overall concordance rate of remote audits was slightly superior than that of site-visit audits, without the difference reaching statistical significance (remote audit concordance, 98.2%; site-visit audit concordance, 98.0%; P = 0.051). Most data accuracies were >95%; however, the accuracy of data on wound class, weight, anesthesia technique, and American Society of Anesthesiologists (ASA) class was <95% in both site-visit audits and remote audits. Data accuracy for all postoperative occurrences were >95%. Measures of mortality at discharge and 30 days after surgery showed an agreement of >99%. Table 3 displays the sensitivity, specificity, PPV, and NPV of postoperative complications and mortality as compared with medical record reviews. Measures of mortality at discharge showed a 99.8% agreement rate, with a sensitivity of 93.9% and specificity of 99.9%.

| Accuracy statistics for mortality and complications
Among the six complications studied, organ/space surgical site infection (SSI) was the most frequent. While specificity was constantly high throughout the whole variable (>98.9%), the sensitivity of organ/space SSI was relatively low (71.1% overall, 74.4% in site-visit audits, and 68.5% in remote audits).

| D ISCUSS I ON
Data verification provides information on completeness and accuracy of data used for investigations aiming to improve clinical practice. In this study, we provide the results of the 2016-2018 gastroenterological section of the NCD data audit. We identified high data accuracy, with an overall concordance rate of 98.1%. This concordance rate was almost equivalent to the rate of our initial audit Mortality is one of the most important surgical outcomes, and mortality risk models have been developed using the NCD data. [7][8][9][10][11][12][13][14] In the present study, measures of mortality at discharge showed a 99.8% agreement rate, with a sensitivity of 93.9% and a specificity of 99.9%. We only identified six cases with data disagreement on mortality at discharge; all six cases of disagreement appeared attributable to simple data entry errors. This result indicates that the NCD data are reliable and characterized by high quality, and that the mortality risk models developed so far are also reliable. There were several variables in which the concordance rates significantly differed (P < 0.05) between the site-visit and remote audits. All differences in the concordance rates were small (within 4%).
Remote audits had better concordance rates in height, wound class, operative time, and urinary tract infection. Data quality improvement could have contributed to these differences because most remote audits were performed at a later time. Site-visit audits had better concordant rates in history of severe COPD, operator and assistant, and surgical procedure. The cause of these differences could not be clearly explained. Hospital selection may have contributed to these differences because the concordance rates of respective hospitals were not stable.
The cost of the audits is of major concern, because the NCD covers all parts of Japan. Although site-visit audits are superior to remote audits in terms of the security of handling personal informa- Ideally, all hospitals should be evaluated, but the cost of the audits was prohibitive. The audit system of NCD was constructed according to that of NSQIP. A total of 45-46 (or 5%) of JSGS-certified hospitals were randomly evaluated each year. The difference in the average concordance rate between selected hospitals was small (not higher than 6.5%). There were only 6 hospitals (4.4% of the selected hospitals) in which the concordance rate was 95% or less. The results of audits in these years should be reflected in future audits to improve data quality.
We did not have a systematic reviewer training system, whereas the NSQIP had an online reviewer training system and examination. 6 All audits were performed under the supervision of a medical doctor who was a member of the quality management subcommittee of the JGSG. The members of the NCD data quality management subcommittee were familiar with the NCD database and the protocol of site-visit and remote audits. The results of audits were discussed and confirmed at the NCD periodical database meeting, and the inter-rater reliability was guaranteed.
In conclusion, this study showed that NCD data are reliable and characterized by high accuracy. Data verification was possible by both site-visit audit and remote audit.

ACK N OWLED G M ENTS
We thank all individuals and academics from the JSGS who extended their cooperation as well as all of the data managers and hospitals that cooperated in these audit activities. We also thank all auditors below as the members of the JSGS database (Hideki Ueno, Susumu