To describe the acute myocardial infarction (AMI) validation project, a test case for health outcome validation within the US Food and Drug Administration–funded Mini-Sentinel pilot program.
To describe the acute myocardial infarction (AMI) validation project, a test case for health outcome validation within the US Food and Drug Administration–funded Mini-Sentinel pilot program.
The project consisted of four parts: (i) case identification—developing an algorithm based on the International Classification of Diseases, Ninth Revision, to identify hospitalized AMI patients within the Mini-Sentinel Distributed Database; (ii) chart retrieval—establishing procedures that ensured patient privacy (collection and transfer of minimum necessary amount of information, and redaction of direct identifiers to validate potential cases of AMI); (iii) abstraction and adjudication—trained nurse abstractors gathered key data using a standardized form with cardiologist adjudication; and (iv) calculation of the positive predictive value of the constructed algorithm.
Key decision points included (i) breadth of the AMI algorithm, (ii) centralized versus distributed abstraction, and (iii) approaches to maintaining patient privacy and to obtaining charts for public health purposes. We used an algorithm limited to International Classification of Diseases, Ninth Revision, codes 410.x0-410.x1. Centralized data abstraction was performed because of the modest number of charts requested (<155). The project's public health status accelerated chart retrieval in most instances.
We have established a process to validate AMI within Mini-Sentinel, which may be used for other health outcomes. Challenges include the following: (i) ensuring that only minimum necessary data are transmitted by Data Partners for centralized chart review, (ii) establishing procedures to maintain data privacy while still allowing for timely access to medical charts, and (iii) securing access to charts for public health uses that do not require approval from an institutional review board while maintaining patient privacy. Copyright © 2012 John Wiley & Sons, Ltd.
In 2007, the US Congress passed the Food and Drug Administration Amendments Act mandating the FDA to establish a postmarket risk identification and analysis system to link and analyze safety data from multiple sources. In May 2008, in response to the Congressional mandate, the FDA launched the Sentinel Initiative, a long-term program designed to create a national electronic monitoring system for medical product safety (the Sentinel System). The Sentinel System is being developed and implemented in stages and, when fully functional, will complement FDA's existing postmarket safety surveillance systems.
The Mini-Sentinel pilot, a contract awarded by the FDA to the Harvard Pilgrim Health Care Institute (HPHCI) to develop the scientific operations needed for the eventual Sentinel System, is being conducted as a collaborative effort between the FDA and a consortium of institutions led by HPHCI.[2, 3] Because accurate and timely identification of health outcomes is an essential component of active safety surveillance, Mini-Sentinel convened a workgroup to establish a process for identification and validation of a selected health outcome: acute myocardial infarction (AMI). This is the first health outcome to be validated under Mini-Sentinel. In addition to developing and validating an algorithm to identify hospitalized AMI cases within the Mini-Sentinel Distributed Database, another goal of the workgroup was to design an efficient validation process that could be used as a model for future validation efforts of other health outcomes of interest.
This article describes how the AMI Validation Workgroup developed the Mini-Sentinel validation process for AMI and discusses the barriers encountered. Additional information can be found in our final report.
The Mini-Sentinel AMI Validation project was a collaboration among multiple entities. The FDA's Sentinel Initiative team contributed input and oversight throughout the validation effort. The Mini-Sentinel Operations Center, composed of researchers and staff from HPHCI, provided a scientific, analytic, and administrative infrastructure. The Operations Center facilitated communication between collaborators, designed programs for chart retrieval, and coordinated the retrieval effort. Academic researchers from Meyers Primary Care Institute and University of Massachusetts Medical School were charged with designing the approach to chart identification, identifying necessary chart components, and performing abstraction and adjudication. Four Data Partners participated in this project, including HealthCore, Inc., Humana, three member health plans within the Kaiser Permanente Center for Effectiveness and Safety Research, and two health plans in the HMO Research Network. Data Partners implemented computer programs written in SAS by the Operations Center to identify likely AMI cases, retrieved, copied, de-identified, and transmitted selected healthcare data to the lead team through the Operations Center via a secure Web portal.
The AMI validation process consisted of four parts: (i) an approach to case identification with the goal of producing an algorithm that would reliably identify AMI cases; (ii) a protocol for case retrieval from the Data Partners, which outlined necessary chart components to confirm the AMI diagnosis and established effective approaches in obtaining and de-identifying chart information; (iii) a parsimonious data abstraction form including relevant elements derived from the medical chart components and completed by trained nurse abstractors; and (iv) an adjudication protocol for confirmation of the AMI diagnosis by cardiologist adjudicators. The culmination of this effort will be a determination of the positive predictive value (PPV) of the algorithm. The following sections focus on the first three parts of this process.
The overarching goals of this project were to validate the diagnostic codes used to identify likely AMI cases and to design an efficient validation process that could be used for future validation of other health outcomes in Mini-Sentinel. It was determined that 100 charts would be sufficient to obtain a reasonable PPV and establish the validation process. To obtain more contemporary findings, we decided to include only patients who were hospitalized for AMI between 1 January 2009 and 31 December 2009 for whom there were records in the Mini-Sentinel Distributed Database, which currently comprises administrative and claims data formatted locally into a common data model. Hospital stays of less than 24 h and observation stays were included. There were no restrictions on age, sex, other diagnoses, or other patient characteristics, but patients were required to be enrollees of the respective health plan for the entire duration of hospitalization.
We had the opportunity to consult with a concurrent Mini-Sentinel workgroup that was charged with developing an active surveillance protocol for AMI, which informed decision making in the early stages. The two workgroups began by reviewing the literature and examining prior completed reviews[7, 8] to identify previously used algorithm components, with a focus on those yielding the highest PPVs (Table 1). We also consulted with cardiologists, cardiovascular researchers, and FDA review staff with expertise in cardiovascular disease. We considered the types of data that would likely be available from the medical records relating to the patient's index hospitalization, as well as the likelihood of access to information both before hospitalization and after hospital discharge for AMI survivors. We reviewed the pathophysiology of AMI and acute coronary syndrome and discussed whether to create a strict definition of AMI or a definition that more broadly captured cardiovascular disease states as part of a continuum.
|Data types||Algorithm components||Algorithm structure||Algorithm performance metrics|
|• Registry||• ICD-8, ICD-9, ICD-10 discharge codes||• Combination of codes by type (ICD-8, ICD-9, ICD-10, and diagnosis related group using OR versus AND||• Sensitivity|
|• Hospital administrative data||• Position of this code (primary position; first position; second position; ‘most responsible diagnosis’)||• Combination by position (primary only versus primary OR first versus primary OR first OR second, etc.)||• Specificity|
|• Claims data||• Diagnosis related group codes||• Combination of code with other criteria (e.g. ICD code AND length of stay >3 days)||• PPV|
|• Other (electronic health record, survey data)||• Other criteria: length of hospital stay (3–180 days); transfer to or from outside hospital; death during hospital stay; previous AMI in past 8 weeks|
In reviewing the literature, we found a wide range of International Classification of Diseases, Ninth Revision (ICD-9), codes in use, with a few studies assessing ICD-8 or ICD-10 codes and several studies combining ICD codes with other criteria.[7-20] We identified ICD-9 code 410 as the code most frequently yielding PPVs in the mid to high 90% range, and we also identified the need to specify the ICD-9 code using two decimal places. Because the number 2 in the second place after the decimal (i.e. 410.x2) indicates a past myocardial infarction, we limited our sample to 410.x0 or 410.x1. Our algorithm therefore identified patients with ICD-9 hospital discharge codes (a principal or primary discharge code only[9, 15, 17, 19]) of 410.x0 and 410.x1. If a Data Partner does not have a diagnosis designated as principal or primary, we used the first-listed discharge diagnosis. Although we reviewed previously studied algorithms that incorporated hospital length of stay, we did not find that this reliably increased the PPV and therefore did not include this requirement in the final algorithm.[10, 12, 16, 17]
The two workgroups held additional meetings with members of both teams to reach consensus on a common AMI definition for both studies. We discussed including in our definition deaths occurring within one day of an emergency department visit for acute ischemic heart disease (ICD-9 code: 411.1, 411.8, 413.x) but decided against including these additional ICD rubrics because of concerns regarding the adequacy of information that would be available to adjudicate these cases. Codes including 412 (old myocardial infarct) and 414 (chronic ischemic heart disease) were excluded in an effort to focus on acute events. Our final algorithm to identify AMI patients identified patients with ICD-9 principal (or first-listed) discharge codes 410.x0 and 410.x1.
On the basis of this algorithm, the Operations Center developed a SAS program, tested it with two Data Partners for accuracy, and then distributed it to all Data Partners participating in the project. To obtain 100 cases for adjudication, efforts were made to identify approximately 150 cases (assuming 67% retrieval) across all participating Data Partners, with each Data Partner pursuing an equal number of cases. To identify a random sample of likely AMI cases and the hospitals in which they received care, participating Data Partners executed the SAS program to query their own locally maintained administrative and claims data (see next section).
The workgroup developed a protocol for retrieving medical chart information. To proceed with chart retrieval, we needed (i) to decide whether chart abstraction would take place centrally or in a distributed manner (i.e. each Data Partner abstracts its own charts) and (ii) to establish protocols for ensuring patient privacy and data security and for helping Data Partners comply with their regulatory responsibilities when supplying data for public health surveillance, which does not require individual consent or privacy authorization.
Because the abstraction process would have major implications in terms of the amount of information transferred, the workgroup held multiple meetings to address the question of centralized versus distributed data abstraction. Before selecting an approach to pursue, the workgroup discussed why a centralized versus distributed approach might be preferred for the purpose of this validation activity and as a model for future Mini-Sentinel validations. Specifically, the workgroup discussed the following issues:
After careful consideration, a centralized approach was ultimately pursued and selected components of medical records were extracted. All charts were redacted of direct identifiers but retained dates of service to reduce the amount of information transmitted to the minimum amount necessary to accomplish the public health purpose of the project. The redacted records were then securely transferred via the Mini-Sentinel Secure Portal to the lead team at Meyers Primary Care Institute for centralized abstraction.
Once a centralized abstraction approach was chosen, the lead team proposed a list of critical chart components and other information they considered important for the validation. This initial list was developed broadly and then narrowed down based on input from Data Partners, the Operations Center, the FDA, and individuals with clinical and epidemiologic expertise relevant to cardiovascular disease.
Abstraction tools from various validation studies were reviewed to inform decisions on the list of critical chart components and other information to be extracted.[10, 20, 22] In response to Data Partners' concerns over the amount of information to be extracted and transferred, the lead team further excluded several chart components (e.g. medications and patient vital signs) to lessen the amount of information to be transferred.
The Operations Center reviewed the revised list in relation to the Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule's minimum necessary standard and confirmed that the critical chart components and other information requested constituted the minimum amount of information for the activity. The chart components that were requested are summarized in Table 2.
|Admission history and physical|
|Cardiology consult notes|
|Autopsy reports/death notes|
|Emergency department notes|
|Copies of all 12 lead EKGs|
|Cardiac catheterization report|
|Percutaneous coronary intervention reports|
|Cardiac bypass surgery report|
|Cardiac stress tests/nuclear stress tests/reports|
Certain items requested remained broad in scope. For example, copies of all laboratory results were requested. This was performed to obtain cardiac biomarker information. Cardiac biomarkers are one of the critical items of interest for AMI validation but only represent a subset of all laboratory results. This decision was made to avoid the need for a highly trained individual at each site capable of determining which specific pages of the laboratory report section of the medical record were required.
After the list of chart components was finalized, Data Partners proceeded to execute the SAS program locally, identifying a random sample of likely AMI cases whose medical records were to be located. Data Partners then asked source data holders (e.g. individual hospital medical records departments) for access to these patients' records. Source data holders either sent the medical records to vendors commissioned to extract, copy, and redact the requested information or allowed Data Partners direct access to records for extraction, copying, and redaction. Redacted chart data were sent to the Operations Center via the Mini-Sentinel Secure Portal.
The Operations Center provided each Data Partner with a privacy packet prepared by the Mini-Sentinel Privacy Panel. This packet included (i) the Mini-Sentinel Privacy Panel White Paper describing data privacy issues in Mini-Sentinel, (ii) a letter from the Department of Health and Human Services Office for Human Research Protections (OHRP) to the FDA stating that the regulations OHRP administers do not apply to the Sentinel Initiative (OHRP oversees all institutional review boards [IRBs]), and (iii) a letter from the FDA to the Mini-Sentinel Principal Investigator explaining FDA's legal authority to obtain data for use in its Sentinel and Mini-Sentinel activities. The privacy packet described the legal basis for determining that the work of the Mini-Sentinel pilot constitutes a public health activity not under the purview of IRBs. Data Partners were strongly encouraged to disseminate the privacy packet to their IRBs and Privacy Boards as well as any other relevant entities.
The Operations Center also supplied a customizable letter template to provider sites, an instructional flowchart, and a list of frequently asked questions. The letter (addressed to each provider site from specific Data Partners) explained the purpose of the project and what was being requested. Letters also explained that the request was being carried out on behalf of Mini-Sentinel and the FDA. The flowchart outlined the array of possible scenarios for chart retrieval and detailed the steps for chart redaction and data transmission. The frequently asked questions included a list of anticipated questions, such as ‘How were provider sites selected to participate in this activity?’, ‘Where will the abstracted data go’, and ‘How can I contact my Data Partner directly?’
The redaction of individually identifiable information was performed in accordance with HIPAA's provisions for a ‘limited data set,’ which is an alternative to using fully de-identified information. Under HIPAA, the creation of a limited data set requires removal of 16 direct identifiers but allows for the inclusion of dates, geographic location (not as specific as street address), and any other code or characteristic not explicitly excluded. Redaction was completed before chart components were transferred to the Operations Center. Each Data Partner assigned a new, de-identified ID unique to each redacted chart and maintained a crosswalk between the newly assigned IDs and the original IDs. The Operations Center did not receive this crosswalk for IDs; it was maintained locally by each Data Partner.
Admission and discharge dates as well as dates corresponding to electrocardiograms (EKGs), laboratory results, procedures, and tests were not redacted. This information was considered crucial for determining whether available EKGs and test results corresponded to the hospital stay of interest and therefore whether an AMI occurred during the identified hospital stay. In addition, for certain tests (EKGs and cardiac biomarkers), the results needed to be assessed by cardiologist adjudicators in chronological order. We considered assigning reference values for every date. Ultimately we opted not to pursue this approach because we felt that this would substantially increase workload and introduce multiple opportunities for error.
Data Partners were provided with credentials to login to the Mini-Sentinel Secure Portal for transferring, managing, or retrieving chart data. Security was managed within the folder structure of the site; the Secure Portal contains private folders accessible only to specified members within each Data Partner site and authorized Operations Center staff as well as common folders defined for all users. Data Partners electronically uploaded redacted charts to their site-specific private folders. The Operations Center verified that all charts were redacted thoroughly and then moved all files to a separate private folder, allowing the lead team access to the data.
A total of 153 charts were requested, of which 143 were successfully retrieved. Approximately 80% of the 143 charts were obtained within 3 months after the request. The total time it took to received all 143 charts was approximately 5 months.
Regarding level of burden, Data Partners were initially concerned that they would be required to obtain information from multiple sources, including outpatient medical records. However, clinical information relevant to the present validation study was to be extracted from medical records relating to only a single hospitalization.
Regarding privacy issues and IRB concerns, Data Partners described several challenges they encountered during the chart retrieval process. Although sometimes causing delays, most source data holder IRBs allowed charts to be located and retrieved after being provided with the privacy packet containing letters and documents that clarified the status of this validation project as a public health surveillance activity undertaken under the auspices of the FDA. However, despite the privacy packet, seven charts requested were not obtained because of IRB concerns and insistence on patient consent before releasing medical records.
Several other issues were brought to the workgroup's attention by the Data Partners. Some redacted charts were sent by mail as opposed to electronic transmission, which led to delays in transferring data. One Data Partner found that including a list of frequently asked questions and answers along with each chart request led to improved turnaround times. Frequent inquiries concerning the disposition of charts and relationship building with hospital staff processing the request were also helpful in obtaining charts more quickly.
The lead team identified and reviewed several AMI abstraction forms and manuals used in past AMI validation activities. The team also consulted individuals with clinical and epidemiological expertise relevant to cardiovascular disease, reviewed the Universal Definition of Myocardial Infarction by the American Heart Association (AHA) [24, 25] and the literature on troponin standardization, and communicated with directors of laboratories on percentile cutoffs for what were considered to be ‘positive’ troponin values. On the basis of clinical consultation and literature review, the lead team created a 36-item abstraction form that included (i) general demographic information; (ii) brief medical history; (iii) cardiac biomarker information; (iv) copies of EKGs; (v) cardiac testing, procedure, and intervention information; and (vi) information on disposition at the time of hospital discharge.
The lead team trained two nurse abstractors to enter abstracted information into a Microsoft Access database and provided an accompanying instruction manual. Both abstractors gathered data from the first 10 cases. These abstractions were reviewed together with both nurse abstractors to ensure high interrater reliability on items critical for the adjudications.
In consultation with the FDA and individuals with clinical and epidemiologic expertise relevant to cardiovascular disease, the lead team created an adjudication protocol on the basis of the AHA Universal Definition of Myocardial Infarction. In addition to abstracted data from the form described earlier, adjudicating cardiologists were provided with copies of EKGs and copies of all cardiac test and procedure reports.
Redacted components of the medical record were sent to the Operations Center via the Mini-Sentinel Secure Portal and then made available through this secure site to the lead team for data abstraction and case adjudication.
One of the more challenging issues that emerged in the design of the abstraction and adjudication forms related to differences in cardiac biomarker reference standards among different hospitals. It was essential to design abstraction materials that could adequately capture both biomarker results and reference standards, even when presented in a variety of ways from different hospital sources.
The workgroup was also challenged with reconciling the biomarker standards described in the published AHA definition of AMI with laboratory values likely to be available in hospital records. Although the published definition we were using defined abnormal biomarker values as falling ‘above the 99th percentile of the upper reference limit’, preliminary reviews of several charts showed that hospital laboratories did not routinely report percentile cutoffs. Through communication with the director of one hospital laboratory, we also found that the reported reference values did not always correspond to this 99th percentile cutoff. We opted to capture any and all available information on reference standards from charts (i.e. from printed laboratory reports) but did not contact laboratory directors at each individual site for any further information.
We believe the following lessons learned will assist us in developing best practices when conducting similar Mini-Sentinel validation activities in the future:
The AMI validation project has established a process for validating medical outcomes within Mini-Sentinel that can serve as a model for future surveillance validation activities. The project has provided important insights into the challenges inherent in conducting health outcome validation across public, academic, and private entities. Key issues identified include (i) the need to determine the scope of health outcome definitions (broad versus more focused) (ii) the need for early assessment regarding centralized versus distributed approaches to chart abstraction, and (iii) the need to have a policy and systematic approach for maintaining patient privacy, data security, and addressing regulatory compliance issues under HIPAA and the Common Rule. In addition, it will be important for future validation projects to anticipate between-hospital differences in laboratory reference standards and between-hospital variations in how these data will be presented to the adjudicators.
The authors declare no conflict of interest.
Mini-Sentinel is funded by the FDA through the Department of Health and Human Services contract number HHSF223200910006I.
Dr Cutrona was supported in part by Award Number KL2RR031981 from the National Center for Research Resources (NCRR). [Correction added on 15 June 2012, after online publication: Information about NCRR support awarded to Dr Cutrona was added in the Acknowledgements section.]
The authors thank the following Data Partners for their input into the validation process and their tireless efforts to obtain and prepare charts. HealthCore, Inc.: Gregory Daniel, Jenni (Jie) Li, and Amanda Rodrigez; HMO Research Network—Group Health Research Institute: Denise Boudreau and Danelle Wallace; Fallon Community Health Plan: Susan Andrade; Humana: Vinit Nair and Mary Costantino; Kaiser Permanente Center for Effectiveness and Safety Research: Daniel Jaynes; Kaiser Permanente Northern California: Daniel Ng; Kaiser Permanente Georgia: Melissa Butler; Kaiser Permanente Hawaii: Cynthia Nakasato and Yee Hwa Daida.
In addition, they thank Dr Karen Hicks for her assistance in developing the abstraction and adjudication tools; Dr Jorge Yarzebski for assistance in obtaining and abstracting charts; Dr Guillermo Talero, Michaela Richardson, and Catherine Emery for their careful chart abstraction work; and Dr David McManus and Dr Joel Gore for their work as adjudicators.