To systematically review algorithms to identify seizure, convulsion, or epilepsy cases in administrative and claims data, with a focus on studies that have examined the validity of the algorithms.
To systematically review algorithms to identify seizure, convulsion, or epilepsy cases in administrative and claims data, with a focus on studies that have examined the validity of the algorithms.
A literature search was conducted using PubMed and the Iowa Drug Information Service database. Reviews were conducted by two investigators to identify studies using data sources from the USA or Canada because these data sources were most likely to reflect the coding practices of Mini-Sentinel data partners.
Eleven studies that validated seizure, convulsion, or epilepsy cases were identified. All algorithms included International Classification of Diseases, Ninth Revision, Clinical Modification code 345.X (epilepsy) and either code 780.3 (convulsions) or code 780.39 (other convulsions). Six studies included 333.2 (myoclonus). In populations that included children, 779.0 (convulsions in newborn) was also fairly common. Positive predictive values (PPVs) ranged from 21% to 98%. Studies that used nonspecific indicators such as presence of an electroencephalogram or anti-epileptic drug (AED) level monitoring had lower PPVs. In studies focusing exclusively on epilepsy as opposed to isolated seizure events, sensitivity ranged from 70% to 99%.
Algorithm performance was highly variable, so it is difficult to draw any strong conclusions. However, the PPVs were generally best in studies where epilepsy diagnoses were required. Using procedure codes for electroencephalograms or prescription claims for drugs possibly used for epilepsy or convulsions in the absence of a diagnostic code is not recommended. Many newer AEDs require no drug level monitoring, so requiring an AED level monitoring procedure in algorithms to identify epilepsy is not recommended. Copyright © 2012 John Wiley & Sons, Ltd.
The US Food and Drug Administration commissioned systematic reviews to identify validation studies of algorithms to identify 20 health outcomes of interest (HOIs) in administrative and claims data as part of its Mini-Sentinel pilot. These reviews provide the foundation for future studies of HOIs in Mini-Sentinel and other administrative and claims data sources. In such studies, it is extremely important to understand the performance characteristics of the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM), International Classification of Diseases, Tenth Revision (ICD-10), and Current Procedural Terminology, Fourth Edition (CPT-4) codes that might be used to identify an HOI, as the presence of a code is not always sufficient to determine that an HOI actually occurred. One HOI selected for a systematic review was seizure, convulsion, and epilepsy.
The outcomes of interest in this review are related but distinct. Acute seizures and convulsions often occur outside the context of epilepsy. In the USA, approximately 10% of the population will experience at least one seizure throughout their lifetime. New onset seizure is the reason approximately 120 of every 100 000 people in the USA seeks medical attention each year. Epilepsy is recurrent unprovoked seizures. There are many causes of and factors that contribute to seizures, including medication use. Most patients with drug-induced seizures present with generalized tonic–clonic seizures. The exact incidence of drug-induced seizures is unknown and is likely to vary according to the classification of the causative drug. The assessment of medication-induced seizures is difficult because the seizures may be due to drug treatment or the underlying medical condition for which the drug was prescribed. Regardless, it is important to distinguish between algorithms to identify seizure or convulsion events and algorithms that identify epilepsy.
This manuscript provides an overview of the seizures, convulsions, or epilepsy algorithm review. Algorithms vary from single diagnosis codes to multiple diagnosis codes or combinations of diagnosis codes, procedure codes, and pharmacy claims. The full report can be found at http://minisentinel.org/foundational_activities/related_projects/default.aspx.
Details of the methods for these systematic reviews can be found in the accompanying manuscript by Carnahan and Moores. In brief, the base PubMed search was combined with the following terms to represent the HOI: ‘epilepsy’ [Mesh], ‘seizures’ [Mesh], ‘spasms, infantile’ [Mesh], ‘epilepsy,’ ‘seizure’, and ‘convulsion’. A search of the Iowa Drug Information Service, a bibliographic database of citations and abstracts from English-language medical and pharmaceutical journals, was also conducted. The details of these searches can be found in the full report on the Mini-Sentinel Web site. Searches of both databases were conducted on 24 June 2010. All searches were restricted to articles published in 1990 or after. Mini-Sentinel collaborators were also asked to help identify any relevant validation studies.
The abstract of each citation identified was reviewed by two investigators. When either investigator selected an article for full-text review, the full text was reviewed by both investigators. Agreement on whether to review the full text or to include the article in the evidence table was calculated using a Cohen's kappa statistic. The data in the evidence table were extracted by one investigator and confirmed by a second.
The PubMed search identified 764 citations, and the Iowa Drug Information Service search identified 39 citations. After excluding duplicates, the total number of unique citations from the combined searches was 774. Mini-Sentinel collaborators provided no reports of validation studies that had been completed by their teams.
Of the 774 abstracts reviewed systematically, 342 were selected for full-text review, 180 were excluded because they did not study the HOI, 131 were excluded because they were not administrative or claims database studies, and 121 were excluded because the data source was not from the USA or Canada. Cohen's kappa for agreement between reviewers on inclusion versus exclusion of abstracts was 0.49. The primary reason for the low agreement was because of uncertainty about whether or not a study used an administrative or claims database. If reviewers disagreed on whether an article should undergo full-text review, the article was included in the full-text review.
After full-text review, 9 studies met the criteria for inclusion in the final evidence tables, 21 studies were excluded because they did not study the HOI, 146 studies were excluded because they were not administrative or claims database studies, 24 studies were excluded because the data source was not from the USA or Canada, 70 studies were excluded because they included no validation of the outcome definition or reporting of validity statistics, and 72 studies were excluded because the HOI identification algorithm was poorly defined. Cohen's kappa for agreement between reviewers on inclusion versus exclusion of full-text articles reviewed was 0.36. The primary reason for the low agreement was that one investigator was more liberal in selecting articles that had poorly defined algorithms or used previously validated methods. If reviewers disagreed on whether a full-text article should be included, agreement was reached by discussion and consensus, with a third investigator consulted if necessary.
In addition to the nine studies from the full-text review, two other studies were identified after abstract and full-text reviews were completed. One of these studies was identified through references of articles that underwent full-text review, and one study that was published after the literature search was conducted was identified by a reviewer of the report. A total of 11 studies[5-15] were included in the evidence table (Table 1). All of the studies were published between 2000 and 2010, and the report on study populations was identified between 1991 and 2008.
|Citation||Study population and time period||Description of outcome studied||Algorithm||Validation/adjudication procedure, operational definition, and validation statistics|
|Vaccine studies in children|
|Barlow et al.||Children (up to age 7 years) enrolled in four large HMOs who received the DTP or MMR vaccine, or no vaccine from 1991 to 1993||Relative risk of febrile and nonfebrile seizures after receipt of the DTP or MMR vaccine||ICD-9-CM codes: 333.2 (myoclonus), 345.X (epilepsy), 779.0 (convulsions in newborn), and 780.3 (convulsions)||Charts were reviewed by medical-record abstractors, and physician investigators determined the final disposition of each case. The analysis of the risk of vaccination included only the first episode of seizure in each child, as confirmed by review of the medical records|
|1094 children were selected for chart review; of those, 716 were confirmed to have had a first seizure during the study period; PPV = 716/1094 = 65.4%|
|In one HMO, potential seizure cases were identified using computerized AED data, referrals to neurology clinics, and EEG records, which makes the algorithm and PPV difficult to interpret|
|Overall, the results of this study are difficult to apply because the authors were not focusing on the algorithm or its performance characteristics in the presentation of data|
|Shui et al.||Children aged 6 weeks to 23 months enrolled in seven MCOs who received at least one dose of pneumococcal vaccine from 2000 to 2005.||PPV of ICD-9-CM codes used to identify seizure visits in the 0- to 30-day period after receipt of a pneumococcal vaccine||ICD-9-CM codes: 333.2 (myoclonus), 345 (epilepsy), 779.0 (convulsions in newborn), 780.3 (convulsions)||Medical records were reviewed by trained abstractors at each MCO; 3223 visits for seizure were identified; a random sample of 1024 cases (257 ED, 236 IP, 355 OP days 1–30, 176 OP day 0) were selected for medical record review and 859 (84%) had records available|
|Visits were stratified by the setting of diagnosis: ED, OP clinic, IP admission. OP visits were stratified by whether they occurred on the same day as the vaccination (day 0) or on days 1–30||PPVs for true seizure events from ED, IP, OP days 1–30, and OP day 0, respectively: all visits—96.6%, 64.0%, 16.4%, and 1.8%; no history of seizure—96.9%, 73.5%, 24.1%, and 6.8%; ICD-9 code 780.3 only—97.0%, 65.8%, 21.5%, and 2.4%; age <1 year—91.6%, 59.7%, 12.7%, and 2.7%; age ≥1 year—99.3%, 79.1%, 25.3%, and 0.0%|
|It is important to note that codes 333.2 and 779.0 accounted for very few of the codes. Thus, it is difficult to generalize the PPVs to these specific codes|
|Zangwill et al.||Infants enrolled in the Southern California Kaiser Permanente Health Care Plan from 2002 to 2005||Incidence of seizures during the 8-day period after the primary doses of DTaP-HepB-IPV compared with the 8-day period after the primary doses of DTaP vaccine in the control cohort.||ICD-9-CM codes: 333.2 (myoclonus), 345.X (epilepsy) , 779.0 (convulsions in newborn), and 780.3 (convulsions)||Charts were reviewed by trained personnel.|
|PPV: 16/41 = 39.0% for the DTaP-HepB-IPV cohort; PPV: 15/51 = 29.4% for the DTaP control cohort|
|DTaP-HepB-IPV cohort: 61 004 infants|
|Historical DTaP control cohort: 58 251 infants|
|Klein et al.||Children aged 12–23 months covered by seven plans participating in the VSD program. During the study period (2000–2008), children must have received their first dose of at least one of the following: MMRV, MMR, or varicella vaccine; 451 charts were reviewed||The first seizure or epilepsy diagnosis within 42 days after vaccination||ICD-9-CM codes: 345.X (epilepsy) and 780.3X (convulsions)||Seizures and febrile seizures were confirmed by chart review, with no specific criteria other than mention in the chart; 87% were febrile seizures.|
|All seizure codes: PPV = 94% (424/451)|
|The proportion of events that were febrile seizures was also examined||Febrile seizures only: days 7–10 after separately administered same-day MMR + varicella vaccine, PPV = 90% (208/230)|
|Days 0–6 and 11–42 after MMR + varicella vaccine: PPV = 83% (184/221)|
|Frost et al.||Children and adults of all ages enrolled in the Lovelace Health Plan in New Mexico from 1995 to 1996 with an epilepsy diagnosis||Comparison of two methods (the epilepsy-attributable cost method and the case-control cost method) for estimating the marginal cost of medical care for epilepsy||The algorithm considered anyone with any of the codes as a potential case||Primary care physicians were sent letters to verified epilepsy status. Patients with confirmed epilepsy status were included in the case cohort. If there was no response or the response needed further investigation, medical records were reviewed. |
2474 persons met one or more criteria and were considered possible cases, and 808 patients were verified as having epilepsy; PPV = 808/2474 = 32.7%. One reason for the low PPV may have been that patients with just a code for EEGs were identified as potential cases, and that procedure may have been used to diagnose conditions other than seizures, convulsions, or epilepsy or might have been negative for seizures. It seems that this algorithm was meant to provide a highly sensitive method of case finding. Thus, the low PPV is not surprising.
|ICD-9-CM codes: 345.X (epilepsy) and 780.3 (convulsions)|
|CPT-4 codes: 18 EEG procedure codes, 9 therapeutic drug assay codes|
|Holden et al.||All members (any age) enrolled in the Lovelace Health Plan in New Mexico from 1996 to 1998 (the exploratory phase) and from 1998 to 2000 (the confirmatory phase)||Prevalence and incidence of epilepsy||A variety of algorithms were explored using combinations of the codes and pharmacy claims for AEDs||Clinicians conducted a review of the medical records to validate a positive case.|
|Exploratory sample results: diagnoses only (PPVs reported for the number of times [1, 2–3, or 4 or more, respectively], a diagnosis was present):|
|Because of the complexity of these models, only the key points are summarized here. For complete models, please see the reference|
|Group A diagnoses: PPV = 38.5%, 69.2%, 100% Group B diagnoses: PPV = 68.8%, 78.0% Groups A and B diagnoses, combined: PPV, 31.9%, 60.0%, 79.0% Group C diagnoses: PPV = 1.9%, 20.4%, 0.0%|
|ICD-9-CM codes: Group A codes: 345.00–345.91 (epilepsy)|
|The exploratory (phase 1) sample included 617 patients.|
|Group B codes: 780.3, 780.31, and 780.39 (convulsions) Group C codes: 333.2 (myoclonic disorders), 779.0 (convulsions in newborns), 779.1 (cerebral irritability in newborns), 780.02 (transient alteration of awareness), and 780.2 (syncope and collapse)||Groups A and B diagnoses (345 or 780.3) plus AED fills or blood level monitoring (PPVs reported for the number of times [0–1, 2–3, or 4 or more, respectively] a diagnosis was present):|
|CPT-4 codes: EEG procedures, four vagus nerve stimulation procedures, eight AED blood level determinations|
|No AED fill and no monitoring: PPV = 0.8%, 10.9%, 20.7%, 40.0% No AED fill and but AED monitoring present: PPV = 10.7%, 83.3%, 100%, 100% AED fill present but no AED monitoring: PPV = 0%, 70.0%, 81.5%, 88.0% AED fill and AED monitoring both present: PPV = 15.0%, 69.2%, 87.5%, 87.5%|
|All groups of diagnoses combined with the presence of at least one AED fills or blood level monitoring code (PPVs reported for the number of times [0–1, 2–3, or 4 or more, respectively] a diagnosis was present):|
|No drug fill or monitoring code: PPV = 0.8%, 10.9%, 20.0%, 40.0% At least one drug fill or monitoring code: PPV = 6.3%, 71.4%, 85.4%, 88.5%|
|Logistic regression models to predict epilepsy: data from the exploratory phase (phase 1) and the confirmatory phase (phase 2) were combined to permit|
|refinement of logistic regression models (phase 3 of the study) and to provide more stable estimates of the parameters. |
According to the authors, the best model was Model 2, which used diagnoses and the presence of specific AEDs (determined by either prescription or blood level monitoring) entered separately in the model: PPV, 83.9%; sensitivity, 81.8%; specificity: 93.8%
|The confirmatory (phase 2) sample included 644 patients.|
|Jette et al.||Patients (mean age, 42 years) from all adult and pediatric acute care sites in the Calgary Health Region in Alberta, Canada||(1) Validity of ICD-9-CM and ICD-10 coding for epilepsy from an emergency room ACCS database and an IP DAD and validity of ICD-10 coding for epilepsy from an SMU database||ICD-9-CM codes: 345.X (epilepsy), 346.X (migraine) , 435.X (transient ischemic attack), 780.2 (syncope), 780.3X (convulsions)||The following performance characteristics were determined for the ICD-10 epilepsy coding from the SMU chart review (127 charts were reviewed): Sensitivity = 99%, specificity = 70%, PPV = 85%, NPV = 97%|
|From the chart review, the PPV and NPV for ICD-9-CM epilepsy codes from the ACCS database were, respectively, 99% and 97% and from the DAD were 98% and 99%. When the convulsion code (780.3) was included, the PPV dropped to 84.0%. The PPV and NPV for ICD-10 epilepsy codes from the ACCS database were, respectively, 100% and 90% and from the DAD were 98% and 99%. When the convulsion code (R56) was included, the PPV dropped to 75.5%.|
|ICD-10 codes: G40.X (epilepsy), G41.X (status epilepticus), G43.1 (classical migraine), G45.x (transient ischemic attack ), R55 (syncope), R56.0 (febrile convulsion), R56.8 (other/unspecified convulsion)|
|During the ICD-9-CM time period (2000–2001) 486 charts were reviewed|
|During the ICD-10 time period (2004–2005), 454 charts were reviewed||(2) Comparison of variations in coding validity between the two ICD systems and between the various hospital settings|
|In ICD-9-CM charts, PPV = 83.9% for 345.3 (grand mal status), PPV = 89.3% for 345.4 (partial epilepsy with impairment of consciousness), and PPV = 45.2% for 780.3 (convulsions excluding epileptic convulsions and convulsions of newborn). Other codes did not perform well or were not found in the sample|
|In ICD-10 charts, PPV = 100% for G41.0 (grand mal status epilepticus), PPV = 83.3% for G41.2 (complex partial status epilepticus), and PPV = 77.6% for G40.2 (localization-related symptomatic epilepsy and epileptic syndromes with complex partial seizures). All other epilepsy codes had PPVs of <40% or were not found in the sample. PPV = 90.9% for R56.0 (febrile convulsions) and PPV = 32.9% for R56.8 (other and unspecified convulsions)|
|Although the PPVs for epilepsy were generally 80% or higher, it was noted that epilepsy was frequently miscoded as convulsions|
|Parko and Thurman||All Navajo tribe members of any age (median age, 23 years) residing in the Navajo Reservation who had at least one medical/dental encounter between 1998 and 2002||Prevalence of epilepsy and seizures||ICD-9-CM codes: 345.0-345.9 (epilepsy), 333.2 (myoclonus), 779.0 (convulsions in newborn), 779.1 (cerebral irritability in newborns), 780.31 (febrile convulsions), 780.39 (other convulsions)||Chart reviews were performed by neurologists, clinical pharmacists, and medical students|
|Across all ages, coding with either 345.X or 780.3X yielded PPV = 90% for clinical diagnosis of epilepsy or seizures, PPV = 62% for a clinical diagnosis of epilepsy (recurrent unprovoked seizures)|
|Pugh et al.||The overall study included national VA patients ≥66 years. National VA patient and pharmacy data (1998–2004) and Medicare data (1999–2004) were used.||New-onset epilepsy in older veterans||ICD-9-CM codes: 345.XX (epilepsy) or 780.39 (other convulsions)||Two researchers abstracted electronic medical records|
|Of the 126 patients identified using administrative and claims data, a diagnosis of epilepsy was confirmed in 119, giving a conservative PPV of 94% (119/126). Three of the remaining seven patients had diagnoses of convulsions (780.39) on multiple occasions, and one had a 345 epilepsy ICD-9-CM code in the administrative and claims data, which improved the investigators' confidence that those three patients had epilepsy. Including the patients as being as being true positive cases, the PPV was 98% (123/126)|
|At least one AED from the VA|
|For validation, a subset of patients with new-onset epilepsy and who received VA care at the South Texas Veterans Health Care System was identified.|
|If the algorithm is to be applied in VA data, it is important to note that 60% of the first seizures were documented in Medicare data, likely because of treatment in non-VA EDs|
|Hardie et al.||Elderly residents 65 years or older in eleven nursing homes managed by Beverly Enterprises (no time period mentioned)||(1) Extent to which epilepsy or seizure disorder documented on the paper MDS agreed with the neurologist's chart review||Paper MDS: ‘seizure disorder’ item was checked or ICD-9-CM codes 345.XX (epilepsy) or 780.3X (convulsions) were listed or the words ‘epilepsy’, ‘seizure’ or ‘convulsion’ were listed in the text.||Records of epilepsy/seizure disorder residents were matched one-to-one with nonepilepsy/seizure disorder residents. |
Abstractors searched medical records, and a neurologist conducted chart reviews.
Overall, agreement between the paper MDS and a neurologist's review was 92.3% (131/142 paper MDS–neurologist pairs). The PPV (likelihood of neurologist finding epilepsy /seizure, when it was documented in the paper MDS) was 87.8% (43/49). The NPV (likelihood of neurologist not finding epilepsy/seizure when it was not documented on the paper MDS) was 94.6% (88/93)
|Computerized MDS: same criteria as paper MDS; however, text documenting epilepsy or seizure disorder was not available in computer documentation|
|Agreement between paper MDS and computerized |
MDS was 97.8% (137/140 MDS paper-computerized pairs). The PPV (agreement with the paper MDS when epilepsy/seizure disorder was documented on the computerized MDS) was 97.9% (47/48). The NPV (agreement with the paper MDS when epilepsy or seizure was not documented on the computerized MDS) was 97.8% (90/92).
|(2) Agreement between the paper MDS and the computerized MDS|
|Gardner, et al.||A cohort of 9218 adult tramadol users and 37 232 concurrent nonusers (mean age, 44.5 years) from 12 UnitedHealth Group-affiliated health plans from 1995 to 1996||The rate and risk of incident tramadol-associated seizures||A case-control study was performed. Cases and controls (noncases) were selected from within the tramadol users cohort||Medical record information was obtained to confirm case status. In the case–control study, medical records abstractions were completed for 51% (38/74) of cases and 55% (101/183) of noncases.|
|8 cases were identified as having epilepsy out of the 38 records that were abstracted; PPV = 8/38 = 21%|
|The algorithm considered anyone with any of the codes as a potential case|
|Of 101 medical records abstracted for claims noncases, 99 were confirmed as not having had a seizure after taking tramadol; NPV = 99/101 = 98%|
|ICD-9-CM diagnosis codes: 780.3 (convulsions), 345.0-345.9 (epilepsy), 333.2 (progressive myoclonus)|
|ICD-9-CM procedure codes: 89.13-89.15, 89.19 (neurologic examination, EEG, other nonoperative neurologic functions tests, or video and radiotelemetered EEG monitoring)|
|CPT-4 codes: 7 codes for EEG procedures|
All 11 of the publications included in the evidence table used ICD-9-CM, ICD-10, or CPT-4 codes, or a combination of one or more of these types to identify patients with seizure, convulsion, or epilepsy. All of the studies that used ICD-9-CM codes included 345.X (epilepsy) alone or in combination with other ICD-9-CM codes. ICD-9-CM code 780.3 (convulsions) was used in combination with code 345.X and other ICD-9-CM codes in all but one study, which used code 780.39 (other convulsions) instead of code 780.3. Other common ICD-9-CM codes were 333.2 (myoclonus) and 779.0 (convulsions in newborn). One study that examined computerized Minimum Data Sets also included records that had the ‘seizure disorder’ item checked. Only one study validated ICD-10 epilepsy coding. This study included G40.X (epilepsy), G41.X (status epilepticus), R56.0 (febrile convulsion), and R56.8 (other and unspecified convulsion).
Nearly all studies included in the report validated administrative coding data through abstraction of medical charts. Documentation of seizures, convulsions, or epilepsy in the medical records was generally based on physician notes. One study first sent letters to the physician to confirm epilepsy status and only consulted the medical chart if the physician did not respond or if the response needed further investigation. A few studies specified criteria from medical records, but there did not seem to be any standardized criteria that were used.
One study classified seizures as simple febrile, complex febrile, or nonfebrile. Another study placed patients into one of five categories: epilepsy, seizure, febrile seizure, no seizure, or missing chart. One study classified seizures as neonatal seizure, febrile seizure, complex febrile seizure, afebrile seizure, symptomatic seizure, or epilepsy.
No algorithms included only a single ICD-9-CM code. All algorithms included code 345.X and either code 780.3 (10 studies[5-12, 14, 15]) or code 780.39 (1 study). Many studies did not specifically state whether subcodes under code 780.3 (i.e. 780.3X) were included, although it is suspected that they were. Six[5-7, 10, 12, 15] of the 11 studies used code 333.2 in the algorithm. In populations that included children, code 779.0 was also fairly common. Several other codes were used, but not consistently. Positive predictive values (PPVs) ranged from 21% (for predicting incident seizure events or epilepsy from tramadol) to 98% (for predicting the agreement between medical records and computerized Minimum Data Sets in nursing homes). Some algorithms included anti-epileptic drug (AED) fills or monitoring for AEDs whereas some included CPT-4 codes.
Because only one study reported validation of ICD-10 codes, it is difficult to comment on the validation statistics between ICD-9-CM and ICD-10 coding algorithms. They found the PPVs from three sources of data—seizure monitoring unit chart review, inpatient (IP) discharge abstract database, and emergency room database—to be 85%, 98%, and 100%, respectively.
In some studies, validation statistics were based on only a subsample of the overall study population. In general, this was because medical records were not available for examination or examining all records was not feasible.
The studies did not restrict the study sample to patients with specific diseases other than seizure, convulsion, or epilepsy. The studies included either the entire health plan membership or all members of a specific age group.
Four studies[5-8] examined the incidence of seizures, convulsions, or epilepsy after several childhood vaccinations, so these studies were limited to children (age range, from birth to 7 years). The tramadol-induced seizures or epilepsy study included only adults (mean age, 44.5 years, range, <15 to ≥ 65 years). One study included only patients 65 years old or older, and one study included patients 66 years or older (because they wanted at least one year of Medicare data). The remainder of studies included patients of all ages. One study found that the PPV for seizure events or epilepsy in children younger than one year of age was lower in all four settings the study evaluated than that in children one year of age or older. One study examined the effects of age and ethnicity, but the only significant demographic variable they found was that patients 0 to 19 years old were less likely than patients 20 to 64 years old to have epilepsy. One study calculated PPVs for adult hospital visits and children's hospital visits and found similar PPVs in both populations, except that the PPVs using epilepsy and convulsion codes (using both ICD-9-CM and ICD-10 codes) were higher in children's hospital visits (96.5% and 85.1%, respectively, for the two different coding systems) as compared with adult hospital visits (85.1% and 72.3%, respectively, for the two different coding systems).
The studies were published between 2000 and 2010 and reported on study populations identified between 1991 and 2008.
Four studies[5-8] identified incidence of seizures, convulsions, or epilepsy after different vaccines, one study identified incidence of drug-induced seizures or epilepsy (after tramadol), four studies[10-12, 14] identified prevalence of seizures or epilepsy, one study identified incident and prevalent epilepsy cases, and one study that examined prescribing trends in the elderly identified new-onset epilepsy cases.
In general, the studies examining incidence of seizures or epilepsy after vaccine administration[5-7] or tramadol prescription had lower PPVs (21%–65.4%), with the exception of patients seen in the emergency department (ED) after pneumococcal administration (96.6%), and another study used only ED or IP claims to identify seizures (94%). Studies that examined prevalent cases[9-12, 14] of seizures or epilepsy tended to have higher PPVs (76.9%–97.9%) with one exception (32.7%).
The studies included in this report examined IP or outpatient (OP) encounters or both to identify seizure, convulsion, or epilepsy outcomes. One study examining epilepsy or seizure after vaccine administration calculated statistics on the basis of setting and found the highest PPV for ED visits (96.6%) and the lowest PPV for OP visits on day 0 (1.8%), the day the vaccine was given. PPVs were 64.0% for IP visits and 16.4% for OP visits days 1 to 30 after vaccination. Another study examining epilepsy or seizures after vaccine administration used only codes from the ED or IP setting and found a PPV of 94%, much higher than that found in studies of epilepsy or seizures after vaccine administration that allowed OP codes. One study focusing on epilepsy calculated PPVs for all charts, IP visits, and emergency visits and found no appreciable differences.
None of the identified studies excluded patients with specific comorbid conditions. Two studies mentioned exclusion criteria.[8, 15] In one, patients were excluded if they had a seizure or convulsion code in 42 days before the vaccination under study. In the other, patients who were not continuously enrolled for at least 90 days before receiving the first prescription for tramadol and for at least 60 days after receiving tramadol (an observation period of at least 151 days) were excluded. This study also excluded patients with a seizure claim or with any prescription for an AED before the index date within the index period and pointed out that this resulted in a slightly altered ratio of nonusers to users.
The studies identified in the systematic review included a broad range of settings, age groups, and algorithms. All algorithms included diagnosis codes 345.X and code either 780.3 (10 studies) or code 780.39 (1 study). Six[5-7, 10, 12, 15] of the 11 studies used code 333.2. In populations that included children, code 779.0 was also fairly common. These latter two codes are most appropriate for identifying single seizure or seizure-like events in children, as opposed to epilepsy (two or more unprovoked seizures). Other diagnosis codes were used (779.1, 780.02, 780.2, and 780.31), in addition to procedure codes for electroencephalograms (EEGs) or AED monitoring (Table 1), but not consistently.
The performance of algorithms was highly variable, with PPVs ranging from 21% (for predicting incident seizures or epilepsy from tramadol) to 98% (for predicting the agreement between medical records and computerized Minimum Data Sets in nursing homes). For these reasons, it is difficult to draw any strong conclusions. However, the PPVs were generally best in studies in which an ICD-9-CM or ICD-10 code for epilepsy was required, whereas those algorithms that used nonspecific indicators such as the presence of an EEG or AED level monitoring without requiring an epilepsy diagnosis code had low PPVs. Including EEG or AED drug level monitoring would be more appropriate for case identification if sensitivity was the goal and all cases would be confirmed by medical record review because they are not specific to seizures or epilepsy. Studies examining the sensitivity of algorithms focused exclusively on epilepsy, as opposed to isolated seizure events. Sensitivity ranged from 70% to 99%.
The study by Holden et al. was the most comprehensive, examining several variables and concluding that the best model to identify epilepsy cases was an algorithm that required one of several ICD-9-CM codes in addition to either a pharmacy fill for an AED or a CPT-4 code for AED monitoring. This model had a PPV of 83.9% and a sensitivity of 81.8%. On the other hand, the study by Parko and Thurman used only ICD-9-CM codes and found a PPV of 90% for a clinical diagnosis of epilepsy or seizures. For epilepsy only, the PPV was 62%, leading them to suggest that a corrective factor is necessary when relying on ICD-9-CM–coded data to estimate the prevalence of epilepsy.
In the Holden et al. study, the PPV of ICD-9-CM code 345.X for identifying epilepsy (recurrent unprovoked seizures) increased substantially as the number of diagnosis codes in a person's records increased. The PPV was only 38.5% if one diagnosis code was present, whereas it increased to 100% if four diagnosis codes were present. In another study, the PPVs for code 345.X exceeded 80% when the code was in the primary diagnostic position. Thus, this code may perform better in the primary diagnostic position or if it occurs multiple times in records. Most studies did not report the PPV of codes separately. When examined, there was variability in the performance of different codes. Epilepsy codes (345.X) performed the best. Less specific codes for convulsions or seizure-like events (333.2, 779.X, and 780.X) performed variably. Epilepsy codes and less specific codes for convulsions or seizure-like events did not perform particularly well in most studies of infants receiving vaccines, although the study by Shui et al. found that for all visits, ED and IP codes were much more reliable (96.6% PPV for ED and 64.0% PPV for IP) compared with OP codes (16.4% PPV for days 1–30 after vaccination and 1.8% PPV for day 0 to the day of vaccination). One study that used only ED or IP codes to identify seizures found a PPV of 94%. This may lead to the conclusion that such studies should rely on only ED or IP diagnoses to identify seizures. One study that focused on epilepsy found no appreciable differences for calculated PPVs for all charts, IP visits, and emergency visits, suggesting that setting may be more important in identifying isolated seizure events than for identifying epilepsy.
The study by Shui et al. also found that in addition to setting of diagnosis (ED, IP, and OP), age of the child and timing of the event influenced PPVs. Codes from the ED, IP, and OP days 1 to 30 after vaccination in patients 1 year or older were more reliable (PPV, 99.3% for ED, 79.1% for IP, 25.3% OP days 1–30 after vaccination) compared with patients younger than one year (PPV, 1.6% for ED, 59.7% for ED, 12.7% OP days 1–30 after vaccination). It was speculated that this may be because parents of younger children are more likely to bring younger infants to medical attention for suspected seizure events that are later ruled out or that parents are more likely to have more follow-up for seizure events. It was also stated that because PPV is a function of disease prevalence and the incidence of all seizure disorders is highest in the second year of life, it would be expected that PPV would be highest in those age groups that are at most risk.
Adding requirements for drugs or procedures to an algorithm that uses diagnostic codes would likely increase specificity and decrease sensitivity; however, this would not necessarily be true for isolated acute febrile seizures. It is clear from the available studies that using procedure codes for EEGs or prescription claims for drugs possibly used for epilepsy or convulsions in the absence of a diagnostic code is not recommended. These procedures and medications are not specific to epilepsy or seizure events. Given that many newer AEDs require no drug level monitoring, requiring an AED level monitoring procedure in algorithms to identify epilepsy is not recommended. This would result in lack of sensitivity of the algorithm.
Gaps in the current literature make it difficult to distinguish which algorithms would be best for identifying epilepsy as opposed seizure or convulsion. However, for epilepsy, more than one ICD-9-CM epilepsy code in the medical history over time seems to improve the PPV. Also, more research is needed to determine how much is gained or lost by adding AED fills, drug monitoring, or other codes to the algorithms, although a requirement of drug level monitoring is not recommended given that drug levels of many newer AEDs are not checked. From the available evidence, the added value of drug indicators seems limited, although addition of an AED indicator to identify epilepsy may be reasonable. More research is also needed to determine the best algorithm for identifying vaccine-related seizures. It is clear that PPVs for codes used to identify seizures or epilepsy after vaccination were influenced by the setting of diagnosis (ED, IP, OP), age of the child (younger than 1 year, 1 year or older), and timing of the event relative to vaccine administration (day 0, days 1–30). On the basis of currently available data, it may be appropriate to only include seizures diagnosed in the ED or IP setting in studies of seizures potentially related to vaccines. More research is also needed on how to identify seizures and convulsions in adults because most studies of seizures and convulsions have focused on children receiving vaccines. In addition, we only identified one study that used ICD-10 codes, so this is another area that needs more research when ICD-10 codes become more widely used.
Our systematic review of studies assessing algorithms to identify seizure events using administrative and claims data found a wide range of variability in algorithm performance. In particular, we found that PPVs varied according to how the specific outcome (epilepsy versus acute seizure events), setting of diagnosis after vaccination in children (ED, IP, or OP), age of children receiving vaccines (younger than 1 year or 1 year or older), and timing of event after vaccination in children (day 0 or days 1–30).
In addition to ICD-9-CM or ICD-10 codes, some studies also required evidence of AEDs or procedure codes for EEGs. Although for epilepsy (recurrent unprovoked seizures) these requirements may increase the specificity of the codes, we did not find conclusive evidence for this. However, because these procedures and medications are not specific to epilepsy or seizure events, and given that many newer AEDs require no drug level monitoring, requiring an AED level monitoring procedure in algorithms to identify epilepsy is not recommended. Inclusion of these drug/procedure requirements may potentially decrease the sensitivity of algorithms, although more study is needed to confirm this suspicion. Finally, it is not recommended that EEG or AED monitoring codes be used to identify seizures, convulsions, or epilepsy without also requiring an appropriate diagnostic code, as these procedures are not specific to seizures. This would only be appropriate if the maximization of sensitivity was the goal, with a plan to confirm all cases through medical record review.
The authors declare no conflict of interest.
Mini-Sentinel is funded by the Food and Drug Administration through Department of Health and Human Services (HHS) Contract Number HHSF223200910006I. The views expressed in this document do not necessarily reflect the official policies of the Department of Health and Human Services, nor does mention of trade names, commercial practices, or organizations imply endorsement by the U.S. government.