• suicide;
  • epidemiology;
  • self-injury;
  • emergency department


  1. Top of page


As part of the Mini-Sentinel pilot program, under contract with the Food and Drug Administration, an effort has been made to evaluate the validity of algorithms useful for identifying health outcomes of interest, including suicide and suicide attempt.


Literature was reviewed to evaluate how well medical episodes associated with these events could be identified in administrative or claims data sets from the USA or Canada.


Six studies were found to include sufficient detail to assess performance characteristics of an algorithm on the basis of International Classification of Diseases, Ninth Revision, E-codes (950–959) for intentional self-injury. Medical records and death registry information were used to validate classification. Sensitivity ranged from 13.8% to 65%, and positive predictive value range from 4.0% to 100%. Study comparisons are difficult to interpret, however, as the studies differed substantially in many important elements, including design, sample, setting, and methods. Although algorithm performance varied widely, two studies located in prepaid medical plans reported that comparisons of database codes to medical charts could achieve good agreement.


Insufficient data exist to support specific recommendations regarding a preferred algorithm, and caution should be exercised in interpreting clinical and pharmacological epidemiological surveillance and research that rely on these codes as measures of suicide-related outcomes. Copyright © 2012 John Wiley & Sons, Ltd.


  1. Top of page

Suicide is widely understood to be an important psychiatric, public health, and public policy concern. It is a significant cause of mortality, particularly among younger, largely healthy, age groups, for whom it accounts for a substantial proportion of deaths. Yet adequate surveillance faces numerous challenges, including the stigma surrounding the event, which may impede information gathering from social networks; the difficulty of inferring intent in some fatal self-injuries (e.g. drug overdose) and accidents; and the relative infrequency of completed suicides.[1] Autopsy use has declined,[2] and recent cross-national comparisons have suggested that autopsy rates may affect the validity of suicide mortality statistics.[3]

US suicide estimates typically rely on figures from the National Vital Statistics System.[4] The need to improve assessment of incidence, prevalence, and risk factors associated with suicide has prompted calls for expanded data collection to provide a foundation for public health policy and prevention efforts.[5] Given that medical care encounters are associated with many (but by no means all) suicide attempts and completed suicides, data from these encounters may offer one strategy to improve monitoring of suicide attempts and suicides. Therefore, it may be valuable to identify algorithms on the basis of medical claims or administrative and claims data (hereafter administrative data) with well-established and acceptable concordance with independent assessments of either outcome.

In the past decade, suicide, and suicidal thoughts and behaviors more broadly, have received attention in the monitoring of health outcomes associated with medication use. Following reports of possible risks of increased suicidal thoughts and suicide attempts associated with pediatric antidepressant use, an advisory from the Food and Drug Administration panel reviewed data from a meta-analysis that found increased suicidality in antidepressant treated children and adolescents compared with those treated with placebo. A boxed warning regarding this risk was issued in 2004,[6] and this warning was expanded to include 18- to 24-year-olds in 2007.[7]

The research reported here originated from a project conducted as a part of the Food and Drug Administration's Mini-Sentinel pilot program, which is intended initially to refine safety signals that emerge for marketed medical products. This report describes the review process and findings relevant to suicide attempts and completed suicide. No research was found regarding administrative data and suicidal ideation.


  1. Top of page

The details of the methods and their rationale can be found in the accompanying article by Carnahan and Moores.[8] The general search strategy was based on prior work by the Observational Medical Outcomes Partnership and its contractors, which was modified slightly. The base search strategy was then combined with PubMed terms representing the health outcomes of interest. Medical subject heading terms were generally preferred as health outcome of interest search terms because of their likely specificity. Text word searches were sometimes used, particularly when the medical subject heading search resulted in a small number of citations for review. The Iowa Drug Information Service (IDIS) database was searched using a similar search strategy. For a limited number of outcomes where very few citations were identified from PubMed and IDIS searches, Embase searches were conducted. Search results were restricted to articles published on or after 1 January 1990. Mini-Sentinel investigators were also surveyed to identify any published or unpublished studies relevant to validation of claims-based algorithms related to suicide. The PubMed search was conducted on 8 May 2010 and the IDIS searches on 12 June 2010. Full PubMed and IDIS search terms and results for suicide articles can be found on the Food and Drug Administration Mini-Sentinel Web site, An additional search was conducted, adding database names to the PubMed search terms (available upon request).

Abstract review

Each abstract was reviewed independently by the first (JW) and second (LT) authors to determine whether the full-text article should be reviewed. Studies were excluded if (i) there was no mention of suicide, suicide attempt, or suicide ideation; (ii) use was not made of an administrative database, such as insurance claims databases as well as other secondary databases that identify health outcomes using billing codes; and (iii) the data sources used were not from the USA or Canada. Exclusion criteria were documented sequentially (i.e. if exclusion criterion 1 was met then the other criteria were not documented). If the reviewers disagreed on whether the full-text should be reviewed, then it was selected for review. Interrater agreement on whether to include or exclude an abstract was calculated using a Cohen's kappa statistic.

Full-text review

Full-text articles were reviewed independently by JW and LT, with a goal of identifying validation studies described in the article or from its reference section. Citations from the article's references were selected for full-text review if they were cited as a source for the suicide or suicide attempt algorithm or were otherwise deemed likely to be relevant. A full-text review study was excluded if (i) the identification algorithm was poorly described such that it would be difficult to operationalize and (ii) there was no validation of outcome definition or reporting of validity statistics. Exclusion criteria were applied sequentially. If there was disagreement on whether a study should be included, the two reviewers attempted to reach consensus on inclusion by discussion. If no agreement was reached, a third investigator (MO) was consulted to make the final the decision.


  1. Top of page

The PubMed search identified 484 citations, and the two IDIS searches identified 39 unique citations. The total number of unique citations from the combined searches was 508. An additional PubMed search subsequently conducted to augment the original search strategy identified 19 citations.

Abstract reviews

Of the 527 abstracts reviewed, 48 were selected for full-text review; 33 were excluded because they did not study suicide or suicide attempt, 248 were excluded because they were not administrative database studies, and 198 were excluded because the data source was not from the USA or Canada.

Full-text reviews

Of the 48 full-text articles reviewed, 6 articles were included in the final evidence tables; 6 articles were excluded because the suicide or suicide attempt identification algorithm was poorly defined, 34 articles were excluded because they included no validation of the outcome definition or reporting of validity statistics, and 2 articles were excluded because the data were not from the USA or Canada (not determined by abstract review). On the basis of reading of the full-text articles, reviewers identified a further 19 citations for review. Of these, 1 article was included in the final report, 3 articles were excluded because they did not study suicide or suicide attempt, 6 articles were excluded because they were not administrative database studies, 6 articles were excluded because the data source was not from the USA or Canada, 2 articles were excluded because the suicide algorithm was poorly defined, and 1 article was excluded because it included no validation of the outcome definition or reporting of validity statistics.

No respondents to the Mini-Sentinel collaborators provided published or unpublished reports of validation studies completed by their teams. They reported they were aware of two published reports with which they had no direct involvement.

Of the six studies included in Table 1, five studies were articles in the group created by the initial search strategy, and one study was from the group identified through references of articles that underwent full-text review, and none were provided by Mini-Sentinel collaborators. All six publications listed in the evidence table used International Classification of Diseases, Ninth Revision (ICD-9) E-codes to identify patients with suicide attempts or completion.[9-13] All but one of the studies[12] used the full range of ICD-9 E-codes (950–959) for intentional self-injury. The one exception excluded E959 (late effects of self-inflicted injury).

Table 1. Evidence table
CitationStudy population and time periodDescription of outcome studiedAlgorithmValidation/adjudication procedure and operational definitionValidation statistics
Blanc et al.[9]Emergency department patients to two urban hospitals with principal diagnosis of suspected drug overdose or poisoning, except alcohol intoxication; October 1989 to December 1990; n = 533; 65.5% men; median age = 32 years; 24.6% admitted to hospitalIntention of overdose (unintentional, suicide/intentional, assault by poisoning, or undetermined intent) determined by medical record reviewE-codes classified into one of four intention categories: unintentional (E800–E869, E880-E929), intentional (E950–E959), assault (E960–E969, 979), or undetermined (E980–E989)Percentage of E-codes and medical record review determinations that agree by intention category32.1% (171 of 533) agreement by intention category
Iribarren et al.[14]HMO patients 15 to 89 years old who underwent multiphasic health examination between 1 January 1979 to 30 September 1985 (n = 157 827), followed until end of 1993 to identify first hospitalization with discharge diagnosis E950–E959Agreement between chart reviews and a sample of 50 E-code identified hospitalizations, a group with selected injury codes plus depression diagnosis, and a group with injury codes but neither depression diagnosis nor E-codes.(a) Inpatient hospital discharge E-codes for deliberate self-harm (E950–E959) in primary or secondary position, or (b) selected ICD-9 injury codes, including poisoning (by analgesics, antipyretics and antirheumatics, 965; by sedatives and hypnotics, 967; by other central nervous system depressants and anesthetics, 969; by psychotropics, 968; by other and unspecified drugs and medicinal substances, 977) and wound diagnoses (of the elbow, forearm, and wrist, 881), in combination with a depression diagnosis (ICD-9 code 296.20–296.82, 300.4, 300.9, or 311), or (c) same injury codes as (b) but without a depression diagnosisPrimary measure was percentage of E-code discharge diagnoses with suicide attempt confirmed by medical chart review. For examination of possible missed cases, an algorithm was constructed of selected injuries plus depression diagnosis (but not E-code) (n = 43).(a) PPV = 86%
(b) PPV = 26%
(c) PPV = 4%
Rhodes et al.[10]Discharges in 1998–1999 (age ≥16 years) with self-poisoning E-codes (969–989) from urban Canadian trauma hospital; n = 181; mean age: 49.4 years; 56.4% menProportion of discharges determined intentional self-poisoning by three experts blind to administrative (E-code) diagnosisDischarge E-codes for deliberate self-harm (E950–E959) as distinct from unintentional (E800-E869, E880-E929) or undetermined (E980–E989) intentPercentage of self-poisoning determined by discharge diagnosis to be intentional compared with percentage determined using consensus of three experts' determinations based on latent class analysis36.5% of discharges indicating self-poisoning coded as intentional by medical record
59.5% of self-poisoning classified as intentional based on expert review
Shevchenko et al.[11]Discharges from 35 Connecticut acute care hospitals; 1979–1993; n = 202 suicides across hospital and mortality databases, age and gender distribution not availableSuicide as cause of death in Connecticut Death RegistryHospital stay ending in death with a discharge diagnosis of E950–E959 from uniform hospital discharge data setsCalculation of sensitivity and PPV with suicide in death registry as criterion standardSensitivity = 65.0% (91 of 140)
PPV = 59.5% (91 of 153)
Simon et al.[12]HMO outpatients starting antidepressants for depression; 1996–2005; n = 109 256 patients; n = 131 788 episodes; 68.7% women; mean age: 42.6 years; no antidepressant prescription I80 days before index script and ≥1 outpatient depression visit (296.2, 296.3, 311, 300.4) within 30 days of index scriptReview of outpatient notes, emergency room notes, and inpatient discharge summaries for documentation of intentional injury(1) An ICD-9 E-code for intentional self-harm (E950–E958) in 90 days before or 180 days after initial antidepressant prescription or psychotherapy visit related to a depression codePP for medical record verified intentional self-harm (PPV1SH) and suicidal intent (PPV1INT) among patients with intentional self-harm codes and undetermined intent self-harm codes (PPV2SH PPV2INT).PPV1SH = 100% (30 of 30)
(2) A self-harm code of undetermined intent (E980–E987) during the same periodPPV1INT = 100% (30 of 30)
 PPV2SH = 80% (24 of 30)
PPV1INT = 70% (21 of 30)
Weis et al.[13]Suicides in the South Carolina Violent Death Reporting System; 2004; n = 491; 80.9% men; age range: 17–84 years, 4.1% ≤ 17 years; 87.4% White race.Suicide as cause of death determined by death certificates, coroner reports, and law enforcement reports(1) Emergency or inpatient visits on date of confirmed suicideProportion of confirmed suicides that have hospital or emergency department visit within 1 day of death (Sensitivity1) and proportion that have such claims for suicide-related events during year (Sensitivity2)Sensitivity1 = 13.8% (68 of 491)
(2) Suicide-related claims (E-codes: 950–959) from statewide hospital and emergency department billing records during yearSensitivity2 = 14.3% (70 of 491)

Validation criteria and method

Suicide attempts

Three of the four studies in this report that focused on suicide attempts validated administrative coding data through the use of medical charts.[9, 10, 12] The documentation of suicide attempts in the medical records was based on review of inpatient, outpatient, and emergency department notes. None of the studies specified criteria from medical records for determining suicide attempts.

Completed suicide

Both studies in the report that focused on completed suicide validated administrative claims through death registries.[11, 13] One study used a statewide death registry[11] and one relied on an integrated violent death reporting system that included death certificates, coroner reports, and law enforcement reports.[13]

Validation algorithms

Suicide attempts

Because each study of suicide attempts used different study methods, it is difficult to make comparisons across studies.

In a study of 30 outpatients who received E-codes of deliberate self-harm (E950–E958) during a new episode of depression treatment, all 30 patients had medical records that were assessed to document intentional injury and suicidal intent.[12]

A large cohort study was conducted in a health maintenance organization (HMO) of patients with a health examination between 1 January 1979 and 30 September 1985. Although the primary purpose of this study was not to validate an algorithm per se, the analytic strategies and variables used as outcomes provide some pertinent information for assessing algorithms. They first identified hospital stays with a primary or secondary discharge diagnosis that included an intentional self-injury code (E950–E959). From a group of 159 qualifying hospitalizations, medical charts were examined for a randomly selected subgroup of 50 identified in the HMO database. Suicide attempt was verified for 43/50, which translates to a positive predictive value (PPV) of 86% for suicide attempt for this set of E-codes.[14] A second outcome was created on the basis of selected poisoning and open-wound codes not designated as self-inflicted, in combination with a depression diagnosis (296.20–296.82, 300.4, 300.9, and 311). Of the 43 hospitalizations identified, suicide attempt was confirmed in 11 cases, one of which was dropped because the individual has a prior hospitalization with an E-code indicating a self-inflicted poisoning or open wound. The 11 cases confirmed in the 43 hospitalizations translate to a PPV of 26%. The third outcome was defined using these same injury codes but without a depression diagnosis. Among 332 qualifying hospitalizations, 50 randomly selected charts were examined. Two suicide attempts were found, which translates to a PPV of 4%.

A study of adult patients presenting for emergency care with a principal diagnosis of suspected drug overdose or poisoning, other than alcohol intoxication, focused on the validity of E-coding intent classification.[9] E-codes were partitioned in four intent-related groups (intentional, E950–E959; accidental, E800–E869; assault, E960–E969, 979; undetermined, E980–E989). Intention was independently determined by expert review of the medical record. The intent determinations of the record review agreed with the E-codes on intent in 171 of 533 patients (32.1%). Separate data were not presented for each intention group.

A study of adults discharged with self-poisoning diagnoses from a Canadian teaching hospital specializing in trauma (n = 181) reported that a substantially lower percentage of patients received E-codes of intentional self-injury (E950–E959) (36.5%) than were determined to be intentional self-injuries by expert record review (59.5%).[10] Because information was not presented on the intersection of the two ratings, it is not possible to determine operating characteristics of the E-codes from this analysis.

One prospective study, which was not included in the table because it lacked sufficient validity information, followed a consecutive sample of outpatients aged 6 years and older who presented to a university-affiliated emergency department.[15] The investigators compared the rate of completed suicide among patients initially receiving emergency department codes for intentional self-injury (E950–E959) to those with other diagnoses. The results indicate that suicide death rates are significantly higher for discharges with self-injury E-codes (345.5 per 100 000 patient years) than other diagnoses (30.9 per 100 000 patient years) (adjusted hazard ratio = 10.45; 95%CI = 7.78–14.04).[15] By comparison, the word “overdose” in the chief complaint or diagnosis field of emergency department records had roughly one half as strong an association with suicide death (adjusted hazard ratio = 5.24; 95%CI = 3.93–7.00).[15]

Completed suicides

The two studies of suicide death also used different research methods, rendering comparisons difficult. One archival analysis was limited to 202 inpatient deaths that were determined as suicides either by hospital E-codes or by a death registry.[11] Hospital E-coding had a PPV for death registry confirmed suicide of 59.5% (91 of 153). However, the kappa, a chance corrected measure of agreement, was 0.37 (considered fair[16] or poor[17]).

In the second study, the validity of suicide codes in a statewide hospital and emergency department billing records was assessed by comparing them with suicide deaths in an integrated violent death registry as a criterion standard.[13] The registry included death certificates, coroner reports, and law enforcement reports. Of the 491 suicide deaths in the violent death registry, 70 appeared in the billing records (sensitivity = 14.3%). The extent to which there was miscoding or to which deaths registered as suicide occurred in the field and were not brought to medical facilities could not be determined. These uncertainties may have a significant effect, inasmuch as many people who complete suicide are not receiving mental health care at the time of their death.[18] Most the individuals who completed suicide in this study (57.4%, 282 of 491) received at least some inpatient or emergency care during the year before their death.[13]

Selected patient populations

The studies were highly heterogeneous with respect to patient populations. They varied by setting (outpatients,[12] inpatients,[11, 14] emergency department patients[9]), geography (e.g. South Carolina,[13] urban Canada[10]), and self-injury status (e.g. completed suicides,[13] nonlethal self-poisoning[10]).

Age of study population

All studies included adults. Two had lower patient age limits, which were 16[10] and 17 years,[13] respectively. Two did not specify an age range for eligibility,[11, 12] and one study included patients without regard to age.[9] None focused specifically on adolescents, a group at high risk for medically injurious suicide attempts.[19]

One study compared rates of E-code and clinician assessed intentional self-poisoning among discharges from an urban teaching hospital by patient age. In the two oldest age groups (55–64 years and 65 years and older), self-poisoning was identified far more commonly by clinician assessment of medical records than by E-codes. The reverse was true among patients aged 16 to 24 years and 35 to 44 years.[10]

Patient gender

In this same study, estimates of self-poisoning on the basis of clinician assessment of medical records were 67.1% higher in men and 58.7% higher in women than rates based on E-codes.[10] None of the other studies provided information relevant to validation stratified by patient sex.

Time period of data collection

The six reviewed studies were published between 1993 and 2007. The earliest data were based on care delivered in 1979,[11] and the most recent data were derived from 2005.[12]

Principal versus secondary diagnosis

None of the studies were limited to patients with E-codes that were in the principal or primary position and therefore most likely responsible for the service utilization. In addition, none included analyses that were limited to E-codes in the principal position. Three studies noted that E-codes were not limited to the principal position.[9, 10, 14] One study did not indicate E-code position.[11]

Hospitalization diagnosis versus outpatient encounter

Included studies examined suicide outcomes on the basis of hospitalization,[10, 11, 13] outpatient visits,[12] or emergency encounters.[13] None separately assessed validation of suicide outcomes from different treatment settings.

Summary of excluded populations and diagnoses

One of the studies was limited to outpatients initiating new treatment episodes for depression.[12] Two studies selected patients with poisoning,[9, 10] thereby excluding patients with other means of self-harm that do not include self-poisoning. These selection criteria may have affected the validation measures. Determining intent may be more straightforward in poisoning than in some other methods of self-injury. In one study comparing agreement on intent of injury of computerized hospital discharge data with expert review of medical records, the kappa for intent was greater for poisoning (kappa = 0.72) than falls (kappa = 0.40).[20]

The design of the studies resulted in substantial differences in excluded populations. Two studies that involved death certificate validation, for example, drew on general community source populations without regard to healthcare seeking behavior.[11, 13] By contrast, three studies of suicide attempt outcomes were limited to patients receiving medical services.[9, 12, 14] Such structural design differences likely affect the observed validation measures, especially estimates of PPV that relate directly to the prevalence of the underlying condition. For example, it would be expected that a self-injury E-code would have a much higher PPV for confirmed suicide attempt within a population of patients receiving treatment for bipolar disorder, depression, or schizophrenia, which are all high-risk conditions, than in a much lower risk general population sample.

None of the studies were based on nationally representative data. In terms of geography, two studies were derived from single state reporting systems,[11, 13] and the others were based on one[10, 12, 14] or two[9] large healthcare facilities or systems. Because the completeness of E-codes likely varies across treatment settings, it is difficult to generalize how the algorithms will perform in different healthcare settings.


  1. Top of page

Although many characteristics of E-codes in administrative data suggest they might make a valuable contribution to monitoring suicide attempts and completed suicide, limited evidence exists to evaluate their validity. Further research is needed to determine the PPV or sensitivity of E950–E959 as indicators of suicide attempts or suicide death across populations and settings. Although E950–E959 may ultimately prove to have sensitivity and PPV adequate to identify attempted and completed suicide in some contexts, insufficient data currently exist to make definitive recommendations regarding a preferred algorithm. Caution should be exercised in interpreting clinical and pharmacological epidemiological surveillance or research that relies on these codes as measures of suicide-related outcomes, and careful attention paid to the ability of an algorithm to address the specific research questions motivating a study. Because of the scarcity of research in this area, the effects of variation in treatment setting, local coding practices, patient populations, and reporting requirements on the completeness and accuracy of E-codes remain poorly defined.

Little is known about the validity of death certificate determinations of suicide.[21] Some evidence indicates acceptable sensitivity. In a study of 446 deaths among Veterans, death certificates were compared with expert physician panel review using information from hospital, law enforcement, autopsy, and medical examiner records indicated a sensitivity of 90% (54 of 60).[22] Neither of the two studies reported here that used completed suicide as an outcome reported an equivalent sensitivity.[11, 13] One derived two sensitivity figures by determining the proportion of confirmed 2004 suicides that have hospital or emergency department visit within 1 day of death (13.8%, 68 of 491) and the proportion that have such claims for suicide-related events in 2003 or 2004 (14.3%, 70 of 491).[13] The other examined hospital stays ending in death with a discharge diagnosis of E950–E959 from uniform hospital discharge data sets, using the state death registry as a criterion standard (65.0%, 91 of 140).[11]

The comparison of PPVs of death certificates and administrative data is likely complicated by a stigma against designation of a death as a suicide.[23] The perfect specificity in the veterans study is consistent with this bias. This absence of false-positives is associated with a PPV of 100%. Only one of the two studies of completed suicide reported here provides the data needed to calculate a PPV, 59.5%.[11] It should be noted that this study used a state death registry as the criterion standard. A conservative classification bias would tend to depress the PPV.

Given the importance of self-injury E-codes, research on their validity remains surprisingly undeveloped. Some researchers have relied on review of medical records.[9, 10, 12] However, the procedures for reviewing and adjudicating cases have not been well defined or standardized. Although medical record reviewers in some studies were masked to the E-code status of cases,[9, 10] this was not uniformly true.[12] Progress has been recently made in the development of a scale to assess suicidal behavior as an adverse event in clinical trials,[24] and models have been developed to characterize nonsuicidal self-injury as a distinct behavioral construct.[25] It is possible that some of these approaches to measurement will prove relevant to research on the validation of self-injury E-codes.

The importance of criterion standards is further underscored by the nature of the E-codes 950–959. These codes designate self-inflicted injury. However, not all self-inflicted injuries are clinically considered as suicidal behavior. Punching a wall in anger, for example, may result in self-injury of the hand that leads to the appropriate use of an E-code (E956) as well as an N-code (833, open wound of fingers), but because there was no intent to die, the injury would not clinically be considered a suicide attempt. Epidemiological research indicates that self-harm events that are associated with high potential lethality also tend to have high suicidal intent and suicidal risk.[26] These events frequently include E-codes related to firearms (E955.0–E955.4) and suffocation (E953.0–E953.9), but not the more common low lethality self-injuries such as cutting or piercing (E956) or even methods associated with moderate risk of lethality such as intentional poisoning (E950.0–E952.9). At the same time, there is evidence to suggest that some emergency department presentations of cutting/piercing or poisoning that receive E-codes of undetermined intent actually represent deliberate self-harm.[27] Some limited number of suicide attempts are coded as injuries, without assignment of an E-code. However, it may be possible to capture these by relying upon select injury codes in combination with a depression diagnosis (although this method is likely to yield many false-positives).[14] Developing criteria standards to measure suicidal intent poses a central challenge to research on the validation of E950–E959 codes. For most pharmacovigilance studies and evaluations, inclusion of International Classification of Diseases, Ninth Revision, Clinical Modification code E959 (late effects of self-inflicted injury) is not appropriate, as the purpose is typically to establish a temporal relationship between an agent and a self-harm event.

One limitation of the available literature is potentially policy modifiable. Incompleteness of E-coding may vary. E-coding is mandatory in approximately one-half of the states. Where it is required, the average rate of injury discharges that include an E-code often surpasses 90%[28] and exceeds rates in states without mandates or regulations for E-code submission on injury records.[29] Similarly high rates of completion have been reported from Canadian administrative databases. In Medicare and private insurance claims databases, however, low rates of E-code completeness are common. This may be because the billing programs used by some hospitals remove E-codes because they do not factor directly in hospital billing and payment. One report indicated that 28% of injury hospitalizations in a Medicare Provider Analysis and Review data set included an E-code.[30] The relatively low sensitivity of death record confirmed suicides coded as suicide E-codes in one of the reviewed studies may partially reflect incomplete E-coding in the hospitals within the study region.[15] More generally, incompleteness poses interpretive challenges regarding concordance between recorded E-codes and any criterion standard because many sources of incompleteness may be correlated with factors likely to affect concordance (e.g. intent ambiguity). To the extent that states, payers, health plans, or other subpopulations can be identified for which completeness of E-coding is high and consistent, it may be possible to substantially improve the utility of these codes for outcomes assessment. However, in data sets where reporting is incomplete and/or heterogeneous, the use of E-codes as a proxy for suicide attempts is more problematic, although such data sets may still have utility for purposes such as identifying high-risk populations for clinical trials, or for hypothesis-generating analyses.

A promising development to address the problem of unpopulated E-codes is the advent of an algorithm to estimate intentional self-harm events from ubiquitous N-codes. This algorithm was developed using a split half design and two large inpatient administrative data sets that are well populated with E-codes (US Nationwide Inpatient Sample, British Columbia Ministry of Health Inpatient Data). E-codes for deliberate self-harm (E950–E958) served as the criterion standard. The preferred algorithm required a diagnosis of selected psychiatric disorders in combination with a diagnosis of poisoning, asphyxiation, or an open wound to the upper extremity.[31] In the US database, the algorithm achieved a sensitivity of 74%, a specificity of 98%, and a PPV of 73%. This algorithm provides a means of examining deliberate self-harm in data sets with incomplete E-coding, although its performance has not been tested with outpatient claims.

In recent years, there has been an increased interest in potential applications of text mining strategies of electronic medical record notes for identifying cases for clinical research, surveillance, or pharmacovigilance.[32] For example, software tools have been developed that reliably identify a diagnosis of diabetes mellitus documented in physician notes of the electronic medical record.[33] If similar tools were developed to identify suicidal behaviors, it might provide a powerful means of case identification.

Possibly encouraging findings in the review derive from samples drawn from prepaid health plans. In one, concordance between database E-codes and medical charts was examined for a group of 50, finding 43/50 (86%) agreed.[14] The other was a small claims-based study of outpatients who were initiating new episodes of treatment for depression.[12] One of the investigators, an academic psychiatrist, reviewed medical records from 30 outpatients who had received E-codes for deliberate self-harm. The medical records of each of the 30 patients clearly documented intentional injury and suicidal intent. The study was set in a large mixed-model prepaid health plan that serves employer-based and private-paying members as well as members covered through capitation contracts with Medicare or Medicaid.

We conclude that postmarket drug safety surveillance would benefit from the availability of well-validated algorithms to identify suicide attempts from electronic health data. This outcome has broad public health and clinical care significance, as concern exists that antidepressants as well as other agents may increase the risk of suicidal behavior in some patient populations.[34-36]


  1. Top of page

The authors declare no conflict of interest.


Research priorities include:

  • The development of expected norms of rates and distributions of E-codes by age, sex, and treatment setting. These could be used to identify healthcare systems with acceptable levels of E-code completeness. A second means of assessing completeness involves evaluating the proportion of patients with an injury principal diagnosis that have an external cause of injury code.
  • The development of standardized procedures for validating E-codes from external sources that include but that are not necessarily limited to medical records.
  • The assessment of the PPV of algorithms to identify suicide attempts and suicide deaths. Research on the validity of algorithms might assess variation in validity by treatment setting (inpatient, emergency department, outpatient), position (principal or primary, secondary), and specific code (e.g. cutting versus ingestion).
  • Because the predictors and lethality of suicide attempts vary markedly by age and gender,[19] assessment is needed on potential variation in the PPV of E-code algorithms by patient age and gender.
  • Research focused on developing software programs capable of reliably identifying suicidal behaviors from the analysis of the text of physician notes in electronic medical records.


  1. Top of page

Mini-Sentinel is funded by the Food and Drug Administration through Department of Health and Human Services (HHS) Contract Number HHSF223200910006I. The views expressed in this document do not necessarily reflect the official policies of the Department of Health and Human Services, nor does mention of trade names, commercial practices, or organizations imply endorsement by the U.S. government.


  1. Top of page
  • 1
    Nock MK, Borges G, Bromet EJ, Cha CB, Kessler RC, Lee S. Suicide and Suicidal Behavior. Epidemiologic Reviews November 1, 2008; 30(1): 133154.
  • 2
    Hoyert D, Kung HC, Xu J. Autopsy patterns in 2003. National Center for Health Statistics. Vital Health Stat 2007; 20(32): 1127.
  • 3
    Kapusta ND, Tran US, Rockett IRH, et al. Declining Autopsy Rates and Suicide Misclassification: A Cross-national Analysis of 35 Countries. Arch Gen Psychiatry 2011; 68(10): 10501057.
  • 4
    Horan JM, Mallonee S. Injury surveillance. Epidemiologic Reviews 2003; 25:2442.
  • 5
    U.S.P.H. Service. The Surgeon General's call to action to prevent suicide. 1999;
  • 6
    Newman TB. A Black-Box Warning for Antidepressants in Children? N Engl J Med 2004; 351(16): 15951598.
  • 7
    Friedman RA, Leon AC. Expanding the Black Box warning -- Depression, Antidepressants, and the Risk of Suicide. N Engl J Med 2007; 356(23): 23432346.
  • 8
    Carnahan RM, Moores KG. Mini-Sentinel's systematic reviews of validated methods for identifying health outcomes using administrative data: methods and lessons learned. Pharmacoepidemiol Drug Saf 2012; 21(S1): 8289.
  • 9
    Blanc PD, Jones MR, Olson KR. Surveillance of poisoning and drug overdose through hospital discharge coding, poison control center reporting, and the drug abuse warning network. Am J Emergen Med. 1993; 11: 1419.
  • 10
    Rhodes AE, Links PS, Streiner DL, Dawe I, Cass D, Janes S. Do hospital E-codes consistently capture suicidal behaviour? Chronic Dis Can 2002; 23: 139145.
  • 11
    Shevchenko IP, Lynch JT, Mattie AS, Reed-Fourquet LL. Verification of information in a large medical database using linkages with external databases. Stat Med 1995; 14: 511530.
  • 12
    Simon GE, Savarino J. Suicide attempts among patients starting depression treatment with medications or psychotherapy. Am J Psychiatry 2007; 164: 10291034.
  • 13
    Weis MA, Bradberry C, Carter LP, Ferguson J, Kozareva D. An exploration of human services system contacts prior to suicide in South Carolina: an expansion of the South Carolina Violent Death Reporting System. Injury Prevention 2006; 12(suppl 2): ii17-ii21.
  • 14
    Iribarren C, Sidney S, Jacobs DR, Jr., Weisner C. Hospitalization for suicide attempt and completed suicide: epidemiological features in a managed care population. Soc Psychiatry Psychiatr Epidemiol 2000; 35(7): 288296.
  • 15
    Crandall C, Fullerton-Gleason L, Aguero R, LaValley J. Subsequent Suicide Mortality among Emergency Department Patients Seen for Suicidal Behavior. Acad Emerg Med 2006; 13(4): 435442.
  • 16
    Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33(1): 159174.
  • 17
    Fleiss J. Statistical Methods for Rates and Proportions. 2nd ed. John Wiley: New York, 1981.
  • 18
    Owens C, Booth N, Briscoe M, Lawrence C, Lloyd K. Suicide Outside the Care of Mental Health Services. Crisis: The Journal of Crisis Intervention and Suicide Prevention 2003; 24(3): 113121.
  • 19
    Spicer R, Miller T. Suicide acts in 8 states: incidence and case fatality rates by demographics and method. Am J Public Health December 1, 2000; 90(12): 18851891.
  • 20
    LeMier M, Cummings P, West TA. Accuracy of external cause of injury codes reported in Washington State hospital discharge records. INJ PREV December 1, 2001; 7(4): 334338.
  • 21
    Donaldson AE, Larsen GY, Fullerton-Gleason L, Olson LM. Classifying undetermined poisoning deaths. INJ PREV 2006; 12(5): 338343.
  • 22
    Moyer LA, Boyle CA, Pollock DA. Validity of death certificate certificates for injury related causes of death. Am J Epidemiol November 1, 1989; 130(5): 10241032.
  • 23
    Rockett IR. Counting suicides and making suicides count. Crisis 2010; 31(5): 227230.
  • 24
    Posner K, Oquendo MA, Gould M, Stanley B, Davies M. Columbia Classification Algorithm of Suicide Assessment (C-CASA): Classification of Suicidal Events in the FDA's Pediatric Suicidal Risk Analysis of Antidepressants. Am J Psychiatry July 1, 2007; 164(7): 10351043.
  • 25
    Nock MK. Why Do People Hurt Themselves?: New Insights Into the Nature and Functions of Self-Injury. Current Directions in Psychological Science 2009; 18(2): 278283.
    Direct Link:
  • 26
    Harriss L, Hawton K, Zahl D. Value of measuring suicidal intent in the assessment of people attending hospital following self-poisoning or self-injury. Br J Psychiatry January 1, 2005; 186(1): 6066.
  • 27
    Bethell J, Rhodes AE. Identifying deliberate self-harm in emergency department data. Health Reports (Statistics Canada, Catalogue 82-003-X), June 2009; 20(2): 19.
  • 28
    Council of State and Territorial Epidemiologists; Data Committee Injury Control and Emergency Health Services Section APHASaTIPDA. How states are collecting and using cause of injury data: 2004 update to the 1997 report 2005.
  • 29
    200. U.S. Agency for Healthcare Research and Quality. Healthcare Cost and Utilization Project (HCUP) E Code Evaluation Report, 2004. Accessed January 9, 2011.
  • 30
    Clark DE, DeLorenzo MA, Lucas FL, Wennberg DE. Epidemiology and Short-Term Outcomes of Injured Medicare Patients. J Am Geriatr Soc 2004; 52(12): 20232030.
  • 31
    Patrick AR, Miller M, Barber CW, Wang PS, Canning CF, Schneeweiss S. Identification of hospitalizations for intentional self-harm when E-codes are incompletely recorded. Pharmacoepidemiology and Drug Safety 19(12): 12631275.
  • 32
    Trifirò G, Pariente A, Coloma PM, et al. Data mining on electronic health record databases for signal detection in pharmacovigilance: which events to monitor? Pharmacoepidemiology and Drug Safety 2009; 18(12): 11761184.
  • 33
    Turchin A Kohane I, Pendergrass ML. Identification of patients with diabetes from the text of physician notes in the electronic medical record. Diabetes Care 2005; 28: 17941795.
  • 34
    Sorensen HT, Mellemkjaer L, Olsen JH. Risk of suicide in users of beta-adrenoceptor blockers, calcium channel blockers and angiotensin converting enzyme inhibitors. Br J Clin Pharmacol 2001; 52: 313318.
  • 35
    Kalinin W. Suicidality and antiepileptic drugs: is there a link? Drug Saf 2007; 30: 123142.
  • 36
    Stone M, Laughren T, Jones ML, et al. Risk of suicidality in clinical trials of antidepressants in adults: analysis of propietary data submitted to US Food and Drug Administration. BMJ 2009; 339: b2880.