Development and Validation of the Excess Mortality Ratio–adjusted Injury Severity Score Using the International Classification of Diseases 10th Edition

Authors

  • Jaiyong Kim MD,

    1. From the Department of Social and Preventive Medicine, Hallym University College of Medicine (JK), Gangwon-do; the Department of Emergency Medicine, Seoul National University College of Medicine (SDS), Seoul; the Department of Emergency Medicine, Hanyang University College of Medicine (THI), Seoul; the Department of Emergency Medicine and Trauma Surgery, Ajou University Hospital (KJL), Kyongki-Do; the Department of Preventive Medicine, Yonsei University Wonju College of Medicine (SBK), Gangwon-Do; the Department of Emergency Medicine, Cheju National University College of Medicine (JOP), Jeju-Do; the EMS Education and Training Center, Seoul Fire Service Academy (KOA), Seoul; and the Department of Emergency Medicine, Seoul National University, Borame Hospital (KJS), Seoul, Korea
    Search for more papers by this author
  • Sang Do Shin MD, PhD,

    1. From the Department of Social and Preventive Medicine, Hallym University College of Medicine (JK), Gangwon-do; the Department of Emergency Medicine, Seoul National University College of Medicine (SDS), Seoul; the Department of Emergency Medicine, Hanyang University College of Medicine (THI), Seoul; the Department of Emergency Medicine and Trauma Surgery, Ajou University Hospital (KJL), Kyongki-Do; the Department of Preventive Medicine, Yonsei University Wonju College of Medicine (SBK), Gangwon-Do; the Department of Emergency Medicine, Cheju National University College of Medicine (JOP), Jeju-Do; the EMS Education and Training Center, Seoul Fire Service Academy (KOA), Seoul; and the Department of Emergency Medicine, Seoul National University, Borame Hospital (KJS), Seoul, Korea
    Search for more papers by this author
  • Tai Ho Im MD,

    1. From the Department of Social and Preventive Medicine, Hallym University College of Medicine (JK), Gangwon-do; the Department of Emergency Medicine, Seoul National University College of Medicine (SDS), Seoul; the Department of Emergency Medicine, Hanyang University College of Medicine (THI), Seoul; the Department of Emergency Medicine and Trauma Surgery, Ajou University Hospital (KJL), Kyongki-Do; the Department of Preventive Medicine, Yonsei University Wonju College of Medicine (SBK), Gangwon-Do; the Department of Emergency Medicine, Cheju National University College of Medicine (JOP), Jeju-Do; the EMS Education and Training Center, Seoul Fire Service Academy (KOA), Seoul; and the Department of Emergency Medicine, Seoul National University, Borame Hospital (KJS), Seoul, Korea
    Search for more papers by this author
  • Kug Jong Lee MD,

    1. From the Department of Social and Preventive Medicine, Hallym University College of Medicine (JK), Gangwon-do; the Department of Emergency Medicine, Seoul National University College of Medicine (SDS), Seoul; the Department of Emergency Medicine, Hanyang University College of Medicine (THI), Seoul; the Department of Emergency Medicine and Trauma Surgery, Ajou University Hospital (KJL), Kyongki-Do; the Department of Preventive Medicine, Yonsei University Wonju College of Medicine (SBK), Gangwon-Do; the Department of Emergency Medicine, Cheju National University College of Medicine (JOP), Jeju-Do; the EMS Education and Training Center, Seoul Fire Service Academy (KOA), Seoul; and the Department of Emergency Medicine, Seoul National University, Borame Hospital (KJS), Seoul, Korea
    Search for more papers by this author
  • Sang Back Ko MD,

    1. From the Department of Social and Preventive Medicine, Hallym University College of Medicine (JK), Gangwon-do; the Department of Emergency Medicine, Seoul National University College of Medicine (SDS), Seoul; the Department of Emergency Medicine, Hanyang University College of Medicine (THI), Seoul; the Department of Emergency Medicine and Trauma Surgery, Ajou University Hospital (KJL), Kyongki-Do; the Department of Preventive Medicine, Yonsei University Wonju College of Medicine (SBK), Gangwon-Do; the Department of Emergency Medicine, Cheju National University College of Medicine (JOP), Jeju-Do; the EMS Education and Training Center, Seoul Fire Service Academy (KOA), Seoul; and the Department of Emergency Medicine, Seoul National University, Borame Hospital (KJS), Seoul, Korea
    Search for more papers by this author
  • Ju Ok Park MD,

    1. From the Department of Social and Preventive Medicine, Hallym University College of Medicine (JK), Gangwon-do; the Department of Emergency Medicine, Seoul National University College of Medicine (SDS), Seoul; the Department of Emergency Medicine, Hanyang University College of Medicine (THI), Seoul; the Department of Emergency Medicine and Trauma Surgery, Ajou University Hospital (KJL), Kyongki-Do; the Department of Preventive Medicine, Yonsei University Wonju College of Medicine (SBK), Gangwon-Do; the Department of Emergency Medicine, Cheju National University College of Medicine (JOP), Jeju-Do; the EMS Education and Training Center, Seoul Fire Service Academy (KOA), Seoul; and the Department of Emergency Medicine, Seoul National University, Borame Hospital (KJS), Seoul, Korea
    Search for more papers by this author
  • Ki Ok Ahn MD,

    1. From the Department of Social and Preventive Medicine, Hallym University College of Medicine (JK), Gangwon-do; the Department of Emergency Medicine, Seoul National University College of Medicine (SDS), Seoul; the Department of Emergency Medicine, Hanyang University College of Medicine (THI), Seoul; the Department of Emergency Medicine and Trauma Surgery, Ajou University Hospital (KJL), Kyongki-Do; the Department of Preventive Medicine, Yonsei University Wonju College of Medicine (SBK), Gangwon-Do; the Department of Emergency Medicine, Cheju National University College of Medicine (JOP), Jeju-Do; the EMS Education and Training Center, Seoul Fire Service Academy (KOA), Seoul; and the Department of Emergency Medicine, Seoul National University, Borame Hospital (KJS), Seoul, Korea
    Search for more papers by this author
  • Kyoung Jun Song MD

    1. From the Department of Social and Preventive Medicine, Hallym University College of Medicine (JK), Gangwon-do; the Department of Emergency Medicine, Seoul National University College of Medicine (SDS), Seoul; the Department of Emergency Medicine, Hanyang University College of Medicine (THI), Seoul; the Department of Emergency Medicine and Trauma Surgery, Ajou University Hospital (KJL), Kyongki-Do; the Department of Preventive Medicine, Yonsei University Wonju College of Medicine (SBK), Gangwon-Do; the Department of Emergency Medicine, Cheju National University College of Medicine (JOP), Jeju-Do; the EMS Education and Training Center, Seoul Fire Service Academy (KOA), Seoul; and the Department of Emergency Medicine, Seoul National University, Borame Hospital (KJS), Seoul, Korea
    Search for more papers by this author

  • Presented at the ACEP Research Forum, October 2007, Seattle WA.

  • This study was conducted by the Injury Statistics Working Group of Korea, which was composed on the basis of the convention for injury research among the Korean Society of Emergency Medicine, the Korean Society of Traumatology, the Korean Burn Society, and the Health Insurance Review Agency. This study was financially supported by the Korean Center for Disease Control and Prevention in 2006.

Sang Do Shin, MD, PhD; e-mail: shinsangdo@snuh.org.

Abstract

Objectives:  This study aimed to develop and validate a new method for measuring injury severity, the excess mortality ratio–adjusted Injury Severity Score (EMR-ISS), using the International Classification of Diseases 10th Edition (ICD-10).

Methods:  An injury severity grade similar to the Abbreviated Injury Scale (AIS) was converted from the ICD-10 codes on the basis of quintiles of the EMR for each ICD-10 code. Like the New Injury Severity Score (NISS), the EMR-ISS was calculated from three maximum severity grades using data from the Korean National Injury Database. The EMR-ISS was then validated using the Hosmer-Lemeshow goodness-of-fit chi-square (HL chi-square, with lower values preferable), the area under the receiver operating characteristic curve (AUC-ROC), and the Pearson correlation coefficient to compare it with the International Classification of Diseases 9th Edition–based Injury Severity Score (ICISS). Nationwide hospital discharge abstract data (DAD) from stratified-sample general hospitals (n = 150) in 2004 were used for an external validation.

Results:  The total number of study subjects was 29,282,531, with five subgroups of particular interest identified for further study: traumatic brain injury (TBI, n = 3,768,670), traumatic chest injury (TCI, n = 1,169,828), poisoning (n = 251,565), burns (n = 869,020), and DAD (n = 26,374). The HL chi-square was lower for EMR-ISS than for ICISS in all groups: 42,410.8 versus 55,721.9 in total injury, 7,139.6 versus 20,653.9 in TBI, 6,603.3 versus 4,531.8 in TCI, 2,741.2 versus 9,112.0 in poisoning, 764.4 versus 4,532.1 in burns, and 28.1 versus 49.4 in DAD. The AUC-ROC for death was greater for EMR-ISS than for ICISS: 0.920 versus 0.728 in total injury, 0.907 versus 0.898 in TBI, 0.675 versus 0.799 in TCI, 0.857 versus 0.900 in poisoning, 0.735 versus 0.682 in burns, and 0.850 versus 0.876 in DAD. The Pearson correlation coefficient between the two scores was −0.68 in total injury, −0.76 in TBI, −0.86 in TCI, −0.69 in poisoning, −0.58 in burns, and −0.75 in DAD.

Conclusions:  The EMR-ISS showed better calibration and discrimination power for prediction of death than the ICISS in most injury groups. The EMR-ISS appears to be a feasible tool for passive injury surveillance of large data sets, such as insurance data sets or community injury registries containing diagnosis codes. Additional further studies for external validation on prospectively collected data sets should be considered.

Despite the debate about the accuracy of diagnosis codes in administrative or insurance claim data, these data are potential resources for the monitoring of population health and health planning programs (such as Healthy People 2010) and for the study of racial and ethnic inequality in health programs, geographic variations, and surveillance.1 Administrative data such as that found in the U.S. Medicare and Medicaid databases and the National Hospital Discharge Survey have been widely used in injury research in the United States. To best study injury, information is needed regarding injury severity. Most administrative databases, however, contain only diagnostic codes rather than directly measured values of injury severity. Among the various methods for measuring injury severity, the New Injury Severity Score (NISS) has been considered a more predictable indicator and superior to the Injury Severity Score (ISS) for the analysis of mortality, multiple organ failure, and functional recovery from multiple trauma.2–4 Despite its excellent performance, the NISS has not been widely used, perhaps because it is derived from the effort-intensive Abbreviated Injury Scale (AIS), which typically is recorded by an independent, qualified recorder.5

The Barell matrix is another approach to gathering information related to injury severity from diagnostic codes. Based on the International Classification of Diseases 9th revision, Clinical Modification (ICD9-CM), it has been favorably compared to the ISS.6 However, the Barell matrix also has not been widely used, perhaps because several diagnostic codes could not be assigned to a matrix cell, which led to incomplete conversion from ICD9-CM to the Barell matrix.7 The ICD9-based Injury Severity Score (ICISS), which is a score derived from the survival probability of trauma patients, was developed as an alternative to the NISS.8,9 Kim et al.10,11 then developed the ICD 10th Edition–based ISS and demonstrated its utility in comparing the performance of emergency medical centers. Despite the feasibility and cost-effectiveness of the ICISS on the basis of either the 9th or the 10th edition, these scales cannot classify an injury into an anatomically injured body area or into an anatomically specified severity scale like the AIS. In addition, the injury severity of non-traumatic injury (such as poisoning, burns, or drowning) cannot be calculated with the present methodologies for measuring injury severity. A valid method of automatically calculating injury severity from existing diagnostic codes of all kinds of injury is needed. This method could then be used in research projects and quality assurance programs for emergency departments (EDs), emergency medical services systems, and trauma systems without consuming additional resources in recoding. In this study, after developing a new methodology for measuring the anatomically based severity of all kinds of injury (both traumatic and nontraumatic), we attempted to validate its discrimination and calibration performance.

Methods

This study, which involved only administrative data not containing individual, private information, was approved by the Korea Center for Disease Control and Prevention.

Database Development

The Korean National Injury Database (NIDB) was constructed from the National Health Insurance (NHI) data set, the Industrial Accident Compensation Insurance (IACI) data set, and the National Death Certificate (NDC) database, using data from 2001 to 2003. The NHI data set is produced by the National Health Insurance Corporation, which is the sole authority for public medical insurance in Korea. The NHI data set includes administrative, diagnostic, and procedural information about injury and disease cases using medical care, except for cases occurring in an occupational setting and a few cases supported by private insurance programs. The IACI data set is managed by the Industrial Accident Compensation Insurance Corporation, which is the authority for public compensation in cases related to occupational and industrial injury or disease. The NDC database is created by the government statistical office.

All injury cases resulting in hospital visits were included in this study. Injury cases were defined as any patient having one or more ICD-10 diagnostic codes of which the initial letter was “S” or “T.” ICD-10 codes S00 through T98 are used to code “Injury, poisoning and certain other consequences of external causes.”12 We merged the NHI data set and the IACI data set to the NDC to derive mortality figures for each ICD-10 injury code, which is essentially the same method that was used in deriving the ICISS system.

Calculation of the Excess Mortality Ratio (EMR)

Using the NHI, we constructed a national cohort consisting of all patients who utilized a medical service in 2001 who had no medical service visits for an injury condition (ICD-10 S00-T98, referred to as “ICD-10/S/T” hereafter) in the prior 6 years (1995 to 2000). From this cohort, we derived observed numbers of deaths for each 5-year age and sex group for each ICD-10/S/T diagnostic code within a 1-year follow-up period. We chose a 1-year follow-up in an effort to measure community-based mortality with longer-term outcomes than ED or hospital mortality data would allow. No consensus exists regarding the proper mortality follow-up interval for such work,13 and recent injury research using administrative data seem to be focusing on longer term outcomes.14

We then calculated baseline expected mortality numbers for each 5-year age and sex group using the NDC database and 2001 national population statistics. The EMR for each ICD-10/S/T code was then calculated as the ratio of the observed number of deaths to the expected number of deaths for each group.15Table 1 shows an example of a diagnostic code–specific EMR derived from this process. There were 15,852 male patients between the ages of 40 and 44 years seen for this particular injury in 2001. Of these, 222 died in the 1-year period following the initial visit for this condition. The expected number of deaths in a group of 15,852 patients of this age and sex is 65, so this particular condition has an EMR of 222/65 = 3.4 for males between the ages of 40 and 44. The calculations at the bottom of the table show that this particular condition has an overall EMR of 1.695, meaning that patients of all ages and sexes with this injury are 1.695 times more likely to die in the following year than are patients without this injury. We calculated EMRs for all injury-related diagnostic codes (ICD-10 S00-T98).

Table 1. 
An Example of Calculation of the EMR of a Specific Diagnosis Code
Age groups (year)MaleFemale
Number of PatientsObserved Number of DeathsExpected Number of DeathsNumber of PatientsObserved Number of DeathsExpected Number of Deaths
  1. Observed number of deaths means the number of deaths with a specific diagnosis code by 5-year age and sex group observed in NHI Database. Expected number of deaths means the number of deaths expected when applying mortality by 5-year age and sex groups derived from general population.

  2. EMR = excess mortality ratio; NHI = National Health Insurance.

0–43181023620
5–93960027110
10–148722067310
15–191,546411,15130
20–241,487321,85341
25–293,3851143,908112
30–346,22036105,226164
35–3911,80997318,103348
40–4415,8522226510,3785514
45–4916,56227610612,3985926
50–5418,10035816316,33612253
55–5920,78457129122,436217108
60–6417,51268836921,354302174
65–6912,13161839317,464416255
70–747,25253037311,882495320
75–794,2594503436,894483343
80–841,3182121632,622329221
≥854329383813151128
Sum140,235  4,172 (A)  2,397 (B)143,998  2,701 (C)1,657 (D)
EMR:Total observed number of deaths (A + C) = 6,873
Total expected number of deaths (B + D) = 4,056
EMR: (A + C)/(B + D) = 6,873/4,056 = 1.695

Severity Measuring of Each Injury Episode

We developed the EMR-based Injury Severity Score (EMR-ISS) by assigning each EMR to one of five grades of severity by quintiles, which we termed minimal, mild, moderate, severe, and critical. The injured body area was categorized as one of the six areas used in assigning an Abbreviated Injury Scale (AIS-90) code: “head and face,”“neck,”“trunk (chest, abdomen, and pelvis),”“upper extremity,”“lower extremity,” and “other and nontraumatic injury.”16 Some ICD-10 codes, however, could not be clearly assigned to one of these six areas. Therefore, we used modified anatomical classes from the AIS code system, such as “trunk” instead of “chest” and “abdomen and pelvis” or “other and nontraumatic injury” instead of “surface.” For example, most of the codes T00–T07 (injuries involving multiple body regions) and T08–T14 (injuries to unspecified part of trunk, limb, or body region) could not be classified as one specific part of the body. Those injuries were assigned to the last category “other and nontraumatic,” except for some codes that could be classified as trunk, upper extremity, or lower extremity (T021, T080, T024, T025, T114, T051, T115, T116, T118, T100, T022, T113, T138, T135, T120, T134, T020, T023, T025, T053, T055). All poisonings, drowning, and burns were coded as “other and nontraumatic injury.” Using the EMR grade and the anatomical classes, we converted each ICD-10/S/T code to the new severity scale and anatomically injured body area (see Data Supplement S1, available as supporting information in the online version of this paper).

Using the EMR grades, we could calculate the EMR-ISS for each injury episode. As with the method for calculating the NISS, the EMR-ISS was calculated as the sum of the squares of the three highest EMR grades among all injury codes: EMR-ISS = (first highest EMR grade)2 + (second highest EMR grade)2 + (third highest EMR grade)2. The sum of squares was used as the ICD-10 injury codes reflect anatomically injured body areas, similar to the body areas in the AIS; the NISS is a sum of the squares of three maximum AIS scores.

All injury episodes were also classified into four groups: mild (1 ≤ EMR-ISS ≤ 8), moderate (9 ≤ EMR-ISS ≤24), severe (25 ≤ EMR-ISS ≤ 74), and critical (EMR-ISS ≥ 75 or death). As in other studies where the ISS or NISS is categorized, we used four severity categories.17

Validation of the EMR-ISS

We used several methods to attempt to internally validate the EMR-ISS. We first calculated the ICISS as the criterion standard of severity measurement, using the survivors risk ratio (SRR) of each patient’s ICD-10 code. The ICISS uses the ICD diagnosis codes for severity measurement and has been regarded as the only validated tool for ICD-10.11 The Hosmer-Lemeshow goodness-of-fit chi-squares (HL chi-squares) of both the ICISS and the EMR-ISS were calculated and calibration curves were generated to compare performance.18 This test divides the data into deciles, creating a 2 × 10 contingency table, and then uses the Pearson chi-square test to compare the predicted to the observed frequency. Lower values and nonsignificance (p > 0.05) indicate a good fit to the data. However, sample size can affect the HL value and significance, and given the extremely large sample size in this study (nearly 30 million), resulting in higher HL values than are typically seen, we should compare only its relative value between the EMR-ISS and the ICISS.

Second, the area under the receiver operating characteristic curve (AUC-ROC) for prediction of death was used and ROCs were generated to compare the EMR-ISS to the ICISS for discrimination power.19 This analysis is the main result of our study, with the AUC-ROC regarded as the criterion standard method for estimating the performance of the new injury severity measuring method. The higher the AUC-ROC of a specific tool, the better its performance to predict outcome.

Third, we estimated the Pearson correlation coefficient (r) for the correlation between the two scales.20 Here a value close to −1 is desirable, because the two scales being compared run in opposite directions.

Fourth, subgroup validation analyses were performed with the traumatic brain injury (TBI), traumatic chest injury (TCI), poisoning, and burn injury data sets as examples. These subgroups were chosen for further analysis due to the high rates of morbidity and mortality associated with TBI and TCI and data showing that poisoning and burns are the second and sixth leading causes of death due to injury, respectively, in Korea.21–23 Patients in these subgroups may have had other injuries besides that of the specific subgroup; for example, a patient in the TBI subgroup might also have chest injuries.

Finally, we externally validated the EMR-ISS using the Korea Discharge Abstract Database (DAD) of 2004. This database is constructed annually by the Korea Center for Disease Control and Prevention. Nationwide, representative discharge abstracts of 150 general hospitals, which were sampled using two strata (hospital size and geographic location), are reviewed by trained medical record reviewers. DAD includes information about injury-related factors, and we thus calculated the ICISS and the EMR-ISS from recorded ICD-10/S/T codes and compared its discrimination power using the AUC-ROC and the Pearson correlation coefficient.

Results

Demographic Findings

Table 2 shows the demographic findings of the 29,282,531 study subjects. Of these, 22.1% were children below 15 years, and 9.1% were adults older than 65 years. By injury severity, the proportions of mild (1 ≤ EMR-ISS ≤ 8), moderate (9 ≤ EMR-ISS ≤ 24), severe (25 ≤ EMR-ISS ≤ 74), and critical injury (EMR-ISS ≥ 75 or death) were 81.0, 17.3, 1.3, and 0.3%, respectively. Overall, 93.7% of patients were managed in outpatient settings (including EDs), and 6.0% were admitted. The overall crude mortality rate was 0.3%. Table 3 shows the anatomical body areas from all involved diagnoses. The total number of diagnosis codes was 34,192,467, which indicates that many patients had more than one injury. The overall proportions of “head and face,”“neck,”“trunk,”“upper extremities,”“ lower extremities,” and “surface and nontraumatic” injuries were 12.7, 5.2, 19.2, 27.1, 22.6, and 13.2%, respectively (Table 3).

Table 2. 
Demographics of the Study Population
 MaleFemaleTotal
  1. Data are reported as n (%).

  2. EMR-ISS = excess mortality ratio–based Injury Severity Score; IACI DB = Industrial Accident Compensatory Insurance database; NDC = National Death Certificates; NHI DB = National Health Insurance database.

  3. *Injury severity was categorized on the basis of the EMR-ISS: mild (1 ≤ EMR-ISS ≤ 8), moderate (9 ≤ EMR-ISS ≤ 24), severe (25 ≤ EMR-ISS ≤ 74), and critical (EMR-ISS ≥ 75 or death).

Total15,731,930 (53.7)13,534,961 (46.22)29,282,531 (100.0)
Age group
 Child (<15 years)4,038,714 (25.7)2,433,869 (18.0)6,472,583 (22.1)
 Adult (≥15 and <65 years)10,729,952 (68.2)9,401,947 (69.5)20,131,890 (68.8)
 Elder (≥65 years)963,264 (6.1)1,699,145 (12.6)2,662,409 (9.1)
Year
 20014,916,985 (31.3)4,274,477 (31.6)9,191,462 (31.4)
 20025,220,607 (33.2)4,444,142 (32.8)9,664,749 (33.0)
 20035,594,338 (35.6)4,816,342 (35.6)10,410,680 (35.6)
Database
 NHI DB15,516,303 (98.6)13,534,961 (100.0)29,001,500 (99.0)
 IACI DB168,642 (1.1)27,599 (0.2)196,241 (0.7)
 NDC46,985 (0.3)22,165 (0.2)69,150 (0.2)
Injury severity*
 Mild12,469,242 (79.3)11,237,659 (83.0)23,706,901 (81.0)
 Moderate2,957,307 (18.8)2,115,373 (15.6)5,072,680 (17.3)
 Severe239,541 (1.5)153,489 (1.1)393,030 (1.3)
 Critical65,840 (0.4)28,440 (0.2)94,280 (0.3)
Outcome
 Outpatient14,609,500 (92.9)12,822,054 (94.7)27,431,554 (93.7)
 Admission1,060,953 (6.7)685,594 (5.1)1,746,547 (6.0)
 Death61,477 (0.4)27,313 (0.2)88,790 (0.3)
Table 3. 
Anatomical Class by Injury Severity
Anatomical classInjury severity, n (%)*
MildModerateSevereCriticalTotal
  1. EMR-ISS = excess mortality ratio–based Injury Severity Score.

  2. *Injury severity was categorized on the basis of the EMR-ISS: mild (1 ≤ EMR-ISS ≤ 8), moderate (9 ≤ EMR-ISS ≤ 24), severe (25 ≤ EMR-ISS ≤ 74), and critical (EMR-ISS ≥ 75 or death).

Head and Face963,241 (22.1)2,959,954 (67.9)394,623 (9.1)40,592 (0.9)4,358,410 (100.0)
Neck1,561,794 (88.4)165,245 (9.4)36,482 (2.1)3,062 (0.2)1,766,583 (100.0)
Trunk5,455,559 (83.2)970,245 (14.8)115,723 (1.8)16,025 (0.2)6,557,552 (100.0)
Upper Extremities8,408,290 (90.9)787,294 (8.5)50,825 (0.5)3,115 (0.0)9,249,524 (100.0)
Lower Extremities6,774,229 (87.6)897,575 (11.6)54,586 (0.7)6,133 (0.1)7,732,523 (100.0)
Other and nontraumatic3,265,498 (72.1)1,009,308 (22.3)194,802 (4.3)58,267 (1.3)4,527,875 (100.0)
Total26,428,611 (77.3)6,789,621 (19.9)847,041 (2.5)127,194 (0.4)34,192,467 (100.0)

Table 4 shows survival and crude fatality rates, the ICISS, and the EMR-ISS for all patients and for the TBI, TCI, poisoning, and burn subgroups. Poisoning had the highest fatality rate (6.1%) and the lowest ICISS (0.962 ± 0.068) compared to those of the other injury subgroups (p < 0.0001 by chi-square for fatality and p < 0.0001 by independent t-test for ICISS).

Table 4. 
The Crude Mortality Rate, the ICISS, and the EMR-ISS of the Study Population
 TotalSurvivalDeathICISSEMR-ISS
  1. Data are reported as n (%) and mean ± standard deviation.

  2. EMR-ISS = excess mortality ratio–based Injury Severity Score; ICISS = International Classification of Diseases 10th Edition–based Injury Severity Score.

Total29,266,89129,178,101 (99.7)88,790 (0.3)0.992 ± 0.0174.534 ± 5.262
Brain injury3,768,6703,749,800 (99.5)18,870 (0.5)0.988 ± 0.02610.094 ± 7.909
Chest injury1,169,8281,164,415 (99.5)5,413 (0.5)0.979 ± 0.0237.385 ± 6.756
Poisoning251,565236,282 (93.9)15,282 (6.1)0.962 ± 0.06810.021 ± 11.078
Burn injury869,020866,437 (99.7)2,583 (0.3)0.992 ± 0.0255.680 ± 5.564

Validation for Discrimination, Calibration, and Correlation

We validated the EMR-ISS through comparison with the ICISS (Table 5). For the overall study population, the HL chi-square was 55,721.9 in the ICISS (p < 0.0001) and 42,410.8 in the EMR-ISS (p < 0.0001). Although these HL values are relatively high (with p < 0.05) and look like poor model fit, the HL value of the EMR-ISS is less than that of the ICISS, indicating better performance of the EMR-ISS. The calibration curves for total injury (Figure 1) and subgroups (Figure 2) show good correlation between expected probability and observed probability. TBI showed a lower HL chi-square (7,139.6) compared to that of ICISS (20,653.9). Poisoning and burn injury showed also lower HL chi-square in EMR-ISS (2,741.2 and 764.4, respectively) versus in ICISS (9,112.0 and 4,532.1).

Table 5. 
Comparison of Discrimination and Calibration for the Prediction of Death of the ICISS and the EMR-ISS and Correlation Coefficient Between ICISS and EMR-ISS
Injury SubgroupSeverity ScaleHL Chi-square†AUC-ROCCorrelation Coefficient (95% CI)
  1. AUC-ROC = area under the receiver operating characteristic curve; EMR-ISS = excess mortality ratio–based Injury Severity Score; HL chi-square = Hosmer-Lemeshow goodness-of-fit chi-square; ICISS = International Classification of Diseases 10th Edition–based Injury Severity Score.

  2. †The lower the HL value is, the more calibration power there is, expressing the difference between the expected and the observed number of deaths in each decile, according to increasing injury severity.

TotalICISS55,721.90.728−0.67876 (−0.67895, −0.67856)
EMR-ISS42,410.80.920
Brain injuryICISS20653.90.898−0.76359 (−0.76401, −0.76316)
EMR-ISS7139.60.907
Chest injuryICISS4531.80.799−0.86020 (−0.86068, −0.85973)
EMR-ISS6603.30.675
PoisoningICISS9112.00.900−0.68999 (−0.69204, −0.68794)
EMR-ISS2741.20.857
Burn injuryICISS4532.10.682−0.57800 (−0.57939, −0.57659)
EMR-ISS764.40.735
Figure 1.

 Estimated and observed probability plot of HL goodness-of-fit test in total injury. EMR-ISS = excess mortality ratio–adjusted Injury Severity Score; HL = Hosmer-Lemeshow; ICISS = International Classification of Diseases 9th Edition–based Injury Severity Score.

Figure 2.

 Estimated and observed probability plot of HL goodness-of-fit test in injury subgroups (TBI, TCI, poisoning, and burn injury). EMR-ISS = excess mortality ratio–adjusted Injury Severity Score; ICISS = International Classification of Diseases 9th Edition–based Injury Severity Score; HL = Hosmer-Lemeshow; TBI = traumatic brain injury; TCI = traumatic chest injury.

The AUC-ROC was 0.728 in the ICISS and 0.920 in the EMR-ISS for total injury (Figure 3). When we performed an analysis of the injury subgroups, TBI and poisoning showed very good discrimination power via EMR-ISS (AUC-ROC = 0.907 in TBI and 0.857 in poisoning) versus ICISS (AUC-ROC = 0.898 in TBI and 0.900 in poisoning). TCI and burn injury showed moderate discrimination power via EMR-ISS (AUC-ROC = 0.675 in TCI and 0.735 in burn injury) versus ICISS (AUC-ROC = 0.799 in TCI and 0.682 in burn injury; Figure 4).

Figure 3.

 Comparison of discrimination and calibration for the prediction of death of the ICISS and the EMR-ISS in total injury. AUC = area under the receiver operating characteristic curve; EMR-ISS = excess mortality ratio–based Injury Severity Score; ICISS = International Classification of Diseases 10th Edition–based Injury Severity Score.

Figure 4.

 Comparison of discrimination and calibration for the prediction of death of the ICISS and the EMR-ISS in injury subgroups. AUC = area under the receiver operating characteristic curve; EMR-ISS = excess mortality ratio–based Injury Severity Score; ICISS = International Classification of Diseases 10th Edition–based Injury Severity Score.

The Pearson correlation coefficient (r; 95% confidence interval [CI]) between the ICISS and the EMR-ISS was −0.67876 (95% CI = −0.67895 to −0.67856; p < 0.0001). The correlation coefficient was −0.76359 (95% CI = −0.76401 to −0.76316) for TBI, −0.86020 (95% CI = −0.86068 to −0.85973) for TCI, −0.68999 (95% CI = −0.69204 to −0.68794) for poisoning, and −0.57800 (95% CI = −0.57939 to −0.57659) for burn injuries (Table 5).

Figure 5 shows the results from external validation using the Korea DAD of 2004. The AUC-ROC was not significantly different (0.850 in EMR-ISS vs. 0.875 in ICISS), with HL values of 28.1 in EMR-ISS versus 49.4 in ICISS. The Pearson correlation coefficient was excellent (−0.75082) between the two methods. These findings show that the EMR-ISS is comparable to the ICISS on calibration and discrimination power to predict mortality.

Figure 5.

 Results from external validation using the Korea DAD of 2004. The AUC-ROC, HL chi-square, and Pearson correlation coefficient were used to compare the EMR-ISS to the ICISS. AUC = area under the receiver operating characteristic curve; DAD = Discharge Abstract Database; EMR-ISS = excess mortality ratio–based Injury Severity Score; HL chi-square = Hosmer-Lemeshow goodness-of-fit chi-square test; ICISS = International Classification of Diseases 10th Edition–based Injury Severity Score.

Table 6 shows the number of injury episodes for each specific injury group. In the TBI subgroup, the total number of injury episodes was 3,768,670, but the number of diagnosis codes was 4,210,020, with 28.7% of these codes coming from patients with multiple diagnosis codes. One TBI-specific code was found in 86.8% of cases, and two or more TBI-related diagnosis codes were found in 13.2%. In other specific injury groups, 109,465 episodes (12.7%) had additional burn-related codes, and 1,795 (0.7%) had other poisoning codes. The proportions of patients with more than two diagnosis codes were 22.8% in TBI, 29.4% in TCI, 7.7% in poisoning, and 16.2% in burn.

Table 6. 
Number of Episodes by Diagnosis Code in Each Specific Injury Group
 TBITCIPoisoningBurn Injury
  1. Data are reported as n (%).

  2. TBI = traumatic brain injury; TCI = traumatic chest injury.

  3. *Number of episodes with only same-type injury codes, such as codes related with TBI, TCI, burn, and poisoning episode, respectively.

  4. †Number of episodes with all kinds of injury type codes as well as codes related with TBI, TCI, burn, or poisoning episode, respectively.

Number of episodes*3,768,670 (100.0)1,169,828 (100.0)251,565 (100.0)869,020 (100.0)
 With one same-type diagnosis code 3,271,014 (86.8)1,104,712 (94.4)249,770 (99.3)759,555 (87.4)
 With two same-type diagnosis codes429,173 (11.4)63,470 (5.34)1,742 (0.7)99,072 (11.4)
 With three same-type diagnosis codes 49.528 (1.3)2,298 (0.20)36 (0.0)7,529 (0.9)
 With four same-type diagnosis codes 14,264 (0.4)329 (0.03)15 (0.0)2,134 (0.3)
 With five same-type diagnosis codes4,691 (0.1)19 (0.00)2 (0.0)730 (0.1)
Number of all codes diagnosed4,210,020 (128.7)1,239,957 (112.2)273,327 (108.7)1,035,025 (118.9)
Number of episodes†3,768,670 (100.0)1,169,828 (100.0)251,604 (100.0)870,428 (100.0)
 With one diagnosis code2,910,094 (77.2)825,339 (70.6)232,284 (92.3)727,829 (83.8)
 With two diagnosis codes653,606 (17.3)267,829 (22.9)17,508 (7.0)124,612 (14.3)
 With three diagnosis codes117,782 (3.1)44,585 (3.8)1,287 (0.5)12,028 (1.4)
 With four diagnosis codes45,074 (1.2)16,388 (1.4)322 (0.1)3,173 (0.4)
 With five diagnosis codes42,114 (1.1)15,687 (1.3)164 (0.1)1,378 (0.2)

Discussion

We developed the new injury severity system, EMR-ISS, which is calculated by a mortality ratio−based model, making EMR-ISS a kind of empirical outcome-based severity measuring method like the ICISS.

The EMR-ISS has great potential compared to other severity measuring tools. Most of the previously used methods for injury severity measurement have variable merits and limitations. In particular, the ICISS has several merits including feasibility, cost-effectiveness, and accurate predication of survival to compare trauma centers or trauma systems.5 However, it cannot be applied to nontraumatic injuries. Also, each injury cannot be classified into a specific injured body area.5 The NISS (using anatomical injury information) and the Revised Trauma Score (RTS; using physiologic information) also demonstrate limitations.24 There has been no diagnosis code–based injury severity measuring method for nontraumatic injury such as burns, poisoning, and so on. Although the Acute Physiology and Chronic Health Evaluation (APACHE) and Simplified Acute Physiology Score (SAPS) have been tested to evaluate severity of paraquat and organophosphate poisonings, they require many more resources for detailed data collection.25–27 The Abbreviated Burn Severity Index (ABSI) as a proxy measure for the severity of burn injury was tested to evaluate its outcome score, but this method is not a diagnosis code–based severity scale.28

The EMR-ISS can automatically classify all types of injuries into six body areas as well as nontraumatic injury, using the mapping table (Data Supplement S1). In the past, when we encountered complex traumatic injuries combined with burn injury or poisoning, we could not find the appropriate method to apply for overall injury severity measurement. In our study, the EMR-ISS has good discrimination statistics and calibration power for the prediction of mortality in such specific nontraumatic injury groups, as well as for overall injury. While the original ICISS can be calculated only for traumatic injuries, we tried to estimate SRRs for all injury codes from the NIDB and calculate the ICISS by multiplying each SRR. The ICISS has good discrimination and calibration for burns and poisonings, as well as for TBI and TCI. The correlation coefficients were also quite good for burn injury (r = −0.58) and poisoning (r = −0.69) between the ICISS and the EMR-ISS.

When we looked to validate the EMR-ISS externally with another nationwide injury database (the Korea DAD 2004), it showed comparable AUC-ROC and HL values to the ICISS and very good correlation with the ICISS. The cases in the DAD are collected from 150 hospitals that are randomly sampled, allowing calculation of representative nationwide admission rates per population.

The EMR-ISS can also be applied to nontraumatic injury. Nontraumatic injury includes poisoning, burns, drowning, environmental injuries, asphyxia, complicated injuries from medical management, and so on. Among these injuries, the ICD-10 codes T36.0 to T65.9 relate to poisoning. The numbers of poisoning-related diagnosis codes with 1, 2, 3, 4, and 5 points of EMR grade were 70, 8, 14, 36, and 98, respectively. Differing from traumatic injuries, poisoning had a peak frequency of 25 points, meaning very severe injury. This characteristic was revealed in the remarkably high crude mortality rate for poisoning (6.1%). In Korea, poisoning is a most serious problem and is the leading cause of death by suicide. It is for this reason that we chose to study this patient population, even though these are traditionally not considered “trauma” patients. The most popular poisoning agents are organophosphates and paraquat.29 The number of deaths due to poisoning in 2001 was greater than 4,000 and has been rapidly increasing. The proportion of intentional (suicide attempt) poisoning is much higher than that of accidental poisoning. Very high poisoning mortality is a unique characteristic of injury epidemiology in Korea and may somewhat limit the external validity of our data when compared to countries with lower rates of poisoning and poisoning deaths.

Nontraumatic injury patients may also have traumatic injuries at the same time, and previous injury severity measuring tools might ignore this point. Injury episodes in our data set can have a maximum of five diagnostic codes (Table 6). The proportions of patients with more than two same-type diagnosis codes were 13.2% in TBI, 5.6% in TCI, 0.7% in poisoning, and 12.6% in burn injury. However, the proportions of patients with more than two diagnosis codes regardless of injury type were 22.8% in TBI, 29.4% in TCI, 7.7% in poisoning, and 16.2% in burn injury. Differences between the two proportions (9.6% in TBI, 23.8% in TCI, 7.0% in poisoning, and 3.6% in burn injury) mean that different body areas should be considered when measuring severity. In particular, poisoning and burn injury had a high frequency of another type of injury code that would not be considered in traditional methods to calculate injury severity. These rates of multiple injuries support calculating overall injury severity across all body areas and injury types, rather than solely by the primary type or site of injury.

To develop this new modality to measure injury severity, we used several national insurance data sets, gathering data from all kinds of practice areas including outpatient clinics, EDs, admission wards, and intensive care units. The NHI and the IACI are obligatory insurance programs for all eligible people in Korea.30,31 Medical institutions and clinics must give clinical information with ICD-10–based diagnosis codes to Korea’s public insurance organizations to charge for medical services. These administrative data are transferred electronically to the national insurance corporations for payment. These data enabled us to create a nationwide injury cohort. Although the NHI and IACI data sets have no information about the exact proportion of scheduled admissions, or admissions to nonmedical acute care hospitals, all cases have information about one major diagnosis code (which means the responsible cause for the medical service visit) and other minor diagnoses (which represent combined injury or comorbidity). The detailed cause of death information in the NDC database allowed us to include only those patients who died as a result of the injuries identified from the NHI database; patients who died within 1 year of the index injury but of unrelated causes were not included. The NDC database, which contains information about cause, date, mechanism, and intent of injuries related to death, as well as individual factors, allowed us to follow-up injury episodes for 1 year to confirm deaths due to injury. Other nations with similar databases may be able to provide additional validation of our findings.

There have been a number of studies using the Korean NHI and IACI databases in the fields of cardiology, oncology, occupational medicine, and neurology.32–35 In these studies, populations in the insurance data sets were used as nationwide cohorts, and diagnosis codes were treated as meaningful outcomes. Routinely collected hospital morbidity data are a valuable source of information for health services research. There have been other studies assessing the quality of diagnostic or procedure codes using the ICD-9-CM. Diagnosis codes showed tolerable validity in injury, while procedure codes or external codes showed lower quality.36–38 In a more specific population such as pediatric emergency patient records, however, the agreement rate between diagnosis codes and discharge clinical and administrative data has been shown to be relatively low (67%).39

Limitations

There are several limitations to our system, as are found with other diagnosis-based injury severity scales. The EMR-ISS can only be derived once the diagnostic work-up is completed and all injuries have been identified. Also, this modality does not account for initial physiologic severity like the RTS does. Therefore, it could not practically be applied to trauma victims in the ED or to dead-on-arrival cases.

The NIDB includes all kinds of injury except injuries that are managed via private Motor Vehicle Accident Insurance (MVAI), because that database uses the AIS instead of the ICD-10 codes. Although approximately 7.1% of MVA victims are managed via private MVAI; the remaining 92.9% are treated via the NHI or the IAIC program, limiting the impact of this limitation on the overall strength of the database. However, since much of the injury severity literature focuses on motor vehicle crash injuries, the exclusion of these patients may limit the external validity of our work.

A previous report assessing the quality of diagnosis and procedure coding in administrative data sets evaluating the ICD-10 code compared to an audit review showed very high validity in motor vehicle injury (98% sensitivity and 98% positive predictive value), but lower validity in poisoning (76% sensitivity and 71% positive predictive value).40 This may be a limitation of our study, given the high number of poisoning cases in Korea.

Conclusions

This study developed a new modality for calculating the injury severity of all types of injuries. The excess mortality ratio–based Injury Severity Score showed good discrimination power compared to the International Classification of Diseases 9th Edition–based Injury Severity Score and was an accurate predictor of mortality. Unlike many other existing injury severity scoring systems, it can be applied to other types of injuries, such as burns and poisoning. The excess mortality ratio–based Injury Severity Score appears to be a feasible tool for passive injury surveillance of large data sets, such as insurance data sets or community injury registries containing diagnosis codes. Additional studies for external validation using prospectively collected data sets should be considered.

Acknowledgments

The authors thank Dr. David C. Cone (Yale University) who helped with the review of the literature and with the preparation and editing of the manuscript.

Ancillary