SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. References
  9. Appendix
  10. Supporting Information

ACADEMIC EMERGENCY MEDICINE 2012; 19:70–78 © 2012 by the Society for Academic Emergency Medicine

Abstract

Objectives:  Lack of adequate risk adjustment methodologies has hindered the progress of emergency medicine health services research. The authors hypothesized that a consensus-derived, diagnosis-based severity classification system (SCS) would be significantly associated with actual measures of emergency department (ED) resource use and could ultimately be used to examine severity-adjusted outcomes across patient populations.

Methods:  A panel of subject matter experts used consensus methods to assign severity scores (1 = lowest severity to 5 = highest severity) to 3,041 ED International Classifications of Diseases (ICD), 9th revision, diagnosis codes. SCS scores were assigned to ED visits using the visit diagnosis code with the highest severity. We tested the association between the SCS scores and measures of ED resource use in three data sets: the Pediatric Emergency Care Applied Research Network Core Data Project (PCDP), the National Hospital Ambulatory Medical Care Survey (NHAMCS), and the Connecticut state ED data set.

Results:  There was a significant association between the five-level SCS and all six measures of resource use: triage category, disposition, ED resource use, Current Procedural Terminology Evaluation and Management (CPT E&M) codes, ED length of stay, and ED charges within the three ED data sets.

Conclusions:  The SCS demonstrates validity in its strong association with actual ED resource use. The use of readily available ICD-9 diagnosis codes makes the SCS useful as a risk adjustment tool for health services research.

The Emergency Medical Services for Children (EMSC) Program mission statement maintains that the EMSC program “aims to ensure state-of-the-art emergency medical care for the ill or injured child and adolescent.”1 To achieve this aim, quality of EMSC care must be assured across settings, patient populations, and practitioners, yet children present to emergency departments (EDs) with illnesses of varying severity and acuity, ranging from rashes and colds to respiratory failure and cardiac arrest. Moreover, patients receive their emergency care in a variety of settings, in various geographic locations, and from practitioners with varying degrees of training and experience, resulting in significant variation in care delivery and quality.2–8

Because EDs provide care to patients with a full spectrum of illnesses and injuries, it is important to have a severity of illness scale that is applicable to all pediatric emergency patients. While several injury scoring systems exist,9–12 they are only applicable to the approximately 25% of pediatric ED visits that occur as a result of trauma.13 Other systems that classify severity of illness focus on adults or inpatients, and not children receiving emergency care.14 Although there are two risk-adjustment tools to assess severity of illness for pediatric ED visits, both tools require significant data abstraction from ED patient records.15,16 This is time-consuming and costly and precludes the application of severity adjustment to large administrative databases. Furthermore, because patient chart review raises personal health information concerns, it is unlikely that these severity measures will be used on a large scale except in carefully controlled research studies and thus has limited applicability for administrative and policy purposes.

A sensible way to classify illness severity for EMSC would be to base such classification on a readily available, familiar, and widely used system. The International Classification of Diseases and Related Health Problems, Ninth Revision, Clinical Modification, commonly referred to as ICD-9, is a system developed to classify diagnosis data for indexing medical records, case reviews, and ambulatory and other medical care programs and to provide basic health statistics. It is familiar internationally, widely used, and often captured in administrative databases. To address the need for a system that can easily be applied to existing ED data, we developed a diagnosis-driven, resource-focused, severity of illness classification system for EMSC using ICD-9 diagnosis codes. We hypothesized that a diagnosis-based severity classification system (SCS) developed using an expert panel and consensus methods would be significantly associated with visit-level measures of resource use (a measure of severity) found in existing administrative ED data sets. Ultimately, such a system could be used to examine severity-adjusted outcomes across patient populations so that EDs may improve the quality of care delivered.

Methods

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. References
  9. Appendix
  10. Supporting Information

Study Design

A panel of subject matter experts used consensus methods, including Delphi surveys and the nominal group technique, to assign severity scores to diagnosis codes. These consensus methods provide a means of synthesizing information where unanimity of opinion does not exist owing to insufficient (or conversely an overload of) information.17–19 The methods assess the extent of agreement (consensus measurement) and resolve disagreement (consensus development). Strengths of these methods include 1) anonymity—dominance is avoided by using private ratings in nominal group and questionnaires in Delphi; 2) multiple iterations—processes occur in “rounds,” allowing individuals to change their opinions; 3) controlled feedback—the distributions of the group’s responses are shown; and 4) statistical group response—judgment is expressed using summary measures of the full group response, giving more information than just a consensus statement.17–19

In the absence of a reference standard for severity of illness, we chose to test construct validity: the association between our severity measure and another measure expected to correlate with severity of illness. For our construct of severity we chose resource utilization, under the assumption that, while actual resources used may not perfectly reflect resources needed for an individual patient, when averaged across large numbers of patients and providers, resource use is a reasonable proxy for resource requirements.16 Therefore, resource use should reflect overall severity across large groups of patients. We performed secondary analyses of existing emergency medicine (EM) data sets to assess the association between the consensus-derived SCS and measures of actual ED resource use. The study was reviewed and determined to be exempt or approved by the institutional review boards of the participating investigators’ hospitals and by the university of the central data center of the research network in which this study was conducted.

Study Setting and Population

The study was performed within the Pediatric Emergency Care Applied Research Network (PECARN), a federally funded national network with diverse hospital types, patient populations, and providers.20,21 We convened an expert panel of 14 general and pediatric emergency physicians, moderated by an experienced consensus group facilitator (JAS). Panel members included physicians from across the United States, practicing in a variety of hospital settings. Specifically, we had representatives from academic and nonacademic hospitals; from small-, medium-, and large-volume EDs; and from urban, suburban, and rural hospitals. This panel previously developed a diagnosis grouping system (DGS) for pediatric ED visits using similar methodology.13 The DGS is a clinically sensible method for describing pediatric ED diagnoses by grouping ICD-9 codes into one of 77 mutually exclusive diagnosis subgroups.

Study Protocol

Principal Data Source.  The data source used to identify ICD-9 codes included in the SCS was the PECARN Core Data Project (PCDP).20 These data were obtained from extant electronic data sets from the 20 EDs within the PECARN network from which ED discharge diagnosis codes could be reliably extracted from 2002 data.21 We included all patients treated at the participating EDs in the year 2002, aged birth to the 19th birthday. Data were collected, cleaned, and stored as previously described by the data coordinating center.20 Because up to 15 ICD-9 codes could be recorded per visit, over 1 million diagnoses were included in the 2002 PCDP. We assigned severity scores to 3,041 diagnosis codes included in the DGS.13 These 3,041 diagnosis codes represent all codes that occurred at least 10 times in the PCDP data set and accounted for 98.9% of all diagnoses given during ED visits.

Definition of Severity.  For this study, expert panel members were instructed to define severity of illness by the intensity of resources needed to diagnose and treat a patient in the ED with a given diagnosis. Resources could include the number and types of personnel at the patient’s bedside, cognitive effort needed, and the number and intensity of diagnostic and therapeutic interventions required. Given the potential variation in the severity of patients with an individual diagnosis, expert panel members were asked to recall all of the children they had seen with that diagnosis over the past year and to use the resources most often required for these children to assign a severity score.

Creating the SCS.  After an initial meeting to discuss the goals and the timeline for the project, electronic Delphi surveys were sent to panel members independently. Each panel member was asked to use his or her expert opinion to assign a severity rating to a random sample of ICD-9 codes from the DGS. Each diagnosis code was assigned a score between 1, minimal resources needed, and 5, maximum resources needed. To assist panel members, examples were provided to anchor the severity scores. The prototype for severity 1 was diaper dermatitis, where only a history and physical examination are needed, and the prototype for severity 5 was anaphylactic shock, where many personnel and diagnostic and therapeutic interventions are required.

Ten of the 14 panel members assigned severity scores to a random sample of approximately 600 diagnosis codes each (20%). They rated only diagnosis codes and were blinded to accompanying resource use. To assure consideration of all diagnoses in relationship to one another, investigators on the expert panel assigned severity ratings to either half (JMC, MHG) or all (EAA, ERA) of the diagnosis codes. Thus, a total of six experts rated each diagnosis code for severity. Consensus was achieved and a final severity score was assigned to a diagnosis when at least four of the six raters assigned the same severity (simple majority), and the range between scores was not more than one (for example, four raters assigned a severity of 4, and two raters assigned a severity of 3; final severity score = 4). Diagnosis codes without severity consensus after the first evaluation were sent for a second Delphi survey round to the same reviewers. In this round, reviewers were asked to consider altering their original response after reviewing the distribution of scores for that code assigned by all six reviewers from the first round. Again, consensus was defined as a simple majority with a range not more than one.

A 1-day facilitated consensus meeting was held to finalize severity scores that could not be determined electronically using our methodology. After the data were reviewed, and to prepare for the consensus meeting, three types of nonconsensus issues were defined. In the first type, there was no majority rating and the difference between scores was greater than one severity category. These diagnoses represented those with the least consensus. In the second type, the range of responses was only one severity category, but there was no clear majority. For example, three raters may have assigned a severity of 2, and three raters assigned a severity of 3. In the third type, diagnoses received a majority rating, but the difference between scores was greater than one severity category; i.e., four raters assigned a severity of 4, and one each assigned a severity of 3 and 2.

Panel members were also given a spreadsheet for each DGS subgroup with all subgroup diagnosis codes placed within their severity category. Codes without a consensus severity were listed in a separate column on the same sheet. This allowed the group to assess consistency of severity ratings across diagnosis codes, since the same diagnosis with different ICD-9 codes should have the same severity. For example, urticaria may be coded as “idiopathic,”“unspecified,” or “allergic.” DGS subgroup spreadsheets also allowed panel members to assess severity trends within a subgroup, ensuring that there is a logical increase in severity and that different diagnoses within a severity category are similar with regard to resource use. For example, within the allergic, immunologic, and rheumatologic subgroup, urticaria requires fewer resources than Henoch-Schonlein purpura, which in turn requires fewer resources than anaphylactic shock. In addition, diagnoses in the same severity category, such as systemic lupus erythematosus and juvenile rheumatoid arthritis, should require a similar level of resources in the ED.

The final consensus meeting used the nominal group technique, and proposals by investigators or panel members were adopted if agreed upon by two-thirds of panel members. Recognizing that all standards are arbitrary,22,23 and balancing the need to move the project forward with the knowledge that these codes had already been through two rounds of review, the investigative team chose two-thirds rather than a more stringent cutoff.

Validating the SCS.  We assessed the association between our expert panel’s assignment of diagnosis severity and our validity construct and the actual measures of ED resource use, for diagnoses found in three data sets in 2002: the PCDP, the National Hospital Ambulatory Medical Care Survey (NHAMCS), and the Connecticut state ED data set. The PCDP included two versions: PCDP I (637,271 visits), electronic administrative data from calendar year 2002 and 3 months of 2003 (February through April); and PCDP II (9,444 visits) data obtained by manual chart abstraction from up to 60 randomly selected pediatric patients per site per day seen on 10 randomly selected dates within February to April 2003 at participating PECARN sites. The NHAMCS is designed to collect data on the utilization and provision of ambulatory care services in hospital emergency and outpatient departments. ED findings are based on a national probability sample of visits to the ED of noninstitutional general and short-stay hospitals and in 2002 represented 28,358,128 ED visits.24 The Connecticut Hospital Association ChimeData Program (= 315,308 visits in 2002) maintains a proprietary health care information system that incorporates statewide clinical, financial, and patient demographic data and is available on a fee-for-service basis (Chime, Inc., http://www.cthosp.org).

We created a visit-level definition of severity called the SCS score, which is based on the highest severity of all diagnoses associated with the visit. Given that each visit could have multiple diagnoses, the SCS score was assigned based on the diagnosis that carried the highest rated severity for that visit. SCS scores were assigned to each ED visit in all study data sets. We then tested the association of the SCS scores with actual measures of ED resource use in each data set.

Outcome Measures of Resource Use.  We tested the association between the SCS score and six distinct measures of resource use: 1) triage category, 2) ED disposition, 3) actual ED resource use, 4) Current Procedural Terminology Evaluation and Management (CPT E&M) code levels, 5) ED length of stay, and 6) ED charges. Triage category was coded as emergent, urgent, or nonurgent in the PCDP II and as visit urgency in NHAMCS (time to be seen <15 minutes, 15 to 60 minutes, >1 to 2 hours, and >2 to 24 hours). Disposition was categorized as discharged to home; admitted, transferred, or observed; admitted to an intensive care unit or for surgery; and died, in both PCDP II and NHAMCS. PCDP I included three categories: discharged, admitted/transferred/observed, or died. The Connecticut database included only two categories: discharged or admitted, transferred, or observed. Categories of resource use were routine nursing and medical care (patient discharged to home from the ED with no diagnostic tests or therapeutic procedures performed), discharged to home after diagnostics performed and/or therapeutics received in the ED, and admitted to the hospital or died in the ED.16 Six CPT E&M codes used to report services in the ED were examined: 99281 (presenting problems are self-limited or minor), 99282 (low to moderate severity), 99283 (moderate severity), 99284 (high severity, require urgent evaluation by a physician but do not pose an immediate significant threat to life), 99285 (high severity and pose an immediate significant threat to life), and 99291/99292 (critical care services provided to the critically ill or injured patient) and were reported in the PCDP I. CPT codes are reported only for hospitals with reliable collection20 and accounted for approximately 50% of ED visits. Length of stay was defined as arrival time subtracted from ED discharge time and was reported in minutes in both PCDP databases as well as the NHAMCS. Charges were included in the Connecticut database only and excluded any inpatient charges.

Data Analysis

For validation of the system, the unit of analysis was the ED visit. Our analysis plan used nonparametric statistics to assess the significance of associations between the ordered five-category SCS score and disposition/resource use variables that were binary, ordered categorical, or continuous. Specifically, the Mantel-Haenszel chi-square statistic was used to assess significance of association of SCS score with binary outcomes, the Jonckeheere-Terpstra statistic was used to assess the significance of association of SCS score with ordered categorical outcomes, and a tau test (of the hypothesis that the Kendall’s tau-b statistic is significantly different from zero) was used to assess the significance of association of SCS score with continuous outcomes. For the continuous outcomes, ED length of stay and charges, we used medians and rank-based measures to eliminate the effect of extreme outliers. ED length of stays longer than 1,440 minutes were excluded, and ED charges were truncated at the 99th percentile, as these values were felt to represent data errors in a majority of circumstances. All analyses were performed in SAS statistical programs (SAS Institute, Cary, NC); p < 0.05 was considered statistically significant. Reported p-values were not formally adjusted for multiple comparisons; magnitudes of significance in all tests performed were sufficiently high that such adjustment would not affect the interpretation of our findings.

Results

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. References
  9. Appendix
  10. Supporting Information

Severity Scores at the Diagnosis Level

A total of 3,041 ICD-9 codes were rated by the expert panel. Consensus was achieved on 1,535 of 3,041 (50.5%) of ICD codes in the first round of Delphi surveys. Of the remaining codes, consensus was achieved on 1,152 of 1,509 (76.3%) in the second Delphi survey round. Overall, after two electronic Delphi survey rounds, 88% of codes had an assigned severity score. The remaining 12% (354 diagnosis codes) of severity ratings were assigned at the consensus meeting. Of these diagnosis codes, 60 (17%) did not have a majority severity score and the range of scores exceeded one; 214 (60%) did not have a majority, but the range was only one; and 80 (23%) had a majority severity score, but the range exceeded one. After the final consensus meeting, all 3,041 diagnosis codes had received a severity designation: 7% of codes were assigned a severity score of 1, 22% were assigned a code of 2, 43% were assigned a score of 3, 25% were assigned a score of 4, and 3% were assigned a code of 5. The most prevalent ICD-9 codes within each severity rating are shown in Table 1.

Table 1.    Most Common Diagnosis Codes (ICD-9) Within Each Severity Category
ICD CodeDescriptionPercentage of All Visits in the 2002 PCDP
  1. PCDP = PECARN Core Data Project.

Severity Rating 1
 6929Contact dermatitis and other eczema, due to unspecified cause0.57
 V583Attention to surgical dressings and sutures0.41
 6910Diaper or napkin rash0.31
Severity Rating 2
 4659Acute upper respiratory infections of unspecified site4.44
 3829Unspecified otitis media3.91
 07999Unspecified viral infection in conditions classified elsewhere and of unspecified site3.73
Severity Rating 3
 7806Fever4.98
 49390Asthma, unspecified, unspecified2.30
 78703Vomiting alone1.83
Severity Rating 4
 78039Other convulsions0.84
 7850Unspecified tachycardia0.31
 49391Unspecified asthma, with status asthmaticus0.26
Severity Rating 5
 7990Asphyxia0.08
 0389Unspecified septicemia0.06
 78603Apnea0.06

Severity Scores at the Visit Level

Figure 1 shows the distribution of visit-level severity derived from the SCS scores in each of the study data sets. More than 80% of visits in each data set were assigned a score of 2 or 3; severity scores of 1 or 5 comprised less than 5% of visits.

image

Figure 1.  Severity Classification System Score.

Download figure to PowerPoint

SCS Validity

Tables 2 through 4 show the association between the SCS scores and actual measures of ED resource use available within each data set. Disposition and ED resource use are reported for all three data sets, CPT E&M codes are only available for PCDP data, and charges are available only in the Connecticut state ED data set. Within each data set, higher SCS scores are associated with increased probability of being triaged as higher urgency. Higher SCS scores are associated with increased probability of hospital admission. Disposition recorded specifically as death in the ED occurred only in visits with a SCS score of 5 in NHAMCS data and PCDP II; 84% of deaths occurred in the SCS category 5 in the PCDP I. Each table shows the relationship between the SCS score and ED interventions. Visits with higher SCS scores are more likely to include diagnostic or therapeutic interventions. In addition, higher SCS scores are associated with higher CPT E&M codes, indicating a higher complexity of medical decision-making. Finally, results demonstrate that higher SCS scores are associated with increased ED length of stay and increased ED charges. Median length of stay in the ED increased from approximately 60 minutes to over 180 minutes as visit severity increased from 1 to 5. Median charges as recorded in the Connecticut database increased steadily from $158 for SCS score 1 visit to over $1,600 for SCS score 5 visits. Two-tailed p-values are <0.0001 for all associations. The electronic version of this article contains stacked histograms that depict the relationship between the SCS scores and ED resource use (Data Supplement S1, available as supporting information in the online version of this paper).

Table 2.    Association Between SCS Score and Measures of Resource Use: PCDP Data Sets
 SCS ScoreTotal (N) Rowp-value
12345
  1. CPT E&M = Current Procedural Terminology Evaluation and Management; PCDP = PECARN Core Data Project; SCS = severity classification system.

  2. *p-value determined by Jonckeheere-Terpstra.

  3. †p value determined by Kendall’s tau b.

PCDP I (administrative data)
 Number of visits (%)17,213 (2.7)207,027 (32.49)333,480 (52.33)73,556 (11.54)5,995 (0.94)637,271 (100) 
 Disposition
  Discharged15,461 (99.28)187,092 (98.62)278645 (89.72)38,503 (56.64)922 (16.55)520,623<0.0001*
  Admitted,  transferred,  observed112 (0.72)2,619 (1.38)31,929 (10.28)29,459 (43.33)4,456 (80.00)68,575
  Died0 (0)1 (0)14 (0)22 (0.03)192 (3.45)229
  Column total  (column %)15,573 (100)189,712 (100)310,588 (100)67,984 (100)5,570 (100)589,427
 ED resource use
  Routine care14,712 (94.47)147,935 (77.98)188,076 (60.56)27,146 (39.94)641 (11.92)378,510<0.0001*
  Diagnostics  and/or  therapeutics749 (4.81)39,157 (20.64)90,569 (29.16)11,357 (16.71)281 (5.22)142,113
  Hospitalized  or died112 (0.72)2,619 (1.38)31,929 (10.28)29,459 (43.35)4,456 (82.86)68,575
  Column total  (column %)15,573 (100)189,711 (100)310,574 (100)67,962 (100)5,378 (100)589,198
 CPT E&M code
  992811,333 (16.4)4,646 (4.53)5,601 (3.71)803 (3.27)15 (1.40)12,398<0.0001*
  992823,189 (39.24)37,980 (37.07)40,796 (27.0)4,374 (17.81)101 (9.45)86,440
  992833,106 (38.22)53,281 (52.0)78,382 (51.88)9,795 (39.88)334 (31.24)144,898
  99284491 (6.04)6,248 (6.1)24,125 (15.97)8,167 (33.25)390 (36.48)39,421
  992857 (0.09)263 (0.26)2,049 (1.36)1,360 (5.54)189 (17.68)3,868
  99291/992921 (0.01)41 (0.04)121 (0.08)64 (0.26)40 (3.74)267
  Column total  (column %)8,127 (100)102,459 (100)151,074 (100)24,563 (100)1,069 (100)287,292
 ED length of stay
  Median in  minutes,  range (N)79, 1–1,440 (13,875)113, 1–1,440 (171,805)151, 1–1,440 (270,999)211, 1–1,440 (53,700)230, 1–1,440 (3,936)514,315<0.0001†
PCDP II (chart abstraction)
 Severity distribution 215 (2.28)3,205 (33.94)5,075 (53.74)878 (9.30)71 (0.75)9,444 (100) 
 Disposition
  Discharged206 (100)3,077 (99.1)4,603 (93.16)608 (71.19)20 (28.17)8,514<0.0001*
  Admitted,  transferred,  observed0 (0)22 (0.71)318 (6.44)205 (24.0)31 (43.66)576
  Admit—  intensive care  or surgery0 (0)6 (0.19)20 (0.4)41 (4.8)18 (25.35)85
  Died0 (0)0 (0)0 (0)0 (0)2 (2.82)2
  Column total  (column %)206 (100)3,105 (100)4,941 (100)854 (100)71 (100)9,177
 ED resource use
  Routine care91 (44.17)1,052 (33.88)698 (14.13)66 (7.73)3 (4.23)1,910<0.0001*
  Diagnostics  and/or  therapeutics115 (55.83)2,025 (65.22)3,905 (79.03)542 (63.47)17 (23.94)6,604
  Hospitalized  or died0 (0)28 (0.9)338 (6.84)246 (28.81)51 (71.83)663
  Column total  (column %)206 (100)3,105 (100)4,941 (100)854 (100)71 (100)9,177
 Triage category
  Nonurgent151 (81.62)1,914 (66.48)2,011 (45.05)219 (28.74)9 (15.52)4,304<0.0001*
  Urgent32 (17.30)903 (31.37)2,142 (47.98)413 (54.20)32 (55.17)3,522
  Emergent2 (1.08)62 (2.15)311 (6.97)130 (17.06)17 (29.31)522
  Column total  (column %)185 (100)2,879 (100)4,464 (100)762 (100)58 (100)8,348
 ED length of stay
  Median in  minutes,  range (N)77, 9–330 (142) 123, 7–893 (2,296)159, 1–1,361 (4,125)213, 9–1,300 (708)221, 33–453 (56)7,327<0.0001†
Table 3.    Association Between SCS Score and Measures of Resource Use: NHAMCS Data
 SCS ScoreTotal (N) rowp-value
12345
  1. NHAMCS = National Hospital Ambulatory Medical Care Survey; SCS = severity classification system.

  2. *p-value determined by Jonckeheere-Terpstra.

  3. †p-value determined by Kendall’s tau b.

Number of visits (%)676,618 (2.39)12,734,135 (44.9)13,153,269 (46.38)1,607,836 (5.67)186,270 (0.66)28,358,128 (100) 
Disposition
 Discharged652,367 (99.78)12,400,000 (99.31)11,960,000 (93.45)1,160,581 (73.36)38,217 (20.52)26,211,165<0.0001*
 Admitted, transferred, observed1,422 (0.22)86,355 (0.69)796,686 (6.23)410,617 (25.95)120,207 (64.53)1,415,287
 Admit— intensive care or surgery0 (0)0 (0)41,070 (0.32)10,844 (0.69)23,867 (12.81)75,781
 Died0 (0)0 (0)0 (0)0 (0)3,979 (2.14)3,979
 Column total (column %)653,789 (100)12,486,355 (100)12,797,756 (100)1,582,042 (100)186,270 (100)27,706,212
ED resource use
 Routine48,301 (7.41)420,555 (3.38)285,179 (2.23)26,141 (1.66)0 (0)780,176<0.0001*
 Diagnostics and/or therapeutics602,354 (92.37)11,930,000 (95.92)1,1650,000 (91.21)1,123,624 (71.51)38,217 (20.52)25,344,195
 Hospitalized or died1,422 (0.22)86,355 (0.69)837,756 (6.56)421,461 (26.82)148,053 (79.48)1,495,047
 Column total (column %)652,077 (100)12,436,910 (100)12,772,935 (100)1,571,226 (100)186,270 (100)27,619,418
Triage category
 >2–24 hour153,730 (27.13)2,349,218 (21.78)1,525,757 (13.37)102,037 (7.44)0 (0)4,130,742<0.0001*
 >1–2 hour164,822 (29.09)2,517,791 (23.34)2,723,534 (23.86)218,428 (15.93)24,695 (15.03)5,649,270
 15–60  minutes186,988 (33.0)4,197,244 (38.91)4,663,276 (40.86)628,686 (45.84)51,967 (31.62)9,728,161
 <15 minutes61,122 (10.79)1,722,943 (15.97)2,500,120 (21.91)422,321 (30.79)87,695 (53.36)4,794,201
 Column total (column %)566,662 (100)10,787,196 (100)11,412,687 (100)1,371,472 (100)164,357 (100)24,302,374
ED length of stay
 Median in minutes, range (N)66, 5–1,424 (614,938)85, 1–1,434 (120,005,730)119, 6–1,429 (12,212,866)165, 5–1,430 (1,421,936)178, 6–1,317 (159,224)26,409,537<0.0001†
Table 4.    Association Between SCS Score and Measures of Resource Use: Connecticut ED Data Set
 SCS ScoreTotal (N) rowp-value
12345
  1. SCS = severity classification system.

  2. *p-value determined by Mantel-Haenszel chi-square.

  3. †p-value determined by Jonckeheere-Terpstra.

  4. ‡p-value determined by Kendall’s tau b.

Number of visits (%)14,194 (4.5)123,369 (39.13)155,970 (49.47)20,614 (6.54)1,161 (0.36)315,308 (100) 
Disposition
 Discharged14,174 (99.86)122,929 (99.64)150,259 (96.34)14,806 (71.82)541 (46.6)302,709<0.0001*
 Admitted, transferred, observed20 (0.14)440 (0.36)5,711 (3.66)5,808 (28.18)620 (53.4)12,599
 Column total (column %)14,194 (100)123,369 (100)155,970 (100)20,614 (100)1,161 (100)315,308
ED resource use
 Routine13,362 (94.14)112,198 (90.95)115,084 (73.79)12,628 (61.26)410 (35.31)253,582<0.0001†
 Diagnostics and/or therapeutics812 (5.72)10,731 (8.7)35,175 (22.55)2,178 (10.57)131 (11.28)49,027
 Hospitalized or died20 (0.14)440 (0.36)5,711 (3.66)5,808 (28.18)620 (53.4)12,599
 Column total (column %)14,194 (100)123,369 (100)155,970 (100)20,614 (100)1,161 (100)315,308
ED charges
 Median in dollars, range (N)158.5, 9–8,429.5 (10,511)263.5, 10–9,179.1 (122,593)398.7, 10–9,187.1 (151,158)785.3, 44–9,194.3 (18,568)1,603.7, 450–9,175 (808)293,618<0.0001‡

Discussion

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. References
  9. Appendix
  10. Supporting Information

The consensus-derived SCS was strongly associated with actual measures of ED resource use in data sets routinely used by EM health services researchers, clinicians, policy-makers, and administrators. Our results suggest that this SCS could be used as the foundation for a risk-adjustment instrument that could affect the evaluation of quality and efficiency of care for children presenting to the ED. The SCS is particularly attractive because it uses ICD-9 codes, which are widely available in billing data, rather than chart review. It is important to note that this system was developed for use with existing data and not for prospective or bedside prognostication.

Categorizing ED diagnoses by severity is a critical step in the advancement of EMSC. It is widely recognized in health services research that comparisons of outcomes, including resource utilization, must take into account important differences in patient populations, especially severity of illness.25 It is imperative to level the playing field prior to comparing quality across settings, populations, and practitioners. Lack of an adequate risk adjustment instrument has hindered the progress of EM health services research in these endeavors to date. We envision that in practice the predicted probability of admission (or other resource consumption measure) based on the severity classification would be used as the covariate in future models.

The SCS reported here is the first general severity measure in EMSC that does not require chart review. Its use of widely available ICD-9 codes may promote widespread use as a broadly applicable risk adjustment tool. Using administrative data to assign severity of illness scores to ED visits for children can assist in measuring quality by providing risk-adjusted comparisons for outcomes of large populations at low cost. Ultimately, researchers and administrators may be able to use this system as the basis for a tool that can be used to compare quality of care provided to emergency patients. Quality of care in this context may include severity-adjusted admission rates, ED length of stay, and other resource-based outcomes. Process measures for specific diseases would not be relevant for application of the SCS. Comparing the quality of care delivered in different systems (e.g., academic vs. community EDs), by different practitioners (e.g., EM board-certified vs. others), or in different patient groups (e.g., disparities by race) is important to inform policy-makers and regulators to maximize quality of health care delivery at the population level. Of note, the SCS correlated with numerous measures of resource use in widely varying settings, with varying distributions of visit severity, and different levels of resource utilization. For example, the proportion of visits resulting in diagnostic and therapeutic interventions was much higher in the NHAMCS data set than in the PCDP, likely reflecting differences in practice patterns among subspecialty physicians in children’s hospitals, who are overrepresented in the PCDP data.2 This supports the general applicability of the SCS.

Limitations

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. References
  9. Appendix
  10. Supporting Information

While we have demonstrated a relationship between SCS and resource use, our study did not directly show a relationship between SCS and severity of illness as measured by degree of physiologic instability or risk of mortality. However, when considered over large populations, resource use reflects severity of illness and may be a reasonable proxy in the pediatric emergency setting where risk of mortality or serious organ failure is rare.15,16 Ongoing exploration of the association between resource use and severity is warranted. In addition, our resource-based definition of severity assumed that, when considering large groups of patients, resource use is equivalent to resource need (e.g., an ordered test was actually needed). While this may be true, on average, over large groups of patients, it may not be true for individual patients.

Furthermore, the reliability of ICD-9 coding, particularly in ambulatory settings, is not perfect.26 It is possible that keystroke errors and other coding mistakes account for some statistical noise. This may explain why some of the visits with a SCS score of 5 were found in low-resource-use groups, such as those triaged as nonurgent or discharged from the hospital. However, there are certainly some potentially life-threatening pediatric illnesses (such as anaphylaxis) that if treated promptly and appropriately may not lead to inpatient admission. In addition, there are some diseases with precipitous progression of illness (e.g., meningococcemia) that may be triaged as nonacute originally, but progress to severe disease within a short period of time. Although the SCS accounted for almost all diagnoses in our data sets, we did not include every existing ICD-9 code. Given the large sample size, it is highly likely that weak as well as strong associations would be reported as statistically significant. We consider the magnitudes of associations found between SCS and resource use, which in many cases were quite strong, the key findings of this report rather than the associated p-values. While the p-values reported above were not formally adjusted for multiple comparisons, the modest number of associations tested in our study, together with the very large magnitudes of significance observed, make the possibility of false-positive associations in this study negligibly small. Furthermore, the system will need to be updated with migration to the ICD-10 system.

Conclusions

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. References
  9. Appendix
  10. Supporting Information

The severity classification system, developed by expert consensus, demonstrates validity in its strong association with actual ED resource use. The use of readily available ICD-9 diagnosis codes, rather than dependence on chart review and clinical data abstraction, makes the severity classification system potentially useful as a risk adjustment tool for health services and comparative effectiveness research.

References

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. References
  9. Appendix
  10. Supporting Information

Appendix

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. References
  9. Appendix
  10. Supporting Information

Appendix A

PECARN

Participating centers and site investigators (in alphabetical order): Children’s Hospital of Philadelphia (Alpern); Children’s National Medical Center (Chamberlain); Cincinnati Children’s Hospital Medical Center (Alessandrini); Medical College of Wisconsin/Children’s Hospital of Wisconsin (Gorelick).

PECARN Steering Committee: N. Kuppermann, Chair; E. Alpern, D. Borgialli, J. Callahan, J. Chamberlain, L. Cimpello, P. Dayan, J. M. Dean, M. Gorelick, D. Jaffe, R. Lichenstein, K. Lillis, R. Maio, F. Moler, D. Monroe, L. Nigrovic, E. Powell, R. Ruddy, R. Stanley, M. Tunik, A. Walker.

MCHB/EMSC liaisons: D. Kavanaugh, H. Park.

Central Data Management and Coordinating Center (CDMCC): M. Dean, R. Holubkov, S. Knight, A. Donaldson, S. Zuspan.

Feasibility and Budget Subcommittee (FABS): T. Singh, Chair; S. Goldfarb, E. Kim, S. Krug, D. Monroe, D. Nelson, H. Rincon, S. Zuspan.

Grants and Publications Subcommittee (GAPS): M. Gorelick, Chair; E. Alpern, D. Borgialli, L. Cimpello, A. Donaldson, G. Foltin, S. Knight, F. Moler, L. Nigrovic, S. Teach.

Protocol Review and Development Subcommittee (PRADS): D. Jaffe, Chair; K. Brown, J. Chamberlain, P. Dayan, M. Dean, R. Holubkov, P. Mahajan, R. Maio, K. Shaw, M. Tunik.

Quality Assurance Subcommittee (QAS): R. Stanley, Chair; E. Alessandrini, R. Enriquez, R. Gerard, R. Lichenstein, K. Lillis, M. Pusic, R. Ruddy, A. Walker.

Safety and Regulatory Affairs Subcommittee (SRAS): W. Schalick, J. Callahan, Co-Chairs; S. Atabaki, J. Burr, K. Call, J. Hoyle, E. Powell, R. Ruddy.

Expert panel members (in alphabetical order): Jonathan Bennett, Dominic Borgialli, David Brousseau, Kathleen Brown, James Callahan, James Holmes, Nathan Kuppermann, Jane Lavelle, Kathleen Lillis, David Monroe, Michele Nypaver, Martin Pusic, Donald Snowdon, Rachel Stanley, Michael Tunik.

Supporting Information

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Limitations
  7. Conclusions
  8. References
  9. Appendix
  10. Supporting Information

Data Supplement S1. Relationship between the SCS scores and ED resource use.

Please note: Wiley Periodicals Inc. are not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.

FilenameFormatSizeDescription
ACEM_1250_sm_DataSupplementS1.pdf80KSupporting info item

Please note: Wiley Blackwell is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.