The authors have no relevant financial information or potential conflicts of interest to disclose.
Comparison of an Informally Structured Triage System, the Emergency Severity Index, and the Manchester Triage System to Distinguish Patient Priority in the Emergency Department
Version of Record online: 15 AUG 2011
© 2011 by the Society for Academic Emergency Medicine
Academic Emergency Medicine
Volume 18, Issue 8, pages 822–829, August 2011
How to Cite
Storm-Versloot, M. N., Ubbink, D. T., Kappelhof, J. and Luitse, J. S. K. (2011), Comparison of an Informally Structured Triage System, the Emergency Severity Index, and the Manchester Triage System to Distinguish Patient Priority in the Emergency Department. Academic Emergency Medicine, 18: 822–829. doi: 10.1111/j.1553-2712.2011.01122.x
Supervising Editor: Clifton W. Callaway, MD, PhD.
- Issue online: 15 AUG 2011
- Version of Record online: 15 AUG 2011
- Received November 1, 2010; revisions received February 9 and February 23, 2011; accepted February 27, 2011.
ACADEMIC EMERGENCY MEDICINE 2011; 18:822–829 © 2011 by the Society for Academic Emergency Medicine
Objectives: The objective was to compare the validity of an existing informally structured triage system with the Emergency Severity Index (ESI) and the Manchester Triage System (MTS).
Methods: A total of 900 patients were prospectively triaged by six trained triage nurses using the three systems. Triage ratings of 421 (48%) patients treated only by emergency department (ED) physicians were compared with a reference standard determined by an expert panel. The percentage of undertriage, the sensitivity, and the specificity for each urgency level were calculated. The relationship between urgency level, resource use, hospitalization, and length of stay (LOS) in the 900 triaged patients was determined.
Results: The percentage of undertriage using the ESI (86 of 421; 20%) was significantly higher than in the MTS (48 of 421; 11%). When combining urgency levels 4 and 5, the percentage of undertriage was 8% for the informally structured system (ISS), 14% for the ESI, and 11% for the MTS. In all three systems, sensitivity for all urgency levels was low, but specificity for levels 1 and 2 was high (>92%). Sensitivity and specificity were significantly different between ESI and MTS only in urgency level 4. In all 900 patients triaged, urgency levels across all systems were associated with significantly increased resource use, hospitalization rate, and LOS.
Conclusions: All three triage systems appear to be equally valid. Although the ESI showed the highest percentage of undertriage and the ISS the lowest, it seems preferable to use a verifiable, formally structured triage system.
Patients arriving at the emergency department (ED) are often confronted with long waiting times. These may be caused by arrival volumes, order of arrival, or clinical urgency. Although EDs always use some form of triage, either formal or informal,1 overcrowding of EDs makes accurate triaging essential to avoid delays in critical patient care, which may result in long waiting times and poor outcomes.2 There are a number of three-level systems that rank patients as emergent, urgent, or nonurgent. However, as five-level triage systems have been shown to be more reliable and valid than three-level systems, they are likely the systems of choice.3,4
Worldwide, there are four five-level triage systems in use: the Australasian Triage Scale, the Canadian Triage and Acuity Scale, the Manchester Triage System (MTS), and the Emergency Severity Index (ESI). The MTS and the ESI are the most commonly used in the Netherlands.5
The MTS was developed in the United Kingdom and is widely used.6 The MTS contains 52 flow charts, each representing a presenting complaint. The presenting complaint determines which flow chart should be followed. Each flow chart is based on a five-step decision process that uses discriminators at each step to assign patients to one of the five triage categories.6 A color indicates the level of urgency and its associated maximum waiting time: red = immediate care by a physician; orange = 10 minutes; yellow = 60 minutes; green = 2 hours; and blue = 4 hours. Interobserver and intraobserver agreement on the MTS has been found to be substantial to excellent.7,8 Under- and overtriage as determined by an expert panel ranges from 5% to 25%.7,8 Validity has been tested in several studies, however, only in children and in patients with chest pain.9–11 Therefore, the overall validity of MTS in daily clinical practice is not supported in the literature.5
The ESI system was developed in Boston, Massachusetts. This system uses one algorithm, with ratings ranging from level 1 (the most acutely ill patients) to level 5 (the least resource-intensive patients). The triage nurse estimates the number of resources needed to discharge the patient from the ED in those patients who do not meet ESI level 1 or 2 criteria.12 The ESI system is valid for both children and adults and has a high interobserver agreement.8,13,14 When triage urgency levels estimated by nurses were compared with the “real” urgency level, version 3 of the ESI had an undertriage rate of 9% and an overtriage rate of 11%.14
The two triage systems have been designed for different purposes. The MTS is meant to place patients in order of priority and to assure that patients do not have to wait longer than is safe, given the presenting complaint. The ESI integrates acuity and estimated resource consumption to determine treatment priority. Both ESI and MTS seem to be useful, but to date no studies have compared the validity of these systems within the same patient mix, nor to an informally structured triage practice. We defined validity as the agreement of classifications by the triage systems with a reference standard and with actual resource utilization. In this prospective observational comparative study, we determined and compared the percentage of undertriage; the validity of both structured triage systems and a local informally structured triage system (ISS); and their relation to resource use, hospital admission, and length of stay (LOS).
This was a prospective, single-center, observational comparative study combined with a retrospective chart review to determine the validity of the ISS, the ESI, and the MTS triage systems. The study was explained to patients, and all gave oral informed consent. Our local institutional review board waived the requirement for written informed consent.
Study Setting and Population
This study was conducted between November and December 2005 at an urban tertiary care academic teaching hospital with a Level I trauma center. The ED sees almost 31,000 patients annually, of whom approximately 1,000 (3%) are seen in a trauma room. Almost 71% of the patients were self-referrals, while 29% of the patients were referred by a general physician (GP) to a specialist. The overall admission rate was approximately 18%, and 15% of the patients were younger than 15 years.
In 2005, when the study began, no formally structured triage system was in use on our ED, but rather an ISS was in place. This system was based on clinical expertise, but not on explicit criteria and information. When patients were registered, the patients’ appearance and presenting complaints were judged, and the nurse or receptionist implicitly answered the question: “Could this patient wait safely before being seen?” Options were: patient could not wait at all, patient should be seen as soon as possible, or patient could wait.
For the purpose of this study, six ED nurses received a 6-hour combination of didactic and practical training in each triage system (ESI and MTS), in accordance with national standards. At random and on different days of the week between 12 noon and 10 PM, the nurses triaged all patients entering the ED consecutively. Patients already triaged before hospital arrival by ambulance staff, and who met the criteria for treatment at the trauma room according to current guidelines, were not triaged again, but classified as level 1 (ESI) or “red” (MTS) patients. Patients who left the ED without being seen by a physician, or whose records were not available, were excluded from analysis.
Patients were first registered by the department receptionist and then prospectively classified by one of the trained triage nurses, using all three systems (ISS, ESI, and MTS). The treating ED nurse and physician were blinded to the classification codes. If the triage nurse classified the patient as needing to be seen immediately, the patient was turned over to the treating ED nurse. Otherwise, the patient was sent to the waiting room and followed the usual procedure.
The information needed to identify the level of acuity is different in each of the triage systems. For this reason, the triage nurse first classified the patient according to informally structured practice, which required the least information, then by means of the ESI system, and finally with the MTS, which required the most information. For triage following the informally structured practice, based on the patients’ appearance and complaints presented, the nurse answered the question: “Under difficult circumstances (e.g., an overcrowded waiting room), what is the maximum possible time that this patient will be able to wait before being seen?” The answer options were: patient could not wait or was able wait up to a maximum of 15 minutes, 1 hour, or 4 hours.
Level 1 was defined as the most urgent category, comprising “patient could not wait” (ISS), level 1 (ESI), and red (MTS). Level 2 comprised the urgency levels “was able to wait up to a maximum of 15 minutes” (ISS), level 2 (ESI), and orange (MTS); level 3 comprised “1 hour” (ISS), level 3 (ESI), and yellow (MTS); level 4 comprised level 4 (ESI) and green (MTS); and level 5 comprised “4 hours” (ISS), level 5 (ESI), and blue (MTS).
Reference Standard. To determine the effectiveness of the triage systems in those patients treated only by an ED physician, the classification of these systems was compared with the reference standard. (Patients referred to meet a specialist were excluded from this analysis, because they had already undergone some prehospital triage and commitment to specific resource utilization.) This reference standard was determined by an expert panel consisting of seven experienced ED physicians. Each physician evaluated all cases individually and was blinded to the conclusions of the other panel members. The physicians evaluated retrospectively what the real degree of urgency would have been, based on the ED data, results of diagnostic tests, and the final diagnosis. Knowing the outcome is prerequisite to determine which patients were really in danger and were misclassified by the systems. Except for age and sex, patient data were deidentified. The main question for the panel was: “Under difficult circumstances, what is the maximum possible time that this patient would have been able to wait before being seen?” The answer options were: patient could not wait or was able to wait up to a maximum of 15 minutes, 1 hour, 2 hours, or 4 hours. In each case the decision of the majority was applied, and this decision was defined as the real degree of urgency. If there were multiple majority decisions, the panel reviewed the case until consensus was reached.
Data Collection and Definitions. The following patient data were collected from case report forms, ED reports, and electronic hospital information systems: patient demographics, mode of arrival, triage ratings by the triage nurse, urgency classification by the expert panel of ED physicians, ED resources used, hospital admission rates (including death), and LOS. LOS was defined as the time in minutes from registration to discharge or admission. Death in the ED was coded as a death and considered as a hospital admission.
The number of resources was counted in accordance with the ESI (version 3) definitions. Resources used included labs, electrocardiogram, radiology, specialist consultation, intravenous (IV) fluids or hydration, IV or imtramuscular medication, simple procedures, and complex procedures. For each patient we documented if one of these resources was used or not.
Descriptive statistics with continuous data are presented either as means with standard deviation (±SD) or as medians, based on the distribution of the data. Categorical data are presented as the percentage frequency occurrence. p values of <0.05 were considered to indicate a statistically significant difference. Differences in distribution of urgency levels were tested by means of the Friedman’s test. In all 900 triaged patients, we determined the number of resources used to diagnose the patient, the number of admissions, and LOS in the ED. To evaluate the relationship between triage classification and these aspects, the Spearman’s correlation coefficient was calculated.
Of the subset (patients treated only by emergency physicians), the data of the reference standard were entered into a text file and imported into SPSS, version 16.0 (SPSS Inc., Chicago, IL), for statistical analysis. To determine the validity of all three systems in patients treated by an emergency physician, we compared the ESI and the MTS triage classifications with the reference standard as both five-level and four-level systems. The latter was achieved by combining the urgency levels 4 and 5. The percentage of patients who were under- or overtriaged was calculated. The sensitivity, specificity, predictive values, and likelihood ratios and their 95% confidence intervals (CIs) for each of the five urgency levels were calculated. Sensitivity and specificity were defined in terms of correct or overtriage classification, and undertriage was defined as a misclassification.
A total of 900 patients were triaged. Of these, 10 patients were lost due to missing ED or triage notes, leaving 890 patients for analysis. Complete triage notes were available for 875 patients (97%) triaged using the ISS, 876 (97%) using the ESI, and 872 (97%) using the MTS.
The characteristics for all 900 patients are presented in Table 1. Patients referred by the GP to the specialist (mean ± SD age = 48 ± 27 years) were significantly older than self-referred patients (mean ± SD age = 33 ± 20 years; p < 0.001). Patients arriving by ambulance (mean ± SD age = 54 ± 28 years) were significantly older than patients arriving by private vehicle (mean ± SD age = 34 ± 21 years; p < 0.001). In self-referred patients, no significant differences in age were found between patients treated by the ED physician only and those referred from ED physician to a specialist.
|All Triaged Patients (n = 900)||Patients Only Seen by ED Physician (n = 428)|
|Male||475 (53)||243 (57)|
|Female||421 (47)||185 (43)|
|Mean (±SD)||37 (±23)||31 (±18)|
|Median (range)||36 (0–102)||30 (0–79)|
|Age distribution, yr|
|<15||163 (18)||81 (19)|
|15–30||212 (24)||134 (32)|
|30–45||207 (23)||116 (27)|
|45–60||170 (19)||69 (16)|
|60–75||82 (9)||20 (5)|
|>75||61 (7)||4 (1)|
|Mode of arrival|
|Private vehicle or otherwise||784 (87)||407 (96)|
|By ambulance||115 (13)||17 (4)|
|Mode of referral|
|Only seen by EP (ED physician)||428|
|Referred (by EP) to specialist||236|
|Referred by GP or otherwise||231 (26)|
|Number of admissions||162 (18)|
|Number of patients triaged out of hospital during measurement period||27|
Distribution of Urgency Levels
In all patients available for analysis, the number of patients in each urgency level in each of the triage systems is shown in Figure 1. When the three triage systems were compared as four-level systems, the distribution of urgency levels was shown to be significantly different (Friedman test, p < 0.006). The same was true when the ESI and the MTS were compared as five-level systems (Friedman test, p < 0.001). In the ISS, more patients were scored “very urgent” than in the ESI and the MTS. Furthermore, according to the MTS, significantly fewer patients belonged to level 5 than according to the other systems.
Number of Resources
In 890 patients available for analysis, the number of resources was strongly associated with the urgency level in all triage systems. The mean number of resources by urgency level and triage system is presented in Figure 2.
Number of Admissions
In 890 patients, the number of patient admissions was strongly associated with the urgency level in all triage systems. The distribution of admission by urgency level and triage system is presented in Figure 3.
In 890 patients, the length of ED stay was strongly associated with the urgency level in all triage systems. Median LOS per urgency level and triage system is presented in Figure 4. Patients in the highest urgency levels had the longest LOS, except for patients assigned to urgency level 1.
Of 890 patients, a total of 428 (48%) were treated by an ED physician only. Seven forms were incomplete and were therefore excluded. Statistically significant differences were found in the percentages of under- and overtriage when all systems were compared as four-level systems. The percentages of undertriage were 8.3% for the ISS, 13.5% for ESI, and 11.2% for the MTS. The highest agreement (64.8%) with the reference standard was found for the ISS, while the highest overtriage (29%) was found for the MTS (Figure 5). When comparing ESI and MTS as five-level systems with the reference standard, agreement decreased and overtriage increased, while significant differences remained.
Details of the sensitivity, specificity, predictive value, and likelihood ratio for each of the five urgency levels are shown in Table 2. Overall, sensitivity and positive predictive values were low for all urgency levels in each system, whereas specificity and negative predictive values were over 95% in urgency levels 1 and 2. The five-level ESI and MTS systems showed significant differences in sensitivity and specificity only in urgency level 4.
|Cutoff||Sensitivity (95% CI)||Specificity (95% CI)||PPV (95% CI)||NPV (95% CI)||LR+ (95% CI)||LR– (95% CI)|
|ISS||40 (5–85)||98 (97–99)||20 (3–56)||99 (98–100)||20.8 (1.7–142)||0.61 (0.15–0.98)|
|ESI||NA||100 (99–100)||NA||99 (97–100)||NA||1 (0.5–1)|
|MTS||17 (4–64)||100 (99–100)||100||99 (97–100)||NA||0.83 (0.4–0.97)|
|ISS||47 (29–65)||92 (89–94)||32 (19–47)||95 (93–97)||5.7 (2.6–11.5)||0.58 (0.37–0.8)|
|ESI||36 (20–55)||95 (93–97)||40 (23–59)||95 (92–97)||7.8 (2.8–19.6)||0.67 (0.46–0.86)|
|MTS||34 (19–53)||95 (93–97)||37 (20–56)||95 (92–97)||7.0 (2.5–17.7)||0.69 (0.48–0.88)|
|ISS||76 (65–85)||73 (68–78)||41 (33–49)||93 (89–95)||2.8 (2.1–3.8)||0.33 (0.20–0.51)|
|ESI||50 (39–61)||74 (69–78)||32 (24–40)||86 (82–90)||1.9 (1.3–2.8)||0.68 (0.50–0.89)|
|MTS||60 (49–71)||66 (61–71)||30 (23–38)||87 (83–91)||1.8 (1.3–2.5)||0.60 (0.41–0.83)|
|ESI||63 (56–71)x||44 (38–50)*||36 (30–42)||70 (63–77)||1.1 (0.9–1.4)||0.84 (0.58–1.17)|
|MTS||98 (94–100)x||2 (1–5)*||34 (29–39)||67 (30–93)||1.0 (0.9–1.0)||0.97 (0.09–7.50)|
In this study we demonstrated that when investigated in an ED setting, the validity of these three triage systems is similar. Undertriage was seen most frequently when using the ESI system. Furthermore, in all triage systems, higher urgency levels were associated with increased resource use, higher rate of hospitalization, and increased LOS. Based on these results, not one of these systems appears superior. However, we do recommend the use of a formally structured triage system in order to obtain verifiable systematic judgments, transparency, and uniformity in triage.
It is difficult to say which level of sensitivity or specificity is acceptable to conclude that a certain triage system is safe. To reach a high sensitivity (i.e., an acceptable degree of undertriage), the specificity will be so low that the potential for saving resources would be marginal at best.
In our study we found that the ESI had a much lower sensitivity than the 75% found by Travers et al.4 They compared ratings by the triage nurse with the triage decisions taken by two reviewers. These decisions were based on the original triage notes. Also, we found the sensitivity of the MTS to be lower than has been reported in previous studies.9,15–17 These differences may be explained by the fact that most of these studies were performed in selected patient groups.9,15,16 In contrast, we studied the systems’ validity in patients treated by an ED physician only, of whom only 30 (7%) were triaged to urgency levels 1 or 2.
Using both formally structured systems, the majority of patients deemed by the expert panel to belong in urgency levels 1 or 2 were undertriaged. This difference between the reference standard and the formally structured systems may be due to the fact that the expert panel of ED physicians knew what happened to the patient. Therefore, they may have retrospectively evaluated such patients as being less (or more) urgent than they would otherwise have done using a formally structured system and before knowing the outcome.
Of the total patient group, only a few patients were categorized to level 1 in accordance with the ESI and MTS, because patients triaged in the prehospital setting by ambulance paramedics, and those who met the current criteria for treatment in the trauma room, were treated immediately. For this reason, these patients were not present in our sample.
In addition, it is possible that the ISS and the reference standard contained other priorities deemed to be more important and that caused the users to classify a patient to a higher (or lower) level than was the case with the formally structured triage systems. For example, patients with cerebrovascular accidents should be seen immediately, to start thrombolysis as soon as possible. Therefore, in the ISS these patients are classified to the highest level of urgency. Using the formally structured systems these patients would be classified as level 2 (ESI) or orange (MTS). To state the issue clearly, decisions made following the ISS are not transparent; relevant information remains implicit and cannot be retrieved from the ED form.
In general, predicting hospital admission is difficult.17 Nevertheless, the lowest levels of urgency determined by our study were associated with very low admission rates. The predictions of resource use, hospital admission, and LOS per urgency level in each of the three triage systems were consistent with research from the ESI group in Boston, as well as other results from within the Netherlands.13,18–20
The limitations of our study design were first, the lack of standardized criteria for determining the reference standard. However, even when working with explicit criteria, a high rate of disagreement among experts has been demonstrated.21 For the purpose of our study, we tried to compensate for possible disagreement by forming a panel of seven experienced ED physicians and settling for a majority consensus. As they did not see the patients, the expert panel had to rely on a verbal description. On the other hand, having the diagnosis and diagnostic test results offered the opportunity to identify potentially urgent patients who were not identified by the triage systems.22
Second, for pragmatic reasons we chose to triage at random on different days and between 12 noon and 10 PM. The distribution of urgency levels within this time frame might differ from other times of the day. However, patient age, sex, and types of condition were similar to a consecutive series of patients in previous research in our department. Additionally, the distribution of urgency levels in our center (according to the ESI and the MTS classifications) was consistent with previous Dutch reports on both systems.18,23
Third, the fact that three triage systems were applied sequentially in every patient might have biased the results. If the MTS, which was always used last, had received the least attention, it would have shown the lowest validity, but this was not the case. Hence, we do not think this has been an important source of bias.
Fourth, data collected from one center may merely reflect that particular institution’s practice. In a previous study comparing inter- and intraobserver agreement between inexperienced and experienced triage nurses, agreement was found to be the same for the ESI triage nurses, but lower than in experienced MTS triage nurses. However, overall the MTS showed a greater inter- and intraobserver agreement than the ESI.8 Still, if the use of nurses extensively experienced in ED practice but inexperienced in triage did result in underestimation of the sensitivity and specificity, this would have affected the results of both formally structured systems equally.
Finally, in our ED we saw a relatively high number of referrals by GPs and percentage of patients sent for specialty services. If this would substantially influence the distribution of the triage levels allocated to these patients, i.e., would lead to generally higher or lower triage levels, this could influence the predictive values we found for the triage systems. In turn, this may be of influence on the generalization of our results. Therefore, the predictive values were determined for the subset of patients treated by the ED physician only. Hence, the reader should check whether this distribution in his or her own ED is similar or dissimilar to ours.
Informally and formally structured triage systems appear to have equal validity, although the Emergency Severity Index tends to undertriage patients. To ensure transparency and uniformity, a verifiable, formally structured triage system for ED patients is advocated.
The authors thank all nurses and physicians who contributed to the realization of this study. Thanks also to the English language editor Daphne Lees for checking the manuscript. Furthermore, we are grateful to the members of the study group: E. R. Schinkel, R. Köhlinger, and D. C. Schutte.
- 5HAN University of applied sciences, the Dutch Emergency Nurses Association (NVSHV), Netherlands Centre of Excellence in Nursing (LEVV). [Guideline triage on the emergency department; 2008.] Available in Dutch at: http://www.levv.nl/fileadmin/sites/LEVV/PDF-PP/richtlijnen/080819_Rapport_Triage_WM.pdf. Accessed May 22, 2011.
- 6Emergency Triage: Manchester Triage Group. London: BMJ Publishing Group, 1997..
- 12Emergency Severity Index, Version 4: Implementation Handbook and DVDs. Available at: http://www.ahrq.gov/research/esi. Accessed May 22, 2011., , , , .
- 16[Triage of cardiac patients at an emergency department] [in Dutch]. Cordiaal. 2005; 1:12–6..
- 18The Emergency Severity Index (version 3): a good predictor of admission, length of stay and mortality in a European emergency department [abstract]. Eur J Emerg Med. 2006; 13:62., , , .