On the computational approaches for supporting triage systems

Triage procedure is used in Emergency Departments (ED) to manage the patient's treatment and prioritise care access. This is a largely resource‐consuming phase and relevant to reduce risk and optimise resource management. Moreover, the presence of patients in the ED (both in treatment rooms and in waiting rooms after triage) may increase the patients' time of stay, thus creating problems for critical patients and for healthcare process management. Moreover, it has been proved that a large fraction of ED incoming patients do not require emergency treatments and might be treated in ambulatory or by family doctors. In such cases, the triage wastes resources and time. In addition, the decision of a low priority or no ED necessity is relevant considering that underestimating treatment necessity may cause errors in patient treatments. Improving triage related decisions is a relevant task. It has been shown that computational methods such as machine learning (ML) may support triage by providing better stratification of patients as well as better results in terms of outcome. We here present a literature review discussing some recent approaches to predict the severity of patients and in particular, we present recent approaches based on ML. We use PRISMA methodology to include works in our analysis. Finally, the future directions of research and open problems are highlighted.

Triage systems are widely accepted methods for screening patients requiring access to EDs to guarantee a prioritisation for examination according to an adequate priority level. In Italy, triage systems are ruled by Ministry of Health guidelines https://www.salute.gov.it/ imgs/C_17_notizie_3849_listaFile_itemName_1_file.pdf.
Patients, in Italy, can usually enter the ED by ambulance or accessing ED waiting rooms. Usually, a patient waiting for ED admission in ED waiting rooms is considered to have access to ED, independently from the triaging.
As stated in the Italian guidelines, Triage is a process by nursing staff before entering the treatment rooms. The pipeline is composed of four phases: 1. Immediate Evaluation Phase (so-called on the door): consists of the rapid observation of the person's general appearance to identify those with assistance problems which need immediate intervention. 2. Phase of subjective and objective evaluation: (i) personal evaluation is carried out through the interview (targeted anamnesis); (ii) an objective evaluation is carried out by detecting clinical signs and vital parameters and analysing the available clinical documentation. 3. Triage Decision Phase: consists of assigning the priority code, implementing the necessary assistance measures and possibly activating diagnostic-therapeutic pathways. 4. Re-evaluation Phase means confirming or modifying the priority code assigned to waiting patients.
Following the guidelines, a severity code is assigned to each patient. Guidelines present some variations in different regions due to the regional organisation of the system. Still, the system's core is a four-level in-hospital triage based on an acuity scale measurement. Table 1 summarises the levels of the Italian Triage System.
The aim of triage systems is mainly to improve the clinical management of patients, even though there is also a positive correlation between ED triage and waiting times of patients. 4,5 Triage is the primary option for managing patient queues during ED overcrowding. We refer to usual clinical settings, while the triage of EDs during some exceptional conditions, like the recent pandemic, has been adapted to such needs. 6 Currently, many strategies adopted in EDs are the socalled streaming, which is based on grouping patients based on the severity of their conditions and referring to separated treatment areas of the ED (e.g. fast-track, see and treat minor injuries). Streaming relies on the quick screening of patients to identify their characteristics and then to group them based on the need for similar healthcare processes. Finally, similar patients are assigned to medical and nursing staff adequate to treat them. 7 Machine learning (ML) and artificial intelligence (AI) algorithms may create predictive clinical models able to handle large heterogeneous datasets, such as electronic medical records. 8 AI models provide better prediction tasks, outperforming classical clinical scoring systems. 9 Consequently, several prediction models have been developed to improve the triage process, providing stratification of patients as well as more sensitivity and specificity to the proper identification of patients at greater risk of mortality. In this study, we present a literature review of some recent state-of-the-art ML models.

EMERGENCY SEVERITY
ED triage systems are diffused worldwide, and among the others, according to literature, 3 the most diffuse and adopted systems are the Australasian Triage Scale (ATS), 10 the Emergency Severity Index (ESI, United States), 11 the Manchester Triage System (MTS), 12 and the Canadian Triage and Acuity Scale (CTAS). 13,14 In parallel, some other indexes are currently in use, such as the Korean Triage and Acuity Scale, 15 the Taiwan Triage Acuity Scale (TTAS), 16 and the South African Acuity Scale. 17 The reliability and validity of such indexes have been compared by Hinson et al., the systematic review. 9 The ESI is one of the four most used triage systems internationally and is most cited as a comparison to evaluate the accuracy of risk prediction systems based on ML. It is a system with five possible levels used for the stratification of the patient who arrives in the emergency room. The assignment of the level takes place by evaluating both the severity of the patient and the available resources. The patient is classified based on assessing the stability of the vital conditions and the potential threat to the patient's life or vital and non-vital organs. The operator assigns the appropriate level by choosing between maximum urgency (ESI level 1 or 2), minimum speed (ESI level 4 or 5) and intermediate urgency level (ESI 3). Compared to other triage systems, ESI considers the need for necessary resources, understood as the number of resources the patient needs based on the assigned level of urgency. 11 The steps of the ESI algorithm are illustrated in Figure 1. 11 They are based on the following questions that guide the correct assignment of the ESI level: � Is the patient life-threatening and requires immediate intervention? � Can the patient wait or is the situation high risk? � How many resources will the patient need? � What are the patient's vital signs?
Each of the steps is characterised by more significant insights relating to the condition of the patient's arrival in the ED, which allows the ESI algorithm to be applied correctly and accurately. For example, in phase A, it is important to recognise the patient's level of consciousness (alert, responsive or unresponsive to verbal stimuli, responsive to other external stimuli, unresponsive).
The MTS is a triage system that guarantees a safe approach for the patient to whom a particular priority is assigned, just like the systems mentioned above. The patient is given a colour ranging from red to orange to yellow to green to blue in decreasing order of urgency. The peculiarity of this system concerns placing the patient at the centre of prioritisation, as much importance is given to what he reports regarding signs and symptoms.

F I G U R E 1
Steps of the Emergency Severity Index algorithm.
The ATS is another triage tool validated and used globally to establish, always based on the patient's condition, the correct waiting time, and the treatment to be implemented. It is an algorithm for describing clinical urgency, which does not allow for describing the quality of care or the workload necessary for the patient. It derives from revising and updating the National Triage Scale. As shown in Table 2, the ATS categorises patients into five categories ranging from category 1, which is needed when the patient is in a life-threatening condition that requires immediate action, to category 5, where the patient's condition is chronic or minor, so that the wait can be up to 2 h. Each class has also been associated with a colour which indicates, similar to the category number, the urgency of the treatment.
The CTAS system, mandatory for all Canadian emergency hospitals, is a five levels triage system. Patients are first analysed and assigned to a class. Similar to the other systems, patients are re-analysed during their stay in the queue to monitor conditions. Levels of CTAS are the following: 1. CTAS I: severely ill, requires resuscitation; 2. CTAS II: requires emergency care and rapid medical intervention; 3. CTAS III: requires urgent care; 4. CTAS IV: requires less-urgent care; 5. CTAS V: requires non-urgent care. Table 3 summarises the CTA levels.

PREDICTING SEVERITY
We queried the Google Scholar and PubMed databases. We are focusing on the use of automatic methods for triaging systems. Considering the evolution in the ML  field, we retain that works presented before 2019 may be outdated and even relevant.
We followed the PRISMA methodology as reported in Figures 2 and 3.
In Google Scholar, an advanced search was driven by using the tips 'allintitle' to limit the search to the articles that contain all the words searched in the title. In addition, the search was restricted to the period that started in 2019 and ended in the first months of 2023. The specified keywords were as follows: � allintitle: ML for triage in ED � allintitle: deep learning for triage in ED � allintitle: AI for triage in ED The results were respectively 7, 3, and 1 for a total of 11 articles.
Similarly, the advanced search was useful for selecting the papers in PubMed. The specified keywords for selecting only articles that contained them in their title were the following:

� ML for triage in the ED [Title] � AI for triage in the ED [Title]
The results were 14 and 15 articles. In addition, other filters were considered: � Results by years: from 2019 to 2023. � Preprints were excluded. � Age: only adults aged 19-80 years and over.
The same filters as the previous searches were implemented for searching in Pubmed all the article types about the application of deep learning in the ED without restriction to a specific article type. From this last search that enables us to find the previous keyword anywhere in the article, the articles were 12, for a total of 41 articles.  After retrieving the results, we grouped them based on the content. Existing approaches in literature may be subdivided based on algorithms aiming at automatic prediction of severity index or an index resulting in an outcome of the model (using a different set of parameters) and algorithms aiming at assessing existing scoring to predict a particular outcome. In particular, the algorithms belonging to the first class learn a ML model to predict the associated severity index for patients. Sometimes, they stratify patients in the same category to better predict the outcome.
The primary objective of this study was to evaluate the accuracy of the ESI in predicting patient outcomes in a retrospective cohort of patients presenting to the ED. The secondary objective is to identify factors that may impact the accuracy of the ESI in this population. Such studies, usually conducted as retrospective cohort studies of patients, integrate the information present into Electronic Health Records (EHRs), such as demographic information, vital signs, chief complaint, and the final disposition, to predict a severity score or to validate the assigned severity score concerning some predefined outcomes (e.g. ICU admission, septic shock etc.). Such studies usually focus on predicting a general purpose index, while in some cases, a narrow scope prediction (e.g. risk of septic shock is made 18,19,20 ).

| Prediction of triage levels
Despite their worldwide adoption and diffusion, some common problems affect the triage systems, such as dependence on subjective medical staff assessment and the possibility of many missing variables.
Levin et al. 21 presented in 2018 a systematic discussion of the open problems and the advantage of the introduction of ML in triage systems.
ML-based triage systems overcome some of these limitations, offering advantages such as (i) predictions are more stable, 21 (ii) the possibility to separate informative from uninformative variables during the evaluation of stable predictions, (iii) the possibility of analysing patient relations (i.e. by modelling the set of patients in a network which evidences patient similarity), to identify subgroups of patients in the same triage level to predicted outcomes and length of stay in the hospital [21][22][23][24] ; (iv) identifying critical patients under triaged into low severity levels 25 ; (vi) absence of racial, gender, age bias. 26 Consequently, many independent studies have presented the introduction of computational approaches for the automatic triage of patients. Table 4 presents a synopsis of these approaches.
In Jiang et al., 27 present a retrospective study evaluating the ability of four ML models to assist in decisionmaking at the triage level. They focus on patients with suspected cardiovascular disease who need admission to the ED after ED triage. Results confirm that trained ML models perform well in discriminating patients, and in particular, they support decisions in differential analysis of high-risk patients considering cardiovascular complications as an outcome.
Liu et al. 28 consider a broad scope to the previous approach. They present an ML algorithm to identify high critical ill patients who have been triaged into level 3 and cannot wait anymore to be admitted to urgent care.
Raita et al. 29 presented a study on electronic health record data. In that work, the author used four ML models: (1) Lasso regression, (2) random forest, (3) decision tree, and (4) deep neural networks. All the models aimed to predict the outcome after triage. All the models demonstrated good performances in predicting outcomes. Moreover, the models also showed good performances in predicting critical outcomes for patients classified as low risk by the ESI system (under-triage) and in predicting over-triage (patients assigned to ESI 3 and 4 without critical outcomes). Klug et al. 31 used patient admission data, including the ESI index, to predict lethal outcomes. They trained a gradient-boosting method to predict mortality data. Results of the validation are then used to stratify patients with high ESI index (3 and 4). A similar scenario, but narrowed to only patients with ESI index 3, has been studied by Chang et al., 32 The authors analysed admission data to train five models: CatBoost, XGBoost, decision tree, random forest, and logistic regression. Models have been used to predict the length of stay of low-severity patients in a population with TTAS level 3.
Goto et al. 33 trained four ML models (lasso regression, random forest, gradient-boosted decision tree, and deep neural network) to predict the outcome of the triage: critical care (admission to an intensive care unit and/or in-hospital death) and hospitalisation (direct hospital admission or transfer). The problem of stratification of patients in the same acuity class considering the outcome has also been investigated by Fernandes et al. 39 Starting from a list of patients assigned to MTS class 3, the authors use ML and Natural Language Processing to integrate and mine clinical variables and the text of EHRs to predict the risk of mortality and cardiopulmonary arrest.
In Yao et al., 35 the authors proposed a novel triage engine based on the synergistic use of a recurrent neural network to model sequential data and of a convolutional neural network to model data extracted from text written by nursing staff. The objective was to predict clinical T A B L E 4 A synopsis of the paper presenting ML approaches for analysing patients accessing ED.

Paper Aim Methods Outcome
Jiang et al. 27 Comparing performances of ML in support triage  38 presented a Lasso regression model for stratifying patients presenting chest pain triaged in the same severity class. The correct triage may also improve the hospital's efficiency and suggest a better resource allocation, considering ICU resources. Klang et al. 40 studied this problem and proposed an ML model, trained on retrospective data, for predicting the risk of the neuroscience intensive care unit. A similar problem is faced in Fernandes et al., 41 with the integration of clinical variables for the stratification of patients in the ESI 3 class for ICU admission.
While all the previous works are retrospective studies, one of the research goals is to develop systems to fully automatise triage, such as the i-TRIAGE system Kipourgos et al. 42 It is a decision support system able to triage patients automatically. Similarly, in Joseph et al., 37 the authors presented an algorithm for a more granular triaging of patients.
In parallel, some independent groups investigated the use of ML methods for characterising some diseases that are difficult to triage with classical indexes. For instance, Song et al., 34 developed a novel deep-learning model to classify retinal images for ophthalmology consultation.
All the previously presented models present common drawbacks to the explainability and interpretability of the trained model. Both concepts are referred to as the possibility to explain (or interpret) the output of the ML models in a way that 'makes sense' to a human being at an acceptable level. Often these models are referred to as white box models compared to the classical black box models.
The problem of explainability of ML models has been explicitly faced by Leung et al. 30 The authors of this paper proposed an interpretable ML model for predicting hospitalisation. Yu et al., 43 developed an interpretable ML model to calculate a Score for Emergency Risk Prediction for each patient. The score is validated as a predictor of mortality in the ED. In Kim et al., 24 an interpretable ML model is introduced to provide automatic triage.

| Problems and challenges
Despite much work, there is still room for research in this area. Here, we present some of the open challenges and possible future research directions.
First, we should note some limitations present in the discussed papers.
Although all the presented papers achieve good prediction outcomes (e.g. in terms of accuracy, specificity and sensitivity), many were based on retrospective data samples collected from patients visiting the ED. Some of these data might be affected by bias or errors. Moreover, usually, all the papers use data collected from a single source (literature dataset or EDs of a single region), while there is the need to collect and analyse multi-source data.
As usual in clinical domains, particularly in clinical decision support systems based on Deep Learning and AI, the explainability of trained models is crucial. 44 In particular, when considering that decisions made by the triage system can result in death (in case of a wrong decision) or the saved life, the legal and ethical implications require that the model be transparent, even to enable medical staff to have counterintuitive choices. We should report that only a few works by Yu et al. 43 and Liu et al. 28 pay attention to the explainability of their models. Consequently, the development of explainable AI models is a promising area.
From a medical point of view, AI models may also support the possible stratification of patients considering age and sex, which recently have been considered as factors in predicting disease severity.
Many works focused on the stratification of patients with ESI-3 levels. Thus, we retain that more work should be done at lower severity levels.
Actual automatic models focus on supporting nurses during the triage by using data inserted by the medical staff. Developing generative AI models, such as GPT-4 Katz et al., 45 may also foster research in developing a fully automated triage system based on direct patient interaction (in the case of low-severity patients).
Many patients in Italy have access to ED through Emergency medical services (EMSs) (the so-called 118 system). Thus, it is important to provide appropriate prehospital management during the transfer to the ED to speed up the admission process and prioritise patients before the physical access to EDs. If the patient needs critical care, the EMS technician must pass through the nearest low-level ED to a high-level ED.
Finally, scalability issues and preparedness for future (undesired) pandemic issues should be considered.

| CONCLUSION
The prioritisation of patients in ED is a crucial requirement for improving the level of care and minimising lethal outcomes. To achieve good prioritisation, a set of international systems have been proposed. Many countries adopt one of these standards in their hospitals. Despite this, it has been shown that such standards may sometimes provide the wrong classification. It has been shown that computational methods such as ML may support triage by providing better stratification of patients as well as better results in terms of outcome. Here, we present recent approaches to predict the severity of patients. Finally, the future direction of research and open problems are highlighted.