SALAD‐BAAR: A numerical risk score for hospital admission or emergency department presentation in ambulatory patients with cardiovascular disease

Abstract Background While many interventions to reduce hospital admissions and emergency department (ED) visits for patients with cardiovascular disease have been developed, identifying ambulatory cardiac patients at high risk for admission can be challenging. Hypothesis A computational model based on readily accessible clinical data can identify patients at risk for admission. Methods Electronic health record (EHR) data from a tertiary referral center were used to generate decision tree and logistic regression models. International Classification of Disease (ICD) codes, labs, admissions, medications, vital signs, and socioenvironmental variables were used to model risk for ED presentation or hospital admission within 90 days following a cardiology clinic visit. Model training and testing were performed with a 70:30 data split. The final model was then prospectively validated. Results A total of 9326 patients and 46 465 clinic visits were analyzed. A decision tree model using 75 patient characteristics achieved an area under the curve (AUC) of 0.75 and a logistic regression model achieved an AUC of 0.73. A simplified 9‐feature model based on logistic regression odds ratios achieved an AUC of 0.72. A further simplified numerical score assigning 1 or 2 points to each variable achieved an AUC of 0.66, specificity of 0.75, and sensitivity of 0.58. Prospectively, this final model maintained its predictive performance (AUC 0.63–0.60). Conclusion Nine patient characteristics from routine EHR data can be used to inform a highly specific model for hospital admission or ED presentation in cardiac patients. This model can be simplified to a risk score that is easily calculated and retains predictive performance.


| INTRODUCTION
Cardiovascular disease (CVD) is the leading cause of death globally as well as in the United States where the prevalence is 1 in 3. 1 According to the American Heart Association, $1 out of every $6 in healthcare spending is spent on the management of CVD. This financial burden is growing and it will outpace capacity if costs of care are not reduced. 2 Inpatient hospital care generally marks further decompensation in disease states and accounts for most of this financial burden. Of the nearly $200 billion in direct costs of CVD care in the United States, emergency department (ED), and inpatient care account for over $100 billion annually, more than the costs of office visits, home health, and medications combined. 3 Estimates of the indirect costs of lost productivity were similarly high at $130 billion annually.
Several groups have designed interventions to reduce inpatient hospitalization with varying degrees of success. 4,5 Successful interventions are typically resource and labor intensive and require careful resource allocation to achieve cost effectiveness compared with routine practice. Therefore, identification of ambulatory patients at high risk for ED presentation or inpatient admission is of high importance.
However, tools to predict ambulatory patients' risk of future admission are lacking. In this project, we developed a computational model that makes use of commonly available electronic health record (EHR) data to predict the risk of hospital admission or ED presentation in ambulatory cardiac patients.

| Study design and population
This retrospective cohort study included consecutive adult patient visits to the outpatient cardiology clinic at an urban academic tertiary referral center from 2013 to 2016. Patients were included in the cohort if they had been seen in the cardiology clinic at least twice in any given year. To avoid inclusion of patients for whom clinical management is atypical, patients with organ transplants, left ventricular assist devices, and home inotropes were excluded. The retrospectively analyzed clinic visits were randomized into a model training set (70% of visits) and a model testing set (30% of visits).
A separate dataset of patients seen in 2017 through 2018 was obtained after the model had been finalized and was used to perform prospective validation.
Data were collected in accordance with an approved Institutional Review Board (IRB) protocol under a general waiver of consent.

| Study outcome
First, we determined whether each clinic visit was followed by a presentation to our ED or admission to our hospital for any reason within 90 days. We did not differentiate between hospital admission or ED presentation, because we felt both represented a notable escalation in level of care. Additionally, differentiating planned from unplanned admission or ED presentation is challenging in secondary analysis of EHR data. While the study population included patients with CVD who were treated in the cardiology clinic, all subsequent ED presentations or hospital admissions were included in the analysis regardless of presenting complaint or final diagnosis.

| Model features
We collected data on each patient available in the EHR at the time

| Statistical analysis
Model features were assessed for collinearity using a Spearman rank correlation. When pairs of variables were found to be collinear, the more clinically relevant was kept in the analysis and the other was removed. Pairs that had a Spearman rank correlation of above 0.7 were removed; these included changes in lowest systolic blood pressure, change in lowest diastolic blood pressure, change in lowest pulse, and chronic obstructive pulmonary disease. When continuous variables (lab values, vital signs, social vulnerability index) were missing, they were imputed. For missing social vulnerability index variables, the mean was imputed. Calculated features (i.e., lab value and vital sign difference-from-median) that could not be calculated due to missing data were imputed to zero. Gender, comorbidities, clinical events, and medications were reported as frequencies and percentages (Table 1). Continuous variables were standardized to a mean of zero and a SD of one.
The retrospective dataset was divided randomly into a derivation set and testing set in a ratio of 70:30. The derivation set was used to

| RESULTS
During the study period a total of 46 465 clinic visits for 9326 unique patients met inclusion criteria and were included in the retrospective analysis. 75 patient characteristics were evaluated for importance as described above. 9096 clinic visits were followed by an ED presentation or hospital admission within 90 days for an event rate of 20%. 4191 clinic visits were followed by an ED or hospital admission within 30 days for an event rate of 10%.
Patient characteristics at the time of their clinic visits are given in Table 1. Median age at the time of the clinic appointment was 68 years, and men and women were near-equally represented. Comorbidities and medications at the time of clinic appointment are also given in Table 1.
The XGBoost model was the most accurate with a receiver operating characteristic AUC of 0.75, compared with the logistic regression AUC of 0.73. Feature importance analysis of the XGBoost model yielded 10 features that were more important than others ( Figure S1).  Figure S2(A-D)), coefficients ( Figure S3(A-D)), and admission rates ( Figure S4(A-D)).  Table 2.

| DISCUSSION
In our study of 46 465 cardiology clinic visits with 9326 patients at a tertiary academic center, we developed the SALAD-BAAR score that predicts the risk of 90-day hospital admission or ED presentation. Our model was prospectively validated using a separate dataset of 22 963 encounters and 6367 unique patients.
Our outcome variable of hospital admission in the 90 days following a cardiology clinic appointment occurred at a rate of 20%, consistent with the highly morbid nature of CVD in our patient population.
Our algorithm was able to predict the likelihood of hospital admission or ED presentation in the 90 days following cardiology clinic visit with sensitivity of 0.58, specificity of 0.68, and AUC of 0.63. These metrics compare favorably to other large cohort numerical prediction scores, such as those used to predict cardioembolic stroke in the setting of atrial fibrillation 8,9 or the likelihood of potentially avoidable 30-day hospital readmission in medical patients. 10 Amongst our prediction variables, we found the use of diuretics, aspirin, and anticoagulant medications were significant markers of increased risk in our patient population. Given that these medications are most often prescribed to manage advanced symptoms (such as heart failure) or reduce the risk of major adverse cardiac events (such myocardial infarction or stroke), it follows that they signal increased risk for acute hospital presentation.
Electrolyte abnormalities also were found to be an important predictor of acute presentation. Hyponatremia is known to be a marker of dysregulation of the renin-angiotensin system in acute and chronic CVD and has been associated with increased mortality in several studies. 11,12,13 In addition, abnormalities of potassium and magnesium are frequent side effects of the high doses of diuretics used to manage symptoms in severe heart failure.
Likewise, patients with severe CVD are more likely to have blood loss and anemia, which can result from acute blood loss related to surgery or procedures but may also be secondary to chronic causes.
Severe aortic stenosis has been associated with acquired von Willebrand deficiency from increased shear forces that can lead to F I G U R E 2 Odds ratios for the logistic regression model and their resulting numerical scores. ED, emergency department F I G U R E 3 Risk of 90-day admission or emergency department (ED) visit versus SALAD-BAAR score chronic blood loss. 14 Finally, anemia is present in one-third of patients with heart failure often secondary to the effects of chronic inflammation, renal insufficiency, and nutritional deficiency. 15 The diagnosis of respiratory disease and the clinical symptom of shortness of breath were also identified as high-powered predictors in our model. Comorbid respiratory disease or symptoms of shortness of breath may reflect chronic secondary remodeling or damage to the pulmonary system due to severe heart disease such as in cor pulmonale. Shortness of breath is also a common clinical manifestation of acute cardiac conditions, such as acute heart failure or myocardial infarction.
Unsurprisingly, prior hospital admission or ED presentation within 90 days was the strongest predictor of future presentation. This finding reflects similar data from other studies, including the HOSPITAL score, a risk stratification tool to predict the risk of hospital readmission following discharge. 10,16 The results of this study must be assessed in the context of their Additionally, while the derivation and initial testing of our model were based on retrospective data, our model's observed performance was consistent when applied to a hold-out prospective validation dataset. While we find these results promising, an external validation is required before this algorithm should be applied more broadly. Specifically, external validation in patient populations disparate from our own would enhance the generalizability of our model.

| CONCLUSION
The SALAD-BAAR Score is a simple tool to predict future 90-day ED presentation or hospital admission in ambulatory cardiac patients. The score performed well when prospectively validated with single-center data. This score, the first of its kind for ambulatory cardiac patients, has the potential to identify patients who may benefit from interventions to reduce their risk of hospital admission.