A systematic review of neurological airway respiratory cardiovascular other‐surgical severity (NARCO‐SS) score as a pediatric perioperative scoring system

To systematically identify and synthesize the available evidence of the neurological airway respiratory cardiovascular other‐surgical severity (NARCO‐SS) score as compared to other pediatric specific perioperative scoring systems.


| INTRODUC TI ON
Neurological, airway, respiratory, cardiovascular, others-surgical severity (NARCO-SS) is a scoring system designed to address the limitations of the American Society of Anesthesiologists classification of physical status (ASA-PS) as a pediatric perioperative risk assessment tool. 1 While the ASA-PS is a widely used scoring system to assess the health status of adult patients and stratify outcomes, it was not initially designed for this purpose. 24][5] In pediatric populations, the ASA-PS has been shown to have poor interrater reliability along with a lack of precision and specificity as a risk assessment tool. 2,4,6,7][10] There is a need for a universally accepted pediatric-specific perioperative risk scoring system to ensure consistency when assessing pediatric patients. 11veral pediatric-specific scoring systems have been studied in the literature including the Pediatric risk assessment score (PRAM), Risk Assessment of Morbidity in Pediatric Surgery (RAMPS) score, NARCO-SS, and the pediatric-adapted ASA-PS classification. 2,4,12,13Both the NARCO-SS and PRAM scores have been externally validated independently. 14,15While some of the scores were specific to organ systems, such as perioperative respiratory adverse events, 16 others were used to assess outcomes in pediatric surgical groups. 17,18The NARCO-SS is a scoring system specifically designed for objective pediatric perioperative risk stratification, which was developed and validated by Malviya et al. at a single institution in the United States.Subsequent external validation studies were conducted in India and Kyrgyzstan. 14,19On preliminary literature review, we did not find a study that focused on the utility of NARCO-SS as compared to other traditional perioperative scoring systems in children.Therefore, the objective of this review was to critically evaluate and synthesize the available literature on the effectiveness of NARCO-SS in pediatric patients undergoing surgery.The findings from this review aim to contribute to recommendations that can be applied to clinical practice and policy decision-making, while also identifying any knowledge gaps for advancing the effective implementation of pediatric risk scoring systems, particularly the NARCO-SS.

| Objective
To systematically identify and synthesize the available evidence on the effectiveness of the NARCO-SS score at predicting perioperative risk of morbidity and mortality as compared to other perioperative scoring systems.

| ME THODS
This systematic review was conducted in accordance with the Preferred Reporting Items for Systematic Review and Meta-analysis (PRISMA) statement. 20The protocol was registered prospectively on PROSPERO (CRD42023409320).S1.To minimize publication and language bias, studies published in any language were included and databases were searched as listed in Table S2.Sources of unpublished studies and gray literature were searched using Google Scholar.Reference lists of studies identified for inclusion were also checked for additional potentially relevant studies that may have been missed in the electronic search.

| Eligibility criteria
This review considered randomized controlled trials, pseudorandomized controlled trials, quasi-experimental trials, observational analytical studies, prospective cohort studies, retrospective cohort studies, and case-control studies.Pseudo-randomized trials were defined as nonrandomized trials which assigned the same treatment to every other included participant. 21Quasi-experimental trials included nonrandomized pre-post intervention studies. 22All clinical studies comparing NARCO-SS with any other pediatric-specific risk assessment systems were included.Editorials, review articles, conference presentations without the full text, and textbook chapters were excluded.
The population of interest included pediatric patients (defined as age from 0 to 18 years) of any gender and completed a NARCO-SS along with a comparative perioperative scoring system prior to undergoing any surgery.NARCO-SS is a composite of NARCO, an organ system-based scoring matrix, and Surgical Severity (SS), which is a score based on the invasiveness of the surgery being performed.The final composite score is reported as NARCO grade I-IV, and SS grade A-D.For example, a healthy patient undergoing superficial foot surgery would be NARCO IA and a child with Down's syndrome with partially corrected VSD undergoing major intraabdominal surgery would be NARCO IIIC.The detailed NARCO-SS template for score generation is provided in Table 1.

| Study selection
All the references searched in the included period were collated and uploaded into Rayyan, a cloud-based systematic review software, and duplicates were removed. 23Titles and abstracts were screened independently by any two of the three reviewers (AM, AU, and LT) using Rayyan for assessment against the inclusion criteria.
Any disagreements that occurred during this process were resolved via consultation of the third reviewer and discussed until consensus agreement was reached.

| Data extraction
Data were extracted from studies included in this review by three independent reviewers (AM, AU, and LT) using a standardized and pilot tested form using Microsoft Excel.The data extracted included specific details about the study demographics, populations, study methods, interventions, outcomes, statistical methods, results with their reported significance, and conclusions.The authors of articles were contacted by email to request missing or additional data.
TA B L E 1 NARCO-SS preoperative risk assessment system for children and the scoring guide.

| Outcome analysis
The included studies were analyzed for predefined intra-and postoperative outcomes related to morbidity and mortality.Examples include desaturation, arrhythmias, airway adjuncts, laryngospasm, and cardiac arrest.Where composite outcomes were used, a list of constituent outcomes was sought.

| Assessment of methodological quality
As the systematic review is assessing and comparing prediction tools, the Prediction model Risk Of Bias ASsessment Tool (PROBAST) was deemed the most appropriate risk of bias (ROB) assessment tool for the studies included. 24PROBAST is systematically organized into the following four domains: participants, predictors, outcome, and analysis.These domains contain a total of 20 signaling questions designed to facilitate structured judgment of both ROB and applicability.Within the PROBAST, ROB and applicability are treated as distinct yet crucial considerations; ROB can arise due to shortcomings in study design, conduct, or analysis potentially leading to systematically skewed estimates of a model's predictive performance.In contrast, applicability concerns can arise when there are discrepancies between the population, predictors, or outcomes of a study and those specified in the review question. 24

| RE SULTS
A total of 1139 studies were identified from the initial electronic database search resulting in 18 potentially relevant studies requiring full text review (Consort diagram-Figure 1).Three studies (with a total sample size of 836) were retrieved that compared NARCO-SS with other perioperative scoring systems, consisting of the original NARCO-SS development study and two external validation studies. 1,14,19During our systematic search, a fourth study was identified as potentially relevant. 25Despite multiple efforts to contact the study's corresponding author via email, we received no response.
Consequently, due to the unavailability of its full text, this study was excluded from our review.Table 2 summarizes study designs and participant characteristics.All studies included the ASA-PS score as the comparator against which NARCO-SS was validated.A comprehensive list of these adverse events and escalation of care events is provided in Table S3.

| Outcome analysis
Udupa et al. 14  prolonged mechanical ventilation, and other related events.The full list of constituent events for "complications" and "treatment escalations" is detailed in Table S4.
Meta-analysis was not possible as per the original protocol as the included studies reported data using composite outcomes that were not uniformly defined.Constituent raw data was either not available or not reported.Hence, a narrative synthesis was performed.

| Statistical tests
Our systematic search included three studies that utilized the NARCO-SS pediatric perioperative scoring system for detailed comparison of the reported statistical results. 1,14,19The key results from the included studies are summarized in Table 3.We did not perform any independent statistical analysis.Notably, all three studies used predefined outcomes (AE and EOC or Complications and TE) to examine their correlation with higher scores of NARCO-SS versus ASA-PS.Overall, higher point estimates were observed with NARCO-SS as compared to ASA-PS when using Spearman's correlation, as evidenced in the studies by Malviya et al. 1 and Udupa et al. 14 In both of these studies, 1,14 Spearman's correlations were used to assess the strength of association between preoperatively assigned scores and outcomes.Specifically, in Malviya et al., 1 NARCO-SS had higher correlation coefficients with EOC, whereas in Udupa et al., 14 both AE and EOC had higher correlation coefficients, with significance noted at p < .01.Malviya et al. 1 did not provide confidence intervals for their Spearman's correlation findings.Additionally, Rybiyanov et al. 19 did not report the Spearman's correlation coefficient.
All the included studies used odds ratio (OR) to measure the association of high scores versus low scores with outcomes.However, the criteria for defining high and low scores varied across studies.
Malviya et al. 1 and Udupa et al. 14 defined high scores as NARCO ≥ III, ASA-PS ≥ 3, SS-B, and SS-C, while Rybiyanov et al. 19 defined high scores as ASA-PS 2-3, and NARCO II-III.The OR for outcomes increased with higher scores for both ASA-PS and NARCO-SS with a higher point estimate for ASA-PS in both Malviya et al. 1 and Udupa et al. 14 Rybiyanov et al., 19 however, found NARCO-SS to be more effective than ASA-PS, evidenced by a higher OR value for both complications and TE.
The ROC contrast estimation test showed the area under the curve (AUC) for NARCO-SS was significantly higher than for ASA-PS with an AUC difference for EOC being 0.068 (p = .0068)and significantly higher for AE with difference being 0.0847 (p = .0020).Rybiyanov et al. 19 found the difference in AUC-ROC for NARCO-SS to be statistically significant with p = .048as compared to ASA-PS for complications and p = .001as compared to ASA-PS for TE.Rybiyanov et al. 19 did not report the contrast estimation values.
Pearson's chi-square was used to test calibration in two included studies. 14,19Udupa et al. 14 found both NARCO-SS and ASA-PS had poor calibration for relationship between scores and the outcomes (AE and EOC) with a p < .001for both outcomes.Rybiyanov et al. 19 found NARCO-SS and ASA-PS had poor calibration for relationship between scores and outcomes with p = .002and p = .183,respectively, for complications, and p < .001and p = .072,respectively, for TE.Two studies tested the interobserver reliability and found moderate (0.6-0.79) interobserver reliability for NARCO, SS, and ASA-PS using kappa statistics. 1,14

| Risk of Bias assessment
Risk of Bias (ROB) analysis for the included studies using PROBAST is summarized in Table 4. 1,14,19 A more detailed analysis is provided in Table S5.On participants enrolment, all three studies had low ROB.
However, the assessment of outcomes revealed high ROB in the studies by Malviya et al. 1 and Rybiyanov et al. 19 All three studies had concerns in analysis which resulted in high or unclear ROB as seen in Table 4. On test of applicability, Malviya et al. 1 and Udupa et al. 14 had low concern for applicability.

| DISCUSS ION
This study aimed to systematically review the evidence for NARCO-SS as a pediatric specific perioperative scoring system.
We conducted this review approximately 10 years post-inception of NARCO-SS to ascertain the quality of evidence in the included studies, and its performance as compared to other perioperative scoring systems.The main results identify a paucity of evidence on the effectiveness of the NARCO-SS score at predicting perioperative risk of morbidity and mortality as compared to other perioperative scoring systems.These results highlight that there is a need for further research to demonstrate the efficacy of NARCO-SS score as an objective pediatric perioperative risk stratification tool.
Our systematic review included three studies 1,14,19 that met inclusion criteria for detailed comparison of the reported results.
Overall, there were higher point estimates with NARCO-SS as compared with ASA-PS using Spearman's correlation.The AUC-ROC values of NARCO-SS for outcomes ranged between 0.6 and 0.8 for the two external validation studies, 14,19 suggesting that the scoring system in its current form may be suboptimal.Interobserver reliability between NARCO-SS categories were moderate and were similar to ASA-PS suggesting that NARCO-SS was no better than ASA-PS in consistency of the ratings.Quality of the available evidence in the three included studies was assessed using the PROBAST.In terms of candidate predictors, there were six candidate predictors in NARCO-SS score, which were used for calculation of events per variable in the analysis component of ROB assessment.The outcomes that were utilized in each study were predefined events categorized under AE and EOC.In multivariate prediction models, it is recommended that each predictor has at least 10 events and an equal number of nonevents. 26The analysis revealed all three studies had adequate events per predictor.However, all studies had unclear or high ROB due to the concerns outlined in detailed analysis in Table S5.Therefore, these findings from our review may reflect the paucity and suboptimal methodological quality of the available evidence rather than a true indicator of intervention effectiveness.
Among other systematic reviews comparing pediatric perioperative scoring systems, Ji et al. included NARCO-SS and conducted a ROB assessment. 8They found a low ROB in the NARCO-SS by Malviya et al. 1 and they used Newcastle-Ottawa Scale (NOS) for ROB analysis.However, the NOS may not be an appropriate tool as it is not designed for evaluation of prediction models, but rather for observational studies of interventions and PROBAST may be better suited in this situation.Nevertheless, they noted that most studies used a binary outcome scale for pediatric perioperative risk TA B L E 2 Study designs and participant characteristics.assessment as compared to NARCO-SS which used an organ system dysfunction approach.Tangel et al. published a systematic review recently where they assessed various perioperative pediatric scoring systems using PROBAST tool for the ROB analysis and they presented the results similar to us, but they did not include NARCO-SS in their comparison. 24,27 is possible that the modification of the categories within the NARCO-SS scoring system may improve its overall predictive performance.The NARCO-SS demonstrated poor correlation with airway and neurological categories when assessed using Spearman's correlation in the studies by Malviya et al. 1 and Udupa et al. 14 In their study, Malviya et al. 1 observed that the airway category in NARCO-SS had the lowest reliability, attributing this to the ambiguous scoring instructions.They recommended modifications with explicit descriptive categories for airway obstruction and presence of hypovolemia to enhance the precision of the NARCO-SS.Furthermore, Malviya et al. 1 proposed that assigning greater importance to the respiratory and cardiovascular components of the NARCO-SS could be advantageous, given that major outcomes were associated with these categories.This uneven distribution of scores raises questions about the limitations of the scoring system.A conclusive evaluation of these limitations necessitates further studies that are robust, methodologically rigorous and appropriately-powered, and includes a broader spectrum of critically ill pediatric patients.
Pediatric perioperative risk prediction tools such as NARCO-SS have been developed with their own patient datasets. 8However, limited validation of these tools in external datasets, particularly in critically ill pediatric patients, limits our ability to generalize and comment on their sensitivity in predicting rare and adverse events.
In our review, two studies demonstrated poor calibration of both NARCO-SS and ASA-PS to predict perioperative outcomes in pediatric patients. 14,19Hence, as noted by Udupa et al., 14 neither scoring system should be relied upon exclusively for clinical decisionmaking or predicting perioperative risks.Instead, clinicians should consider a range of factors, including type of surgery, patient characteristics, and clinical judgment alongside these scoring systems. 14ere were several limitations to this review.Firstly, despite our review being conducted a decade after the inception of NARCO-SS, our review could only identify three pertinent articles, a factor that significantly limits the robustness and comprehensiveness of our analysis.Indeed, there appears to be a noticeable scarcity of literature on pediatric risk stratification tools, as found by Ji et al. in their systematic review of all pediatric-specific perioperative risk prediction tools. 8This paucity may be attributed, at least in part, to the perceived complexity of such pediatric scoring tools, which poses challenges for their practical implementation, especially in the demanding and fast-paced settings of pediatric perioperative care. 1,28One method of improving usability could be via automated calculations and integration into electronic anesthesia information management systems.
Secondly, the included studies had substantial differences that limited us to only qualitative analysis.Specifically, these studies provided inconsistent definitions for the composite outcomes AE and EOC, and raw outcome data for constituent outcomes were not provided and could not be obtained.Thirdly, we used a commercially available third-party website for translation of Rybiyanov et al. 19 However, the standard of translation thus obtained was considered adequate by the reviewers for the comparison and ROB assessment.

| CON CLUS ION
Due to the limited evidence available and the high risk of bias in the included studies, definitive conclusions could not be drawn from this review.Given the well-recognized limitations of ASA-PS, we advocate for further high-quality research into NARCO-SS, which, despite showing promise, appears to be limited in its current form.
Addressing its limitations in predictive accuracy and improving its usability through refinement of the descriptive categories of NARCO-SS could increase its adoption into clinical practice.

ACK N OWLED G M ENTS
We thank Assoc Prof Dr. John Moran and Prof Dr. Sandra Peake for their guidance.We also thank chief librarian Ms. Anna Holasek for

CO N FLI C T O F I NTER E S T S TATEM ENT
No conflict of interests declared.

A
comprehensive search strategy was conducted across several electronic databases including MEDLINE, Embase, Scopus, and CINAHL from inception to 9 February 2023.The database search was conducted from 29 January 2023 to 9 February 2023.The search strategy was developed in consultation with an academic medical librarian, and all identified keywords and index terms were adapted for each included database.An example of the EMBASE search strategy is provided in Table Predefined intra-and postoperative outcomes were extracted from each study.Malviya et al.,1 the original developers of the NARCO-SS score, correlated the preoperative score with the incidence of various adverse intra-and postoperative outcomes.1 Their analysis utilized two composite outcomes: adverse events (AE) and Escalation of care (EOC).Examples of adverse events include difficult airway, bronchospasm, arrhythmias, cardiac arrest, seizures, and bleeding requiring transfusion.Examples of escalation of care events include use of specialist airway equipment, invasive monitoring, unplanned cardioversion, and use of antiarrhythmic agents or vasoactive agents.
used the same definitions in their external validation study.Similarly, Rybiyanov et al.19 used a comparable definition for composite outcomes, which encompassed "complications" and "treatment escalations" (TE).The term "complications" were defined by the study authors to include a range of issues including difficult airway management, intraoperative and postoperative desaturation, postoperative apnea, arrhythmias, hemodynamic instability, postoperative pain, postoperative delirium, postoperative nausea and vomiting, neurological complications, and bleeding.Similarly, 'treatment escalations' were defined by outcomes such as the administration of vasoactive and antiarrhythmic drugs, postoperative intubation,

F I G U R E 1
Flow diagram showing the review process and reason for exclusions.PICO, patient/population intervention comparison outcomes.
Abbreviations: AE, Adverse events; ASA-PS, American Society of Anesthesiologists-Physical status; AUC-ROC, area under the receiver operating characteristic curve; CI, confidence interval; EOC, escalation of care; NARCO, neurological, airway, respiratory, cardiovascular, other; SS, surgical severity.a NARCO, NARCO-SS in Udupa et al. and Rybiyanov et al. b On application of AUC contrast estimation test between NARCO-SS and ASA-PS, AUC values were significantly higher for NARCO-SS at the level of p < .05,N/A = The values were not available, OR calculated low versus high scores for NARCO and ASA-PS (refer discussion for details).c Spearman's correlation (r) is a nonparametric measure of strength of association between scores and outcomes.dOdds ratio is a measure of association which represents the odds that an outcome will occur given a particular exposure compared to the odds of an outcome occurring in the absence of that exposure.
assistance with search strategy.Open access publishing facilitated by The University of Adelaide, as part of the Wiley -The University of Adelaide agreement via the Council of Australian University Librarians.
Abbreviations: PROBAST, prediction model risk of bias assessment tool; ROB, risk of bias.Note: + indicates low ROB/low concern regarding applicability; − indicate high ROB/high concern regarding applicability; ?indicates unclear ROB/ unclear concern about applicability.
TA B L E 4 PROBAST Summary.