High inter‐rater reliability between physicians and nurses utilising modified Downes' scores in preterm respiratory distress

To assess the inter‐rater reliability of modified Downes' scores assigned by physicians and nurses in the Ethiopian Neonatal Network and to calculate the concordance of score‐based treatment for preterm infants with respiratory distress.


| INTRODUC TI ON
In 2020, 2.4 million deaths, or roughly 47% of all deaths under 5 years of age, occurred in newborns. 1Most neonatal deaths occur in the first day and week translating to 6700 newborn deaths every day of which 98% occur in low-and middle-income countries (LMIC). 1 Approximately 80% of these deaths occurred in neonates with a birthweight less than 2500 g and two-thirds of these deaths occurred in neonates born before 37 weeks' gestation, 2 identifying low-birth weight and preterm neonates as groups at high risk for mortality.
Complications of preterm birth are specifically listed as the leading primary cause of death among neonates (36%) as well as among children younger than 5 years (17.7%). 3study at five tertiary centres in Ethiopia, a low-income country, used post-mortem examinations and expert clinical case review to determine the primary and contributory causes of mortality among preterm infants admitted to neonatal intensive care units (NICUs). 4ring the neonatal period, up to 28 days postnatal age, 29% of preterm infants admitted to participating NICUs in Ethiopia died.
Respiratory distress syndrome (RDS) was the primary cause of death identified in nearly half (45%) of the cases.Understanding the pathology and factors contributing to neonatal deaths is a prerequisite to identifying and prioritising interventions to reduce the current neonatal mortality rate of 28 per 1000 live births in Ethiopia. 4rmont Oxford Network (VON) 5  (CPAP) for infants with birth weights >1000 g and with respiratory distress (Figure 1).In the development of this algorithm, the Downes' score 6 was modified as blended oxygen was not reliably available for use in assessments to determine if cyanosis was relieved with FiO 2 of 0.4.The modification (Figure 2) uses "cyanosis relieved with oxygen" and "cyanosis not relieved with oxygen" for assessments.A planned national implementation of this guideline with on-site coaching visits to neonatal units was halted due to the COVID-19 pandemic.
Results: Of the 1151 eligible infants admitted, 817 infants (71%) had scores reported concurrently and independently by nurse and physician.The kappa statistic for modified Downes' score components ranged from 0.88 to 0.92 and was 0.89 for the total score.There was 98% concordance for score-based treatment.

F I G U R E 1
Guideline for the initiation of CPAP in newborns.This guideline recommends the use of blended oxygen and continuous pulse oximetry, and was designed for use with Diamedica Baby CPAP devices.When these resources are not consistently available, or other resources such as nasal interfaces differ, this guideline will need to be adapted.

| Population
The study population included all infants with gestational ages <37 weeks' admitted to the four participating hospital NICUs with respiratory distress present on admission, with hospital disposition (discharge, transfer or death) between the time period of June 2020 to July 2021.
Gestational age was recorded by the clinical care team as the best estimate of gestational age in weeks and days using the following hierarchy: obstetrical measures based on prenatal ultrasound, obstetrical parameters or last menstrual period as recorded in the maternal chart followed by postnatal examination by NICU care providers utilising the Ballard score. 7Small for gestational age was defined within categories of sex as birth weight below the 10th percentile for gestational age based on the 2013 Fenton curve. 8Nurses and physicians at the four participating hospitals received training on the modified Downes' score and the national guideline for CPAP initiation prompting treatment for scores of 4 or greater combined with an oxygen requirement (Figure 2).The modified Downes' score was incorporated into admission assessments.When feasible, concurrent and independent assessments were completed and recorded by a nurse and physician.

| Statistics
We calculated the kappa statistic to determine the nurse and physician inter-rater reliability for each component of the modified Downes' score and the total score.We determined the concordance of nurse and physician scores above and below the treatment threshold of 4. We also described the in-hospital mortality by modified Downes' score, using the average of nurse and physician scores completed on admission assessments.

| RE SULTS
Of the 5357 infants admitted to the four participating NICUs during the study period, 1867 were <37 weeks' gestational age of which 1151 (62%) had respiratory distress.Of the eligible preterm infants with respiratory distress, 817 (71%) infants were concurrently and independently assessed by both nurse and physician, 186 infants (16%) received only a physician assessment with modified Downes' score, 8 (0.7%) received only a nurse assessment with modified Downes' score, and 140 (12%) were not objectively assessed with a modified Downes' score.Three hospitals contributed infants who were assessed by both nurses and physicians (Figure 3).
Maternal and infant characteristics for the 1151 eligible preterm infants with respiratory distress and the 817 infants with nurse and physician-modified Downes' scores completed on admission are found in Table 1.There were not any clinically meaningful differences noted between these two groups.The majority of infants in this cohort were born at the same hospital as the inpatient neonatal unit to which they were admitted (inborn), had a delivery provider trained in Helping Babies Breathe (HBB) or Neonatal Resuscitation Program (NRP) and were born in the setting of preterm labour.The percentage of infants exposed to antenatal corticosteroids was 41.7%.Among this cohort of preterm infants with respiratory distress on admission, 5% were extremely preterm (<28 weeks' gestational age), 31.5% very preterm (28 to 31 weeks' gestational age) and 63.6% moderately preterm (32-36 weeks' gestational age).Half (49.8%) of infants were very low birth weight (≤1500 g), 46.4% with birth weight 1501-2500 g, and 43% of infants recorded as small for gestational age.Among the 817 preterm infants who received both physician and nurse assessments, 43.8% died during their NICU stay.
In-hospital mortality increased with increasing average modified Downes' scores (Figure 4).
The mean overall and component-modified Downes' scores with kappa statistics for the 817 infants who received assessments from both providers are found in Table 2.The overall mean score for nurses was 4.68 and for physicians was 4.83 (K = 0.89).The kappa statistic for the modified Downes' score components ranged from 0.88 to 0.92, consistent with near-perfect agreement.The mean modified Downes' score for the 186 patients only assessed by physicians was 5.2, with a median of 5. Of the 817 infants concurrently assessed and scored, 98% had concordance above or below the score-based treatment threshold of 4 (Table 3). the continued use of this objective score in their setting. 9When their research team compared the Downes' score to the Silverman Andersen score in this Indian NICU, the Silverman Andersen score had lower inter-rater reliability, and participants found the Downes' score to be comparatively easier to use and assessments were quicker to complete. 9The use of the Downes' score or a modified Downes' score has also been implemented and its use published in multiple other LMIC settings, including China, 10 Indonesia, 11 and Nigeria. 124][15] An increase in Downes' scores and Silverman Andersen scores has also been associated with increased risk of death due to respiratory failure, 12,15 which is consistent with our findings.Among very low birth weight infants (birth weight 500-1500 g) admitted to tertiary Nigerian NICUs, an increase in Downes' score was independently associated with adjusted odds of in-hospital mortality (aOR 1.27, 95% CI 1.14-1.41). 12An area of future research is evaluation of further simplifying these objective scores 16 and the inclusion of current objective scores into risk estimators with the addition of patient-level risk factors, such as gestational age and birth weight, 17 with the goal of supporting multi-disciplinary care teams, standardising and improving team communication, and empowering bedside care providers.
In our Ethiopian cohort, 43.8% of infants died as patients in the NICU.This magnitude of in-hospital mortality is similar to what has been published from other LMIC NICUs.Fajolu et al. examined rates and predictors of mortality of VLBW infants in three Nigerian tertiary hospitals.They reported that nearly half (47%) of all VLBW infants admitted to their sites died during hospitalisation, with respiratory distress noted as an independent predictor of death. 12In a rural Ugandan NICU, McAdams et al. described the implementation of bubble CPAP, noting that approximately half (52%) of infants treated with CPAP survived to hospital discharge. 13Acknowledging that the magnitude of in-hospital mortality in our Ethiopian cohort is not an outlier among LMIC NICUs calls attention to the burden of neonatal mortality in LMICs and opportunities for improvement in outcomes with cost-effective, evidence-based care.
Standardisation and improving the quality and equity of care provided to infants and their families is important globally, emphasised by the World Health Organization's Quality of Care Network 18 and the Every Newborn Action Plan. 2,19,20An objective respiratory assessment of all preterm infants on admission is an opportunity to standardise care and potentially reduce disparities if assessments and score-based treatment algorithms, such as our example in Ethiopia, are approached with an equity-intentional lens 21 and adopted into practice. 22Timeliness is an important factor in quality of care.Delays in the appropriate initiation of respiratory treatment with CPAP have been documented relating to lack of effective team communication between physicians and nurses, as well as due to rigid division of roles and responsibilities among healthcare providers. 23In a systematic review of barriers and facilitators to implementing bubble CPAP to improve neonatal health in sub-Saharan Africa, staffing shortages, high turnover and effectively informing and engaging caregivers were noted barriers in addition to reliable availability of equipment. 24Reinforcing the importance of clinical mentorship as well as advocacy for multidisciplinary teams, Tooke et al. highlighted that neonatal nursing was recognised as a specialty in only 57% of the 49 African countries represented in a continental survey of respiratory care practices. 25It is critical to view an objective respiratory assessment score in the context of nursing and medical education and the system of care in which infants are treated.To realise an impact in TA B L E 1 Maternal and infant characteristics among eligible infants <37 weeks' gestational age with respiratory distress on admission and infants with modified Downes' scores completed by nurse and physician on admission.improved care and outcomes, identified knowledge gaps, such as in nursing knowledge and practice relating to the care of neonates with respiratory distress, 26 and quality gaps, such as hospitals failing to meet the national maternal-newborn health care quality standards, 27 must be addressed.Implementation of an objective respiratory assessment such as the modified Downes' score represents one process in a complex system with the potential to catalyse system improvement.

N
A limitation of our study design was lack of video recording As signs of respiratory distress are dynamic, we took a pragmatic approach in requesting concurrent and independent mod- with assessments by both nurse and physician.However, more than 800 preterm infants were concurrently and independently assessed, representing a spectrum of times and days of the week for admission, birth weights, physical appearances, degrees of illness and comorbidities, strengthening the findings of close alignment of scores and score-based treatment.
The Sustainable Development Goals seek to reduce neonatal mortality to 12 per 1000 live births or less in all countries, 28 which will not be met without a focus on mortality reduction in preterm infants. 2,19,29CPAP is an evidence-based treatment for RDS, [30][31][32] the leading cause of mortality of preterm infants in many LMIC NICUs. 4,12The consistent availability of physicians dedicated to patient care in NICUs in low-resourced settings, let alone practising neonatologists, is challenging. 25 has partnered with Addis Ababa University and Tikur Anbessa Specialized Hospital in Ethiopia since 2008, when the Tikur Anbessa NICU project was jointly established and supported the fellowships of the first three Ethiopian-trained neonatologists.Building upon that strong foundation and meeting the demand for locally relevant and actionable data paired with quality improvement initiatives tailored for resource-limited settings, VON partnered and collaborated with the Ministry of Health, Ethiopia and the Ethiopian Paediatrics Society to establish the Ethiopian Neonatal Network in 2018.The Ethiopian Neonatal Network has a membership of 22 Ethiopian inpatient neonatal units caring for small and sick newborns.The Ethiopian Neonatal Network teams collect data on all neonatal admissions and work on quality improvement projects aimed at reducing preventable causes of neonatal mortality.In November 2019, as a response to national investments in medical devices for neonatal care, VON volunteers and Ethiopian Neonatal Network leaders partnered with the Ministry of Health, Ethiopia and UNICEF to develop a national guideline for the initiation of continuous positive airway pressure

:
Incorporation of the modified Downes' score on admission for preterm infants with respiratory distress was feasible in tertiary NICUs in Ethiopia.The kappa statistics showed near-perfect agreement between nurse and physician assessments, translating to a very high degree of concordance in score-based treatment recommendations.These results highlight an opportunity for task-shifting assessments and empowering nurses.K E Y W O R D S continuous positive airway pressure, downes' score, neonatal mortality, neonatal nursing, respiratory distress syndrome Key Notes • The modified Downes' score is used in Ethiopia to evaluate preterm neonates and direct treatment yet the inter-rater reliability between nurses and physicians completing this score on admission had not been established.• We found near-perfect agreement between nurse and physician assessments of components of the modified Downes' score and total score, translating to 98% concordance in score-based treatment recommendations.• These results highlight an opportunity for task-shifting assessments and empowering nurses.In June 2020, four tertiary NICUs in the Ethiopian Neonatal Network embarked on a study to establish the incidence of presumed RDS using a clinical definition appropriate to LMIC settings without chest x-ray findings or blood gas results.One of the goals of the project was to establish the inter-rater reliability of the modified Downes' score between nurses and physicians in recognition of the inclusion of this score as the objective respiratory assessment utilised in the Ethiopian national guideline for CPAP initiation and to calculate the concordance of scores above and below the scorebased treatment cut-off of four. 2 | PATIENTS AND ME THODS 2.1 | Setting This study took place at four tertiary NICUs participating in the Ethiopian Neonatal Network: Tikur Anbessa Specialty Hospital in Addis Ababa, St. Paul's Hospital Millennium Medical College in Addis Ababa, Gondar University Hospital in Gondar and Ayder Comprehensive Specialized Hospital in Mekelle.The inpatient neonatal units have a capacity of 40-60 beds, with an overall average daily census of 42 patients.Three of the four hospitals consider neonatal nursing as a specialty, with an average of 11% of nursing staff overall considered specialty nurses.None of the four hospitals practice scheduled rotation of nursing staff across maternal-newborn-child health units.Three of the four hospitals employ neonatologists and all employ paediatricians to staff the inpatient neonatal units.

A
master trainer from each participating hospital completed handson training with VON and Ethiopian Neonatal Network faculty on the use and implementation of the modified Downes' score, the national guideline for CPAP initiation and CPAP optimisation.Training sessions included didactic sessions, case scenarios, the use of standardised video clips for trainee scoring and feedback and hands-on patient assessments in the NICU.Master trainers led dissemination at their individual hospitals utilising the same training package, with remote support from VON and Ethiopian Neonatal Network faculty.
Local staff collected data on infants admitted to the NICUs using uniform definitions until death, discharge home, or transfer to other hospitals.Data collected in REDCap (Research Electronic Data Capture) underwent automated checks for quality and completeness at the time of and after submission.All four Ethiopian university hospitals obtained institutional review board (IRB) approval (Addis Ababa University Reference: PD/MF/087/20, St. Paul's Hospital Millennium Medical College Reference: OM23/512, Mekelle University Reference: ERC 1491/2020, Gondar University Reference: V/P/ RCS/05/1581/2020).The IRB at the University of Vermont (USA) reviewed this study and determined it was exempt (Study00001462).

4
| DISCUSS ION In this large Ethiopian prospective cohort involving preterm infants admitted to tertiary neonatal intensive care units, physicians and nurses trained on a modified Downes' score as an objective respiratory assessment had near-perfect agreement in component and total scores.While previous studies have evaluated inter-rater reliability in neonatal units, this study, which included over 800 preterm infants with respiratory distress admitted to tertiary NICUs in a low-income country, which is, to our knowledge, the largest published cohort to date addressing concordance of Downes' or modified Downes' scores, adds clinical relevance by framing the scores in the context of the national algorithms with score-based treatment thresholds.Overall, 98% of infants concurrently and independently scored by a nurse and physician had concordance in score-based treatment recommendations.The high degree of concordance between nurses and physicians assessing preterm infants with respiratory distress in a low-income country NICU utilising the Downes' score is consistent with previous work done in Bangalore, India.In a small, single-centre study, Shashidhar et al. found acceptable inter-rater reliability between nurses and medical interns utilising the Downes' score, supporting

F I G U R E 3
Flow diagram for cohort.Flow diagram of population screened, inclusion in cohort, assessment with modified Downes' score on admission, type of neonatal care providers (nurse and/or physician) that completed the assessment and contribution of each hospital to the cohort.
or direct observation by study staff of the concurrent and independent evaluation of infants.Neonatal nurses and physicians recorded their modified Downes' scores on separate paper tools and then transposed both answers to the data collection form for entry and analysis.Team discussions of objective and serial assessments with modified Downes' scores were encouraged following independent evaluations and documentation.However, it is possible that team conversations focused on clinical care influenced the independence of the nurse and physician assessments.All nursing and physician team members received the same multi-disciplinary training on use of the modified Downes' score including the use of simulation to practice assigning the modified Downes' score.Although in a different context, simulation-based neonatal resuscitation team training has been associated with improved team performance and technical performance including decision-making and team interaction processes used during the team's management of a scenario in simulation-based evaluations. 24The participating Ethiopian Neonatal Network teams, working together as a multi-disciplinary collaborative quality improvement community since 2018, incorporated the modified Downes' score assessment and national CPAP algorithm for preterm infants into a foundation of team training with emphasised values of teamwork, trust and respect.

5 |
This challenge represents an opportunity for a frameshift.The evaluation to complete a modified Downes' score represents a skill that is shown in this study to not be defined by siloed disciplines, such as medicine or nursing, and furthermore has the potential to enhance communication while empowering a multi-disciplinary team and improving multiple domains of quality of care.The use of the modified Downes' score particularly by neonatal nurses to initiate CPAP therapy represents a positive disruption and may serve as an example for re-designing a system of care for improvement and to better serve our patients and families.CON CLUS ION Incorporation of an objective clinical assessment on admission for preterm infants with respiratory distress was feasible in tertiary NICU settings of the Ethiopian Neonatal Network.The kappa statistics of the components of the modified Downes' score and total score showed near-perfect agreement between nurse and physician assessments, translating to a very high degree of concordance in score-based treatment recommendations.As LMIC NICUs prioritise improving RDS care and quality of care, these findings reinforce nursing empowerment and task-shifting standardised assessments and initiation of CPAP within the context of a multi-disciplinary team.AUTH O R CO NTR I B UTI O N S DEYE, AD, AH, GM, YH, MA, BW, RK, KK, MD, MS, JB, AG and MD participated in the modification of the Downes' score and the implementation of the Ethiopian national guideline.DEYE, AD, AH, YH, GM, JDH, EME, BW, MD and MA conceptualised the development of this study and the analytic framework and were involved in data collection and the writing of the manuscript.EME had primary responsibility for data analysis and data interpretation.All authors approved the final manuscript as submitted and agreed to be accountable for all aspects of the work.
In-hospital mortality by modified Downes' score on admission.In-hospital mortality (%) by the averaged nurse and physician-modified Downes' score completed on admission among the 817 infants with both assessments.
ity to complete two provider assessments.With this approach, 71% of eligible infants in this cohort had concurrent and independent assessments by nurse and physician.It is possible that the infants that received an assessment by either physician or nurse or lacked assessment with a modified Downes' score had different presentations with potential for more variability in scoring compared to those F I G U R E 4 TA B L E 2 Mean scores and Kappa statistics for modified Downes' score components and overall for infants assessed by both nurses and physicians (n = 817).TA B L E 3 Concordance of nurse-recorded and physicianrecorded modified Downes' scores above and below the scorebased treatment threshold of four.N Nurse <4 and physician <4 175