Measuring respiratory syncytial virus infection severity in hospitalized children using the Pediatric Respiratory Syncytial Virus Electronic Severity and Outcome Rating System (PRESORS)

The Pediatric Respiratory Syncytial Virus Electronic Severity and Outcome Rating System (PRESORS) was developed to assess the severity of respiratory syncytial virus (RSV) infection in children. Because young children cannot report how they feel or function, ratings are based on observations by the child's caregiver (Observer‐Reported Outcome questionnaire [ObsRO]) and clinician (Clinician‐Reported Outcome questionnaire [ClinRO]). This prospective study aimed to evaluate the psychometric properties of the PRESORS.


| INTRODUCTION
Respiratory syncytial virus (RSV) is the major cause of lower respiratory tract illness in young children. [1][2][3][4] Most children will have experienced at least one RSV infection by 2 years of age. 5 Globally, RSV is the most common cause of bronchiolitis and pneumonia in children <1 year of age. 3,6 In the absence of specialized care, RSV infection can be fatal in neonates, as witnessed in developing countries. 3,6 There exists a high unmet need for pediatric RSV vaccines and treatments to prevent infection, reduce the severity of symptoms and disease, and reduce the time to recovery. 4,5 Several treatments are in development; however, no widely accepted and validated tool yet exists to measure RSV disease severity in young children, and the development of such a tool to assist in assessing treatment efficacy is of high priority. 7  Outcome questionnaire [ClinRO]) and the child's caregiver (the PRE-SORS Observer-Reported Outcome questionnaire [ObsRO]). 8 The ClinRO and ObsRO measures were developed in parallel based on ethnographic and qualitative interview studies with clinicians and caregivers of children with RSV and consultations with pediatric clinical experts. [8][9][10][11] Other tools in development include the Respiratory Syncytial Virus Network (ReSViNET) Scale, 12,13 the Global Respiratory Severity Score (GRSS), 14 and the Gilead RSV Caregiver Diary (GRCD). 7 For clinician ratings, an RSV severity rating scale needs to be reliable such that different clinicians provide the same rating for the same child. This characteristic is measured by inter-rater reliability and is a key aspect of any scale in which the patient cannot self-report outcomes. 15 To assess inter-rater reliability, this study required multiple clinician ratings using the ClinRO of disease severity for the same child.
The PRESORS version 6 (PRESORSv6) ClinRO is a clinician-rated assessment of the severity of signs of RSV infection. ClinRO ratings are based on information that clinicians would ordinarily use to evaluate a pediatric patient: observed presentation of the child (assisted with clinical case notes), discussion with other medical staff, and interaction with the child's parents or caregivers. The ClinRO includes 12 items reflecting signs of RSV infection that had been identified in earlier development steps: activity level, sleep, feeding, dehydration, apnea, retractions, tachypnea, breathing problems, cyanosis, cough, nasal secretions, and wheezing (Table S1). 8 For each item, possible responses reflect potential diagnoses or other key elements. For example, the clinician can indicate for the item retractions whether any subcostal, intercostal, or tracheosternal retractions were observed in the relevant time window. By avoiding subjective Likert-type responses (e.g., mild, moderate, or severe), it was anticipated that choices based on diagnosis would increase reliability and validity. In addition to RSV severity items, the ClinRO includes questions regarding the degree of concern with the child's present health status, a global impression of the child's health, and a global impression of change in the child's health from baseline.
The PRESORSv6 ObsRO is an assessment of the severity of signs of RSV infection from the perspective of the parent/caregiver. It comprises 16 items: activity level, sleep, crying, feeding, urination, dehydration, vomiting, apnea, retractions, tachypnea, tachycardia, breathing problems, cyanosis, cough, nasal secretions, and wheezing (Table S2).
Items were generated and refined through focus group interviews involving concept elicitation and cognitive debriefing. [9][10][11] Meaningful responses were selected for easily understandable items. [9][10][11] Like the ClinRO, the ObsRO included questions on a global impression of the child's health and a global impression of change in the child's health from baseline.
Points for each response were initially assigned based on clinical guidance (Tables S1 and S2). These scoring rules were to be evaluated based on study results. It was hypothesized that RSV severity consists of two main dimensions, respiratory signs and illness behavior, with overall RSV severity resulting from the combination of these two domains.
This study aims to develop the PRESORS further based on its psychometric properties.

| Study design
This was a prospective study conducted in children aged ≤36 months with a confirmed RSV infection requiring hospitalization. No study medication or experimental treatment was administered. The RSV diagnosis could be established by a rapid antigen assay or a polymerase chain reaction--based assay and was to have an onset of symptoms within 5 days of enrollment. A coinfection with another respiratory pathogen was allowed for inclusion, but the use of an investigational (RSV) drug or vaccine was a reason for exclusion.
At each study site, three to five health care providers were trained to complete the ClinRO. The primary ClinRO rating was completed as soon as possible after admission. Independent of this assessment, a second clinician would complete an additional ClinRO to assess inter-rater reliability. This second assessment was ideally performed within 30 min before or after the primary ClinRO to reduce potential changes in the child's condition. Further evaluations were performed daily while the participant was hospitalized and on day 7, preferably by the same health care provider throughout. Participant demographics and medical history were captured at enrollment, and clinical evaluations (physical examination, feeding/hydration, vital signs, and oxygen saturation) were performed during hospitalization and on day 7. The parent/caregiver completed the ObsRO twice daily during the assessment period regardless of whether the child was an inpatient or discharged from the hospital and once on day 7. The ObsRO was preferably completed by the same parent/caregiver throughout the study, but if another person was caring for the child during an interval, this person was to assess the ObsRO.
For the assessment of convergent validity, or how closely a scale relates to other measures of the same construct, 16 the ReSViNET was included, which was considered the most established alternative scale.
The ReSViNET also has clinician and caregiver versions, each including seven items related to symptoms and clinical management (feeding intolerance, medical intervention, respiratory difficulty, respiratory frequency, apnea, general condition, and fever). 13 The same clinician who completed the ClinRO throughout the study also completed the clinician ReSViNET on day 1. The parent/caregiver completed the ReSViNET caregiver version on day 1.
To determine the sample size for the study, the objective was set to exclude an inter-rater reliability <0.65 for the ClinRO. Assuming a true inter-rater reliability of 0.75 and two to three raters per child, a sample size of 124 children would provide 80% power to exclude an inter-rater reliability <0.65 at an α level of 5%. 17 A sample size of 124 was also considered sufficient to provide meaningful information for other study objectives.

| Statistical analysis
This study aimed to develop the PRESORS further based on the psychometric properties of the ClinRO and ObsRO. Methods used in the first step were exploratory factor analysis (EFA), variability assessment, reliability based on internal consistency (Cronbach α), and itemrest correlations. In this step, items were to be iteratively removed based on poor performance on these properties (in particular, based on low variability and/or low item-rest correlations). 18 Simultaneously, the domain structure was established by examining EFA solutions with up to three factors. In a second step, confirmatory factor analysis (CFA) was used to confirm the domain structure by assessing the goodness of fit and to investigate domain structure and item behavior over time. Next, item response theory (IRT; graded response model) was applied to each dimension of the ClinRO and ObsRO to evaluate the scoring of item responses. 19  Inter-rater reliability of the revised ClinRO was investigated using the intra-class correlation (ICC) in a one-way random-effects model. 20 Statistical analyses were performed using R Statistical Software version 4.1.0 (http://www.r-project.org, RRID:SCR_001905); the R psych package (https://cran.r-project.org/web/packages/psych/index.html, RRID:SCR_021744) was used to estimate the ICC and corresponding confidence intervals (CIs). A two-way random-effects model was investigated as a sensitivity analysis but gave similar results, which are therefore not presented. Bland-Altman plots were used to assess whether score differences between raters depended on the mean score and whether the mean scores and score differences varied between sites.
Psychometric properties of the revised scales were investigated by examining internal consistency, convergent and divergent validity, and floor and ceiling effects. The correlation between the ClinRO and ObsRO scale was used to evaluate the correspondence between clinician and caretaker assessments. Both Spearman and Pearson correlations were used but generally yielded similar results.

| RESULTS
A total of 124 participants were enrolled across 10 sites in Argentina, Chile, and the United States. Baseline characteristics and features of hospitalization are summarized in Table 1. Approximately half of the participants were male (53.2%), the mean age was 7.6 months, and the mean duration of RSV symptoms from onset until day 1 was 3 days. Two participants (1.6%) were admitted to the pediatric intensive care. Completion information on ClinRO and ObsRO is available in Figure S1.

| Evaluation of the ClinRO
The items dehydration, apnea, and cyanosis were rarely reported as present and showed little variability. In EFA, cyanosis correlated poorly with other items and did not fit well in either domain. Clinical experts noted that dehydration and apnea may be difficult to detect, and cyanosis is typically present only in more serious cases. Based on these findings, it was decided to remove these three items from the revised ClinRO for these analyses. EFA revealed that the items retractions, breathing problems, tachypnea, cough, nasal secretions, and wheezing loaded on one factor that was interpreted as respiratory signs; the items activity level, feeding, and sleep loaded on a second factor that was interpreted as illness behavior. This result supports the hypothesized structure that RSV severity has two distinguishable dimensions: respiratory signs and illness behavior.
IRT analysis was used to evaluate the scoring system, and few adjustments were made (Table S3). The switch points for responses to all items linked to the level of the latent trait are shown in Figure S2.
The revised PRESORSv6 ClinRO is shown in Table 2. The domain scores are obtained by summating the corresponding signs and rescaling this value to obtain a domain score of 0 to 8 points. Figure 1 (left) presents the factor analysis loading plot of the results of the primary rater on day 1. The two-factor CFA model consistently provided a good fit to the ClinRO data, as shown in item loadings (Table S4) and fit statistics (Table S5) from days 1 to 3.

| Evaluation of the ObsRO
As seen in the ClinRO, illness behavior and respiratory signs factors explained most of the variance in the ObsRO data. Model fit was improved when rarely observed signs of severe RSV infection (apnea, cyanosis, and vomiting) were removed; for the purpose of these analyses, these items were omitted from the ObsRO scoring.
In IRT analysis of the original ObsRO scoring system, several adaptations were made in each domain based on the switch points of each of the items in the illness behavior domain and respiratory signs domain ( Figure S3 and Table S6).
The revised ObsRO scale and scoring are shown in Table 3. The domain scores are obtained by summating the corresponding signs and rescaling the summed value to obtain a domain score between 0 and 8. Figure 1 (right) presents the factor analysis loading plot of the results on day 1. A two-factor CFA on days 1 to 3 was performed for the 13 items in the revised scale (Tables S7 and S8). The items loaded on each of the two factors corresponding to the hypothesized domain structure (illness behavior and respiratory signs). It was observed that dehydration, nasal secretions, and cough did not load well on day 1 but performed better in the hypothesized structure on days 2 and 3. It was concluded that the two-factor structure provided a satisfactory model fit across the first 3 days for the ObsRO. The results and interpretation were in line with those for the ClinRO.

| Inter-rater reliability of the PRESORS ClinRO
The multi-rater ClinRO was obtained for 121/124 participants Summary scores based on primary rater assessments were generally comparable to those based on the second rater's assessments (Table S9). Scatterplots for all summary scores demonstrated a moderate association between the primary rating and the second rating ( Figure S4). In the Bland-Altman plot of the overall RSV severity scores, the scores of the primary rater were not consistently higher or lower than those of the second rater, and differences between ratings were independent of the mean scores ( Figure 2).
Based on the revised ClinRO scoring, the estimated ICC for the overall RSV severity summary score was 0.66 (95% CI, 0.55-0.75), with ICCs for respiratory signs and illness behavior of similar magnitudes (0.60 vs. 0.62, respectively; Table 4). One study site in the United States deviated markedly from the other sites ( Figures S5 and   S6). With this outlier site excluded, a sensitivity analysis yielded an estimated ICC for the overall RSV severity summary score that was Overall RSV severity is the combination of the illness behavior and respiratory signs items

| Psychometric properties of the ClinRO and ObsRO
The psychometric properties (internal consistency, convergent validity, and floor and ceiling effects) of the revised ClinRO and ObsRO scoring were evaluated (Tables 5 and S10).
Convergent validity between the revised ClinRO overall RSV severity score and clinician ReSViNET 14 total score was moderate  (Table S11).

| DISCUSSION
This prospective multi-center study studied the use of the PRESORS for clinician-observed (ClinRO) and parent/caregiver-observed Although reliability and validity are typically thought of as characteristics of scales, these attributes are not fixed. A homogeneous population implies a relatively low "true" score variance, which produces weaker reliability results (defined as "true" score variance divided by "true" score variance plus error variance). Validity results, which are based on correlations, are likewise weakened by homogeneous populations. The relative homogeneity of severity on day 1 was reflected in internal consistency and convergent validity associations that generally increased for days 2 and 3, which in turn also reflect positively on the ability of the PRESORS to reliably assess RSV severity. Estimates of the inter-rater reliability using the ICC in a one-way randomeffects model.

ACKNOWLEDGMENTS
This study was funded by Janssen Pharmaceuticals (Beerse, Belgium).
We thank the participants and site personnel involved with this study.
Writing and editorial assistance for the development of writing-original draft; writing-review and editing.

PATIENT CONSENT STATEMENT
Informed consent for participation was obtained from each participant's parent or caregiver prior to screening and enrollment in the study.

SOURCES
Does not apply to this study. The authors declare that this manuscript does not contain any previously published material (including figures/ diagrams, or short extracts, or content taken from websites), and all figures and tables are original.

PEER REVIEW
The peer review history for this article is available at https://publons. com/publon/10.1111/irv.13028.

DATA AVAILABILITY STATEMENT
The data sharing policy of Janssen Pharmaceutical Companies of