Prediction of serious RSV‐related outcomes in older adults with outpatient RSV respiratory illness during 12 consecutive seasons

Abstract We developed and evaluated a model to predict serious outcomes among 243 adults ≥60 years old with medically attended respiratory illness and laboratory‐confirmed respiratory syncytial virus (RSV); 47 patients had a serious outcome defined as hospital admission, emergency department (ED) visit, or pneumonia diagnosis. The model used logistic regression with penalized maximum likelihood estimation. The reduced penalized model included age ≥ 75 years, ≥1 ED visit in prior year, crackles/rales, tachypnea, wheezing, new/increased sputum, and new/increased dyspnea. The optimal score cutoff yielded sensitivity and specificity of 66.0% and 81.6%. This prediction model provided moderate utility for identifying older adults with elevated risk of complicated RSV illness.

criteria varied by season, but fever/feverishness or cough were required during most seasons. Multiplex reverse-transcription polymerase chain reaction (RT-PCR) was performed on archived samples to detect RSV and other viruses. The multiplex panel included RSV A and B, human rhinovirus, human metapneumovirus, parainfluenza viruses 1-3, influenza A (H3, H1, and H1N1pdm09), influenza B, and adenoviruses B/E and C. Symptoms were assessed during the enrollment interview, and medical records were abstracted for all RSV cases. Serious RSV outcomes were defined as acute care hospital admission, emergency department (ED) visit for acute illness, or pneumonia occurring within 28 days after enrollment. A physician independently validated all serious outcomes. All participants provided informed consent for influenza study participation, and subsequent multiplex testing was approved by the institutional review board with a waiver of informed consent.

| Statistical modeling
The modeling dataset contained information from 243 adults with RSV illness, including 47 who had a serious outcome and 196 with a non-serious outcome. A model to predict presence of a serious RSV outcome was built via logistic regression using penalized maximum likelihood estimation (PMLE). 6-8 Models employing PMLE, a form of shrinkage estimation, effectively use fewer degrees of freedom (df) than standard MLE models and hence have a higher threshold for what would constitute overfitting. In this context, overfitting translates to the number of candidate predictors evaluated being high relative to the number of patients with serious RSV outcome events. PMLE directly adjusts for "over-optimism," the fact that predictive performance measures derived from the dataset on which the model was built will generally be more favorable than would be expected when the model is applied to a new data source.
Penalized maximum likelihood estimation is well suited for situations with a high potential for overfitting as described above.
However, since the modeling dataset included only 47 serious outcome events, a univariate screening of candidate predictors was performed prior to multivariable modeling. The set of candidate variables included signs and symptoms (both self-reported and physical examination findings) as well as other clinical and demographic factors.
A goal of the PMLE analyses was to impose shrinkage such that the df from the standard MLE-based logistic regression model (one df for each parameter required to represent the terms in the model) were reduced to the point where overfitting was not present. We implemented the shrinkage by selecting a penalty factor and applying this factor in the process of estimating regression coefficients for the full penalized model. Optimization of the corrected Akaike information criterion, a goodness of fit measure, was the basis for selecting the penalty factor. To further reduce the effective df, a reduced penalized model containing a parsimonious set of important predictors was established by applying a backward stepdown procedure to the full penalized model. 6 We assigned a score to each patient that represents the likelihood of a serious RSV outcome using a system based on the estimated regression coefficients. A score of 100 was assigned to the predictor with the largest regression coefficient and the remaining scores reflected the importance of other predictors relative to the predictor with a score of 100. For each patient, scores across predictors were summed to generate a total score. The final step in the predictive modeling process was to establish an "optimal" cutoff for dichotomizing the score to represent either presence or absence of a serious RSV outcome. We evaluated two well-known criteria for selecting cutoffs: the "closest to ideal" criterion and the Youden index. 9,10 In this context, the term "optimal" corresponds to selecting a cutoff with a desirable tradeoff between sensitivity and specificity. We generated an empirical receiver op-

| D ISCUSS I ON
This predictive model was moderately successful in predicting serious RSV outcomes in older adults with RSV infection. It was based on patient characteristics readily available in clinical practice and may be useful for identifying and risk stratifying patients with an increased likelihood of serious complications of RSV infection. It could also be useful for constructing clinically meaningful efficacy endpoints for prelicensure clinical trials of RSV vaccines. While the aim of such vaccines would be to prevent serious outcomes such as hospitalizations, ED visits or pneumonia caused by RSV infection, capturing these outcomes typically requires prohibitively large clinical trials. Use of clinical endpoints based on prediction scores derived from readily captured data could be a more efficient approach to measuring vaccine efficacy against serious outcomes of interest.
The strengths of this analysis include systematic screening and testing of patients in the original study, recruitment from a stable community cohort, and inclusion of 12 consecutive seasons with access to outpatient and inpatient electronic medical records.
Weaknesses included the small number of serious outcomes captured among adults seeking outpatient care, lack of racial/ethnic diversity, and inability to examine predictors of hospitalization or more severe hospitalized illness. The number of coinfections was small, and it is possible that a larger study would have identified influenza coinfection as a significant predictor of serious outcomes. Validation of this prediction model in larger populations is needed to determine if data collected during an acute respiratory illness visit is useful to predict the likelihood of a subsequent hospital admission or other serious outcome.
In conclusion, a prediction model using clinical information available during the initial evaluation of RSV illness was moderately useful for assessing the risk of a serious outcome. F I G U R E 1 Empirical receiver operating characteristic (ROC) curve for the reduced penalized model