Implementing the World Health Organization Pandemic Influenza Severity Assessment framework—Singapore's experience

Abstract Background We report our experience in evaluating the severity of local influenza epidemics using the World Health Organization Pandemic Influenza Severity Assessment framework. Methods We assessed the severity of influenza by monitoring indicators of influenza transmissibility, seriousness of disease and impact on healthcare resource utilisation. Indicators were described by various parameters collected weekly from eight government hospitals, 20 government and 30 private primary care clinics, and the national public health laboratory. Transmissibility and seriousness of disease indicators were each represented by multiple parameters, and alert thresholds were set at the 70th and 90th percentile of a parameter's past 2‐year surveillance data. We derived a collective measure for each indicator using the average percentile rank of the related parameters. Alert thresholds for the single impact parameter were set at predefined values and evaluated for its sensitivity, specificity and positive predictive value. Results For the transmissibility and seriousness of disease parameters, calculation of the percentile rank was simple and independent of a parameter's underlying distribution. For the impact parameter, predefined alert thresholds had high sensitivity and specificity (>80%) but low positive predictive value (15%‐30%). Assessment scales were used to qualitatively classify the activity of an indicator as low, moderate or high together with a confidence level. Conclusion We applied different methods for threshold setting depending on the attributes of each parameter and indicator. For indicators represented by multiple parameters, an aggregated assessment of the indicator's level of activity and confidence level of the assessment was needed for effective reporting.

Indicators were described by various parameters collected weekly from eight government hospitals, 20 government and 30 private primary care clinics, and the national public health laboratory. Transmissibility and seriousness of disease indicators were each represented by multiple parameters, and alert thresholds were set at the 70th and 90th percentile of a parameter's past 2-year surveillance data. We derived a collective measure for each indicator using the average percentile rank of the related parameters. Alert thresholds for the single impact parameter were set at predefined values and evaluated for its sensitivity, specificity and positive predictive value.
Results: For the transmissibility and seriousness of disease parameters, calculation of the percentile rank was simple and independent of a parameter's underlying distribution. For the impact parameter, predefined alert thresholds had high sensitivity and specificity (>80%) but low positive predictive value (15%-30%). Assessment scales were used to qualitatively classify the activity of an indicator as low, moderate or high together with a confidence level.

Conclusion:
We applied different methods for threshold setting depending on the attributes of each parameter and indicator. For indicators represented by multiple parameters, an aggregated assessment of the indicator's level of activity and confidence level of the assessment was needed for effective reporting.

| BACKG ROU N D
Early severity assessment of pandemic influenza is helpful for guiding pandemic response actions. However, during the 2009 H1N1 pandemic, severity assessment was not standardised across countries, making it difficult to evaluate the local or global situation as the pandemic evolved. 1 The lack of a consistent measure of severity also posed a challenge to calibrate pandemic response, which is dependent on geographical spread, clinical severity and public interest, among other factors. 1 Through the lessons learnt from the 2009 H1N1 pandemic, the World Health Organization (WHO) has developed a framework for pandemic influenza severity assessment (PISA). 2 PISA is a structured way of tracking influenza epidemics or pandemics. The three recommended indicators for monitoring severity were the transmissibility of the influenza virus, the seriousness of the disease and the impact of influenza on healthcare resource utilisation (referred to as transmissibility, seriousness of disease and impact, in the subsequent sections). By assessing severity from multiple dimensions, this encourages countries to establish surveillance at different levels of the healthcare system to create a holistic picture of an influenza epidemic or pandemic.
Using virological and surveillance data from different sources, the severity of each indicator can be represented by more than one type of data, or parameter. The choice of parameters may vary across countries due to different data availability, of which some require substantial resource to collect. While the challenge of data comparison remains, PISA plays an essential role-to promote enhanced TA B L E 1 Parameters considered for assessing severity of influenza surveillance and increase information sharing among public health officials during an influenza epidemic or pandemic.

| Influenza surveillance in Singapore
Singapore, a city-state in South East Asia, is a major global travel hub with over 18 million tourist arrivals 3  In this paper, we document Singapore's experience in developing and evaluating the PISA indicators and parameters, and this would provide other countries with suggestions that they can use in developing their own indicators.

| Data sources
A wide range of parameters were reported weekly to the Ministry of Health (MOH) and considered for PISA (Table 1). Influenza transmission in the community was monitored using the average daily attendance for acute respiratory infection (ARI) and the average daily attendance for influenza-like illness (ILI) at the government primary care clinics. An ARI diagnosis was made when a case had at least one acute respiratory symptom such as cough, sore throat and coryza, while an ILI diagnosis was made when a case had a fever of ≥38.0°C and cough, with onset within the last 10 days. The average daily attendance for ARI and average daily attendance for ILI at the government primary care clinics were used, instead of the weekly attendances, to offset the effect of public holidays and clinic closure on weekends. As not all ILI attendances at the government primary care clinics were attributed to influenza, we explored using the product of the average daily attendance for ILI and weekly proportion of respiratory samples positive for influenza to estimate the average daily number of influenza-positive ILI cases at the government primary care clinics. We also collect parameters from the eight acute govern-

| Assessing the transmissibility and seriousness of disease indicators' level of activity
As the transmissibility and seriousness of disease indicators were represented by more than one parameter, an overall measure of each indicator's level of activity and the confidence of the indicator was necessary for weekly reporting.  Furthermore, the distance of the average percentile value from the cut-offs percentiles provided a measure of confidence-the further away, the average percentile is from an alert threshold, the greater the confidence in the assessment of an indicator's level of activity and vice versa.

| Assessing the impact indicator's level of activity
The weekly number of laboratory-confirmed influenza cases who were admitted to the intensive care unit (ICU) or died is the only impact parameter, and we used data from January 2011 to December 2017 for threshold setting due to the absence of reporting artefacts over the years. The discrete data had a small range of observed values, and hence, we used a different approach to set the alert thresholds and to ensure that alert thresholds had integer values.
A sustained high (moderate) influenza activity is said to occur when the impact parameter values remain above the high (moderate) alert thresholds for 2 weeks after the first alert week. We set alert thresholds at predefined values and tested two different scenarios. In the first scenario, the moderate and high alert thresholds were set at three and six, respectively. In the second scenario, they were revised to four and six, respectively. We evaluated key performance metrics of sensitivity, specificity and positive predictive value (PPV) of a threshold to assess the threshold's ability to provide early warning prior to the peak of an influenza season. 6 The sensitivity was the proportion of sustained high influenza activity with a moderate alert raised in at least one of the 2 weeks prior to crossing the high alert threshold. The specificity was the proportion of weeks with no alerts during the baseline influenza periods. The PPV for high (moderate) influenza activity was the proportion of true high (moderate) alerts among all high (moderate) alerts.

| Parameters selected for PISA reporting
Time series plots of the parameters in Table 1 are shown in Figure 1.
The average daily attendance for ARI at the government primary care clinics ( Figure 1A) exhibits a multimodal distribution as it is in-  Figure 1K).

| Performance of the impact parameter alert thresholds
The weekly number of laboratory-confirmed influenza cases that were admitted to ICU or died ranged from 0 to 24 ( Figure 1K). When the moderate and high alert thresholds were predefined at an inte-    Key challenges remain in achieving a representative indicator for seriousness of disease in Singapore. The weekly proportion of ARI or pneumonia ED attendances that were hospitalised were chosen to illustrate the severity of each condition, but the absence of hospital laboratory surveillance data limits our ability to verify the infection status of each patient. Spikes in the weekly proportion of ARI ED attendances that were hospitalised ( Figure 1G) could be attributed to changes in health-seeking behaviour, reporting habits of physicians and higher tendency to admit a patient during a pandemic, though extent of influence has yet to be studied.

| D ISCUSS I ON
The cumulative number of patients tested positive for influenza admitted to ICU is a component to some WHO recommended parameters in Table 1  integer values. The PPV of the thresholds was poor and implied that in many occasions, there was no sustained moderate or high influenza activity occurring after a moderate or high alert was triggered.
The moderate threshold was eventually set at four as about 70% of the historical data was below this value, and a moderate alert was triggered before the onset of all sustain high influenza activity.
In this paper, we also presented an assessment scale, which provides a combined measure of an indicator's level of activity and the confidence level of the assessment. With more than one parameter serving as proxies for an indicator, the method of providing an aggregated assessment for an indicator remains undocumented in PISA. Furthermore, the confidence of an indicator's assessment is part of PISA reporting, but its interpretation is multifaceted. It is dependent on, but not limited to, reporting biases, timeliness and agreement between the parameters. The first two factors are re- where the average daily attendance for ARI was high but the same was not observed for ILI surveillance data. However, it is still important to track the ARI attendances at the government acute hospitals and primary care clinics as it potentially informs us of any changes in the clinical representation of influenza cases. One possible way of overcoming this challenge is to assign weights to each parameter based on its importance in assessing the local influenza situation.
The weighted average percentile rank could be computed to represent an indicator's level of activity.
In addition, when a parameter is higher (or lower) than the historical maximum (or minimum), the percentile of that parameter's data was capped at 100 (or zero). Taking

| CON CLUS ION
We share Singapore's practices in the weekly assessment of PISA indicators. For indicators represented by multiple parameters, a collective assessment of the indicator's level of activity and the confidence level of this assessment were necessary. Here, we have introduced an assessment scale to accomplish both objectives. We placed priority in creating a simple collective assessment for a com-