Assessment of the benefits of seasonal influenza vaccination: Elements of a framework to interpret estimates of vaccine effectiveness and support robust decision‐making and communication

Abstract Systematic reviews and meta‐analyses confirm that influenza vaccination reduces the risk of influenza illness by between about 40% and 60% in seasons when circulating influenza stains are well matched to vaccine strains. Influenza vaccine effectiveness (IVE) estimates, however, are often discordant and a source of confusion for decision makers. IVE assessments are increasingly publicized and are often used by policy makers to make decisions about the value of seasonal influenza vaccination. But there is limited guidance on how IVE should be interpreted or used to inform policy. There are several limitations to the use of IVE for decision‐making: (a) IVE studies have methodological issues that often complicate the interpretation of their value; and (b) the full impact of vaccination will almost always be greater than the impact assessed by a point estimate of IVE in specific populations or settings. Understanding the strengths and weaknesses of study methodologies and the fundamental limitations of IVE estimates is important for the accuracy of interpretations and support of policy makers’ decisions. Here, we review a comprehensive set of issues that need to be considered when interpreting IVE and determining the full benefits of influenza vaccination. We propose that published IVE values should be assessed using an evaluative framework that includes influenza‐specific outcomes, types of VE study design, and confounders, among other factors. Better interpretation of IVE will improve the broader assessment of the value of influenza vaccination and ultimately optimize the public health benefits in seasonal influenza vaccination.

Epidemiological factors account for some of these differences but often study design 3 and the assessment of different outcomes make it difficult to compare studies within and across different seasons. 4 The strengths and weaknesses of IVE study design have been extensively reviewed elsewhere. [5][6][7][8][9][10][11] But methodological deficiencies in the evaluation of influenza vaccine effectiveness, and misinterpretation of the outcomes, 12,13 pose a serious challenge to the use of seasonal influenza vaccination as a public health tool. 14 This paper reviews the critical considerations that should be made when interpreting IVE and proposes an evaluative framework to be used to interpret the outcomes of IVE studies as a basis for determining the full benefit of influenza vaccination. In sum, the contextual variability of IVE calls for the use of a common evaluative framework to ensure the consistency of the assessment of IVE studies outcomes and to ensure that the strengths, weaknesses, and limitations of IVE data are fully appreciated by the reader.

| Proposed element s of a fr amework for the assessment of seasonal influenz a vaccine ef fec tiveness
To account for differences in reported IVE estimates, WHO recommends that reporting of IVE studies include sufficient details on study participants, data collection, and analyses to enable readers to judge the validity of each study. 14 With this in mind, and expanding on these ideas, we propose that an evaluative framework could be developed, which would include the following elements (as summarized in Table 2 and discussed below), to ensure the limitations of estimates of IVE, as an indicator of public health benefit, are fully appreciated and effectively communicated: For each of these outcomes, IVE may differ. This is evidenced by the variability of IVEs against different endpoints, from different systematic reviews or meta-analyses, as shown in Table 1.

| The outcomes measured
IVE studies of asymptomatic influenza are important for assessing vaccination effectiveness against disease dynamics but are difficult and expensive to conduct. IVE studies of symptomatic influenza are important for assessing the impact of vaccination on the burden of disease, since even for those not seeking medical attention influenza is socially and economically burdensome. However, these studies are expensive and time-consuming since they often require active surveillance and testing. Medically attended IVE studies are the most common because they can be the least logistically challenging, but they are prone to bias and often do not/cannot capture the effectiveness that influenza vaccine may have on other population outcomes.
Likewise, IVE studies against hospitalizations and severe outcomes tend to show the highest IVE, but they are subject to selection bias because hospitalized individuals may not be representative of the entire population. It is important to note that studies usually focus on outcomes clearly related to influenza (influenza-like illness, hospitalization for influenza/pneumonia), but ignore the broader impact of influenza and vaccines on outcomes where the role of influenza may be less apparent, such as exacerbations/destabilization of non-communicable diseases (NCDs) (eg, COPD, diabetes) or as a trigger of serious events, such as myocardial infarction or stroke.
To recap, IVE is specific to an outcome, and each outcome assesses a different impact of vaccination. When evaluating IVE, it is important to be mindful of the outcomes reviewed and thus the relevance of the outcome to the population or setting when subsequently communicated.

| Experimental, observational, and hybrid methods
Experimental In observational studies, VE is estimated by rate ratios or hazard ratios of events occurring in vaccinated versus unvaccinated persons over time5. But these methods cannot control for bias to the same degree as prospective randomized clinical trials. 30 However, observational studies can estimate the impact of vaccination programs on the entire exposed population.
In other words, experimental studies answer the binary question 'does the vaccine work?' (yes or no), whereas observational studies address 'how well a vaccine works'.
A hybrid experimental-observational method, referred to as "pragmatic clinical trial" (PCT) 31 has also recently been used to estimate vaccine effectiveness. This study design investigates randomized groups prospectively, but measures endpoints from routinely collected data or vital statistics. The primary advantage of pragmatic/ hybrid clinical trials is that they can be designed to be more reflective of real-world vaccine experience, with research questions (such as outcomes, patients populations, and so on), that are more relevant to policy makers, clinical decision makers, and others as they seek to optimize immunization programs. This design is currently being used to compare the effectiveness against LCI of licensed egg-based inactivated influenza vaccines against two other types of licensed vaccines (cell-culture inactivated and recombinant). 32

| Strengths and weaknesses of study designs
The large number of study designs used to assess IVE underscores the imperfect nature of each.

| The randomized controlled trial (RCT) or Group Randomized Trial (GRT)
The RCT is the only design that controls for selection bias and con-

| Cohort studies
Cohort studies, the gold standard design for estimating incidence rates, relative risks, and attributable risks, proceed in a logical sequence from exposure to outcome. They allow for hypotheses about causality of the exposure on the outcome and can provide an estimate of the VE. However, for influenza disease, at an attack rate of 5 to 10%, the duration and cost of studies can be prohibitive.

| Case-control studies
Case-control studies identify cases (outcome) and then ascertain exposure status. This design is the most appropriate for diseases with low incidence rates or with long duration between exposure and outcome incidence. The main advantages of case-control studies are that they require a smaller sample size than cohort studies, they can be relatively inexpensive, and they can be relatively short in duration. They can also be used to assess VE in real time, from sentinel screening and surveillance. But since case-control studies do not measure incidence rates, VE is estimated from the odds ratio.
When the relative risk of exposure is small (<5%), the odds ratio can approximate the relative risk.
The main weaknesses of case-control studies are that the exposure status is subject to recall bias, and the strength of causal association (odds ratio) is not as not as deterministic as the relative risk.

| Other designs
With large databases of patient registries, electronic health records, insurance data, web/ social media, or records of pharmaceutical products sales, observational data not collected under experimental conditions can be used to estimate "real-world" effectiveness.
Advantages and weaknesses of these estimates are mainly linked to the quality and completeness of the data collected.
A summary of the advantages and disadvantages of main study designs are summarized in Table 3.

| Confounders and other factors that influence IVE
Identifying factors that are strongly associated with vaccination and with disease risk is key to interpreting IVE estimates. At least two sets of factors play an important role in determining the likelihood of IVE against illness: 1) host characteristics (such as age, underlying health conditions, and level of pre-existing immunity to circulating strains of influenza), and 2) vaccine characteristics, including the match between circulating stains of influenza virus and influenza vaccine strains. 33 In years when vaccine strains are mismatched with circulating strains, IVE will be lower. These factors alone can explain most of the observed inter-season variability in benefit. 21 Several factors that confound comparisons between studies must be considered. The main factors that can confound IVE are identified in Table 4. Some of these are discussed below.

| Strain match
IVE is dependent on the match between vaccine strains and circulating strains. IVE will be lower in years when there is a mismatch.

| Virus growth in eggs and cell culturepotential for mutation
Generally, reassortant viruses must be adapted in eggs in order to produce high-yield candidate vaccine viruses. 38 During this process, mutations of influenza viruses may result in altered antigenicity.
While antigenicity is continually tested, it can be difficult to accurately predict and these mutations may affect vaccine match with circulating strains. Vaccines are now available that are not manufac-

| Virus type/subtype
Vaccine effectiveness is viral type/subtype specific. VE is typically highest for H1N1 and lowest for H3N2 (Table 1) to reduced VE. [41][42][43] VE for B types may be similar to H1N1 or slightly lower (see Table 1).

TA B L E 4 Potential factors that can confound IVE
Viral/ epidemiological Strain match-circulating strains can be genetically diverse, differing between seasons, geographic regions, and countries. Antigenic drift can occur after vaccine strain selection.
Potential for adaptive mutations in HA during virus growth and passage in eggs or cell culture.
Type/subtype-specific effectiveness-predominant circulation of a subtype as effectiveness is highest for H1N1 and lowest for H3N2. Transmissibility and virulence of viruses can vary (from mild to severe disease).
Vaccine-type effectiveness-VE is brand-specific as production methods (split virus, sub-unit, etc) and processes (purification, inactivation, etc) differ between manufacturers.
Sampling/ methodological Healthy user effect.
Timing of enrollment into study during influenza season in relationship to vaccine uptake and incidence of influenza.
Accuracy of link between record of immunization in databases and cases.

| Vaccine type/origin
Vaccines are produced by several manufacturers, and none are identical. Differences in production methods (split virus, sub-unit, etc) and processes (purification, inactivation, etc) may mean that the nature of the vaccine itself may impact VE, making VE brandspecific. The use of quadrivalent vaccines versus trivalent vaccine may also affect the IVE estimates since more strains (2 lineages of B) are covered by quadrivalent vaccines. The DRIVE project1 seeks to achieve high quality, brand-specific effectiveness estimates for all influenza vaccines used in the EU each season. However, with frequent use of a number of different influenza vaccines in any given population, few vaccine registries and reliance on patient recall, developing valid brand-specific IVE estimates will be challenging.

| Healthy user effect
One of the main confounders in observational studies is the healthy user effect. This refers to a situation where patients in better health conditions are more likely to adhere to the annual recommendation for influenza vaccination. If not adjusted for comorbidities or health seeking behavior, the healthy user bias can overestimate IVE.
Several approaches to control for this effect, such as by using propensity scores and other means, exist in the literature.

| Waning seroprotection
Post-vaccination antibody titers decline over time, and although titers usually remain at levels above those associated with protection for the duration of an influenza season, 48

| Vaccination history
Vaccination history has become one of the latest confounders to come under scrutiny in test-negative design studies. Vaccine protection from influenza is not binary (all or nothing) but an odds ratio will not account for an incomplete protective effect of vaccination. 55 There are also questions about whether differences in estimates of IVE can be explained by factors like immune or vaccine history. For example, an unvaccinated individual may be afforded some protection from influenza due to prior exposure to influenza or previous vaccination history; thus, estimates of IVE are relative and not absolute measures of effectiveness.

| Accuracy of vaccination history
The accuracy of immunization records from clinical or administrative databases that are associated with cases and comparison groups is critical. Invalid immunization history may compromise the validity of a study, and reliance on patient recall may also confound estimates.

| CON CLUS IONS
While the need to monitor vaccine performance by estimating IVE is recognized, no single methodology is perfect, and the headlinegrabbing point estimates communicated during and at the end of each season hide the considerable public health benefits of vaccination, even in years with modest vaccine effectiveness. Annual studies of IVE have limitations, are difficult to generalize, and, if considered in isolation, will not provide a comprehensive picture of the public health impact of influenza vaccines. However, development and implementation of a common evaluative framework may offer a consistent and objective approach to the assessment of IVE estimates, and help to ensure that the strengths and weaknesses of the data can be appreciated. We recommend that before VE estimates are used to support policy, they should be evaluated against the criteria outlined Table 2.
Using a framework to support the development of vaccine policy will improve understanding of the impact of seasonal vaccination programs and thus may drive much-needed urgency in influenza prevention efforts. Similar frameworks and approaches could also be applied to assess VE for other vaccines (eg, SARS-CoV-2).

ACK N OWLED G EM ENTS
The authors wish to acknowledge Shawn Gilchrist, President of S Gilchrist Consulting Services Inc, for providing his professional writing services in the development of this manuscript.