WHO guide on the economic evaluation of influenza vaccination

Abstract Influenza is responsible for substantial morbidity and mortality across the globe, with a large share of the total disease burden occurring in low‐ and middle‐income countries (LMICs). There have been relatively few economic evaluations assessing the value of seasonal influenza vaccination in LMICs. The purpose of this guide is to outline the key theoretical concepts and best practice in methodologies and to provide guidance on the economic evaluation of influenza vaccination in LMICs. It outlines many of the influenza vaccine‐specific challenges and should help to provide a framework for future evaluations in the area to build upon.


| INTRODUCTION
Influenza is responsible for substantial morbidity and mortality across the globe, with a large share of the total disease burden occurring in low-and middle-income countries (LMICs). 1 The costeffectiveness of seasonal influenza vaccination programmes has been widely assessed in high-income countries. 2,3 The value for money estimated for programmes targeted at children, 4 the elderly 5 and those at high risk of infection and/or severe complications 6 has been most favourable, whereas results for healthy adults have been less consistent. 7 A recent literature review on the topic found that in LMICs relatively few economic evaluations have assessed the value of seasonal influenza vaccination. 8 Nine economic evaluations were identified in middle-income countries, with none identified from low-income countries. 8 The review found important methodological limitations in several studies and called for greater standardization of methods for economic evaluation of influenza vaccination, and thus for the need of global guidance on the economic evaluation of influenza vaccination in LMICs.
In response, WHO commissioned the Guidance on the economic evaluation of influenza vaccination, 9 which outlines the key theoretical concepts and best methodological practice, aiming to offer high-level guidance on influenza vaccination assessment in LMICs (see Box 1).
The guide is aligned to existing vaccine introduction guidance 10 and key documents for the assessment of influenza vaccination (Table 1).

| ESTIMATING THE DISEASE BURDEN AND ASSOCIATED HEALTHCARE USE
Estimating the disease burden from influenza using routinely collected data can be challenging. This is the case even in high-income countries with comprehensive surveillance networks and national electronic healthcare records (eg, for hospitalization episodes). One major reason for this is that laboratory confirmation is not routinely requested in suspected influenza cases. Estimation is further complicated because patients may present with secondary complications potentially triggered by influenza (eg, acute myocardial infarction). 11 WHO's Manual for estimating disease burden associated with seasonal influenza 12 outlines various methods that can be applied in LMICs to evaluate the disease burden attributable to influenza. However, other sources of data will be required to estimate the full range of influenza disease burden (see Figure 1). Using the definitions set out in this manual, the estimated disease burden is divided into 2 main categories: (i) influenza-associated ILI, which represents an estimate of the outpatient/primary care clinic visits due to influenza illness, and (ii) influenza-associated severe acute respiratory infections (SARI), which represents an estimate of the hospitalization visits due to influenza illness. Both categories use laboratory confirmation (on at least a subset of cases) to estimate the proportion of events suspected to be due to influenza. Evaluating mortality in SARI cases can also be used to estimate the case fatality rate in hospitalized influenza-positive cases. 12 However, such an estimate is likely to be a conservative as some deaths from influenza infection will not occur in a hospital setting.
An alternative approach is to use statistical modelling techniques to estimate the influenza-attributable burden. [13][14][15] These methods involve time series analyses of non-specific disease outcomes, such as respiratory deaths, to estimate a non-influenza baseline burden above which any excess disease may be considered attributable to influenza. While these methods are a useful way to estimate influenzaattributable burden, they have specific data requirements (eg, complete and accurate data on the non-specific disease outcomes), can involve relatively complex technical analysis and may be more difficult to apply in (sub)tropical regions where influenza does not always show a clear seasonal pattern of circulation.
Year-to-year variation in the influenza disease burden should be considered in the analysis. This variation is due to changes in the circulating virus over time which can impact on influenza virus T A B L E 1 WHO documents and tools that may be relevant to the different subsections of an economic evaluation of influenza vaccination • These data will not be avaliable from the manual. Estimations of incidence may need to be based on population influenza attack rates (excluding medically attended cases).

Non-medically attended influenza burden
• Potential to use "influenza-associated ILI". However, estimation of incidence is possible only if denominator data are avaliable. Data on the catchment area are often unavailable.
Outpatient (primary care) influenza burden • Potential to use "influenza-associated SARI". It is often possible to estimate the catchment area for hospitals which can then be used to help estimate incidence.

Inpatient (hospitalized) influenza burden
• Potential to estimate a case fatality rate for hospitalized cases identified. However, the avaliable data may be incomplete and will not capture deaths in the community.

Influenza mortality burden
transmissibility and virulence. 4 It is suggested that data from at least 5 years are used to estimate the existing influenza disease burden 12 ; where not available, a minimum of a single calendar year can serve as a starting point provided that appropriate caution is taken when interpreting the results. 12 In all cases, but particularly when dealing with imperfect data, care should be taken to conduct an appropriate sensitivity analysis across a range of plausible values (see Section 8).
Age must also be considered when estimating influenza disease burden, with age-specific rates used whenever possible.
Non-medically attended influenza cases have been found to be influential in many economic evaluations of influenza vaccination in high-income settings. 2  • Ideally, at least 5 years of data should be used to estimate the existing influenza disease burden. 12 However, a shorter period (minimum of a single calendar year) can serve as a starting point.
• WHO's A manual for estimating disease burden associated with seasonal influenza 12 can be used to estimate some key outcomes, but further sources are required (eg, to estimate non-medically attended influenza).

Economic burden
• Evaluations should ideally adopt a societal perspective, including all relevant costs and consequences irrespective of who incurs them. However, costs borne by different entities should be reported separately where possible. 17 • If productivity costs are included, they should be reported separately from other costs and cost-effectiveness results should be presented with and without indirect costs.
Programme costs • The vaccine administration strategy should be carefully considered and outlined in detail. Key choices include who administers the vaccine, in what setting and whether it is delivered opportunistically or at a separate encounter.
• Where possible, estimates of adverse events following immunization (AEFI) should be included in economic evaluations of influenza vaccination.
Programme impact • Efficacy against confirmed influenza disease from a meta-analysis will often be the most appropriate estimate. This can be applied to all estimates of influenza-specific outcomes (eg, influenza death).
Modelling approach • Table 2 summarizes when to consider each modelling approach. Discounting/horizon • Costs and effects should be discounted at the appropriate level indicated in relevant guidelines for the setting under evaluation, but results should also be reported applying WHO-CHOICE recommendations in sensitivity analysis.
• A 1-year time horizon may be appropriate in most cases; however, long-term consequences from prevented influenza mortality that occur outside the 1-year time frame must be fully incorporated into model results.

Results/presentation
• In most cases, strategies for each different target group (eg, pregnant women) should be compared only to alternative strategies for that group.
• Total costs and outcomes should be presented for each strategy, as well as the incremental results comparing the strategies. These results should be further disaggregated to show the factors driving the results.

Uncertainty
• Key types of uncertainty, including parameter, methodological and structural, should be explored. Interyear variation also needs to be considered. 33 Other recommendations • Consideration should be given to specific issues that may arise when evaluating a particular population subgroup (see 2 for a summary of potential issues).
an understanding of their impact on evaluation results. Estimates of the total symptomatic attack rate can be used to calculate the nonmedically attended symptomatic disease rate by excluding estimates of medically attended cases. 12 It is generally recommended that a cost-utility approach be used for economic evaluations. 17 All of the cost-utility analyses identified in LMICs to date have used a quality-adjusted life years (QALYs) framework. 8 However, in some cases, disability-adjusted life years (DALYs) may be a more appropriate outcome measure than QALYs for LMICs as estimates may be more consistently available across all countries. 17 Local guidelines and the availability and/or transferability of qualityof-life weights to the setting under evaluation should help inform these decisions. Careful thought must also be given to whether DALYs or QALYs are the more appropriate measure to value (uncomplicated) acute influenza illness.  Severe adverse events following immunization (AEFI) for influenza vaccination are very rare and mild adverse events usually resolve after a short period of time. [23][24][25][26] Due to the relatively minor consequences from these events, economic evaluations have not always included AEFI and, if included, they have generally not been found to be influential in determining cost-effectiveness. 4

| ASSESSING THE IMPACT OF VACCINATION EFFORTS
Influenza vaccine efficacy estimates should generally be obtained from randomized clinical trial evidence, ideally from meta-analyses that appropriately synthesize all the relevant available data rather than from a single vaccine trial. 2 Estimates of efficacy should incorporate multiple influenza seasons because of the year-to-year variation in vaccine match as well as in transmissibility and prior immunity in the population. 4 Efficacy estimates may differ depending on the population group being targeted, for example by age and for those with underlying medical conditions. [22][23][24] Distinctions should also be made between the different types of influenza vaccines (eg, between LAIV and TIV, and adjuvanted and quadrivalent vaccines). There may also be differences in vaccine efficacy between regions. 29 Efficacy estimates against non-specific outcomes have sometimes been used in economic evaluations of influenza vaccination programmes, 2,4 but the most straightforward estimate is efficacy against confirmed influenza disease. It is reasonable to apply a vaccine efficacy calculated against confirmed influenza disease to all estimates of influenza-specific outcomes, including influenza hospitalizations and influenza deaths. This involves simplifying assumptions, for example, that the prevention of influenza infection will prevent all subsequent disease outcomes from this infection. Efficacy estimates specifically against severe influenza outcomes cannot easily be measured in clinical trials because of the large sample sizes required to detect an adequate number of events. 30 The probability of vaccine match to the circulating strains must be accounted for when evaluating the cost-effectiveness of influenza vaccination strategies. While the vaccine may match in any given year, over the longer term the match to the predominate strains will not always be successful. 31 The vaccine match is important as it has been shown that a poor match will reduce the efficacy of the vaccine to prevent influenza illness. [23][24][25] Economic evaluations can apply a vaccine efficacy estimate from a meta-analysis that already incorporates both matched and poorly matched seasons (eg, they may apply a single estimate of vaccine efficacy derived from trials run over multiple years).
In many cases, this method may be a reasonable approach; however, it can be problematic when using modelling techniques that seek to estimate herd protection effects. 32 Another important factor that impacts population protection from influenza vaccination is the vaccine uptake in the targeted groups.
The degree of uptake impacts the total direct protection and may also (depending on those targeted) have a substantial impact on any indirect protection of the community through herd protection. In models not incorporating herd protection, uptake will have an impact on the absolute benefits of vaccination and the total cost of the programme but the impact on the cost-effectiveness ratio may not always be as substantial. This is because the costs of the vaccination programme and the health benefits that accrue through the programme may both increase (approximately) proportionally with the uptake. 17 However, with substantial fixed programme costs or economies of scale (eg, when purchasing large orders), obtaining higher coverage may improve the cost-effectiveness of the programme. When herd protection is modelled, vaccination uptake can become more influential in determining cost-effectiveness 36 (see Section 6).

| ALTERNATIVE MODELLING APPROACHES
The simplest approach to evaluate the cost-effectiveness of influenza vaccination is to apply a "decision tree" model ( Table 2). In these models, each pathway through the "tree" represents a sequence of events and is associated with costs and consequences. 37 Decision trees are often used when the costs and consequences of an intervention occur over a short period of time, as is the case for influenza vaccination.
This is because decision tree models cannot explicitly account for time. However, as is discussed in Section 7, this may not be essential in influenza models as the impact of long-term consequences from mortality can be incorporated through a discounted pay-off attached to specific endpoints where required.
In most circumstances, "Markov" state-transition models with a static (fixed) force of infection irrespective of the proportion of the population that is infectious have limited advantages over decision tree models in the context of influenza evaluations. This type of statetransition model allows for time-or age-dependent transition probabilities to be specified and is therefore often appropriate when costs and consequences occur over an extended period (eg, as in chronic diseases). 37 However, the duration of influenza vaccine protection is typically modelled as lasting only for a single season because of strain changes that occur from season to season. There are some situations where this type of model may be advantageous; for example, it can be used to explore alternative options for the timing of vaccination, where vaccination uptake can be modelled as a gradual process.
Dynamic transmission models can incorporate herd protection into economic evaluations by having the risk (force) of infection vary (being dynamic rather than static) on the basis of the proportion of the population that is infectious over time. 38,39 These models are increasingly used in economic evaluations of influenza vaccination 4 ; for example, recently a dynamic model was used to assess the cost-effectiveness of influenza vaccination of children in Thailand. 40 However, dynamic models are often more complex, time-consuming and costly to produce than static models and have additional data requirements (eg, information on contact patterns between individuals). 32 As such, these models may not be the most appropriate choice for evaluations in LMICs in all circumstances.
Dynamic transmission models are most applicable when evaluating programmes targeting a substantial proportion of those responsible for influenza transmission (eg, vaccinating all eligible children 34 ) and are less likely to be required when evaluating programmes that are less likely to result in substantial herd effects (eg, when targeted on relatively small population subgroups). In most cases, the use of static models will bias an analysis towards conservative estimates of costeffectiveness. 17 The WHO guide for standardization of economic evaluations of immunization programmes provides an informative decision chart to identify what type of model may be appropriate in different circumstances. 17 This guide also provides important information on model validation and collaboration. 17 Economic evaluations alongside clinical trials provide another avenue to assess the cost-effectiveness of influenza vaccination strategies. 8 However, there may be several important limitations to this approach 2,4 (see Table 2).

| DISCOUNTING AND ANALYTICAL HORIZON
The majority of the costs and consequences resulting from influenza vaccination occur within a single year, making discounting less influential compared to vaccination programmes with a longer delay between upfront costs of the vaccination programme and the ben-  should also be applied.
The analytical horizon for economic evaluations should be long enough to account for differences in costs and consequences between the various strategies being evaluated 37 (eg, between "no influenza vaccination in group X" and "influenza vaccination targeted at group X"). As the majority of costs and consequences resulting from influenza vaccination occur in a single year, a 1-year time horizon may be appropriate for the economic evaluation. However, it is important that the long-term consequences from prevented influenza mortality that occur outside of this single-year time frame are fully incorporated into model results. One simple way to do this is to apply discounted pay-off/s in the model that incorporate the full benefits of prevented influenza mortality. It should be noted that longer time horizons are often required in more complex modelling approaches (Section 6), such as those which follow populations through time to account for the build-up of immunity and herd protection.

| ESTIMATING AND PRESENTING RESULTS OF THE ECONOMIC EVALUATION
The influenza strategy being considered for implementation should be compared to an appropriate alternative. For example, the alternative for comparison may be the costs and consequences of "no vaccination" (i.e do nothing) for this group. This will allow an incremental cost-effectiveness ratio (ICER) to be calculated, representing the difference in costs between the alternatives divided by the difference in health outcomes. 37 However, there may be more than 2 strategies that should be considered for this target group; for instance, one may also want to consider immunization with an alternate vaccine (eg, LAIV or TIV) in this group. The ICERs should then be calculated by comparing each strategy to the next best alternative, after excluding dominated strategies (see 17 for detailed advice on this process). Only strategies that are mutually exclusive would generally be considered as comparators within a single economic evaluation. 37 In most situations in LMICs, a practical approach to economic evaluation is to treat each target group as independent (eg, pregnant women, children aged 6 months to 2 years, children aged 2-5 years). While cost-effectiveness ratios are undoubtedly informative in assessing value for money, countries should be encouraged to develop a context-specific process for decision-making that is supported by legislation, has stakeholder buy-in and is transparent, consistent and fair.

| ASSESSMENT OF UNCERTAINTY AND INTERYEAR VARIABILITY
It is vital to assess and appropriately present uncertainty when estimating the cost-effectiveness of influenza vaccination programmes.
Uncertainty in influenza models can be placed into 3 main categories: parameter, methodological and structural (see 43 for a detailed discussion of these categories). Parameter uncertainty is the most frequently discussed form of uncertainty. 43  Alongside uncertainty regarding the true average (numerical) value of parameters in influenza vaccination models, there is also variation between influenza seasons in many values. This variation results from the interyear variation in influenza virus transmissibility, virulence, prior immunity and vaccine match. 4 However, as economic evaluations generally seek to make decisions on whether to implement vaccination programmes for several years, one simple approach is to use of the average input values calculated from data collected over several years as the base-case value (eg, for hospitalization rates). The most appropriate approach to use should be carefully considered and decisionmakers should be aware that cost-effectiveness in any given year may vary substantially.

| CONCLUSIONS
Influenza vaccination strategies in LMICs offer substantial scope to reduce both morbidity and mortality. However, there are currently few published economic evaluations for LMICs that can help decisionmakers understand the value for money that may be offered by different influenza vaccination strategies. 8 As many economic evaluations have been conducted in high-income countries, 2 some of the lessons learned through this process can help to inform future evaluations for LMICs. However, there are important differences that also need to be taken into account when assessing vaccination programmes for LMICs. This guide has outlined many of the influenza vaccine-specific challenges and should help to provide a framework for future evaluations in the area to build upon.