Validation of Health Economic Models: The Example of EVITA

Authors


Thomas Hammerschmidt, GlaxoSmithKline GmbH & Co. KG, Theresienhoehe 11, D-80339 Munich, Germany. E-mail: Thomas.Hammerschmidt@gsk.com

ABSTRACT

Objectives:  The credibility of models rests on their validity. An age-structured decision analytic model, Economic Varicella VaccInation Tool for Analysis (EVITA), has been developed to examine the epidemiologic and economic effects of universal varicella (chicken pox) vaccination in Germany. EVITA combines a varicella transmission module describing the spread of infection in a population over time with a second module describing the course of disease in case of an infection. Any vaccination strategy can be assessed dependent on coverage levels and targeted age group. Model input data include epidemiologic, clinical, and economic information, which were mainly derived from actual varicella cases (retrospective survey). The objective of this study was to illustrate the efforts undertaken to validate the EVITA model.

Methods:  We assess the descriptive validity, i.e., whether the model provides an adequate picture of the reality and covers all relevant aspects of the spread of varicella and the course of disease. Analyzing the consistency of the model results with observable data does technical verification. Face validity, i.e., the consistency with the underlying theoretical basis of the spread of varicella, is analyzed with respect to results on possible age shifts and elimination of varicella. Tests of corroboration, or convergent validity, are performed by comparisons with other models.

Results:  Without vaccination, the EVITA model predicts undiscounted, indirect costs of €154 million, nearly 40,000 complications and 5,700 hospitalizations per year owing to varicella. These results, especially the distribution of complications and hospitalizations, fit well with population-based survey data. The development of the EVITA model is based on an established epidemiologic model and on real-life data from the survey, ensuring descriptive validity. Results on age shifts and elimination show face validity. Although other models differ considerably with respect to methods applied, the economic results of EVITA, i.e., a benefit–cost ratio of 4.12 when vaccinating young children, lies in the range found in other studies. This underscores its convergent validity. Comparable with other studies, discount rates and price of vaccine proved to be most sensitive variables.

Conclusions:  EVITA provides a powerful tool to simulate the highly complex processes associated with varicella infections and the impact of vaccination. The results of EVITA provide a reliable tool for informed decision making and should enhance the acceptance of such models.

Introduction

Models are increasingly used in the field of health economics. They are developed to assist decision makers in evaluating options and making choices [1]. Modeling allows one to extrapolate beyond the follow-up period of clinical trials to assess longer lasting effects, and it is used to predict costs and consequences of alternative courses of action, e.g., therapeutic choices, based on epidemiologic baseline data [2]. The value of models is not restricted to the results they produce, but they also help to reveal the logical connection between assumptions, inputs, and outputs [3]. Modeling is often unavoidable in economic evaluations as health economists are interested in the cost-effectiveness of applying health technologies under almost real-life conditions rather than solely in efficacy in a controlled clinical trial setting. Modeling is, thus, used to extrapolate beyond data observed in a trial, to link intermediate clinical end points to final outcomes or to generalize results to other settings [4]. A good example for the necessity of modeling is the evaluation of immunization programs where efficacy data such as seroconversion rates provide only limited evidence on the power of vaccination in preventing infectious diseases in a population or community [5]. This is especially true for highly contagious infectious diseases like measles or varicella where vaccination programs can generate indirect, herd immunity effects on the population.

Whereas models are established in a variety of domains, which also involve decisions about health and life such as environmental protection, their use in health care decision making is still controversial [6]. Although the majority of problems with health economic analyses relates to uncertainty in the estimates of comparative clinical efficacy data that are due to unavailability of randomized trials, poor-quality trials, or flawed analyses and interpretation of trials, and not to economic modeling issues [7], there is concern about the credibility of models per se [1]. A decision maker's consideration of model analysis, therefore, rests on the understanding of the model's workings and its logical connections. This understanding, in turn, depends on the transparency in the presentation of a model and its results as well as its overall quality. Quality assessment comprises three criteria: structure, data, and validation [3]. It has been suggested that sections on validation and quality control should be included in any reporting format for modeling studies [8]. A model's structure and data inputs can generally be assessed in peer-reviewed publications. Validation efforts generally form a considerable part of modeling and are carried out often, e.g., for submissions to authorities for approval, pricing, or formulary access. Nevertheless, most publications of modeling studies can and do devote only limited space and attention to illustrating these efforts [9].

The objective of this article is to present validation efforts for a health economic model on varicella (chicken pox) vaccination. This model named EVITA (Economic Varicella VaccInation Tool for Analysis) has been developed to analyze the clinical, epidemiologic, and health economic effects of varicella vaccination strategies in Germany, including the strategy of no universal vaccination. The no-vaccination arm provides insight in the population-based burden of varicella, whereas the model's results on the cost-effectiveness of vaccination strategies can aid decisions on resource allocation. EVITA consists of two modules. The first module is an infectious disease model, which describes the epidemiologic aspects of varicella infection in a population over time. The second module depicts the clinical course of the disease and its economic implications. Both modules, individually and jointly, will be evaluated.

The next section describes the evaluation criteria used to assess the validity of the EVITA model. The following section provides a brief description of the model, which has been published elsewhere [10]. The validity of the model is then assessed accord-ing to the evaluation criteria, and conclusions are drawn.

Evaluation Criteria for Validity Assessment

Several sets of criteria for assessment of a model's validity have been developed [1,3,6,9,11–13] and applied [9]. Although the terminology differs between these sets, and different methodologies have been used to test the criteria, the sets cover mostly the same aspects, but are weighted differently. The criteria used to assess the validity of the EVITA model cover the range of criteria that can be found in the literature.

The first criterion is descriptive validity [12]. Models shall provide a simplified, but adequate picture of reality. A model shall consider all relevant aspects and omit only those aspects that do not alter its results and conclusions significantly. The second criterion is the verification, or the technical validity, of the model [6,9]. Verification includes processes to ensure the model's proper functioning. This can be achieved, for example, by means of debugging. Calibration is an additional step in the verification of a model. Here, the consistency of the model with observable data is tested. The third criterion is face validity [3,6,9]. Face validity is given when a model produces outputs that are consistent with the theoretical basis of the disease and the medical intervention or can be explained intuitively on this basis. These three criteria can be interpreted as tests of the internal validity of a model. The fourth criterion is corroboration or convergent validity. Corroboration is a test of between-model validation and deals with uncertainties in the modeling process. Different independent models addressing the same question should give the same or similar results and, more importantly, lead to the same conclusions. If results or conclusions differ, those differences should be explainable [3,6,9].

The strongest criterion for the validity of a model is the so-called predictive validity, which relates the modeling results to real-life outcomes [3,6,9]. Tests of predictive validity are only possible if the modeled situation is observable, measurable, constant in its structure over time, and constant across variations of conditions not specified in the model. In reality, such a situation is not usually given. Tests of predictive validity are judged to be valuable, but not absolutely essential [3,6]. Sendi et al. [9] performed a test of predictive validity by cross-validation, i.e., by basing their model analysis on a subset of their data and by comparing the outputs of the model with the remainder of the data set. With regard to the EVITA model, such an approach was not feasible, as no data are available on the effects of universal varicella vaccination in Germany. The predictive validity of the EVITA model could, therefore, not be assessed.

Description of the EVITA Model

The decision analytic model EVITA has been developed to compare the potential clinical, epidemiologic, and economic effects of varicella vaccination strategies to no general vaccination. Figure 1 illustrates the model. The model combines two modules. The first module is based on a dynamic infectious disease model that was developed by Halloran et al. [14,15]. This module is an age-structured, deterministic model where the population is divided into groups of susceptible, infectious, and immune individuals. Transition between these groups is calculated with a set of partial differential equations that model the spread of varicella infection over time in the population. The spread of infection depends on the age-specific infection rates, i.e., the age-specific force of infection, combined with the number of susceptible, infectious, and immune individuals in each of the subgroups. Herd immunity effects are considered in this dynamic model as the spread of infection slows as more people become immune. Immunity can be induced either by a previous contact with the pathogen or by successful immunization with a vaccine. In contrast to this approach, a static model would not cover herd immunity effects because the spread of infection solely depends on infection rates and the number of susceptible individuals.

Figure 1.

Simplified model framework.

To adapt the dynamic model to Germany, input data for the spread of infection were collected in a German seroprevalence study, conducted for this purpose, which will be described briefly later in this article [16]. The age structure of the population is drawn from official statistics. Within this module, the effect of different, defined vaccination strategies on the spread of varicella can be modeled for the German population. Differing vaccination coverage rates and age targets for the vaccinees can define vaccination strategies. A strategy of no universal vaccination has also been assessed. Vaccine efficacy was assumed to be 86% based on clinical trial data. Immunity was assumed to wane at a rate of 0.5% per year.

The second module of the EVITA model describes the course of varicella and its potential complications as well as the associated health-care resource utilization (see Fig. 1). Input data on the medical management, epidemiology of varicella complications, hospitalization rates, number of inpatient days, and work days lost were taken from a German epidemiologic survey that was specifically designed to collect the data for the development of this module [17,18]. This survey is also described briefly later in this article.

Outputs from the EVITA model are the age-specific incidence of varicella and related complications over time associated with a chosen vaccination strategy and the costs of varicella as well as benefit–cost ratios (BCR). The BCR is defined as the quotient of the savings induced by the vaccination strategy and the costs of carrying out the vaccination strategy. BCRs above one express net savings and are an indicator for the return of investment of the vaccination strategy. BCRs are widely used in economic evaluations of vaccines. The BCR allows better comparisons across different vaccination programs and different countries than the measure of net benefits of those programs that depend on the number of vaccinees, for instance. The EVITA model has been extensively described elsewhere [10].

Input data of the model were taken mainly from two large epidemiologic surveys [16–18] and were complemented by data from official statistics and the literature. The first, a seroprevalence survey [16], is based on a cross-sectional, age-stratified, representative sample of the German population aged between 0 and 79 years. The sample size was calculated on the basis of previous studies to reach 95% confidence intervals (CIs) with an accuracy of 6% for prevalence rates below 80%, 4% for prevalence rates above 80 and 2% when the prevalence rate is above 95%. Based on this calculation, 4602 sera from two serum banks that have been collected between 1995 and 1999 by the Robert Koch-Institute, Berlin, Germany, were analyzed with an indirect enzyme immunoassay (EIA), Enzygnost anti-VZV/IgG (Dade Behring, Marburg, Germany). Samples with EIA levels below 500 IU/L were tested additionally with an in-house fluorescent antibody to membrane antigen assay, which served as the gold standard in case of discrepant results.

For the second, epidemiologic survey [17,18], a sample size of 1190 cases was calculated on the basis of the results of a pilot study to obtain an accuracy of 1% for the 95% CI of the overall complication rate. A countrywide, representative sample of 1334 unvaccinated varicella cases from 1999 was acquired from randomly selected pediatricians, GPs, and internists. Data have been collected retrospectively from medical files via structured telephone interviews. These data include sociodemographic data such as age at diagnosis, sex, number of siblings, epidemiologic data on the severity of diseases, complications, hospitalizations, and medical resource utilization data like physician contacts, medication, duration of hospitalization and work days lost. The latter can be divided into workdays lost by patients or by parents who are absent from work to care for their sick child. In Germany, sickness funds compensate parents’ loss of earnings by paying so-called “childcare benefits” in line with the Social Security Act. Comparable legal requirements do not exist in other countries. Apart from the epidemiologic data, the indirect costs of varicella were compiled in this survey and were calculated to have amounted to €150 million for the German population in 1999 [17].

Validation of the EVITA Model

This section reports the efforts undertaken to validate the EVITA model and the results of the validation process. We start with the internal validation covering the aspects of descriptive validity, technical verification, and face validity. We then detail the results of the external validation, i.e., the comparison of the EVITA model with other models analyzing varicella vaccination.

Descriptive Validity

The varicella transmission module is based on an established and peer-reviewed model. Thus, it is consistent with accepted theory on the spread of varicella infection in a population and adequately describes the spread of infection over time. The adaptation to German conditions was performed using German data that was specifically collected for the modeling purposes as described above [16]. The second module of the model was developed on the basis of the large, representative survey that provided real-life data from Germany on the course of the disease and related resource utilization [17,18]. The data and the development of this module were reviewed by a German expert panel and deemed to provide an adequate picture of the course of varicella and its clinical management in Germany. Given the acceptance of the data sources and the model structure, the EVITA model fulfills the conditions of descriptive validity.

Verification

Verification relates to the consistency of the model results with known inputs and outputs and the model's proper technical functioning, in particular the absence of coding or calculation errors. The technical functioning was tested by means of an extensive sensitivity analysis. Extreme values of the input variables were used, and the model's actual outputs were compared with expected outcomes. Two examples of these sensitivity analyses illustrate these efforts: For instance, as would have been expected, extreme values like varicella death rates of zero yielded no varicella-related death. Similarly, when assuming zero vaccine efficacy or zero coverage rates the model calculated the same outcomes in the “vaccination” and in the “no-vaccination” arms in line with expectations. As the model calculates both the vaccination and the no-vaccination arms separately, this latter sensitivity analysis is comparable to a double implementation strategy. In such a strategy, a model is independently implemented twice and the results of both versions are checked for their consistency. This analysis was performed in the validation process described by Sendi et al. [9].

Calibration of the model was carried out by calculating the force of infection, i.e., the age-specific annual attack rate of varicella, using the data from the  seroprevalence  study  [16] and  by  comparing the result with the number of cases observed in Germany. The number of varicella cases fluctuates from year to year. The model calculates an average of 739,000 varicella cases per year, with the average being dependent on the length of the observation period. This figure underestimates the mean number of 756,000 cases per year, fluctuating between 737,000  and  778,000  cases,  which  were  observed in Germany during the years 1998 to 2001 [19]. Nevertheless, owing to the natural fluctuations of varicella incidence, the model was assessed to satisfactorily fit reality, as its imputed average number of cases lies within the range of actual observations in Germany and is, if anything, a conservative estimate. The model predicts an average of 39,700 complications per year, 5,700 of which require hospitalization. The distribution of these complications and hospitalizations fits very well with the distribution derived from the epidemiologic survey [17] (see Fig. 2).

Figure 2.

Distribution of complications and hospitalization taken from the epidemiologic survey [17,18] and as calculated by the EVITA model.

According to the epidemiologic survey, about 1 million days of work were lost for both varicella-infected individuals and caregivers in 1999 in Germany, resulting in overall costs of €150 million [17]. The model calculates an average indirect cost of €154 million per year, which also corresponds well to known facts. Overall, the model can, therefore, be successfully verified.

Face Validity

The consistency of the results with the underlying theory can be illustrated by examining the question whether varicella can be eliminated by any of the vaccination strategies. In general, elimination is achieved when the prevalence falls to below a predetermined, very low level [20]. The so-called “critical coverage levels” needed to eliminate varicella are usually cited in the literature to lie between 86 and 91%[21]. Those figures rest on two important issues. The first one is the assumption that the so-called reproduction number is constant and not influenced by the immunization program. The reproduction number is the number of new infections occurring for each single infection in a completely susceptible population. It is defined as the product of the number of contacts of an infectious individual per day, the probability of transmission of a pathogen, and the duration of infectiousness measured in days. The higher the reproduction number, the higher the critical coverage rate for the elimination of an infectious disease [21]. The aim of an immunization program is to lower the reproduction number to below one. Populations are never fully susceptible, either because of immunity caused by the previous contact with the pathogen or by vaccination, if a vaccination program is in place. Furthermore, vaccinated people are less likely to transmit a pathogen, as the duration of their infectiousness may be lowered. Therefore, the real reproduction number in a population differs from the one assumed when determining the critical coverage rate [15]. Second, the age at infection plays a crucial role. The critical coverage rates for varicella elimination of 86% to 91% do not take into account the age structure of infections and the age at vaccination. If vaccinees are immunized earlier than at the median age of infection, the coverage rate necessary to achieve elimination will be lowered [15]. Based on this theoretical consideration, an early vaccination of children may result in a lower required coverage needed to eliminate varicella than the figures cited in the literature. The EVITA model predicts that by vaccinating children aged 15 months a coverage rate of at least 75% would be required to eliminate varicella within 30 years. Figure 3 shows the relationship between the target age of vaccinees and the associated coverage rate necessary to achieve elimination as derived by the model. In Germany, the median age of infection with varicella-zoster virus (VZV) is 5 years [17,18]. Figure 3 shows that, according to the EVITA model, if the target vaccination age is 5 years, coverage rates of about 90% are necessary to eliminate varicella. This result corresponds well to the range of 86% to 91% that is cited in the literature [21]. When vaccinating much earlier in childhood, elimination can be achieved with much lower coverage rates, highlighting that a young age of vaccinees is the most important factor in achieving elimination. EVITA model results are, therefore, in line with theoretical considerations.

Figure 3.

Age of vaccines and minimum required coverage rate to eliminate varicella in Germany within the analytic time horizon of 30 years.

Face validity of the EVITA model can also be illustrated with respect to the possibility of a shift of varicella to older age groups reflected in a higher incidence in adolescents or adults. As varicella infections are more likely to be severe and associated with complications in adolescents and adults, an age shift would be a negative external effect of a vaccination program. The reason for this possible phenomenon can be explained as follows: If only a part of each birth cohort is immunized, the spread of infection will slow, because less individuals are susceptible within a population. As the pathogen remains in the population, susceptible individuals are still likely to become infected, but, given the slowdown in the spread of infection, the infection occurs at a later age. Such an age shift was observed in the Greek population with regard to rubella, because vaccination coverage was lower than 50% for a relatively long period of time [22]. The EVITA model shows that within the first years after the start of an immunization program covering young children,  the  number  of  cases  decreases  in  every age group independently of the coverage rate. Of course, the greatest decrease occurs in the group of preschool children if the age at immunization is 15 months. The finding can be explained by the effects of herd immunity, i.e., the reduced rate of infection in an unvaccinated population owing to the immunization of a part of the population. The pattern of these results is supported by evidence from surveillance of varicella after the introduction of the vaccination program in the United States in the mid-1990s [23]. The age distribution of varicella cases has already changed in the United States and the mean age of infection increased because the reduction in the incidence was more pronounced in children than in adolescents and adults. Nevertheless, this is no age shift in the sense of an increase in the absolute number of cases in any of the older age groups.

When looking further into the future, the picture can be expected to change, as depicted by the EVITA analysis. After 30 years, an increase of the total number of cases over this whole period can be observed in adolescents and adults at coverage rates below 50%. The greatest increase can be expected in the group of 21- to 30-year-old persons (see Fig. 4). These dynamics of varicella can be explained as follows: Today, most adolescents and adults have already contracted varicella and are immune. When vaccinating only a portion of children, these immune adolescents and adults will not be at a higher risk of infection. Therefore, no age shift can be expected in the short run. Nevertheless, in the long run, the group of unvaccinated children develops partly into a group of susceptible adolescents and adults. Thus, only in the long run an age shift can occur, and the model calculates that it will occur only at coverage rates lower than 50%.

Figure 4.

Relative number of cases* over a 30-year period in different age groups. *The relative number is defined as number of cases under a vaccination program divided by the number of cases without a vaccination program, both numbers within the analytic time horizon of 30 years, times 100. A relative number of cases above (below) 100 indicate that more (less) cases occur under a vaccination program compared to the situation without a vaccination program.

The examples of model-derived coverage rates required for varicella elimination as well as the analysis of the threat of an age shift demonstrate that the EVITA model results correspond well to theoretical considerations. Face validity of the varicella transmission model can therefore be judged as good. Overall, based on the criteria of descriptive validity, verification, and face validity, the model fulfills the criteria of internal validity.

Corroboration (Convergent Validity)

Corroboration tests were performed by comparing results of other models with the result of the EVITA model. Because most models are static models and, thus, do not cover effects of herd immunity, elimination of varicella was not reported. This is also true for the dynamic model by Coudeville et al. [24] in which the spread of infection after vaccination is described over a comparable period of time. In this model a different vaccination strategy was analyzed in contrast to the EVITA model. Coudeville et al. vaccinate only a part of the birth cohort in the second life-year while the remaining children are vaccinated during the preschool age (third to sixth life-year). Because early vaccination is crucial for elimination of varicella, one cannot expect elimination in the model by Coudeville et al. under this immunization program. Following effect of the immunization was reported by Coudeville et al.: The prevalence of varicella falls after starting the immunization program, followed by a slight raise to an endemic equilibrium, which is much lower than the equilibrium before starting the immunization program. When analyzing a similar immunization strategy with the EVITA model like the one analyzed by Coudeville et al., the resulting prevalence over time coincides with the one reported by Coudeville et al. (see Fig. 5). This provides evidence for corroboration.

Figure 5.

Projected number of varicella cases in Germany following the introduction of a immunization program: 65% of the children of each birth cohort are vaccinated at the age of 15 months, and 75% of the remaining unvaccinated and still susceptible children of these cohorts are vaccinated at the age of 4 years.

Several health economic models have assessed the costs and benefits of routine varicella vaccination [24–31]. In the following, the corroboration of the EVITA model is assessed by a comparison with these models. The models differ with respect to the methods that have been applied and with respect to the vaccination strategies. Most studies applied static modeling, i.e., the force of infection is assumed to be constant such that no effects of herd immunity occur. It was argued that dynamic models covering herd immunity effects are more suitable for evaluations of vaccination programs [5]. Only the models in the studies by Coudeville et al. [24], Brisson et al. [30], and Lieu et al. [31] used dynamic modeling like the EVITA model. In most studies, the analytic time horizon for the vaccination program was 30 years, in line with EVITA. Exceptions are the studies by Diez-Domingo et al. [27], Scuffham et al. [28], and Beutels et al. [26] with 20, 25, and 70 years, respectively. Future costs were discounted at a rate of 5% in all but the study of Brisson et al. [30] in which a discount rate of 3% was used. Table 1 provides an overview on the vaccination strategies, the assumption on the vaccine efficacy, and the BCRs. From a societal perspective, BCRs range from 1.61 to 5.40, reflecting net-savings. The result of the EVITA model fits well in this range. Differences in the magnitude of the results can be explained by differences in the assumptions made as well as the epidemiologic situations and the organization of the health-care systems in the countries studied.

Table 1.  Varicella vaccination models
StudyCountryAge of
vaccinees
Coverage (%)Efficacy (%)BCR
Societal
perspective
Payers’
perspective
  • *

    Maximum to be reached after 5 years, starting at 30% in the first year.

  • Abbreviation: BCR, benefit–cost ratio.

EVITA [10]Germany12–18 months  85*864.121.75
Huse [25]United States12–18 months100952.38NA
Beutels [26]Germany12–18 months 70904.600.82
Diez-Domingo [27]Spain12–18 months 95901.610.54
Scuffham [28]New Zealand12–18 months 80952.790.67
Scuffham [29]Australia12 months 8095NA0.25
Brisson [30]Canada1 year 90935.240.61
Lieu [31]United Statesup to 6 years 97905.400.90

When examining BCRs from a payer's perspective, differences in health-care systems produce even more differentiated results, with the EVITA model generating the highest BCR. This is caused by the fact that in Germany, indirect costs are partly covered by the sickness funds, i.e., parents’ absence from work to care for a sick child is paid for by the sickness funds and not by the employer. In the other countries that were analyzed in the studies, no such regulation exists resulting in lower costs to payers. Even the study by Beutels et al. [26] that also evaluates varicella vaccination in Germany did not take these childcare benefits into account.

When focusing only on directs costs to payers without the “childcare benefits,” the BCR would be 0.74 and is, thus, in the range reported in the other studies. The difference in results from the payers’ perspective can therefore be explained by the specific legal situation in Germany.

Generally, there is a high consistency between the EVITA model [10] and the results of the sensitivity analyses reported in the other studies with regard to the influential variables, i.e., discount rates, costs of vaccine and costs of work loss, as well as less sensitive variables, i.e., efficacy and waning of immunity.  Overall,  these  considerations  illustrate  that the model demonstrates a good convergent validity with the other models that have been used to evaluate varicella vaccination programs.

Conclusions

Health economic models are intended to assist health-care decision-making. The acceptance of models is highly dependent on their validity. The process of validation is not usually described extensively in scientific publications of models. In this article, we reported on the validation process undertaken in the case of the EVITA model that has been developed to analyze the effects of varicella vaccination programs in Germany with the aim of assisting decision making on future vaccination strategies. Based on the criteria of descriptive validity, face validity, verification, and corroboration, also known as convergent validity, we provided evidence that the EVITA model can be judged to be a valid model. The EVITA model provides a powerful tool to simulate the highly complex process associated with varicella infections and the impact of vaccination strategies. The results of the EVITA model provide a sound basis for scenario analysis and informed decision making and should further increase the acceptance of such models by health care decision makers.

This study was supported by a grant from GlaxoSmithKline.

Ancillary