Background. The framework outlined above is illustrated using a stylized example which is based on the NICE appraisal of orlistat in the treatment of obesity undertaken in 2001 . The guidance imposed restrictions and conditions on continued use in terms of minimum body mass index and weight loss using dietary control. In addition, it required that the patients should only continue with therapy beyond 3 months if they lose at least 5% of body weight, and only continue beyond 6 months if they lose at least 10% of body weight. The guidance suggests that patients are not expected to be on therapy beyond 12 months.
The appraisal of orlistat and its guidance was based on the independent assessment report  which identified 14 randomized controlled trials (RCTs) and two economic models. None of the RCTs measured changes in HRQL or utilities, resource use, regain in weight after cessation of treatment or any longer-term consequences for mortality of morbidity. Clearly, RCT evidence alone was not sufficient to establish the cost-effectiveness of this technology, and there were a number of extrapolations and generalizations which needed to be made. These included establishing how many patients continue with treatment, which effects both outcomes to patients and resource use, how changes in body weight translate into changes in HRQL and some assessment of the long-term impact on weight, HRQL and mortality and morbidity after 12 months of treatment.
Decision model. The structure of this decision problem can be represented as a simple decision tree (Fig. 1). The structure compares orlistat with dietary management and reflects the nature of the NICE guidance outlined above. It involves two chance nodes: the probability of greater than 5% weight loss at 3 months and, if this is achieved, the probability of greater than 10% weight loss at 6 months. The key end points are weight loss compared with dietary control, changes in HRQL given weight loss and the additional cost of 3, 6, and 12 months of treatment. In this simple case study, long-term mortality and morbidity, and the possibility of sustained weight loss, have not been modeled. This is in order, first, to simplify the example and, second, because no evidence seems to exist to support these effects which were not considered credible by NICE. The RCT evidence is a crucial source for parameter estimates for the model, particularly the magnitude of treatment effects (weight loss and probability of weight loss at 3 and 6 months). Other sources of data, however, have been taken from nontrial sources detailed in the assessment report . This applies, for example, to changes in HRQL and the resource implications of weight loss.
All decisions about the cost-effectiveness of interventions are based on uncertain information. The extent of the evidence available, for each of the inputs, can be reflected in probability distributions assigned to these estimates, where less information and more uncertainty about an input will be reflected in greater variance from the distribution assigned. The quality and exchangeability or relevance of the evidence available may also be represented by linking the uncertain estimate to model inputs through additional uncertain parameters which can represent potential bias or exchangeability. These may be based on evidence of the potential bias of alternative designs or on expert judgment. Without access to patient-level data, these distributions are assigned based on secondary sources (e.g., published literature reviewed in the assessment report ) and some judgment about which type of distribution would be appropriate . The parameter estimates, their distributions and data sources are described in Table 1. In general, the quality of this evidence is very low, and the shape of the distributions assigned to these parameters attempts to represent the substantial uncertainty surrounding these estimates. For an overview of the process of selecting and fitting probability distributions in decision models, see Briggs et al. (2002) .
Table 1. Data inputs for the orlistat case study
|Input parameter||Distribution used to characterize parameter uncertainty||Source|
|Probability of 5% weight loss at 3 months||Beta (286, 214)||Meta-analysis of trials with 3-month follow-up (n = 500) |
|Probability of 10% weight loss at 6 months||Beta (170, 230)||Meta-analysis of trials with 6-month follow-up (n = 500) |
|Weight loss at 12 months||Normal (95% CI 2.19–3.69)||Meta-analysis of trials with 12-month follow-up (n = 548) |
|Health value (utility) gain per 10 kg weight loss||Log-normal (95% CI 0.0767–0.26)|||
|Total costs per annum||Log-normal (95% CI £554–£887)|||
Results of the case study—the adoption decision. The model indicates that orlistat is more effective but more costly than dietary control alone, with an incremental cost per additional QALY of £21,400. Hence if the decision-maker's threshold willing to pay is more than £21,400 per QALY, orlistat should be adopted given existing evidence. Nevertheless, there is uncertainty in cost-effectiveness, and this is shown in the cost-effectiveness acceptability curve (CEAC) in Figure 2. Detailed descriptions of the derivation and interpretation of CEACs are available elsewhere [25–28] In brief, it shows the proportion of the simulations in which (i.e., the probability that) orlistat is considered cost-effective for a given maximum willingness to pay on the part of the decision maker. That is, the proportion of simulations in which the orlistat has an ICER which is less then the maximum willingness to pay. One minus this probability reflects the decision uncertainty around adoption. That is, the probability that, in adopting orlistat on current evidence, a “wrong” decision would have been made. The figure shows that, unless the cost-effectiveness threshold is very high, there will be substantial decision uncertainty surrounding this decision to adopt. For example, at a threshold willingness to pay of £30,000, the probability that orlistat is cost-effective is 0.758, giving an error probability of 0.242. Although this probability is strictly Bayesian, it is possible to interpret this in terms of a conventional (“frequentist”) P value on a one-tailed test on a null hypothesis of no difference in expected cost-effectiveness . As such, this probability is much greater than the traditional rules of inference and statistical significance of 0.05 or 0.1.
Results of the case study—the decision about further research. How can this error probability be interpreted? If the wrong decision about adoption is made, there will be costs in terms of health benefits and resources forgone. Therefore, the expected cost of uncertainty is determined jointly by the probability that a decision based on existing information will be wrong and the consequences of a wrong decision. The expected costs of uncertainty can be interpreted as the expected value of perfect information (EVPI) because perfect information can eliminate the possibility of making the wrong decision. This is also the maximum that the health-care system should be willing to pay for additional evidence to inform this decision in the future, and it places an upper bound on the value of conducting further research [10,16].
More formally, EVPI is simply the difference between the payoff with perfect and current information. The payoff can be seen in terms of expected net benefit—for example, expected net monetary benefit which, for a given option, is: (expected QALYs × λ) − expected costs, where λ is the decision maker's threshold willingness to pay . More specifically, if there are two alternative interventions (j = 1, 2), with unknown parameters θ then, given the existing evidence, the optimal decision is the intervention that generates the maximum expected net benefit [maxj Eθ NB(j, θ)]. This is the maximum net benefits over all the iterations from the Monte Carlo simulation, because each iteration represents a possible future realization of the existing uncertainty (a possible value of θ). With perfect information, we would know how the uncertainties would resolve, which value θ will take, before making a decision and could select the intervention that maximizes the net benefit given a particular value of θ[maxj NB(j, θ)]. Nevertheless, the true values of θ are unknown; we don’t know which value θ will take. Therefore, the expected value of a decision taken with perfect information is found by averaging these maximum net benefits over the distribution of θ[Eθ maxj NB(j, θ)]. The EVPI for an individual patient is simply the difference between the expected value of the decision made with perfect information about the uncertain parameters θ, and the decision made on the basis of existing evidence:
- EVPI = Eθ maxj NB(j, θ) − maxj Eθ B(j, θ)(1)
This provides the EVPI surrounding the decision problem for each time this decision is made and for an individual patient or individual episode. Nevertheless, once information is generated to inform the decision for an individual patient or a patient episode, then it is also available to inform the management of all other current and future patients. If this “population” EVPI exceeds the expected costs of additional research, then it is potentially cost-effective to conduct further research, current evidence is not sufficient and additional research should be undertaken.
Figure 3 illustrates the population EVPI for the orlistat guidance. At a cost-effectiveness threshold of £30,000, the population EVPI is just more than £1.5 m. This may well exceed the costs of further investigation and suggests that further research is needed to support the adoption of orlistat. When the threshold for cost-effectiveness—maximum value of health outcome—is low (much less than £21,400), the technology is not expected to be cost-effective and additional information is unlikely to change that decision (the EVPI is low). Similarly, when the threshold willingness to pay is higher (i.e., much higher than £21,400), the ICER is much lower than the threshold, oralist would be considered cost-effective in terms of expected costs and QALYs and this decision is unlikely to be changed by further research. In this case the population EVPI reaches a maximum when the threshold is equal to the expected ICER; that is, where there is most uncertainty about whether to adopt or to reject orlistat based on existing evidence. Nevertheless, EVPI does not always reach a maximum at this point. This is because, although the probability of error falls as the threshold increases, the value of changing the decision (the cost of error) also increases, so the maximum point is determined by the balance of these two factors.
The value of reducing the uncertainty surrounding individual input parameters in the decision model can also be established. This type of analysis can be used to focus further research by identifying those inputs for which more precise estimates would be most valuable. In some circumstances, this will indicate which end points should be included in further experimental research; in others, it may focus research on inputs which may not necessarily require experimental design and can be provided relatively quickly.
This analysis of the value of information associated with each of the model inputs (parameter EVPI) is, in principle, conducted in a very similar way to the EVPI for the decision as a whole [29,30]. The EVPI for a parameter or group of parameters (ϕ) is again simply the difference between the expected net benefit with perfect information about the parameter(s) ϕ and the expected value with current information. The expected value with current information is the same as before [maxj Eθ NB(j, θ)]. With perfect information, the decision maker would know how the uncertainties about ϕ would resolve (which value ϕ will take) before making a decision and could select the intervention that maximizes expected net benefit, which must now be calculated over all the other remaining uncertain parameters (ψ) the model [maxj Eψ|ϕ NB(j, ϕ, ψ)]. As before, the true value of ϕ is unknown so these maximum expected net benefits must be averaged over the distribution of ϕ[Eϕ maxj Eψ|ϕ NB(j, ϕ, ψ)]. The EVPI for ϕ is the difference between the expected net benefit with perfect information about ϕ and the expected value with current information:
- EVPI for ϕ= Eϕ maxj Eψ|ϕ NB(j, ϕ, ψ) − maxj Eθ NB(j, θ)(2)
This does require substantial additional computation for models where the relationship between the model's inputs and expected cost and outcomes is not linear, for example, in Markov models [19,30]. It should also be noted that, in general, the EVPIs for individual model inputs will not sum to the EVPI for the decision problem. This is because both decision and parameter EVPI depend entirely on whether additional research would be predicted to change the decision about the preferred option. In the simulation process undertaken to estimate this, if a value of a specific parameter is drawn some distance from its mean, it may be insufficient in itself to change the decision. Nevertheless, when that value is drawn together with similar extreme values for other parameters, this combination may well be enough to change the decision. So there is no simple relationship between individual parameter and decision EVPI.
Figure 4 illustrates the EVPIs for individual parameters associated with the overall population EVPI at a cost-effectiveness threshold of £21,400. In this example, it is the EVPI associated with the changes in HRQL, due to modification in body weight, which is highest. This should not be surprising as there was limited evidence to link changes in weight to HRQL, but it is this relationship which is crucial to establishing the cost-effectiveness of orlistat. The EVPIs associated with resource use parameters are also relatively high for the same reasons. Although the EVPI analysis in Figure 3 suggests that further research may be required to support the adoption of orlistat, the analysis of the EVPIs for individual parameters indicates that this may not need to have an experimental design. This is because more precise estimates of HRQL changes and elements of resource use can be established without an additional clinical trial and could be based on an observational survey.
There remains, however, substantial value of information associated with the expected loss in body weight at 12 months, and more precise estimates of this input would require experimental design. Its also interesting to note that the probability of remaining on treatment is not associated with the highest values of information. This is partly because substantial evidence from the previous trials exists already. The other reason is that, when patients come off treatment, the potential gains in HRQL are lost, but these are offset by reduction in the intervention costs. It should be noted that the relative value of information associated with model inputs will also change with the cost-effectiveness threshold. Specifically, those inputs which are more closely related to differences in expected costs will be relatively more important at low threshold values, and those more closely related to differences in outcomes will be more important at high values.
The case study highlights the fact that economic considerations are central, not only to establishing how much evidence is required to support the adoption of a technology, but also what type of evidence will be required and the appropriate research design. Given an objective to maximize health gain from limited resources, this framework demonstrates that, for a particular technology, the amount and type of evidence required depends on decision uncertainty and economic decision rules, rather than on rules of statistical significance applied to the trial end point. It is also clear that different amounts and types of evidence will be required for different types of technology relevant to different patient populations.