### ABSTRACT

- Top of page
- ABSTRACT
- Introduction
- Methods
- Results
- Discussion
- References

**Objective: ** Value of information (VOI) analysis informs decision-makers about the expected value of conducting more research to support a decision. This expected value of (partial) perfect information (EV(P)PI) can be estimated by simultaneously eliminating uncertainty on all (or some) parameters involved in model-based decision-making. This study aimed to calculate the EVPPI, before and after collecting additional information on the parameter of a probabilistic Markov model with the highest EVPPI.

**Methods: ** The model assessed the 5-year costs per quality-adjusted life year (QALY) of three bronchodilators in chronic obstructive pulmonary disease (COPD). It had identified tiotropium as the bronchodilator with the highest expected net benefit. Total EVPI was estimated plus the EVPPIs for four groups of parameters: 1) transition probabilities between COPD severity stages; 2) exacerbation probabilities; 3) utility weights; and 4) costs. Partial EVPI analyses were performed using one-level and two-level sampling algorithms.

**Results: ** Before additional research, the total EVPI was €1985 per patient at a threshold value of €20,000 per QALY. EVPPIs were €1081 for utilities, €724 for transition probabilities, and relatively small for exacerbation probabilities and costs. A large study was performed to obtain more precise EQ-5D utilities by COPD severity stages. After using posterior utilities, the EVPPI for utilities decreased to almost zero. The total EVPI for the updated model was reduced to €1037. With an EVPPI of €856, transition probabilities were now the single most important parameter contributing to the EVPI.

**Conclusions: ** This VOI analysis clearly identified parameters for which additional research is most worthwhile. After conducting additional research on the most important parameter, i.e., the utilities, total EVPI was substantially reduced.

### Introduction

- Top of page
- ABSTRACT
- Introduction
- Methods
- Results
- Discussion
- References

Decision-analytic models are commonly used to analyze the costs and cost-effectiveness of pharmaceuticals. Originally, these models were mostly deterministic and only considered uncertainty around model parameters in sensitivity analyses. In later years, these models developed into probabilistic models in which uncertainty around input parameters was considered simultaneously by entering prespecified distributions for these parameters [1,2]. These probabilistic models allowed displaying the resulting uncertainty around costs and effects on cost-effectiveness planes and by means of cost-effectiveness acceptability curves (CEACs) [3] and frontiers [4]. Recently, value of information (VOI) analysis has received increasing attention in the area of economic evaluations in health care [5–9] and can be seen as a valuable extension of probabilistic cost-effectiveness analysis, because, unlike CEACs, it provides information on the consequences of adopting the wrong treatment strategy.

In probabilistic cost-effectiveness models, the treatment strategy to adopt is identified as the strategy with the highest expected net benefit. Net monetary benefit is calculated as the total number of health effects, in this case quality-adjusted life years (QALYs), multiplied by the willingness to pay (WTP) for a QALY minus the total costs: (QALY × WTP) − C [10,11]. Expected net benefit is defined as the mean of the net benefits across all model iterations. VOI analysis is a Bayesian decision analytic approach which acknowledges that the decision to adopt and reimburse the strategy with the highest expected net benefit is based on currently available information that is surrounded by uncertainty. As long as there is uncertainty, there will always be a chance the wrong decision is made. Making the wrong decision comes with a cost that is equal to the benefits forgone because of the wrong decision. The expected costs of uncertainty can be determined by: 1) the probability that a decision based on mean net benefit is wrong; and 2) the size of the opportunity loss if the wrong decision is made. A VOI analysis informs decision-makers about the expected costs of uncertainty and, hence, the value of collecting additional information to eliminate or reduce uncertainty [9]. The total expected value of perfect information (EVPI) estimates the value of simultaneously eliminating all uncertainty on all parameters involved in taking a decision [7]. A VOI analysis may also provide information on the parameters for which additional research is most useful. Estimates of partial EVPI (EVPPI) can identify the parameters which uncertainties contribute most to the overall decision uncertainty. This information is valuable because a decision-maker does not only have to decide which treatment strategy to adopt but also whether more research regarding the decision is desirable. Since 2004, the National Institute for Clinical Excellence “Guide to the Methods of Technology Appraisal” states that candidate topics for future research may be best prioritized by considering the value of additional information in reducing the degree of decision uncertainty [12].

Value of information has been developed and successfully applied outside the health-care sector [13–16]. Health Technology Assessment (HTA)-researchers have adopted and further developed the concept for application in health-care decision-making [5,17]. Several authors have presented methods to calculate partial EVPI in probabilistic decision analytic models [8,17,18]. There has been some confusion because these different computational approaches all differ slightly. Hence, we have recently seen the publication of a few articles further clarifying the methodology [19,20]. Although the number of case studies is increasing [9,21–24], the number of actual applications of VOI in health care is still limited.

This article describes a VOI analysis based on a previously published probabilistic Markov model to assess the cost-effectiveness of bronchodilator therapy in chronic obstructive pulmonary disease (COPD) [25,26]. The objective of this article is to determine the impact that actually collecting additional data on one key parameter of the model, namely utilities, has on the overall model uncertainty. Hence, our study is an application of the VOI methodology to the realistic problem of choosing between bronchodilators in COPD. In the methods section, we first present the model, the prior utility data, and the posterior utility data that were obtained after a Bayesian update of the prior data with the newly collected data. Then, we present the VOI methodology and the sampling algorithms that we have used to calculate the EVPI and partial EVPI. We introduce a notation that can be easily understood by researchers without a mathematical background. In the Results section, we present the value of collecting additional information before and after collecting new utility data.

### Discussion

- Top of page
- ABSTRACT
- Introduction
- Methods
- Results
- Discussion
- References

The results of the VOI analysis before collecting additional data showed that the overall EVPI for the choice between tiotropium, salmeterol, and ipratropium was €1985 per patient at a ceiling ratio of €20,000. This is the absolute limit of the value of further research that would completely eliminate the uncertainty around the parameters in the model. Partial EVPI analyses showed that the costs of uncertainty were highest for the utility values, followed by the transition probabilities between COPD severity stages. Hence, collecting additional data on utilities was potentially of most value; the expected value of eliminating uncertainty in this subset of parameters was €1081. After having collected additional information on utilities, the overall EVPI was considerably reduced. In this posterior analysis, the remaining overall EVPI was €1070 and almost entirely because of uncertainty around disease state transitions. The partial EVPI of the utilities was reduced to almost zero. Collecting additional data on exacerbation probabilities or resource utilization did not appear to be of value.

That it would be most beneficial to collect additional information on utilities and transitions between disease states was something that we had not expected beforehand, because we had found in our previous studies that exacerbations and the costs of these exacerbations were important drivers of the cost-effectiveness of bronchodilator treatment [25,26]. Scenarios in which we had assumed a complete absence of a difference between the treatments in terms of exacerbation rates had a relatively great impact on the cost-effectiveness ratios. Nevertheless, such extreme uncertainties are unrealistic and not represented by the model. The model does include all the uncertainty around exacerbation rates that was observed in the clinical trials. Moreover, it is important to note that, although sensitivity analyses have shown that exacerbation rates influence the cost-effectiveness rates of tiotropium compared to its alternatives, the cost-effectiveness ratios remained below the decision threshold of €20,000 per QALY. This difference in observations from sensitivity analyses and EVPI analyses clearly illustrates the benefit of doing an EVPI analysis. The value of collecting additional information for a particular parameter depends not only on the association with cost-effectiveness, but also on the prior uncertainty about this parameter. Trying to interpret the joint impact of the strength of the association and the uncertainty without doing a formal VOI analysis is difficult and may easily lead to false conclusions about the parameters for which additional data collection is most useful.

The crucial question after an EVPPI analysis on currently available information is whether the expected value of additional information outweighs and justifies the cost of collecting additional information. The actual costs of this utility study are hard to estimate because the EQ-5D data were collected within the context of a large clinical trial that was designed to measure the decline in lung function over time [47]. Hence, the EQ-5D data could be collected at relatively little additional costs. If we were to set up a new study of the same size just for the purpose of collecting utility values, the costs would probably have been higher. Nevertheless, to get a sufficiently precise estimate of utilities by GOLD stage would probably require far fewer patients and thus, a much cheaper study. But in some jurisdictions, cost-effectiveness models can only be filled with country-specific data. Consequently, data from patients in each separate country are needed. In that case, a large multinational trial is a good opportunity to collect these data.

It is interesting to note, from the CEACs in Fig. 2, that even though the EVPI was strongly reduced by the use of new utility estimates with substantially lower standard errors, the decision about which treatment to adopt does not change. In both cases, tiotropium has the highest probability of being optimal as well as the highest expected net benefit over the whole range of threshold ICERs studied, and should therefore be adopted. Nevertheless, before additional data collection, the partial EVPI of utilities was high and this partial EVPI was our best estimate of the change in expected net benefit that we could get by doing additional research. It is important to stress that acceptability curves show just one element of the EVPI, namely the probability that a decision based on the mean net benefit is correct. Thus, the probability of making the wrong decision is the complement of the curve. The curves do not show the second element of the EVPI, which is the magnitude of the opportunity loss or, in other words, the consequence of making the wrong decision. It is precisely this magnitude of the opportunity loss that is considerably reduced by doing the additional utility study, which has considerably reduced the SE of the QALY outcome. This limitation of the acceptability curve is, for instance, discussed by Groot Koerkamp et al. [48] In general, there is no one on one relationship between the probability that an alternative is the “true” preferred alternative and the VOI. When the acceptability curve decreases, the EVPI necessarily increases, but the reverse is not true [49]. Thus, the acceptability curve on its own might lead to wrong conclusions by policymakers, as many people would be inclined to think that at 95% certainty of making the right decision, there is minimal value in additional research, whereas at 65% certainty this value from research would be high, whereas in truth the opposite might be the case.

A word of caution is necessary about the uncertainty incorporated in the model and about EVPI analysis in general. An EVPI analysis only provides information about the values of eliminating uncertainty around the probabilistic parameters included in the model. The characterization of the parameters and the different types of uncertainty is a major challenge and some forms of uncertainty may not have been taken into account. For example, the new EQ-5D utilities were obtained in a multinational trial and we are uncertain how well these utilities represent the heterogeneity of the COPD population in The Netherlands. Another example relates to the disease state transitions. We incorporated all uncertainty around these transitions as observed in clinical trials, but we are also uncertain as to what extent these trials reflect real life. To fully represent uncertainty and to establish a full EVPI analysis would require a parameterization of these types of uncertainty.

One could argue that the high EVPI for utilities is partly due to the fact that we have defined independent beta distributions for moderate, severe, and very severe COPD with and without exacerbations. Hence, it may occur that the utility value that is drawn for severe COPD is better than the value drawn for moderate COPD. We have not built in an association between the utilities by disease severity, in the sense that, when a low utility is drawn for moderate COPD, a lower value should be drawn for severe COPD. Such an association would reduce the uncertainty and the EVPI. Nevertheless, we doubt whether it reflects reality, because it is well known that the association between lung function and quality of life, though present, is rather weak [50], even at a group level. Therefore, we have chosen to let the uncertainty around the utility data speak for itself and not build in, a priory, a hierarchy for which the evidence is not strong. At the number of model runs in the current analyses, the average utility value for moderate COPD is better than for severe COPD, which in turn is better than for very severe COPD.

The results of the EVPPI analyses presented in Table 4 clearly show that the sum of the partial EVPIs does not add up to the total EVPI. This feature of partial EVPI analysis has also been explored in other publications [19,20]. The EVPI indicates the value of perfect information. Perfect information means that we have perfect information about the parameter of interest *θ*_{i}, and perfect information about the complementary parameters, *θ*_{c}, at the same time. The partial EVPI for *θ*_{i} is the value of perfect information about *θ*_{i}, given the uncertainty about *θ*_{c}, whereas the partial EVPI for *θ*_{c} is the value of perfect information about *θ*_{c}, given the uncertainty about *θ*_{i}. Summing the partial EVPIs for *θ*_{i} and *θ*_{c} does not return the EVPI. For this to happen, we should sum the partial EVPI for *θ*_{i} and the value of perfect information about *θ*_{c}, given perfect information about *θ*_{i}. The difference between the EVPI and the sum of the partial EVPIs complicates straightforward interpretation of a VOI analysis. Obtaining perfect information on one parameter of interest does not reduce the EVPI to the same extent as may have been expected (falsely) from the value of the partial EVPI for that parameter. Hence, the importance of a parameter for further research should not be judged by the reduction in EVPI [19]. The added value of a partial EVPI analysis is to set priorities with regard to the parameters for which additional data collection is most beneficial by ranking them according to their expected value of research that would eliminate the uncertainty.

In this study, we compared the outcomes of a partial EVPI analysis obtained with a two-level analysis with a one-level analysis. Theoretically, the two-level analysis provides the correct values of the EVPPI. The required numbers of inner and outer loop iterations are not known beforehand and, among other factors, depend on the distribution and the number of uncertain model parameters. In our analysis, the results of the two-level analysis may have been biased because of the limited number of iterations. Nevertheless, repeating some of the two-level analyses showed that the consistency of outcomes across analyses was good. For example, the EVPPIs of exacerbation probabilities and costs were consistently around zero and the EVPPI of utilities in five posterior analyses varied between €843 and €868. When a model is perfectly linear and no correlation exists between input parameters, the one-level sampling algorithms will provide estimates of the EVPPI that are equal to the two-level sampling algorithms. In this case, these assumptions are not fulfilled. The Dirichlet distribution that was used for transition probabilities, by definition, introduces correlation in the transition rates. Moreover, an inherent characteristic of a Markov simulation is the multiplication of matrices with transition probabilities over subsequent cycles, causing the transitions to be nonlinear. Hence, for that reason, the results of the one-level sampling approach should be interpreted with care. Nevertheless, the results of the EVPPI analysis with the one-level and the two-level sampling approach both indicated that the EVPPI was highest for utilities and transition probabilities although in absolute terms the EVPPIs of especially the utilities differ.

In conclusion, this study has clearly shown the benefits of doing a VOI analysis. Before any additional data collection, the VOI analysis at a ceiling ratio of €20,000 estimated the EVPI to be €1985 per patient and identified the utilities as the subset of parameters with the highest EVPI. After additional research on utilities was performed and a formal Bayesian update of the utilities was conducted, the EVPPI of utilities was reduced to almost zero. The EVPI was still €1070 and largely driven by the uncertainty in transition probabilities, which would be the next best candidate for doing additional research in the future. Such research should focus on estimates of lung function decline over time, because this decline drives the transition probabilities between COPD severity stages as defined by GOLD.