SEARCH

SEARCH BY CITATION

Keywords:

  • cost-effectiveness acceptability curve (CEAC);
  • cost-effectiveness acceptability frontier (CEAF);
  • cost-effectiveness plane;
  • expected value of perfect information (EVPI);
  • uncertainty

ABSTRACT

  1. Top of page
  2. ABSTRACT
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. References

Objective:  To demonstrate how the optimal decision and level of uncertainty associated with that decision, can be presented when assessing the cost-effectiveness of multiple options. To explore and explain potentially counterintuitive results that can arise when analyzing multiple options.

Methods:  A template was created, based on the assumption of multivariate normality, in order to replicate a previous analysis that compared the cost-effectiveness of multiple options. We used this template to explain some of the different shapes that the cost-effectiveness acceptability curve (CEAC), cost-effectiveness acceptability frontier (CEAF), and expected value of perfection information (EVPI) may take, with changing correlation structure and variance between the multiple options.

Results:  We show that it is possible for 1) an option that is subject to extended dominance to have the highest probability of being cost-effective for some values of the cost-effectiveness threshold; 2) the most cost-effective (optimal) option to never have the highest probability of being cost-effective; and 3) the EVPI to increase when the probability of making the wrong decision decreases. Changing the correlation structure between multiple options did not change the presentation of results on the cost-effectiveness plane.

Conclusion:  The cost-effectiveness plane has limited use in representing the uncertainty surrounding multiple options as it cannot represent correlation between the options. CEACs can represent decision uncertainty, but should not be used to determine the optimal decision. Instead, the CEAF shows the decision uncertainty surrounding the optimal choice and this can be augmented by the EVPI to show the potential gains to further research.


Introduction

  1. Top of page
  2. ABSTRACT
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. References

The cost-effectiveness acceptability curve (CEAC) was introduced as a method to represent uncertainty, and is often used to present the results of cost-effectiveness analysis [1–4]. However, it is possible that the CEAC may be misinterpreted [5,6]. This possibility is highlighted by the fact that one might (incorrectly) infer, from one paper [7], that CEACs show the value at which each option becomes the optimal choice, and from another paper [8], that they show the probability that one option dominates another option. There is a similar possibility that the purpose of the cost-effectiveness acceptability frontier (CEAF) may also be misinterpreted. This is highlighted by the fact that, when purportedly constructing the CEAF, some analysts [8,9] have plotted the probability of cost-effectiveness for many options, rather than just the optimal option, and used the term CEAF to describe this mode of presentation. In light of these issues, we seek to demonstrate how the CEAC and CEAF should be constructed, what they should and should not be used for, and in doing so, how the optimal decision can be presented, along with levels of uncertainty.

In order to illustrate the above, we present (and make available online) a spreadsheet-based template that can be used to represent uncertainty in the decision among a number of mutually exclusive treatment options. We then employ this template to explain two entirely possible results that may nonetheless appear counterintuitive. First, that an option that was subject to extended dominance [10,11] can have the highest probability of being cost-effective. Second, that the optimal option (as it provided the greatest benefit for a given cost) may not have the highest probability of being cost-effective for any value of the cost-effectiveness threshold. Both of these situations were apparent in the results presented by Goeree et al. [1], and this study serves as a practical example to show why such results are in fact valid, and how decision uncertainty in such situations should be presented and interpreted.

The paper is structured as follows: First, we provide an overview of the main principles of cost-effectiveness analyses and illustrate the importance of the CEAC, CEAF, and expected value of perfect information (EVPI). Second, we outline a template that can be used by others and is used here to demonstrate how to calculate CEACs, the CEAF, and the EVPI in the presence of multiple options. Finally, this template is used to illustrate how different assumptions about the correlation structure and the variances of the model outputs can influence the shape of the resulting CEACs, CEAF, and EVPI, thus demonstrating why the two potentially counterintuitive results occurred in the study by Goeree et al. [1].

Principles of Cost-Effectiveness Analyses

When making decisions about the allocation of scarce health-care resources, it has been argued that two questions are fundamental [12–14]. First, which option is estimated to be cost-effective, on the basis of existing evidence? Second, should further research be undertaken in order to reduce the level of uncertainty associated with that decision?

In order to answer the first question, one seeks to identify the option that is expected to provide the highest level of benefit for a given level of cost, i.e., the aim is to maximize health subject to a budget constraint [15–17]. In the case of mutually exclusive options, this involves the calculation of the incremental cost-effectiveness ratio (ICER) of each option relative to the next best treatment option, and the selection of the option that has the highest ICER below the cost-effectiveness threshold as the optimal option. When calculating the ICER, it is important to identify, and exclude, those options that are dominated (more costly and less effective than another option) and those that are subject to extended dominance (combinations of other options can provide a higher level of benefit for the same cost) [18].

One of the most common ways of presenting the results of cost-effectiveness analysis is on the cost-effectiveness plane [19], where the effects are measured on the horizontal axis and costs are measured on the vertical axis. By plotting the mean cost and mean effect of each option and connecting each cost-effective (i.e., nondominated) option with the next less effective option, it is possible to display the efficiency frontier for these options [1,20]. Thus, though the cost-effectiveness plane can answer the first fundamental question, by identifying the option that provides the highest level of expected benefit at different levels of cost, it has been argued that the plane fails to fully represent the level of uncertainty associated with the estimated cost-effectiveness of an option in a graphical manner [21]. As a consequence of this, the CEAC was developed. The CEAC is constructed by plotting (for each option) the proportion of the cost and effect pairs that are cost-effective for a range of values of the cost-effectiveness threshold [22]. Thus, uncertainty is characterized by estimating the probability that an option is cost-effective at different levels of the cost-effectiveness threshold, where this is undertaken for all possible options. Fenwick et al. [17], however, have shown that an option that had the highest probability of being cost-effective need not necessarily have the highest expected net benefit, and for this reason, the CEAC should not be used to identify the optimal treatment option. Instead, they propose the use of the CEAF [17], which plots the uncertainty associated with the optimal option, for different values of the cost-effectiveness threshold. This is equivalent to plotting each CEAC over the range of values for the cost-effectiveness threshold for which each option is estimated to be the most cost-effective, i.e., has the highest ICER that falls below the threshold [17].

Finally, in order to inform the second fundamental question concerning the decision of whether to undertake further research, the EVPI can be calculated. The EVPI gives an upper bound on the value of undertaking further research in order to eliminate the uncertainty surrounding the decision about which option is optimal for different levels of the cost-effectiveness threshold [20,23]. The EVPI is dependent upon both the probability of a wrong decision being made (as shown by the CEAF), but also the consequences of that wrong decision [23,24]. Further research costing less than the EVPI only has the potential to be efficient; a sufficient condition for determining the worth of further research requires comparison of the expected value of sample information and the costs (see [23] and [25] for more details).

Methods

  1. Top of page
  2. ABSTRACT
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. References

Probabilistic methods can be used to describe the uncertainty associated with input parameters in a model, and in turn, can estimate the uncertainty in the model outputs of cost, effect, and cost-effectiveness [20]. This equates to a Bayesian approach to cost-effectiveness analysis because the parameters of interest are ascribed a distribution in order to reflect the uncertainty concerning the true value of the parameter [26]. For the purposes of exploring potential scenarios relating to the costs and effects of multiple options, we developed a template (in Excel, Microsoft, Redmond, WA) that approximates the joint distributions of the costs and effects of multiple treatment options under the assumption of multivariate normality. As such, we seek to provide a generic characterization of the uncertainty associated with model outputs for the purposes of illustration and ease of manipulation. Readers are directed to Briggs et al. [20] for further direction on how to characterize the uncertainty of input parameters using probability distributions. The template, which can be downloaded at http://www.dph.gla.ac.uk/heat, works with up to seven different options and requires the user to input the means and standard errors for all options. In addition, the user also specifies the correlation structure between the costs and effects both within and between options. From this information, the variance-covariance matrix is generated and (through the use of the Cholesky decomposition technique [20]) the template draws 1000 simulations of both the cost and effect parameters of each option from the specified multivariate normal distribution. Finally, using the methods described later, the template provides estimates of the CEACs, CEAF and the EVPI.

CEAC

In the case of multiple options, a separate CEAC can be plotted for each option, where each CEAC represents the (essentially Bayesian) probability of each option being cost-effective at different levels of the cost-effectiveness threshold [20,27]. This probability was estimated in the following manner: In each of the 1000 iterations, the total cost (C) and total effect (E) was estimated for each option, and for a particular cost-effectiveness threshold (λ), the net monetary benefit (NMB) was estimated: NMB = λ × E − C. In each of the 1000 iterations, the option with the highest net benefit was then identified. The probability of being cost-effective was then equivalent to the proportion of the 1000 iterations for which each option had the highest net benefit. CEACs were estimated by plotting these proportions (y-axis) for different λ-values (x-axis). With regard to the claim that the CEAC shows the probability that one option dominates another option [8], it can be seen that this is untrue because the CEAC is determined by the net benefit of each option and it is possible for an option to have a higher net benefit without dominating another option.

CEAF

Fenwick et al. [17] have shown that the probability of being cost-effective cannot be used to determine the optimal option, and that if the societal objective is to maximize health gain, then decisions should be taken on the basis of expected net benefit, regardless of the uncertainty (probability) associated with the decision. Thus, the option with the highest probability of being cost-effective (highest CEAC) at any value of the cost-effectiveness threshold need not be the optimal option [28]. The solution to the fact that CEACs cannot identify the optimal option is to plot the CEAF [17].

In contrast to the CEACs, the CEAF plots only the probability that the optimal option is cost-effective (at different λ-values). Thus, it is first necessary to identify which option is optimal at each level of the cost-effectiveness threshold. To do this, it is necessary to identify the mean cost and mean effect for each option (across each of the 1000 simulations), and to calculate which option is optimal (has the highest expected net benefit) at different levels of λ. Thus, the range of λ-values over which an option is optimal can be calculated, and the “switch points” (when there is a change in the optimal option) correspond to the ICER between different options [17]. The lower value of the range of values for the cost-effectiveness threshold for which each option was optimal denotes the ICER for that particular option, and the upper value denotes the ICER for the next most costly option.

Once the optimal option has been identified for each level of λ, the probability of this option being cost-effective can then be plotted (y-axis) for different λ-values (x-axis). This probability of being cost-effective is determined by the CEAC. One minus the probability given by the CEAF at any value of the cost-effectiveness threshold is equivalent to the probability that a “wrong” decision will be made (the error probability) [13]. Finally, it should be noted that because only optimal options are represented, the CEAF may not always correspond to the options with the highest probability of being cost-effective.

EVPI

The EVPI (for an individual patient) is determined, for a given λ, by the difference between the expected value with perfect information and the expected value with current information. After calculating the expected net benefit for each option across all iterations, the first step is to identify the optimal option based on the current level of information. This is the option with the maximum expected net benefit for the given level of the cost-effectiveness threshold, as identified by the CEAF. The second step is to identify the decision with perfect information. This is done for every possible realization of uncertainty (every iteration) by identifying the option that maximizes net benefit. Within each iteration, the value of perfect information is equal to the difference between the net benefit of the optimal option (from step one) and the maximum net benefit (from step two), and will equal zero when the optimal option has the highest net benefit. Finally, the EVPI is calculated by taking the expectation over the values of perfect information. This is equivalent to the value of the decision when made with perfect information averaged across all possible realizations of uncertainty [14]. The EVPI was then calculated for differing λ-values and plotted on the same diagram as the CEAF; it shows the maximum amount that one would be willing to pay to eliminate uncertainty.

Replication of Previous Analyses

By replicating the comparison of multiple options by Goeree et al. [1], we sought to depict and explain both the optimal decision and the level of uncertainty associated with that decision. Goeree et al. [1] estimated the cost-effectiveness of seven different management strategies for gastroesophageal reflux disease, where costs were estimated at year 2000 levels (US$), and effects were measured in terms of quality adjusted life years (QALYs). In order to replicate the results, for each of the seven options, we extracted the mean cost (and associated standard error) and mean effect (and associated standard error) from the results of their probabilistic analysis (which were reported for a cohort of 1000 patients). From the results provided by the original authors, the correlation matrix was determined and employed to construct the variance–covariance matrix for the analysis. Initially, we undertook a replication of the analysis undertaken by Goeree et al. [1] (the base-case analysis). By altering the assumptions underlying this analysis, we then ran a series of sensitivity analyses in order to explain why the two potentially counterintuitive results occurred (i.e., why an option that was subject to extended dominance had the highest CEAC while an optimal option never had the highest CEAC), and showed that those results were entirely appropriate. In the first of these analyses, we assumed that the costs and effects of different options were independent, but that the relationship between costs and effects within options remained the same. In the second analysis, we used the original correlation matrix but made the assumption that the variance associated with the mean cost and mean effect of each option was one-fifth of that in the base-case (which may have occurred if the original analysis had involved a larger sample size). Finally, the third analysis repeated that of the second, but with the assumption that the variance was five times greater than that in the base-case (representing a smaller sample size).

Results

  1. Top of page
  2. ABSTRACT
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. References

Replication of Previous Analyses

Base-case.  After constructing the variance–covariance matrix for the analysis by Goeree et al. [1], the template was used to replicate their analysis. The cost and effect pairs for each of the 1000 iterations, for each of the seven options, were first plotted on the cost-effectiveness plane (Fig. 1a). After determining which options were dominated (none) and subject to extended dominance (options C, D, and G), it was possible to estimate the efficiency frontier. In Figure 1a, it can be seen that this is composed of options B, A (ICER = $7755, compared with B), F (ICER = $12,183, compared with A), and E (ICER = $110,845, compared with F)—the efficiency frontier is presented for the mean costs and mean effects, rather than for each of the 1000 simulations.

image

Figure 1. Replication of the analysis by Goeree et al. [1]. Results are presented in terms of the cost-effectiveness plane (a), cost-effectiveness acceptability curves (b), and the cost-effectiveness acceptability frontier and the expected value of perfect information (EVPI) (c).

Download figure to PowerPoint

The corresponding CEACs are also plotted in Figure 1b, where it can be seen that one of the optimal options (A) never had the highest probability of being cost-effective. Instead, B initially had the highest probability of being cost-effective (λ < $7970), followed by D ($7970 = λ < $12,540), C ($12,540 = λ < $14,850), F ($14,850 = λ < $112,120), and E (λ = $112,120). Thus, options that were subject to extended dominance (C and D) were also associated with the highest probability of being cost-effective at certain levels of ë. This provides an illustration of why CEACs should not and cannot be used to identify the optimal option, as has been previously shown by Fenwick et al. [17].

In order to simultaneously present the optimal option and the level of uncertainty associated with that option, the CEAF was plotted (Fig. 1c). The ICERs for options A, F, and E are plotted as vertical lines in order to demonstrate how the optimal option was identified. Prior to the first vertical line (λ < $7755), option B was optimal, between the first two vertical lines ($7755 = λ < $12,183), option A was optimal, followed by option F ($12,183 = λ < $110,845), and finally, option E (λ = $110,845). It should also be noted that the CEAF may be disjointed. This can be seen at the first vertical line (λ = $7755), where the probability of option B being cost-effective was 32.7%, whereas the corresponding probability for option A was 25.8%. Moreover, at this point, as well as the optimal option (A) not having the highest probability of being cost-effective, there was a 74.2% probability that option A was not the most cost-effective option. This arose because the optimal option was determined by the highest expected net benefit, whereas the CEAC simply represents the proportion of iterations over which each option had the highest net benefit.

By replicating the Goeree et al. [1] analysis, it was also possible to estimate the EVPI for an individual patient (Fig. 1c). The EVPI was relatively low when there was a high probability that the optimal option was cost-effective, with local maxima when the optimal option changed (and the probability of a wrong decision was relatively high), e.g., at a λ of approximately $12,000 and $111,000. However, this was not always the case, as when the optimal option changed from B to A there was a point of inflexion, but the EVPI continued to increase. This arose, despite there being a local maxima in the error probability, because the falling probability of making an incorrect decision was outweighed by the increasing consequences of a wrong decision. Finally, it should be noted that at high levels of the cost-effectiveness threshold, the EVPI fell to virtually zero. This was because of virtually nonexistent uncertainty at these levels of λ, where option E had a high probability of being cost-effective because of strong evidence that it was the most effective option.

Sensitivity analysis.  When the analyses by Goeree et al. [1] were replicated, with the assumption that the costs and effects between options were independent, the cost-effectiveness plane (Fig. 2a) was visually the same as in the base-case analyses (see Fig. 1a), because near identical cost and effect pairs (within options) were drawn in both of these probabilistic analyses. Thus, the ICERs were very similar to that in the base-case analyses. Conversely, the CEACs changed, with the curves tending to converge—those options that previously had the highest probability of being cost-effective, for a particular λ, tended to have a lower probability of being cost-effective, and other options tended to have a higher probability of being cost-effective. Importantly, this meant that option A (which was the optimal option between a λ of $7755 and $12,183) now had the highest probability of being cost-effective for a small range of λ-values, and option C (which was subject to extended dominance) no longer had the highest probability of cost-effectiveness over any threshold value. Here, for reasons of brevity, rather than plotting the CEACs in the traditional manner (as in Fig. 1b), they have been plotted on the same diagram as the CEAF and EVPI. In Figure 2b, the CEACs are plotted, where the (bold) continuous section of the CEAC denotes the range of λ-values over which each option was optimal, and the (lighter) dashed section of the CEAC denotes the range of λ-values over which each option did not provide the highest expected net benefit.

image

Figure 2. Replication of the analysis by Goeree et al. [1], with the assumption that the costs and effects between options were independent. Results are presented in terms of the cost-effectiveness plane (a), cost-effectiveness acceptability curves, cost-effectiveness acceptability frontier, and expected value of perfect information (EVPI) (b).

Download figure to PowerPoint

In the Goeree et al. [1] analysis, these two potentially counterintuitive results arose because all the options were positively correlated. Consequently, when, for example, option B had a low net benefit in a particular iteration, it was likely that in the same iteration, other options would also have a low net benefit, and at low levels of λ, this meant that option B had the highest net benefit in a high proportion of iterations (and thereby a high CEAC). Conversely, when the options were assumed to be independent, when option B had a low net benefit, then there was a higher chance of another option having a higher net benefit, in the same iteration, and option B thereby had the highest net benefit in a lower proportion of iterations.

The assumption that the costs and effects between options were independent also changed the CEAF (Fig. 2b), where the optimal options tended to have a lower probability of being cost-effective, and the EVPI (Fig. 2b), which also tended to be higher. Here, as in Figure 1c, the CEAF was again constructed by plotting the CEAC, as a continuous line, over the range of λ-values over which each option had the highest expected net benefit. It should also be noted that the EVPI even increased when the probability of the optimal option being cost-effective increased, e.g., at a λ of $12,000; in the base-case the EVPI was 41.4 when option A had a 17.9% probability of being cost-effective, yet it rose to 107.4 when the options were assumed to be independent and the probability of option A being cost-effective increased to 19.8%. This change in the EVPI can be explained by the fact that when option B, for example, had a high net benefit in a particular iteration in the base-case (independent) analysis, it was more (less) likely that other options would also have a high net benefit. Thus, in the base-case the consequences of the wrong decision (represented by the incremental net benefit) were estimated to be less than in the analysis, where the options were assumed to be independent.

The final two analyses had the same correlation structure as the base-case, but had differing levels of variance. In both analyses, the cost and effect pairs for each option were centered around the same means, and thereby ICERs, but the pairs were more condensed when the level of variation was reduced by a factor of five (Fig. 3a) and more spread out when the variation was inflated by a factor of five (Fig. 4a). Looking at the CEACs, reducing the level of variation tended to increase the probability of optimal options being cost-effective, at the expense of the probability associated with other options (Fig. 3b). Indeed, option A now had the highest probability of being cost-effective over a reasonable range of λ-values, and options C/D only briefly did so ($11,690 ≤ λ < $12,950).

image

Figure 3. Replications of the analysis by Goeree et al. [1] with the assumption that the variance associated with the mean cost and mean effect of each option was one-fifth of that in the base-case analysis. Results are presented in terms of the cost-effectiveness plane (a), cost-effectiveness acceptability curves, cost-effectiveness acceptability frontier, and expected value of perfect information (EVPI) (b).

Download figure to PowerPoint

image

Figure 4. Replications of the analysis by Goeree et al. [1] with the assumption that the variance associated with the mean cost and mean effect of each option was five times that in the base-case analysis. Results are presented in terms of the cost-effectiveness plane (a), cost-effectiveness acceptability curves, cost-effectiveness acceptability frontier, and expected value of perfect information (EVPI) (b).

Download figure to PowerPoint

In the Goeree et al. [1] analysis, the two potentially counterintuitive results arose because options C and D, which were subject to extended dominance, had relatively high levels of variation in net benefit, compared with option A. For example, at a λ of $12,000, the mean net benefits for options A, C, and D were $9844, $9841, and $9840, and the variances were $7426, $10,967, and $11,027. Thus, as the probability distributions surrounding the mean net benefit of options C/D were more spread out than for option A within a particular iteration, despite the marginally lower mean values of net benefit for C/D, there was a higher probability of a higher net benefit being drawn for options C/D than for option A. When the level of variation was reduced, the distributions became more tightly centered around the mean, and within a particular iteration, option A had a higher probability of having a higher net benefit than options C/D (see CEACs in Fig. 3b). Increasing the level of variation had the opposite effect on the CEACs (Fig. 4b), as those options that had a relatively high level of variation (e.g., options C and D) had the highest net benefit in a greater proportion of iterations, and thereby a greater probability of being cost-effective than in the base-case, at the expense of the those options that had a relatively low level of variation (e.g., options A and B).

Reducing the level of variation increased the proportion of iterations where optimal options had the highest net benefit. Thus, as optimal options tended to have the highest probability of being cost-effective, the CEAF was more often equivalent to the uppermost CEAC, and the probability of making a wrong decision tended to be lower (Fig. 3b). This, combined with the lower level of variation in net benefit, which reduced the consequences of making a wrong decision, meant that the EVPI was also lower when the variation was reduced (Fig. 3b). However, the converse occurred when the level of variation increased—the CEAF was less frequently equivalent to the uppermost CEAC, the probability of making the wrong decision tended to increase, as did the consequences of making a wrong decision, and in turn the EVPI also increased (Fig. 4b).

Discussion

  1. Top of page
  2. ABSTRACT
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. References

In order to show how to present the optimal decision and the uncertainty associated with that decision when assessing multiple options, we replicated the analysis of a previous paper and performed sensitivity analysis on the assumptions of that analysis. In doing so, we demonstrated that CEACs can sometimes show 1) that options that are subject to extended dominance have the highest probability of being cost-effective; or 2) that optimal options do not have the highest probability of being cost-effective for any value of the cost-effectiveness threshold. Thus, we have shown that CEACs should not be used to determine the optimal decision, as previously suggested by Kamath et al. [7]. In addition, we have explained that these two effects can arise because of the correlation structure and differences in the level of variation between multiple options. Furthermore, in order to present the optimal decision and the level of uncertainty associated with that decision, we have demonstrated that the CEAF, along with the EVPI, should be calculated. Finally, it has also been shown that it is, for example, possible for uncertainty to increase (as assessed by the EVPI) when the probability of the optimal option being cost-effective also increases.

Implications

We have demonstrated, as previously suggested [17,21,22], that instead of using the CEAC to estimate which option is optimal, the sole purpose of presenting the CEAC is to demonstrate the level of uncertainty associated with an option. Moreover, we have also shown that if our objective is to maximize the expected net benefit regardless of the level of uncertainty, as argued previously by Claxton [15], then this can result in choosing options where we know there is a high probability of making a wrong decision. For example, in the study by Goeree et al. [1], at a λ of $12,000, option A was optimal despite the fact that the probability of this being the wrong decision exceeded 80% (see Fig. 1c). The first implication of our results, therefore, is that when presenting the results of cost-effectiveness analyses, it should be made clear that the CEACs are presented in order to depict the level of uncertainty and not the optimal decision.

We also demonstrated that though the cost-effectiveness plane can depict the optimal decision, it cannot depict the level of uncertainty, because it is insensitive to the correlation structure between options. Thus, the second implication of our results is that in order to simultaneously present the optimal decision, along with the uncertainty associated with that decision, the CEAF should be plotted. Indeed, the only time it may be appropriate to present uncertainty on the cost-effectiveness plane is when two options are being compared. This arises because in contrast to the situation for multiple options, the incremental cost and incremental effects can be plotted, thus directly revealing the correlation structure.

The CEAF is, however, constructed from the CEAC of the optimal option(s), and therefore only depicts uncertainty through the probability of not selecting the most cost-effective option. The CEAF does not portray the consequences of the decision; as such, it provides only a first step in determining the value of further research. Indeed, in order to determine the latter, the third implication of our research is that the EVPI should be calculated. Moreover, because the correlation structure between options was shown to affect the consequences of making the wrong decision, this shows that the EVPI can provide further information that is not captured by the cost-effectiveness plane or the CEAF.

Strengths and Weaknesses

A crucial assumption within this paper is that optimal decisions will be made when options with the highest expected net benefit are implemented (i.e., when benefits are maximized for a given cost). However, Coast [5] has argued that such an objective does not comply with how society would wish to allocate scarce health-care resources, because for example, it ignores the notion of equity. To reflect this, the objective could be adapted to include an equity weighting, but this would not change the fact that it is possible for an optimal option (unless optimal is defined in terms of probability) not to have the highest probability of being cost-effective, and thus the CEAC should not be used to determine the optimal allocation of resources.

We are only aware of one previous possible explanation as to why an optimal option may not have the highest probability of being cost-effective, namely, that the incremental net benefit was positively skewed [17]. However, as this was not the case in the Goeree et al. [1] analysis, positive skewness could not account for why it had previously been shown that an option that was subject to extended dominance had, at certain λ-values, the highest probability of being cost-effective, nor could it account for why an option that was cost-effective never had the highest probability of being cost-effective [1]. Thus, the main strength of this article is in demonstrating that these instances can be brought about by particular correlation structures, or by differences in the relative levels of variance between multiple options. Our results are also therefore supported by others [29,30] who have shown that the correlation structure between the costs and effects of different options can greatly influence the level of uncertainty. By constructing a template, we have further demonstrated that the shapes of the CEACs, the CEAF, and the EVPI are highly dependent upon assumptions made about the correlation structure and the relative levels of the variance.

The template we created was based on the assumption of multivariate normality; thus, it may not be appropriate to use it in replicate studies where there is substantial departure from this distribution. That said, it is possible to use the Cholesky decomposition technique to incorporate correlations between other types of distributions, as has been explained by Briggs et al. [20]. Moreover, the template can be similarly adapted to extract the mean cost and mean effect from other probabilistic models, and thereby construct the CEACs, CEAF, and EVPI for other studies.

Conclusion

  1. Top of page
  2. ABSTRACT
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. References

The cost-effectiveness plane has limited use in representing uncertainty in the multiple-option case as it cannot represent correlation between the options, which can play a very important role. CEACs can represent decision uncertainty, but should not be used to determine the optimal decision. Instead, the CEAF shows the optimal choice and this can be augmented using the EVPI plot to show the potential gains of further research to reduce uncertainty.

We thank the authors of a previous publication (Goeree R, O'Brien B, Blackhouse G, Marshall J, Briggs A, and Lad R) for providing us with the results of their analyses, which were used to inform some of the analyses in this paper.

Source of financial support: No specific funding was provided for this piece of research, which was undertaken while Garry Barton undertook a PhD sponsored by the UK Economic & Social Research Council (ESRC) (PTA-037-2004-00051) at the University of Nottingham. The manuscript was written during an ESRC sponsored visit to the Health Economics Appraisal Team at the University of Glasgow. Andrew Briggs is supported by the William R Lindsay Chair in Health Policy & Economic Evaluation.

References

  1. Top of page
  2. ABSTRACT
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. References