SEARCH

SEARCH BY CITATION

Keywords:

  • cost-effectiveness;
  • economic;
  • guidelines;
  • randomized clinical trial

ABSTRACT

  1. Top of page
  2. ABSTRACT
  3. Introduction
  4. Initial Trial Design Issues
  5. Data Elements
  6. Database Design and Management
  7. Analysis
  8. Reporting the Methods and Results
  9. Conclusions
  10. References

Objectives:  A growing number of prospective clinical trials include economic end points. Recognizing the variation in methodology and reporting of these studies, the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) chartered the Task Force on Good Research Practices: Randomized Clinical Trials—Cost-Effectiveness Analysis. Its goal was to develop a guidance document for designing, conducting, and reporting cost-effectiveness analyses conducted as a part of clinical trials.

Methods:  Task force cochairs were selected by the ISPOR Board of Directors. Cochairs invited panel members to participate. Panel members included representatives from academia, the pharmaceutical industry, and health insurance plans. An outline and a draft report developed by the panel were presented at the 2004 International and European ISPOR meetings, respectively. The manuscript was then submitted to a reference group for review and comment.

Results:  The report addresses issues related to trial design, selecting data elements, database design and management, analysis, and reporting of results. Task force members agreed that trials should be designed to evaluate effectiveness (rather than efficacy), should include clinical outcome measures, and should obtain health resource use and health state utilities directly from study subjects. Collection of economic data should be fully integrated into the study. Analyses should be guided by an analysis plan and hypotheses. An incremental analysis should be conducted with an intention-to-treat approach. Uncertainty should be characterized. Manuscripts should adhere to established standards for reporting results of cost-effectiveness analyses.

Conclusions:  Trial-based cost-effectiveness studies have appeal because of their high internal validity and timeliness. Improving the quality and uniformity of these studies will increase their value to decision makers who consider evidence of economic value along with clinical efficacy when making resource allocation decisions.


Introduction

  1. Top of page
  2. ABSTRACT
  3. Introduction
  4. Initial Trial Design Issues
  5. Data Elements
  6. Database Design and Management
  7. Analysis
  8. Reporting the Methods and Results
  9. Conclusions
  10. References

Clinical trials evaluating medicines, medical devices, and procedures now commonly assess economic value of these interventions. The growing number of prospective clinical/economic trials reflects both widespread interest in economic information for new technologies and the regulatory and reimbursement requirements of many countries that now consider evidence of economic value along with clinical efficacy. In recent years, research has also improved the methods for the design, conduct, and analysis of economic data collected alongside clinical trials. Despite these advances, the literature reveals a great deal of variation in methodology and reporting of these studies. Improving the quality of these studies will enhance the credibility and usefulness of cost-effectiveness analyses to decision makers worldwide.

To foster improvements in the conduct and reporting of trial-based economic analyses, the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) chartered the ISPOR Task Force on Good Research Practices: Randomized Clinical Trials-Cost-Effectiveness Analysis (RCT-CEA). The task force cochairs were selected by the ISPOR Board of Directors, and the cochairs invited the other panel members to participate. The panel was first assembled in January 2004, communicated monthly, and agreed on an outline and preliminary content that was presented for comment by ISPOR membership at the May 2004 annual meeting. The draft report was then written and presented to the ISPOR membership at the October 2004 European meeting. A volunteer reference group of ISPOR members provided valuable comments on the draft report, which supported the completion of the final report in February 2005.

The purpose of this report is to state the consensus position of this Task Force (Table 1). The goal for the panel was to develop a guidance document for the design, conduct, and reporting of cost-effectiveness analyses alongside clinical trials. The intended audiences are researchers in academics, industry, and government who design and implement these studies, decision makers who evaluate clinical and economic evidence for formulary and insurance coverage policies, and students of the area. The panel recognizes that advances in methodology for joint clinical/economic analyses will continue and that clinical/economic trials are heterogeneous in nature. Therefore, the report highlights areas of consensus, emerging methodologies with a diversity of professional opinions, and issues where further research and development are needed.

Table 1.  Core recommendations for conducting economic analyses alongside clinical trials
  1. QALYs, quality-adjusted life-years.

Trial design
Trial design should reflect effectiveness rather than efficacy when possible.
Full follow-up of all patients is encouraged.
Describe power and ability to test hypotheses, given the trial sample size.
Clinical end points used in economic evaluations should be disaggregated.
Direct measures of outcome are preferred to use of intermediate end points.
Data elements
Obtain information to derive health state utilities directly from the study population.
Collect all resources that may substantially influence overall costs; these include those related and unrelated to the intervention.
Database design and management
Collection and management of the economic data should be fully integrated into the clinical data.
Consent forms should include wording permitting the collection of economic data, particularly when it will be gathered from third-party databases and may include pre- and/or post-trial records.
Analysis
The analysis of economic measures should be guided by a data analysis plan and hypotheses that are drafted prior to the onset of the study.
All cost-effectiveness analyses should include the following: an intention-to-treat analysis; common time horizon(s) for accumulating costs and outcomes; a within-trial assessment of costs and outcomes; an assessment of uncertainty; a common discount rate applied to future costs and outcomes; an accounting for missing and/or censored data
Incremental costs and outcomes should be measured as differences in arithmetic means, with statistical testing accounting for issues specific to these data (e.g., skewness, mass at zero, censoring, construction of QALYs).
Imputation is desirable if there is a substantial amount of missing data. Censoring, if present, should also be addressed.
One or more summary measures should be used to characterize the relative value of the intervention.
Examples include ratio measures, difference measures, and probability measures (e.g., cost-effectiveness acceptability curves).
Uncertainty should be characterized. Account for uncertainty that stems from sampling, fixed parameters such as unit costs and the discount rate, and methods to address missing data.
Threats to external validity—including protocol-driven resource use, unrepresentative recruiting centers, restrictive inclusion and exclusion criteria, and artificially enhanced compliance—are best addressed at the design phase.
Multinational trials require special consideration to address intercountry differences in population characteristics and treatment patterns.
When models are used to estimate costs and outcomes beyond the time horizon of the trial, good modeling practices should be followed. Models should reflect the expected duration of the intervention on costs and outcomes.
Subgroup analyses based on prespecified clinical and economic interactions, when found to be significant ex post, are appropriate. Ad hoc subgroup analysis is discouraged.
Reporting the results
Minimum reporting standards for cost-effectiveness analyses should be adhered to for those conducted alongside clinical trials.
The cost-effectiveness report should include a general description of the clinical trial and key clinical findings.
Reporting should distinguish economic data collected as part of the trial vs. data not collected as part of the trial.
The amount of missing data should be reported. If imputation methods are used, the method should be described.
Methods used to construct and compare costs and outcomes, and to project costs and outcomes beyond the trial period should be described.
The results section should include summaries of resource use, costs, and outcome measures, including point estimates and measures of uncertainty. Results should be reported for the time horizon of the trial, and for projections beyond the trial (if conducted).
Graphical displays are recommended for results not easily reported in tabular form (e.g., cost-effectiveness acceptability curves, joint density of incremental costs and outcomes).

The focus of this report is cost-effectiveness analysis conducted alongside randomized clinical trials designed to test the efficacy or effectiveness for drugs, devices, surgical procedures, or screening interventions, including pragmatic trials. Clinical trials are artificial treatment environments and do not provide all the economic information needed by decision makers. Trial populations do not commonly reflect patient groups treated in clinical practice, and the time horizon for trials often does not reflect the duration of impact of the intervention. These issues are commonly addressed with modeling. The reader is referred to an earlier guidance document addressing these issues [1].

There are also some common issues in cost-effectiveness analysis that are fundamental to all studies of this nature that will not be addressed in this article. These include study perspective, choice of discount rate for costs and outcomes, type of analysis (e.g., cost–utility, cost–benefit), types of costs that will be included (direct medical, nonmedical, etc.), and marginal versus average costing. These issues apply to all economic analyses, not just economic studies within clinical trials, and are well described in the literature.

Initial Trial Design Issues

  1. Top of page
  2. ABSTRACT
  3. Introduction
  4. Initial Trial Design Issues
  5. Data Elements
  6. Database Design and Management
  7. Analysis
  8. Reporting the Methods and Results
  9. Conclusions
  10. References

The quality of economic information that is derived from trials depends on the attributes of the trial's design. Those economic analyses often described as being conducted “alongside” clinical trials are indicative of an important practical design issue. Economic analysis is rarely the primary purpose of an experimental study. Nevertheless, it is important that the analyst contributes to the design of the study to ensure that the structure of the trial will provide the data necessary for a high quality economic study.

Appropriate Trial Design

The distinction between pragmatic and exploratory trials and the corresponding distinction between effectiveness and efficacy is well understood [2,3]. It is generally acknowledged that pragmatic effectiveness trials are the best vehicle for economic studies; however, it is usually necessary to undertake economic evaluations earlier in the development cycle where the focus is on efficacy, including phase III or even phase II drug trials, to provide timely information for pricing and reimbursement. Our report is meant to apply to both types of trials.

Large simple trials [4] are efficient for addressing clinical questions because they capture the main effects of treatments that have small to moderate impacts on large potential populations. They will also be efficient for answering economic questions for diseases or treatments where the bulk of costs derive from primary outcomes that are measured in the trial and for which the quality of life impacts are persistent, and thus can be measured infrequently.

An ideal follow-up period for an economic study is independent of the occurrence of clinical events, whether study-related or not. All patients should be followed for a common length of time or the full duration of the trial. Discontinuation of data collection because of a clinical event will fail to capture the important aspects of the disease under study: the adverse effects of the event on quality of life, resource use, and cost.

Sample Size and Power

In an ideal world, the economic appraisal would be factored into sample-size calculations using standard methods [5,6] based on asymptotic normality, or by simulation [7]. Nevertheless, it is common for the sample size of the trial to be based on the primary clinical outcomes alone. As a consequence, it is possible that the economic comparisons will be underpowered. Analysts should calculate the likely power of the study at the design stage to ascertain whether, given the proposed sample size, it makes sense to undertake an economic appraisal. In many cases, sample-size restrictions will necessitate focus on estimation rather than hypothesis testing of the economic outcomes. In cases where researchers wish to set up formal hypotheses for the economic analyses, these should be stated a priori including the thresholds (e.g., $50,000 or $100,000/quality-adjusted life-year [QALY]) and the power to detect when incremental analysis meets or exceeds those thresholds [8].

Study End Points

The choice of primary end point in a clinical study may not correspond with the ideal end point for economic evaluation. For example, the use of composite clinical end points is common in clinical trials (e.g., fatal events and nonfatal events combined) to provide greater statistical power. Nevertheless, cost per composite clinical end point is often an unsatisfactory summary measure for an economic analysis, in part because the different outcomes are rarely of equal importance. It is recommended that clinical end points used in economic evaluations be presented in disaggregated form. We recommend weighting end points (e.g., by utilities) so that they yield a measure of QALYs in the case of cost-effectiveness analysis, or a monetary benefit measure in the case of cost–benefit analysis. Alternatively, quality of life values may be obtained within the trial at regular intervals and the QALYs estimated as one of the outcomes of the trial.

If possible, one should avoid using intermediate end points (e.g., percent low-density lipoprotein reduction) as the measure of benefit; however, intermediate outcome measures are often employed when the costs of conducting a long-term trial are prohibitive. When use of intermediate outcomes is unavoidable, additional evidence is needed to link them with long-term costs and outcomes. If such a link is not reliable or is unavailable, the analyst should argue for follow-up sufficient to include clinically meaningful disease end points.

Appropriate Follow-up

Economic analyses ideally include lifetime costs and outcomes of treatment. Nevertheless, clinical trials rarely extend beyond a few years and are often conducted over much shorter periods. In practice, consideration of the follow-up period for the trial involves the relationship between intermediate end points gathered in the short run and long-term disease outcomes—the stronger that relationship, the more a reliance on intermediate end points can be justified.

Data Elements

  1. Top of page
  2. ABSTRACT
  3. Introduction
  4. Initial Trial Design Issues
  5. Data Elements
  6. Database Design and Management
  7. Analysis
  8. Reporting the Methods and Results
  9. Conclusions
  10. References

The design issues discussed above will impact decisions about which resource use and outcome measures to collect, how to collect them, and how to value them. To begin, we recommend developing a description of the clinical processes for the intervention and how the intervention may impact resource use in the short and long term [9]. In this process, study perspective affects the types of resource use—both medical and nonmedical—that should be considered for inclusion in the study. For example, the societal perspective might include patients’ costs for transportation; time spent undergoing treatment, caregiver time, and nonmedical goods and services attributable to the disease or treatment.

After resources have been identified, logistical and cost considerations often require prioritization as to which data elements will be collected. We recommend that analysts focus on “big ticket” items as well as resources that are expected to differ between treatment arms [10]. For the items chosen, the study should collect information on all resource use not just that considered to be disease- or intervention-related [11,12]. If necessary, the distinction between costs related and unrelated to the disease can be attempted at the analysis phase.

For each resource, the level of aggregation desired should be prospectively determined. As an example, hospitalizations could be considered in disaggregated units, such as nursing time, operating-room time, and supplies, or in highly aggregated units, such as numbers of hospitalizations or days in the hospital. The decision is typically driven by the characteristics of the intervention under study, resource use patterns expected, and availability of unit costs, also called unit prices or price weights. For practical reasons, the level of aggregation may be varied by whether the resource use is thought to be disease- or intervention-related.

In some trial settings, secondary data such as hospital bills or claims data are available. These data sources can provide an inexpensive, detailed accounting of some resources consumed by patients, and should be used when available [13].

Valuing Resource Use

Unit costs should be consistent with measured resource use, the study's perspective, and its time horizon. For example, if the level of resource aggregation is hospital days that include intensive care unit (ICU) and non-ICU days, the unit costs should reflect the costs of each type of service [13,14]; if the study is conducted from the societal perspective, unit costs should reflect social opportunity cost. In selecting a costing approach, analysts should weigh issues of accuracy/bias, cost, feasibility, and generalizability [15]. For a more thorough discussion on costing issues, we refer readers to Drummond et al. [16] and Luce et al. [9].

Unit cost estimates are rarely derived via direct observation of patients in trials. Most often they are derived from substudies that are divorced from the trial itself. Sometimes, unit costs will be estimated from trial centers, but more commonly they are derived from national data [17–20]. If a reliable method for cost imputation exists (e.g., diagnostic group weights), one can combine the two methods by collecting a limited set of unit costs in a number of countries and imputing the remainder of costs [21]. Ideally, unit costs to be used for resource costing should be finalized prior to unblinding the trial data.

Because relative costs can affect resource use, in general one should use unit cost estimates that are specific to the precise intervention under study and specific events of interest in the trial but which are generalizable to the population that the study is intended to inform.

When unit costs from more than one country are used in an analysis (e.g., the pooled analysis, when resource use in a country is multiplied by unit costs from that country), results have to be converted to a common currency if they are to be compared appropriately. Purchasing power parity adjustments are recommended for such conversion [22,23].

Selecting and Tracking Measures of Outcomes

Because cost–utility analyses are widely accepted, we recommend that analysts collect preference weights as part of clinical trials. The most common method of assessing preferences is the use of a preference-weighted health state classification system such as the EuroQol-5D [24,25], one of the three versions of the Health Utilities Index [26–29], or the Quality of Well-Being Scale [30]. Analysts may also consider the inclusion of a rating scale to measure patient-based preferences [31]. Frequency and timing of these assessments should capture changes in patients’ quality of life that may be affected by the treatment but will be influenced by the disease severity of the study population, the study duration, the timing of trial visits, and patient burden [32].

Other options for collecting preference data include direct-elicitation methods such as standard gamble or time-tradeoff exercises. These methods have certain theoretical advantages; however, their use in clinical trials is often difficult. Trained interviewers or computerized applications are routinely used to conduct such exercises [16,33]. Also, many respondents have difficulty understanding and completing the exercises [34–36]. Finally, there is some evidence that these measures are generally unresponsive to changes in health status [37–40]. At present, the balance between the feasibility and desirability of using direct-elicitation methods in clinical trials remains an issue to be decided on a case-by-case basis.

Database Design and Management

  1. Top of page
  2. ABSTRACT
  3. Introduction
  4. Initial Trial Design Issues
  5. Data Elements
  6. Database Design and Management
  7. Analysis
  8. Reporting the Methods and Results
  9. Conclusions
  10. References

Ideally, collection and management of the economic data should be fully integrated into the clinical data. As such, there should be no distinction between the clinical and economic data elements. As with any prospective study, there should be a plan for ongoing data quality monitoring to address missing and poor quality data issues immediately. Queries should be managed on an ongoing basis rather than at the end of the trial to maximize data completeness and quality, and the timeliness of final analysis.

Informed consent for clinical studies does not routinely include provisions for collection of economic data, particularly from third-party databases. Therefore, explicit language should be included in trial consent documents. The consent forms should allow for capture of pre- and post-trial economic data if such data are necessary for the economic analysis.

Collection of economic data may reveal adverse events, such as hospitalization, not otherwise found in the clinical data. Data handling procedures are necessary to maintain consistency between economic and safety databases.

In reality, the clinical analysts are usually separate from the economic analysts. The clinical data elements and data formatting procedures needed for the economic analysis should be prespecified such that transfer of all necessary data for the economics study is timely and complete.

Analysis

  1. Top of page
  2. ABSTRACT
  3. Introduction
  4. Initial Trial Design Issues
  5. Data Elements
  6. Database Design and Management
  7. Analysis
  8. Reporting the Methods and Results
  9. Conclusions
  10. References

Guiding Principles

The analysis of economic measures should be guided by a data analysis plan. A prespecified plan is particularly important if formal tests of hypotheses are to be performed. Any tests of hypotheses that are not specified within the plan should be reported as exploratory. In addition, the plan should specify whether regression or other multivariable analyses will be used to improve precision and to adjust for treatment group imbalances. The plan should also identify any selected subgroups and state whether a no intention-to-treat analysis will be conducted.

Although the specific analytic methods used in the analysis of resource use, cost, and cost-effectiveness are likely to differ, there are several analysis features that should be common to all economic analyses alongside clinical trials:

  • • 
    The intention-to-treat population should be used for the primary analysis.
  • • 
    A common time horizon(s) should be used for accumulating costs and outcomes; a within-trial assessment of costs and outcomes should be conducted, even when modeling or projecting beyond the time horizon of the trial.
  • • 
    An assessment of uncertainty is necessary for each measure (standard errors or confidence intervals for point estimates; P values for hypothesis tests).
  • • 
    A (common) real discount rate should be applied to future costs and, when used in a cost-effectiveness analysis, to future outcomes.
  • • 
    If data for some subjects are missing and/or censored, the analytic approach should address this issue consistently in the various analyses affected by missing data.

Trial Costs

The purpose of clinical trial cost analysis is to estimate costs, cost differences associated with treatment, the variability of differences, and whether the differences occurred by chance.

Once resources have been identified and valued, differences between groups must be summarized. Arithmetic mean cost differences are generally considered the most appropriate and robust measure [41]. Nevertheless, cost data often do not conform to the assumptions for standard statistical tests for comparing differences in arithmetic means [42–44]. They are usually right-skewed and truncated at zero because of small numbers of high-resource-use patients, many patients who incur no costs, and the impossibility of incurring costs less than zero. In most cases, the nonparametric bootstrap is an appropriate method to compare means and calculate confidence intervals [45,46]. Other common nonparametric tests (e.g., Wilcoxon) compare medians and not means and thus are not appropriate [47–49]. Transformations to normalize the distribution are not straightforward and are often sensitive to departures from distributional assumptions. Retransformation to the original scale of costs must include transformation of error terms [50–54].

The same distributional issues that affect univariate tests of costs also affect use of costs as a dependent variable in a multivariate regression analysis. The underlying distribution of costs should be carefully assessed to determine the most appropriate approach to conduct statistical inference on the costs between treatment groups [55]. The choice of the multivariate model requires careful consideration: ordinary least squares and generalized linear models perform differently in terms of bias and efficiency of estimation, depending upon the underlying data distribution [51]. If differences in resource use or subsets of costs are to be estimated, similar considerations regarding the appropriateness of statistical tests based on distributional assumptions should be applied.

When study participants use large amounts of medical services that are unrelated to the disease or treatment under study, it may be difficult to detect the influence of the treatment on total health-care costs. One approach to addressing this problem is to conduct secondary analyses that evaluate costs that are considered related to the disease or treatment under study. If such analyses are performed, it is important to identify services that were deemed “disease-related” versus those deemed “unrelated,” and to display costs for each component in the treatment and control arms.

Outcomes

When one of the trial's clinical end points is also used as the outcome for the cost-effectiveness analysis (e.g., in-trial mortality), it is generally most transparent to adopt the methods used in the clinical analysis for the primary analysis plan, particularly if the clinical result is cited in product labeling or a publication. In some cases, the clinical analysis methods are not appropriate for economic analysis (e.g., the clinical analysis may focus on relative treatment differences, while the economic analysis needs absolute treatment differences); if other outcomes are used for the economic analysis, the linkage between the clinical and economic measures should be clearly specified. Analyses of outcomes for the cost-effectiveness study may employ multivariable or other methods that are consistent with the cost analysis or otherwise appropriate for the data [56–60]. Cost-effectiveness analysis should still be performed if the clinical study fails to demonstrate a statistically significant difference in clinical end points. In situations where cost-minimization analysis is conducted, the analyst should also conduct joint analysis of costs and outcomes to convey information about the likelihood of an intervention being cost-effective.

Using nonclinical effectiveness end points, such as QALYs, involves both construction and analysis. Health state utilities, either collected directly from trial patients or imputed based on observed health states, can be transformed into QALYs using standard area-under-the-curve methods [16,61]; a recent consideration involves adjusting for changes in health [62]. Simple analysis of means is the usual starting point; refinements may include adjusting for ceiling effects [63] and modeling of longitudinal effects [64,65].

Missing and Censored Data

Missing data are inevitable in economic analyses conducted alongside trials. Such data can include item-level missingness and missingness because of censoring. In analyzing data sets with missing data, one must determine the nature of the missing data and then define an approach for dealing with the missing data. Missing data may bear no relation to observed or unobserved factors in the population (missing completely at random), may have a relationship to observed variables (missing at random), or may be related to unobserved factors (not missing at random) [66]. Eliminating cases with missing data is not recommended because it may introduce bias or severely reduce power to test hypotheses. Nevertheless, ignoring small amounts of missing data is acceptable if a reasonable case can be made that doing so is unlikely to bias treatment group comparisons.

Imputation refers to replacing missing fields with estimates. If one chooses to impute missing data, most experts recommend multiple imputation approaches, as they reflect the uncertainty that is inherent when replacing missing data [67–69]. Most commonly used statistical software packages include programs for imputation of missing data. A review of these programs can be found at http://www.multiple-imputation.com[70].

Censoring can be addressed with a number of approaches. Most assume that censoring is either completely at random [71] or at random [72–76]. Nevertheless, nonrandom censoring is common, and external data sources for similar patients may be required to both identify and address it.

Summary Measures

One or more summary measures should be used to characterize the relative value of the treatments in the clinical trial. Three general classes of summary measures are available that differ in how the incremental costs and outcomes are combined into a single metric:

  • • 
    Ratio measures (e.g., incremental cost-effectiveness ratios) are obtained by dividing the incremental cost by the incremental health benefit.
  • • 
    Difference measures (e.g., net monetary benefits) rely on the ability to define a common metric (such as monetary units) by which both costs and outcomes can be measured [77–79].
  • • 
    Probability measures (e.g., acceptability curves) characterize the likelihood the new treatment will be deemed cost-effective based on the incremental costs and outcomes [80,81].

The difference measures and probability measures are calculated for specific values of “willingness to pay” or cost-effectiveness thresholds. Because these values may not be known and/or vary among health-care decision makers, one should evaluate the summary measure over a reasonable range of values.

Uncertainty

Results of economic assessments in trials are subject to a number of sources of uncertainty, including sampling uncertainty, uncertainty in parameters such as unit costs and the discount rate, and—when missing data are present—imputation-related uncertainty.

Sampling uncertainty.  Because economic outcomes in trials are the result of a single sample drawn from the population, one should report the variability in these outcomes that arises from such sampling. Variability should be reported for within-group estimates of costs and outcomes, between-group differences in costs and outcomes, and the comparison of costs and outcomes. One of the most common measures of this variability is the confidence interval.

Policy inferences about adoption of a therapy should be based on one's level of confidence that its cost for a unit of outcome, for example, a QALY, is less than one's maximum willingness to pay. Thus, one should report ranges of ceiling ratios for which one: 1) is confident that the therapy is good value for the cost; 2) is confident that the therapy is not good value; and 3) cannot be confident that the therapies differ from one another. Policymakers can then draw inferences by identifying their maximum willingness to pay and determining into which of the ranges it falls.

These ranges of ceiling ratios where one can and cannot be confident about a therapy's value can be calculated by use of confidence intervals for cost-effectiveness ratios [82,83], confidence intervals for net monetary benefit, or the acceptability curve. One advantage of the confidence interval for the cost-effectiveness ratio is that its limits directly define the boundaries between these ranges. One advantage of the acceptability curve is that it defines the boundaries between these ranges for varying levels of confidence that range from 0 to 100%.

Parameter uncertainty.   Uncertainty related to parameter estimates such as unit costs and the discount rate should be assessed by use of sensitivity analysis. For example, if one uses a discount rate of 3%, one may want to assess the impact of this assumption by repeating the analysis but using a 1% or a 5% rate. Analysts should evaluate all parameters that, when varied, have the potential to influence policy decisions. Measures of stochastic uncertainty and sensitivity analysis for parameter uncertainty are complements, not substitutes. Thus, when conducting sensitivity analysis, one should report both the revised point estimate and revised 95% confidence intervals that result from the sensitivity analysis.

Imputation uncertainty.   Finally, some methods employed to address missing or censored data (e.g., use of an imputed mean) may artificially reduce estimates of stochastic uncertainty. One should make efforts to address this shrinkage when reporting stochastic uncertainty, for example, by bootstrapping the entire imputation and estimation process.

Identifying and Addressing Threats to External Validity/Generalizability

Because of the “artificiality” of most clinical trials, they have high internal, but may have low external validity. The threats to external validity come from:

  • • 
    protocol-driven resource use (which could bias costs in each treatment arm upwards if included and downwards if excluded, but it is generally difficult to know how this will bias the difference between treatments);
  • • 
    unrepresentative recruiting centers (e.g., large, urban, academic hospitals);
  • • 
    inclusion of study sites from countries with varying access and availability of health-care services (e.g., rehabilitation, home care, or emergency services);
  • • 
    restrictive inclusion and exclusion criteria (patient population, disease severity, comorbidities);
  • • 
    artificially enhanced compliance.

The external validity can best be increased by making the trial more naturalistic during the design phase of the trial [16,84].

Additional threats arise with international trials, as treatment pathways, patient and health-care provider behavior, supply and financing of health care, and unit costs (prices) can differ tremendously between countries [85–92]. Pooled results may not be representative of any one country, but the sample size is usually not large enough to analyze countries separately.

It is common to apply country-specific unit costs for pooled trial resource use to estimate country-specific costs. In practice, this approach yields few qualitative differences in summary measures of cost-effectiveness among countries of similar levels of economic development but may not adjust for important country-specific differences [93,94]. Rather, intercountry differences in population characteristics and treatment patterns are more likely to influence summary measures between countries rather than differences in unit costs. Recommended approaches to address this issue include [93,95–97]:

  • • 
    hypothesis tests of homogeneity of results across countries (and adjusting the resource use in other countries to better match those seen in country X);
  • • 
    multivariate cost or outcome regressions to adjust for country effects (e.g., include country dummies or adjusted gross national product per capita as covariates);
  • • 
    multilevel random effects model with shrinkage estimators.

Modeling Beyond the Time Horizon of the Trial

The cost-effectiveness observed within the trial may be substantially different from what would have been observed with continued follow-up. Modeling is used to estimate costs and outcomes that would have been observed had observation been prolonged. When modeling beyond the follow-up period for the trial, it is important to project costs and outcomes over the expected duration of treatment and its effects.

Direct modeling of long-term costs and outcomes is feasible when the trial period is long enough, or if at least a subset of patients are observed for a longer time and provide a basis for estimating other patients’ outcomes. Parametric survival models estimated on trial data are recommended for such projections. In cases where such direct modeling is not feasible, it may be possible to “marry” trial data to long-term observational data in a model. In either case, good modeling practices should be followed. The reader is referred to the consensus position of the ISPOR Task Force on Good Research Practices—Modeling Studies for discussion of modeling issues [1].

Cost-effectiveness ratios should be calculated at various time horizons (e.g., 2, 5, 10 years, or as appropriate for the disease), both to accommodate the needs of decision makers and to provide a “trajectory” of summary measures over time. The effects of long-term health-care costs not directly related to treatment should be taken into account as well as possible [98]. As always, assumptions used must be described and justified, and the uncertainty associated with projections must be taken into account.

Subgroup Analysis

The dangers of spurious subgroup effects are well-known. For example, the probability of finding a difference due solely to random variation increases with the number of differences examined unless the alpha-level is scrupulously adjusted. Yet economics requires a marginal approach, so proper subgroup analysis can be vital to decision makers. The focus should be on testing treatment interactions on the absolute scale, with a justification for choice of scale used. In cases where prespecified clinical interactions are significant, subgroup analyses may be justified. Subgroup analysis based on prespecified economic interactions should also be reported.

Reporting the Methods and Results

  1. Top of page
  2. ABSTRACT
  3. Introduction
  4. Initial Trial Design Issues
  5. Data Elements
  6. Database Design and Management
  7. Analysis
  8. Reporting the Methods and Results
  9. Conclusions
  10. References

We anticipate that the results of an economic analysis will have a variety of audiences. Correspondingly, detailed and comprehensive information on the methods and results should be readily available to any interested reader. Journal word limits often necessitate parsimony in reporting; therefore, we recommend that detailed technical reports be made available on the World Wide Web.

A number of organizations have developed minimum reporting standards for economic analyses (e.g., study perspective, discount rate, marginal vs. average outcomes and analyses) [99,100]. The principles in these should be adhered to in all economic studies. Here, we highlight issues particular to economic studies conducted alongside clinical trials.

The report should include these elements:

Trial-Related Issues

  • • 
    General description of the clinical trial, including patient demographics, trial setting (e.g., country, tertiary care hospital), inclusion and exclusion criteria and protocol-driven procedures that influence external validity, intervention and control arms, and time horizon for the intervention and follow-up;
  • • 
    Key clinical findings.

Data for the Economic Study

  • Clear delineation between patient-level data collected as part of the trial versus data not collected as part of the trial:
    • • 
      Trial: health related quality of life survey instruments, data sources, collection schedule (including the follow-up period), etc.
    • • 
      Nontrial: unit costs, published utility weights, etc.
  • • 
    Amount of missing and censored data.

Methods of Analysis

  • • 
    Construction of costs and outcomes;
  • • 
    In cases where the main clinical end point is used in the denominator of the incremental cost-effectiveness ratio and different methods were used to analyze this end point in the clinical and economic analyses, any differences in the point estimates should be explained;
  • • 
    Methods for addressing missing and censored data;
  • • 
    Statistical methods used to compare resource use, costs, and outcomes;
  • • 
    Methods and assumptions used to project costs and outcomes beyond the trial period;
  • • 
    Any deviations from the prespecified analysis plan and justification for these changes.

Results

  • • 
    Resource use, costs, outcome measures, including point estimates and measures of uncertainty;
  • • 
    Results within the time horizon of the trial;
  • • 
    Results with projections beyond the trial (if conducted);
  • • 
    Graphical displays of results not easily reported in tabular form (e.g., cost-effectiveness acceptability curves, joint density of incremental costs and outcomes).

When there are economic analyses alongside several clinical trials for a given intervention, attempts may be made to estimate a summary cost-effectiveness ratio across trials (although the methods for this are not perfectly straightforward, i.e., clearly not a simple average of the individual incremental cost-effectiveness ratios). Data from economic analyses performed in the context of trials may also be used in independent cost-effectiveness models based on decision analysis or meta-analyses [84]. To facilitate synthesis of economic information from multiple trials, authors should report means and standard errors for the incremental costs and outcomes and their correlation.

Conclusions

  1. Top of page
  2. ABSTRACT
  3. Introduction
  4. Initial Trial Design Issues
  5. Data Elements
  6. Database Design and Management
  7. Analysis
  8. Reporting the Methods and Results
  9. Conclusions
  10. References

As decision makers increasingly demand evidence of economic value for health-care interventions, conducting high quality economic analyses alongside clinical studies is desirable because they provide timely information with high internal validity. This ISPOR RCT-CEA Task Force Report is intended to provide guidance to improve their quality and consistency. The task force recognizes that there are areas where future methodological research could further improve the quality and usefulness of these studies. Examples here include (among many): new approaches for pooling and analyzing data from multinational trials; issues related to multiple trial analysis, such as Bayesian learning designs, pooling of clinical trial data, or meta-analysis; design and analysis in trials where outcomes are valued in monetary units (i.e., willingness-to-pay studies); methods for projecting trial findings; appropriate methods for a priori selection of items of resource use to be included in trial protocols (e.g., whether to include outpatient services, nonstudy drugs); and selecting levels of aggregation of resources necessary for discriminating between intervention and control (e.g., counts of hospitalizations vs. length of stay). As these methods are identified and validated, they will be included in future versions of this guidance document.

Source of financial support: Support for this project was provided by ISPOR.

References

  1. Top of page
  2. ABSTRACT
  3. Introduction
  4. Initial Trial Design Issues
  5. Data Elements
  6. Database Design and Management
  7. Analysis
  8. Reporting the Methods and Results
  9. Conclusions
  10. References