Address correspondence to: Michael Drummond, Center for Health Economics, University of York, Heslington, York YO10 5DD, UK. E-mail: email@example.com
Objectives: Despite the growing number of pharmacoeconomic (PE)/health economic (HE) studies, very little is known about their use by decision makers. The objectives of the Task Force were to ensure that the good research practices of PE/HE studies pay attention to the needs of health-care decision makers and to develop a “toolbox” for the health-care decision maker wanting to interpret and use PE/HE studies.
Methods: The membership of the Task Force consisted of individuals involved in making decisions about the availability or use of medicines and researchers into the use of economic evaluations. The group communicated by E-mail and face-to-face meetings. A literature review of decision makers’ attitudes toward PE/HE studies and published economic evaluation guidelines was undertaken. In addition, a focus group discussion was held with opinion leaders in managed care pharmacy.
Results: The literature review identified 16 surveys of decision makers’ attitudes toward PE/HE studies and 15 published guidelines that outlined reporting requirements for economic evaluations. These were reviewed and classified. Based on the published literature and comments from decision makers, seven additional reporting requirements for studies were specified.
Conclusions: While the Task Force's additional reporting requirements may be helpful to decision makers, they raise a number of issues. These include the feasibility of meeting the additional requirements, whether decision makers should receive more education in economic evaluation, and whether there should be more study of health-care decision-making procedures themselves.
Despite the growing number of pharmacoeconomic/health economic (PE/HE) studies and the methodological developments in recent years, very little is known about the attitudes of decision makers toward such studies. That is, are studies used and, if so, do they have any impact on decision making?
Although PE/HE studies can be undertaken for academic interest, their main purpose is to help those making decisions about the allocation of health-care resources. Therefore, the objectives of the ISPOR Task Force on the use of PE/HE in health-care decision making were to ensure that the good research practices of PE/HE studies pay attention to the needs of health-care decision makers and to develop a “toolbox” for the health-care decision maker wanting to interpret and use PE/HE studies.
This report is organized in the following manner: First, the contexts for health-care decision making and the uses of PE/HE studies are discussed. Second, the Task Force's methods of working are outlined. Third, the current literature on decision makers’ attitudes toward PE/HE studies is reviewed. Fourth, elements of good practice in the reporting of PE/HE studies are specified. Finally, several issues for further research are outlined.
Contexts for Decision Making and the Uses of PE/HE Studies
There are several contexts for health-care decision making, which may vary from place to place. First, at the central level, decisions are made about policies and programs for the populations of particular countries or regions. In some jurisdictions these include centralized procedures for the pricing and reimbursement of pharmaceuticals (e.g., Australia, Ontario). In a wider range of jurisdictions there are national programs for prevention of disease, including screening and immunization.
Second, many policy decisions are made at the local level, namely the health plan, hospital, or practice. These may include the adoption of treatment guidelines or the inclusion of drugs on the local or regional formulary of that organization. In some countries, such as the United States, the majority of health-care resource allocation decisions are made at the local level.
Finally, health-care resource allocation decisions are made in all health-care systems at the patient level. However, in general the main application and relevance of PE/HE studies is at the central and local levels, although these decisions undoubtedly condition the treatment decisions taken by doctors on behalf of their patients. For example, if a given drug is not on the local formulary, or is at the third tier attracting high patient copay, physicians are less likely to recommend it for their patients if they are aware of the situation.
Whereas the same elements of good practice apply to PE/HE studies irrespective of the level of decision making they seek to inform, there are critical differences between the central and local levels that bear on the work of this Task Force. First, at the central level, expertise is generally available to evaluate the methodological quality of the studies. This may not always be the case at the local level. Second, at the central level there is usually a prescribed process for presenting data and a set of methodological guidelines that need to be followed. Again, this may not always be the case at the local level, although the Academy of Managed Care Pharmacy (AMCP) format is one attempt to introduce such a process .
The main implication of these differences is that, at the local level, there is much more concern over whether analyses can be trusted and whether there are potential biases in research sponsored by manufacturers. By contrast, at the central level, a company submission is, by definition, advocacy for the product and there is usually sufficient expertise available to undertake a detailed critical appraisal.
This suggests that, while elements of good practice in the conduct and reporting of PE/HE studies are relevant in all decision-making contexts, decision makers’ needs for assistance in interpreting studies are greater at the local level.
Methods of Working
The membership of the Task Force Core Group was drawn from two constituencies: 1) those involved in making decisions about the availability or use of medicines at the central or local level, and 2) those who had previously undertaken research into the use of economic evaluations. The Core Group was supported by a broader group of ISPOR members (the Reference Group) who offered feedback on the Task Force's suggestions. Finally, the Task Force had access to a researcher, who undertook the literature review of decision makers’ attitudes toward PE/HE studies and the classification of existing reporting guidelines.
The Core Group communicated mainly by E-mail but in addition met face-to-face at three consecutive ISPOR meetings in May 2001, November 2001, and May 2002.
Also, four members of the Core Group attended the AMCP Educational Meeting in Dallas, Texas, in October 2001. This provided the opportunity to organize a focus group discussion with 10 opinion leaders in managed care pharmacy, the outcome of which was very influential in shaping the Task Force's recommendations.
The Task Force's draft report was posted on the ISPOR website toward the end of November 2001. A wide range of comments was received, in particular from the ISPOR Special Interest Group on Managed Care/Pharmacy Benefit Management. Where relevant, these comments were taken into account in drafting the final report.
Finally, the recommendations of the Task Force were presented at the Fourth ISPOR European Congress in Cannes, France, in November 2001 and the Seventh Annual ISPOR International Meeting in Washington, DC, in 2002.
Review of Decision Makers’ Attitudes toward PE/HE Studies
The literature review was conducted by screening the references identified by a recent systematic review of the use of health technology assessment in health-care decision making, plus articles identified in previous literature reviews undertaken by members of the Core Group . The systematic review, being broad in scope, identified 1040 references. However, the vast majority were commentaries or dealt with aspects of health technology assessment other than economic evaluation. A much smaller number of papers (n = 16) reported empirical research (i.e., surveys) on decision makers’ attitudes toward PE/HE studies. These are summarized in Table 1. The general commentaries were used more generally to inform the Task Force's debate about the uses and limitations of PE/HE information in health-care decision making.
Table 1. Surveys of decision makers’ attitudes toward PE/HE studies
Main barriers to use of studies are: 1) short-term nature of the decision- making process; 2) problems in interpreting studies; 3) lack of timeliness in study results; 4) importance of other factors in decision making.
Decision makers from hospitals, HMOs, and third-party payers
Range and sophistication of decision-making processes vary. Hospitals focus on traditional financial analysis, with the exception of pharmacy committees, which conduct socioeconomic analyses. HMOs undertake outcomes assessments but exclude economics.
Respondents rated clinical effectiveness and cost-effectiveness assessments more useful than quality-of-life assessments. Most plans were considering establishing a partnership with a drug company for disease management and would support some form of regulation of pharmacoeconomic claims.
Members of GHAA, HIAA, and BCBS and other insurers
MDirs have final coverage authority in only 27% of plans. Barriers to making optimal decisions were lack of timely evidence on effectiveness and cost- effectiveness. HMO and small and nonprofit plans were two to three times more likely to list lack of cost-effectiveness data than their counterparts.
Members of GHAA, HIAA, and BCBS and other insurers
The majority of respondents were willing to cover technologies that are more effective and more costly (89%). Cost-effectiveness was listed as reason to support or deny coverage twice as often as cost alone.
Cost-effectiveness was a minor tool in pharmaceutical decision making. Reasons for not using CEA more often were lack of information on the potential cost offsets, lack of independent sponsorship, and inadequate expertise in economic evaluation.
Government officials, health-care managers, pharmacists, and physicians
Mail questionnaires, interviews, and focus groups
Studies are not widely used. Institutional problems, such as transferring budgets and lack of credibility of studies, are important barriers. A better explanation of practical relevance of results and more training in health economics are needed.
Government officials, health-care managers, and purchasers
Barriers in the use of studies are: 1) overriding concern with cost rather than quality; 2) difficulties in accessing clinical and cost-effectiveness data; 3) insufficient training in interpretation and use of studies; 4) lack of skills in translating evidence into practice.
Most managed care decision makers believe pharmacoeconomics information is important. The main barriers to using study results are lack of relevance, drug silo mentality, lack of credible information, lack of resources, no focus on long-term costs, and lack of expertise.
The main conclusions from the review were as follows: for a study to be useful in a given decision, the decision maker needs to be convinced of its reliability and relevance. A reliable study would be one giving accurate estimates that are free from bias. A relevant study would be one containing results that apply to the decision maker's own setting. (Some authors use the terms “internal” and “external validity” to refer to the same concepts.) The main findings, in relation to reliability and relevance, from the various surveys are summarized below.
Issues Relating to Reliability
A major general concern of decision makers is the lack of transparency in the reporting of PE/HE studies. This concern applies to all studies, but is probably greatest in the case of modeling studies, with which most health-care decision makers are less familiar. The lack of transparency also partly fuels decision makers’ concerns about the potential sponsorship bias in PE/HE studies.
In addition to the relative lack of transparency in modeling studies, decision makers are often concerned about the extensive use of assumptions and the extrapolation of benefits over a time scale not directly observed in the clinical trials themselves. For example, in one economic evaluation of cholesterol-lowering therapy, only 10% of the benefit (in life extension) was observed during the trial itself . The remainder came from an extrapolation, over the lifetime of patients, from events (such as angina) observed during the trial follow-up period.
Whereas most economists would prefer the use of extrapolation, especially if this leads to the consideration of a more relevant time horizon or more relevant outcome, decision makers tend to prefer the observed to the unobserved. The same is true of the analysis of practice patterns and treatment costs, where the estimates from physician expert panels are usually considered inferior to data from a clinical trial, patients’ charts, or an administrative database.
Finally, several of the surveys indicate that decision makers are less comfortable than economists with the methods for calculating quality-adjusted life-years (QALYs) and willingness-to-pay. First, they find the concepts behind these benefit measures a little difficult to understand. Second, they have some concerns about the reliability of the estimation methods themselves, and third, in respect of QALYs, they have a more general concern about the aggregation of health benefits in a single index. Therefore, decision makers often prefer to see the various components of benefit presented in a cost-consequences analysis.
Issues Relating to Relevance
Probably the main issue relating to relevance is that typically PE/HE studies do not explore budgetary impact. Whereas the cost-effectiveness ratio gives an indication of the value for money from a therapy, it says nothing about total cost. On the other hand, the decision maker is often more concerned about affordability, which obviously depends on the overall volume of patients likely to benefit from the therapy and on whose budget the costs are likely to fall. Therefore, decision makers often prefer to see various budgetary perspectives explored (along with the societal perspective), as well as an estimate of overall budgetary impact.
In the case of managed care in the United States, this may represent a real challenge, owing to the diversity of plans. However, it may be possible to develop a “reference case” for managed care, or to show budgetary impact, under different assumptions, for various time frames (e.g., 2 years, 5 years, etc.) .
A common justification for investments in higher cost therapies is that savings will be made elsewhere in the health-care system (on other budgets) or in the future. Even those decision makers not adopting a “silo” mentality (i.e., concern only for their own budget) sometimes doubt whether many of the savings will actually be achieved. For example, this may be dependent on changes in the behavior of individual physicians. Of course, in many cases economists refer to freed resources rather than financial savings. That is, the benefit from a shorter length of hospital stay is that the vacated bed can be used in the treatment of another patient. Although decision makers understand these arguments, it is often difficult for them to take these on board when living within a financial budget constraint. Indeed, a hospital or a health plan could get into financial difficulties by adopting too many cost-effective interventions.
One way of bringing together the value-for-money and budgetary considerations would be to explore the cost-effectiveness ratios and budgetary impacts of treating different subgroups of patients within the total patient population; subgroups are often defined by indication, or by various pretreatment risk factors. The current consideration of these issues in PE/HE studies is, at best, erratic. However, it is also controversial, because there may be insufficient patient numbers in clinical studies to demonstrate differences between subgroups of the overall patient population at the conventional level of statistical significance.
Finally, some decision makers doubt whether PE/HE studies undertaken in other locations apply in their settings. There is a substantial literature on the transferability of PE/HE studies and economists have developed methods to adapt study results from one location to another . However, the individual decision maker may still require a presentation of study results that reflects the local situation. Often this can be dealt with by the use of sensitivity analysis or interactive models.
Recommendations for the Reporting of PE/HE Studies
There are now a number of guidelines for the conduct and reporting of PE/HE studies. Some originate from decision-making bodies, in those jurisdictions where there is a formal requirement for cost-effectiveness evidence. Others originate from groups of academic researchers or related groups interested in maintaining methodological standards in this field of research.
Twenty-five existing guidelines have recently been reviewed by Hjelmgren et al. . They concluded that, while there were differences among the published guidelines, there was substantial harmonization of methodological standards. The level of agreement on methodological aspects was slightly higher for the formal guidelines than for informal guidelines or general guidelines on health economic methods.
Not all the available guidelines for PE/HE studies specify a standard reporting framework or template. Details of 15 that do are given in Fig. 1. It can be seen that there is a fair amount of agreement between the different guidelines in terms of reporting requirements. In the main they are aimed at increasing transparency (e.g., state what comparator was used), although sometimes they embody elements of methodological prescription (e.g., present the results with costs and effects discounted at 5%). Our analysis indicates that there is considerable agreement on what should be reported, even if the methodological prescriptions differ slightly from guideline to guideline. The article by Hjelmgren et al.  includes detail of the methodological prescriptions.
Relevance of Other Task Force Reports
Four of the other ISPOR Task Forces deal with aspects of study methodology, namely, those on modeling, prospective studies, retrospective (database) studies, and quality-of-life. Some of the reports of these Task Forces make methodological recommendations, many of which are carried forward as requirements for reporting of results. Therefore, we refer the reader to the reports of the relevant Task Forces, particularly in relation to any requirements for reporting.
Additional Reporting Requirements for Decision Makers
Because of the concerns raised by decision makers about the reliability and relevance of PE/HE studies, we propose some additional reporting requirements below. This represents an ideal list and we recognize that it may not be possible for the authors of PE/HE studies to address all these points in a published article. However, serious consideration should be given to addressing them in formulary submissions and other direct communications to decision makers.
Description of relevant patient population(s).
The value for money of a given therapy depends on the patient population(s) in which it is used. The size of the patient population also affects budgetary impact. Therefore, the study report should clearly identify the relevant patient population(s) and, if possible, their size in the jurisdiction concerned.
Budgetary perspectives and budget impact.
Decision makers are interested in the cost of adopting the new therapy on their own budget and other budgets in their organization. Therefore, the study report should clearly identify the relevant budgets and the impact on each of adopting the new therapy.
Decision makers appreciate a disaggregated presentation of the study costs and outcomes, prior to any aggregation in an incremental cost-effectiveness or cost–utility ratio. Outcomes (consequences) could include changes in survival, quality of life, or indicators of patient satisfaction. Therefore, the study report should include disaggregated costs and outcomes, comparing the new therapy with the existing one (i.e., the most widely used therapy in the setting concerned).
Costs, consequences, and cost-effectiveness by patient subgroup.
Where there are relevant subgroups of the patient populations, decision makers are interested in how value-for-money varies by subgroup. Subgroups may be defined by clinical indication, risk factors, or previous exposure to treatment. Therefore, where relevant and feasible, the study report should present costs, consequences and incremental cost-effectiveness ratio by patient subgroup.
Practical implications of adopting the new therapy.
Decision makers sometimes find it difficult to understand the practical implications of adopting a therapy with a given incremental cost-effectiveness ratio. An alternative way of presenting results would be to explain what the adoption of the new therapy might mean in terms of budgetary impact and implications for the health of the relevant patient population(s). The analyst might also attempt to explain how and when savings in the use of other health-care resources may be achieved, although we recognize that much of this is context-specific.
Therefore, the study report should attempt to explain the impact, in practical terms, of adopting the new therapy.
Listing of key assumptions and data sources.
A key concern of decision makers is transparency in the reporting of PE/HE studies. At the local level, in particular, decision makers may not have the time or expertise to undertake detailed critical appraisals of studies. Therefore, the study report should list all the key assumptions and data sources.
Sensitivity analyses using the decision maker's own data and assumptions.
Economic data do not easily transfer from place to place, and it is known that a number of factors are likely to affect the cost-effectiveness of health-care interventions. In particular, it would be useful to know which parameters have the biggest impact on study results. Therefore, the study report (or model) should facilitate sensitivity analyses, using the decision maker's own data and assumptions.
The Task Force's recommendations for additional reporting requirements are summarized in Table 2.
• The study report should clearly identify the relevant patient population(s) and, if possible, their size in the jurisdiction concerned.
• The study report should clearly identify the relevant budgets, and the impact on each, of adopting the new therapy.
• The study report should include disaggregated costs and outcomes, comparing the new therapy with the existing one.
• Where relevant, the study report should present costs, consequences, and incremental cost-effectiveness ratio by patient subgroup.
• The study report should attempt to explain the impact, in practical terms, of adopting the new therapy.
• The study report should list all the key assumptions and data sources.
• The study report (or model) should facilitate sensitivity analyses using the decision maker's own data and assumptions.’
The Task Force's recommendations for additional reporting requirements raise a number of issues. First, will it be possible to meet these requirements in all study reports? It is unlikely that the space restrictions imposed by journals will allow such detailed reporting in published articles, although it would be useful if authors explored the practical implications of their findings to a greater extent than at present. However, in the main, we expect that the full implementation of our additional requirements will be much more feasible in the context of submissions to major payers, such as large managed care groups or government agencies.
Second, given the burden on manufacturers of making submissions, is there scope for harmonization of decision makers’ requirements? The article by Hjelmgren et al.  and our own review suggests that there are already considerable similarities between the various official requirements, such that it is possible for manufacturers to compile a core economic dossier for a product that can then be adapted slightly to meet individual requirements. Over time, as more payers request economic submissions, the pressure from manufacturers for greater harmonization is likely to increase, but some adaptation will always be required, given variations in patient populations, resource availability, and current clinical practice from one location to another.
Third, should decision makers at different levels receive more education in the methodology of PE/HE studies? The EUROMET survey by Hoffmann et al.  showed that the levels of sophistication and knowledge among decision makers varied greatly. Certainly, knowledge is greatest in those settings where there is a formalized procedure for considering economic data and less throughout the health-care system more generally. Therefore, in conjunction with the improvements in study reporting recommended here, more attention should be given to improving decision makers’ understanding of PE/HE studies.
Fourth, should there be more study of health-care decision making procedures themselves? In a recent editorial, Hutton and Brown  point out that, while decision makers often claim that PE/HE studies are irrelevant or unhelpful, the basis on which decisions are made is often unclear. If studies of the relative cost-effectiveness of treatment options are not relevant, what objectives are decision makers trying to fulfill when they make decisions?
Finally, what additional research could be conducted into decision makers’ needs for PE/HE information? One approach would be to undertake new surveys at the time of methodological advances in pharmacoeconomics. For example, do decision makers understand, or indeed prefer, the presentation of PE/HE results in the form of a cost-effectiveness acceptability curve ? Do they find the results of discrete choice experiments more informative than those of QALY or willingness-to-pay estimations ?
The other main approach, as opposed to more surveys, would be to undertake some controlled experiments, where one group of decision makers would receive various types of PE/HE information and the other not. One could then explore the impact that various types of data have on the outcome (i.e., decision). While more difficult to undertake, such research would not be reliant on decision makers’ survey responses and whether these reflect what they would actually do in practice.
Therefore, while we believe that our recommendations for additional reporting are a useful contribution, there is much more to be done in “bridging the gap” between the practitioners of PE/HE and the needs and concerns of those decision makers they seek to inform.
We are especially grateful to the health-care decision makers, members of the ISPOR Special Interest Group on Managed Care/Pharmacy Benefit Management, and other members of ISPOR that have taken time to give us their views on PE/HE studies and how the practice of reporting studies can be improved.
The Task Force is grateful to GlaxoSmithKline for an unrestricted grant. We are also grateful to François Schubert for his contribution to the work of the Task Force. M.B. also received financial support from Schering-Plough Inc. for a fellowship at the Centre for Health Economics.