Effectiveness is a concept central to Health Technology Assessment (HTA). Effectiveness is the measure of whether and how well an intervention works to mitigate disease and to enhance and extend human life in normal practice. Effectiveness is “the what” or the noun of HTA. In the current issue, Ridyard and Hughes have conducted an evaluation of how well researchers have studied costs within a selection of 100 primary research articles published in the British journal—Health Technology Assessment[1]. Their evaluation concerns “the how” of economic evaluation alongside clinical trials. They evaluated the methods and means of collecting, describing, and analyzing economic information. As a descriptive article concerning the methods of economic evaluations alongside clinical trials, this work will be useful for health outcomes researchers seeking to optimize the collection of economic information. Clearly, good research techniques serve to strengthen economic findings and make them more credible. But a potentially more important question, not addressed by Hughes, is whether a trial should lead to a companion economic evaluation.

The methods and means of effectiveness research are pivotal considerations in HTA. Effectiveness includes an assessment of efficacy, but it also considers whether a clinical intervention works as well in real world settings as it does in controlled trials. Both evaluative settings are needed for robust and credible estimates of benefit and risk in actual practice. Methodological concerns are central to effectiveness estimates. Regulatory authorities, including the Food and Drug Administration and the International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use, have issued the E10 Guidance (Choice of Control Group in Clinical Trials) [2]. The guidance describes the types and appropriateness of trial designs and provides a wealth of methodological considerations. Furthermore, the CONSORT initiative ( has issued guidance to enable readers and researchers to critically appraise and interpret clinical trials, and they have updated their recommendations to include pragmatic trials—or effectiveness considerations [3]. Why are these references important to economic evaluations of clinical trials and HTA? If one thinks of effectiveness as a noun and economic evaluation as an adjective, then they should align to optimize clarity and perhaps appropriateness. A proper estimate of effectiveness is the foundation for sound HTA decisions.

In 2008, a position paper was issued, entitled: National Institute for Health and Clinical Excellence (NICE): How does it work and what are the implications for the United States? The article was authored by Sorenson, Drummond, Kanavos, and McGuire and is available through Michael Drummond [4]. This article has a section on “assessment processes” and the evidence normally preferred and accepted by NICE. In the “hierarchy of preferred study designs,” as a reliable basis for relative therapeutic effects, the lowest level is “expert opinion,” level 3 is “observational studies without control,” level 2 is “controlled observational studies,” and level 1 is “randomized controlled trials” (RCTs). “The RCT is the ideal method for measuring treatment effects.”[5] Even though RCTs minimize the potential for bias and generate the most credible estimates of therapeutic effect, the D'Agostinos (Junior and Senior) argue that observational and nonrandomized studies have a role in codifying effectiveness and real-world experiences. They recommend statistical methods involving matching, stratification, and/or covariance adjustments to minimize bias in effectiveness research. Good research designs and statistical methods enable generation of the most credible estimates of effectiveness, which comprehends all levels of clinical evidence.

The story of drug-eluting stents provides an insightful case of how randomized and nonrandomized evidence evolves in an iterative fashion to address clinically important therapeutic questions of effectiveness [6]. Percutaneous coronary intervention with drug-eluting or bare metal stents to support optimal blood flow within the heart has been the subject of significant research over the past three decades. Stenting technology has proved therapeutically meaningful to reduce disease secondary to a heart attack, but not without surprises. The stent devices work best when adjunctive medical therapy is used. It was demonstrated in observational research that discontinuing antiplatelet medical therapy too soon after the stent placement was associated with late stent thrombosis leading to another myocardial infarction [7,8]. After a host of observational research was conducted to codify the late effect, JAMA selected two observational (registry) investigations by Win et al. and Beohar et al. and commissioned Drs. Harrington and Ohman to explore the contradictory and somewhat confusing findings [9,10]. One of their observations was: “While iterative and exploratory science is not suitable for development of clinical guidelines, mature science always is.”[6] They also explored the potential biases in the Win and Beohar investigations and recommended ways to minimize bias to derive a more mature estimate of therapeutic effect.

While demand for economic evaluations alongside clinical trials may be increasing [2,11], serious attention should be exercised concerning whether a trial warrants a companion economic evaluation. The clinical trial may not have a credible design or the observational study appropriate adjustments, and therefore, the associated economic evaluation will be flawed. Investigations with credible effectiveness estimates, well conceived, conducted, and characterized along the lines described in the E10, CONSORT and D'Agostinos publications, will probably deserve economic evaluations. These characteristics are associated with the clinical investigations, not with the economic evaluations. The economic findings are informative or hypothesis-generating. The clinical investigations provide the foundation for effectiveness estimates of benefit and risk. To my knowledge, no primary investigation, designed, sized, and powered to explore comparative economic effects, has ever been conducted; that is why I think of the economic evaluation as an adjective, rather than a noun.


  1. Top of page
  2. References
  • 1
    Ridyard CH, Hughes DA. Methods for the collection of resource use data within clinical trials: a systematic review of trials funded by the UK Health Technology Assessment programme. Value Health 2010;13:In press.
  • 2
    Department of Health and Human Services, Food and Drug Administration. International Conference on Harmonisation: choice of control group in clinical trials. Fed Regist 1999;64:5176780.
  • 3
    Zwarenstein M, Treweek S, Gagnier JJ, et al. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ 2008;337:18.
  • 4
    Sorenson C, Drummond M, Kanavos P, McGuire A. National Institute for Health and Clinical Excellence (NICE): how does it work and what are the implications for the U.S.? Available from:[Accessed April 2008.
  • 5
    D'Agostino RB Jr, D'Agostino RB Sr. Estimating treatment effects using observational data. JAMA 2007;297:3146.
  • 6
    Harrington RA, Ohman EM. The enigma of drug-eluting stents: hope, hype, humility and advancing patient care. JAMA 2007;297:202830.
  • 7
    McFadden EP, Stabile E, Regar E, et al. Late stent thrombosis in drug-eluting coronary stents after discontinuation of antiplatelet therapy. Lancet 2004;364:151921.
  • 8
    Spertus JA, Kettellamp R, Vance C, et al. Prevalence, predictors, and outcomes of premature discontinuation of thienopyridine therapy after drug-eluting stent placement: results from PREMIER registry. Circulation 2006;113:28039.
  • 9
    Win HK, Caldera AE, Maresh K, et al. Clinical outcomes and stent thrombosis following off-label use of drug-eluting stents. JAMA 2007;297:20019.
  • 10
    Beohar N, Davidson CJ, Kip KE, et al. Outcomes and complications associated with off-label and untested use of drug-eluting stents. JAMA 2007;297:19922000.
  • 11
    Reed SD, Anstrom KJ, Bakhai A, et al. Conducting economic evaluations alongside multinational clinical trials: toward a research consensus. Am Heart J 2005;149:43443.