Comparative Assessment for Medications and Devices: Apples and Oranges?


  • Scott Ramsey has no conflicts to declare.

Scott Ramsey, University of Washington, Member, Fred Hutchinson Cancer Research Center, Seattle, WA 98109-1024, USA. E-mail:


Over the past two decades, there have been multiple attempts by government to fund comparative effectiveness research (CER) in the US. Unfortunately, these efforts have been stymied, typically by Congress refusing to commit the scale of funding necessary to address the magnitude of clinical problems for which such information would influence practice. As a result, to date, the drug and device regulatory approval system has been the primary driver of the type of evidence that is generated to support a product launch. The great majority of evidence generated for regulatory approval could not be construed as comparative effectiveness research.

In an unprecedented move, new legislation passed by Congress known as the American Recovery and Reinvestment Act (ARRA) that was aimed at overcoming the economic downturn, includes $1.1 billion dollars to support CER in the US; the size of this budget is orders of magnitude more than has ever been committed to this field. These funds are being distributed via three main routes: the Office of the Secretary of Health and Human Services, the director of the National Institutes of Health, and the Agency for Healthcare Research and Quality.

Because of the ongoing debate around what constituted comparative effectiveness research, the ARRA legislation specifically called on the Institute of Medicine (IOM) to define comparative effectiveness and to set CER priorities. Ironically, comparative effectiveness research funds were already flowing out of the National Institutes of Health (NIH) before the IOM published its definition of CER (subsequently published August 4, 2009), or the IOM priority areas established. The IOM Committee working definition of CER is given below:

CER is the generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition, or to improve the delivery of care. The purpose of CER is to assist consumers, clinicians, purchasers, and policymakers to make informed decisions that will improve health care at both the individual and population levels (adapted from [1])

The NIH had set its own priorities and released more than $200 million dollars to fund several hundred research programs.

In light of the debate about what CER was, it is noteworthy that the IOM definition of CER includes consumers and purchasers as beneficiaries of the research, implying that costs or cost-effectiveness are valid metrics of comparison.

Regulation of Drugs and Devices in the US

The process of regulation that brings new drugs and devices to market illustrates the reason why many researchers and policymakers felt that comparative effectiveness research was needed. One of the first decisions the Food and Drug Administration (FDA) makes about a new technology is whether it is a drug or a device as defined in the Food and Drug and Cosmetic Act (Table 1). The regulatory requirements set by the FDA are quite different for these categories. After determining whether a product is a drug or device, meeting these requirements is key to accessing the health-care market.

Table 1.  Food and Drug Administration working definitions of drugs and devices
DrugsArticles that are recognized in the Official United States Pharmacopoeia, or Official National Formulary, or any supplement to any of them; they are articles (other than food) that are intended for use in the diagnosis, cure, mitigation, treatment, or prevention of disease in man or other animals that are intended to affect the structure or any function of the body.
DevicesArticles used in the diagnosis, cure, mitigation, treatment or prevention of disease or medical condition that affect the structure or function of the body, but that do not achieve their intended effect through chemical reaction or metabolism.

It is important to note that neither of these definitions requires that a drug or device show equivalence or superiority to best alternative care. Indeed, most products come to market with studies comparing them to placebo or best supportive care (i.e., care that does not intend to cure the condition).

Drug Labeling and Promotion

The drug label (stated indication) drives the use of drugs and devices in the US; it is limited to proven efficacy and safety. Although drugs may be prescribed off-label, by law, the drug or device cannot be promoted beyond the bounds of what is on the label.

Drug Development

Drug development is a very long and costly process that starts from basic research, preclinical and the investigational new drug application, and goes from Phase One to Phase Three (often called “pivotal”) clinical trials. Phase One clinical trials are the first human trials of a new drug. They typically involve testing the drug in a small number of healthy individuals to determine whether it is metabolized as expected, and what side effects occur. Phase Two trials are then conducted to test drug efficacy. They typically involve a small number of patients and are also often designed to establish an appropriate dosage, as well as to continue to build the safety profile. Phase Three trials are large controlled trials that assess safety and efficacy in the general patient population. They typically involve one or more large cohorts of patients, with international representation. The comparator may be a placebo, an active comparator, or current standard of care.

If a drug shows significant clinical improvement in outcomes and reasonable safety in two double-blind, controlled studies, the FDA may grant approval to market the drug, after which the product is launched. A typical time horizon to get from concept to approval is 8 to 10 years, although it may be longer. The estimated cost of bringing a new chemical entity to the market, including the legislated fee for the FDA review, is in the order of about $800 million dollars [2].

Limitations of the Regulatory Process for Drugs

Trial-based evaluations of safety and efficacy have several limitations for clinicians and other decision-makers. First, the trial based populations typically represents a select patient population rather than patients in the “real world;” as such, the results may not be generalizable. Second, as noted above, the comparator is typically not the best alternative treatment. Placebo or the weakest alternatives tend to be selected, rather than standard of care. Other limitations are that study endpoints are typically short-term, and the studies are often underpowered to determine population-wide safety endpoints.

Device Development

The FDA classifies devices that are submitted to them for approval into three classes:

  • 1Class I: Low-risk devices; e.g., band-aids, crutches. These types of devices are subject to general controls with low stringency.
  • 2Class II: Medium-risk devices; e.g., syringes. These types of devices are subject to general controls plus additional special controls that are often related to manufacturing processes (quality control).
  • 3Class III: High-risk devices; e.g., drug-eluting stents. These types of devices are subject to general controls and a system of premarket approval (PMA) that requires evidence generation.

There are two routes to getting a device approved through the FDA; premarket notification or 510(k), and premarket approval (PMA). The vast majority of devices get approved through the 510(k) process. PMA only applies to Class III devices. If it is permitted by the FDA, device manufacturers much prefer the 510(k) route, as it simply requires proving “substantial equivalence” to a currently marked device, and therefore is much less costly and faster than PMA (Table 2).

Table 2.  Regulatory approval processes for devices
 Premarket notification 510(k)Premarket approval
  • *

    For first application.

Standard fees$3,693$200,725
Small business fee$1,847 (≤$100 million in gross receipts or sales)≤$30 million in gross receipts or sales − fee is waived*
≤$100 million in gross receipts or sales − $50,181*
Review time90% completed within 90 daysAverage of 335 days

Premarket Notification, 510 (k)

The 510(k) process requires the manufacturer to demonstrate that the new device is substantially equivalent to an existing device that is already marketed in the US, with substantial equivalence being defined as meaning that the device is as safe and as effective as the existing device. It is worth noting that setting the regulatory standard at proving that a new device is as good as existing technology creates somewhat of a conundrum for purchasers and clinicians, as after gaining approval, the manufacturer may market their device as the best new thing.


This process applies when the FDA decides that the technology is not substantially equivalent to an existing technology because it is a novel entity; as such, it is considered to be of high-risk. In these cases, the manufacturer must prove reasonable assurance of safety and effectiveness. As shown in Table 2, the standard fee for PMA is much higher and the review time much longer than for 510(k). However, the major expense for the manufacturer is in cases where the FDA requires a prospective study (although not necessarily controlled) that demonstrate the safety and effectiveness of the device.

Limitations of the Regulatory Process for Devices

There are a number of limitations of the regulatory process for devices. First, there is currently no requirement by the FDA for randomized controlled trials of devices versus best alternative care. In fact, the vast majority go through the 510(k) process, which requires a comparison to an older device, but no trial of any sort. A second very important limitation is that operator skill is not part of the regulatory process. Evidence suggests that variations in operator skill can lead to significantly different patient outcomes. As a well-described example, perioperative mortality associated with lung volume reduction surgery with bovine pericardium (the device) was reduced from around 12–15% prior to the pivotal trial of this procedure to 5% when the procedure was carried out by skilled operators. However, bovine pericardium obtained FDA device approval without attention to the role of operator competence or its ultimate use in care. This example draws attention to the lack of focus on definitive outcomes and patient oriented outcomes.

Closing Remarks

The FDA applies very different processes to evaluating drugs and devices. Although the regulatory process for drugs stimulates the production of good quality evidence of whether the drug improves outcomes for a particular condition, this may imply that it is better than best alternative care or that the results from the clinical trial population are generalizable to the “real world.” Getting drugs from discovery to market takes years or even decades. The regulatory process for devices, on the other hand, is usually much faster. Because controlled clinical trials are rare for devices, most devices come to market with evidence that falls far short of drugs.

The FDA's approach to evaluating and regulating drugs and medical devices is dictated by statute. Changes in its approach for evaluating new medical devices—for example, to fit a comparative effectiveness framework—would require federal legislation. Because of the uncertainty it would create for the drug and device industries, wholesale adjustments to the FDA's current process of evaluating medical devices is highly unlikely in the foreseeable future. Therefore, one can expect that the bulk of CER will continue to be conducted on drugs and devices that have already been approved for marketing. Conducting CER outside of the FDA regulatory structure does have the advantage of allowing for novel study designs and comparisons that could address important questions in an efficient and potentially timely manner.

On moving forward, we should have great optimism that the new CER funding offers a unique opportunity to improve the knowledge base for medications and medical devices. First and foremost, it is hoped that CER improves patient care. It is also hoped that the information will help control health-care costs.

Source of financial support: Oxford Outcomes, the National Pharmaceutical Council, and Shire Pharmaceuticals.