A critical review of methods to evaluate the impact of FDA regulatory actions

Authors

  • Becky A. Briesacher,

    Corresponding author
    1. Meyers Primary Care Institute, a joint endeavor of Fallon Community Health Plan, Reliant Medical Group and the University of Massachusetts Medical School, Worcester, MA, USA
    • Division of Geriatric Medicine, University of Massachusetts Medical School, Worcester, MA, USA
    Search for more papers by this author
  • Stephen B. Soumerai,

    1. Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, MA, USA
    Search for more papers by this author
  • Fang Zhang,

    1. Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, MA, USA
    Search for more papers by this author
  • Sengwee Toh,

    1. Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, MA, USA
    Search for more papers by this author
  • Susan E. Andrade,

    1. Division of Geriatric Medicine, University of Massachusetts Medical School, Worcester, MA, USA
    2. Meyers Primary Care Institute, a joint endeavor of Fallon Community Health Plan, Reliant Medical Group and the University of Massachusetts Medical School, Worcester, MA, USA
    Search for more papers by this author
  • Joann L. Wagner,

    1. Meyers Primary Care Institute, a joint endeavor of Fallon Community Health Plan, Reliant Medical Group and the University of Massachusetts Medical School, Worcester, MA, USA
    Search for more papers by this author
  • Azadeh Shoaibi,

    1. Center for Drug Evaluation and Research, U.S. Food and Drug Administration, Silver Spring, MD, USA
    Search for more papers by this author
  • Jerry H. Gurwitz

    1. Division of Geriatric Medicine, University of Massachusetts Medical School, Worcester, MA, USA
    2. Meyers Primary Care Institute, a joint endeavor of Fallon Community Health Plan, Reliant Medical Group and the University of Massachusetts Medical School, Worcester, MA, USA
    Search for more papers by this author

  • Mini-Sentinel is a pilot project sponsored by the U.S. FDA to inform and facilitate development of a fully operational active surveillance system, the Sentinel System, for monitoring the safety of FDA-regulated medical products. Mini-Sentinel is one piece of the Sentinel Initiative, a multifaceted effort by the FDA to develop a national electronic system that will complement existing methods of safety surveillance. A related White Paper was developed as part of this initiative and can be found at: http://mini-sentinel.org/methods/methods_development/details.aspx?ID=1038?

Correspondence to: B. A. Briesacher, Division of Geriatric Medicine, University of Massachusetts Medical School, Biotech Four, Suite 315, 377 Plantation Street, Worcester, MA 01605, USA. E-mail: Becky.Briesacher@umassmed.edu

ABSTRACT

Purpose

To conduct a synthesis of the literature on methods to evaluate the impacts of FDA regulatory actions and identify best practices for future evaluations.

Methods

We searched MEDLINE for manuscripts published between January 1948 and August 2011 that included terms related to FDA, regulatory actions, and empirical evaluation; the review additionally included FDA-identified literature. We used a modified Delphi method to identify preferred methodologies. We included studies with explicit methods to address threats to validity and identified designs and analytic methods with strong internal validity that have been applied to other policy evaluations.

Results

We included 18 studies out of 243 abstracts and papers screened. Overall, analytic rigor in prior evaluations of FDA regulatory actions varied considerably; less than a quarter of studies (22%) included control groups. Only 56% assessed changes in the use of substitute products/services, and 11% examined patient health outcomes. Among studies meeting minimal criteria of rigor, 50% found no impact or weak/modest impacts of FDA actions and 33% detected unintended consequences. Among those studies finding significant intended effects of FDA actions, all cited the importance of intensive communication efforts. There are preferred methods with strong internal validity that have yet to be applied to evaluations of FDA regulatory actions.

Conclusions

Rigorous evaluations of the impact of FDA regulatory actions have been limited and infrequent. Several methods with strong internal validity are available to improve trustworthiness of future evaluations of FDA policies. Copyright © 2013 John Wiley & Sons, Ltd.

Ancillary