Empirical assessment of methods for risk identification in healthcare data: results from the experiments of the Observational Medical Outcomes Partnership

Authors

  • Patrick B. Ryan,

    Corresponding author
    1. Johnson & Johnson Pharmaceutical Research and Development LLC, Titusville, NJ, U.S.A.
    2. Observational Medical Outcomes Partnership, Foundation for the National Institutes of Health, Bethesda, MD, U.S.A.
    3. UNC Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, NC, U.S.A.
    • Patrick B. Ryan, Johnson & Johnson 1125 Trenton-Harbourton Road PO Box 200 MS K304 Titusville, NJ 08560, U.S.A.

      E-mail: ryan@omop.org

    Search for more papers by this author
  • David Madigan,

    1. Observational Medical Outcomes Partnership, Foundation for the National Institutes of Health, Bethesda, MD, U.S.A.
    2. Department of Statistics, Columbia University, New York, NY, U.S.A.
    Search for more papers by this author
  • Paul E. Stang,

    1. Johnson & Johnson Pharmaceutical Research and Development LLC, Titusville, NJ, U.S.A.
    2. Observational Medical Outcomes Partnership, Foundation for the National Institutes of Health, Bethesda, MD, U.S.A.
    Search for more papers by this author
  • J. Marc Overhage,

    1. Observational Medical Outcomes Partnership, Foundation for the National Institutes of Health, Bethesda, MD, U.S.A.
    2. Regenstrief Institute and Indiana University School of Medicine, Indianapolis, IN, U.S.A.
    Search for more papers by this author
  • Judith A. Racoosin,

    1. Observational Medical Outcomes Partnership, Foundation for the National Institutes of Health, Bethesda, MD, U.S.A.
    2. Center for Drug Evaluation and Research, U.S. Food and Drug Administration, Silver Spring, MD, U.S.A.
    Search for more papers by this author
  • Abraham G. Hartzema

    1. Observational Medical Outcomes Partnership, Foundation for the National Institutes of Health, Bethesda, MD, U.S.A.
    2. College of Pharmacy, University of Florida, Gainesville, FL, U.S.A.
    Search for more papers by this author
    • At the time of this work, Dr. Hartzema was on sabbatical at the U.S. Food and Drug Administration.

  • This article expresses the views of the authors and does not necessarily represent those of their affiliated organizations.

Abstract

Background: Expanded availability of observational healthcare data (both administrative claims and electronic health records) has prompted the development of statistical methods for identifying adverse events associated with medical products, but the operating characteristics of these methods when applied to the real-world data are unknown.

Methods: We studied the performance of eight analytic methods for estimating of the strength of association-relative risk (RR) and associated standard error of 53 drug–adverse event outcome pairs, both positive and negative controls. The methods were applied to a network of ten observational healthcare databases, comprising over 130 million lives. Performance measures included sensitivity, specificity, and positive predictive value of methods at RR thresholds achieving statistical significance of p < 0.05 or p < 0.001 and with absolute threshold RR > 1.5, as well as threshold-free measures such as area under receiver operating characteristic curve (AUC).

Results: Although no specific method demonstrated superior performance, the aggregate results provide a benchmark and baseline expectation for risk identification method performance. At traditional levels of statistical significance (RR > 1, p < 0.05), all methods have a false positive rate >18%, with positive predictive value <38%. The best predictive model, high-dimensional propensity score, achieved an AUC  =  0.77. At 50% sensitivity, false positive rate ranged from 16% to 30%. At 10% false positive rate, sensitivity of the methods ranged from 9% to 33%.

Conclusions: Systematic processes for risk identification can provide useful information to supplement an overall safety assessment, but assessment of methods performance suggests a substantial chance of identifying false positive associations. Copyright © 2012 John Wiley & Sons, Ltd.

Ancillary