Challenges in the design and analysis of sequentially monitored postmarket safety surveillance evaluations using electronic observational health care data
Article first published online: 19 JAN 2012
Copyright © 2012 John Wiley & Sons, Ltd.
Pharmacoepidemiology and Drug Safety
Supplement: The U.S. Food and Drug Administration's Mini-Sentinel Program
Volume 21, Issue Supplement S1, pages 62–71, January 2012
How to Cite
Nelson, J. C., Cook, A. J., Yu, O., Dominguez, C., Zhao, S., Greene, S. K., Fireman, B. H., Jacobsen, S. J., Weintraub, E. S. and Jackson, L. A. (2012), Challenges in the design and analysis of sequentially monitored postmarket safety surveillance evaluations using electronic observational health care data. Pharmacoepidem. Drug Safe., 21: 62–71. doi: 10.1002/pds.2324
- Issue published online: 19 JAN 2012
- Article first published online: 19 JAN 2012
- FDA through the Department of Health and Human Services. Grant Number: HHSF223200910006I
- observational study;
- sequential testing;
- study design;
- vaccine and drug safety
Many challenges arise when conducting a sequentially monitored medical product safety surveillance evaluation using observational electronic data captured during routine care. We review existing sequential approaches for potential use in this setting, including a continuous sequential testing method that has been utilized within the Vaccine Safety Datalink (VSD) and group sequential methods, which are used widely in randomized clinical trials.
Using both simulated data and preliminary data from an ongoing VSD evaluation, we discuss key sequential design considerations, including sample size and duration of surveillance, shape of the signaling threshold, and frequency of interim testing.
Results and Conclusions
All designs control the overall Type 1 error rate across all tests performed, but each yields different tradeoffs between the probability and timing of true and false positive signals. Designs tailored to monitor efficacy outcomes in clinical trials have been well studied, but less consideration has been given to optimizing design choices for observational safety settings, where the hypotheses, population, prevalence and severity of the outcomes, implications of signaling, and costs of false positive and negative findings are very different. Analytic challenges include confounding, missing and partially accrued data, high misclassification rates for outcomes presumptively defined using diagnostic codes, and unpredictable changes in dynamically accessed data over time (e.g., differential product uptake). Many of these factors influence the variability of the adverse events under evaluation and, in turn, the probability of committing a Type 1 error. Thus, to ensure proper Type 1 error control, planned sequential thresholds should be adjusted over time to account for these issues. Copyright © 2012 John Wiley & Sons, Ltd.