SEARCH

SEARCH BY CITATION

Keywords:

  • study design;
  • safety monitoring;
  • pharmacoepidemiology;
  • exposure characteristics;
  • outcome characteristics

ABSTRACT

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. SAFETY MONITORING DATA AS AN OPEN COHORT EVALUATION
  5. SAMPLING FROM THE OPEN COHORT
  6. FACTORS TO CONSIDER IN CHOOSING APPROPRIATE DESIGNS
  7. DESIGN RECOMMENDATIONS
  8. CONFLICT OF INTEREST
  9. ACKNOWLEDGEMENTS
  10. REFERENCES

Active medical product monitoring systems, such as the Sentinel System, will utilize electronic healthcare data captured during routine health care. Safety signals that arise from these data may be spurious because of chance or bias, particularly confounding bias, given the observational nature of the data. Applying appropriate monitoring designs can filter out many false-positive and false-negative associations from the outset. Designs can be classified by whether they produce estimates based on between-person or within-person comparisons. In deciding which approach is more suitable for a given monitoring scenario, stakeholders must consider the characteristics of the monitored product, characteristics of the health outcome of interest (HOI), and characteristics of the potential link between these. Specifically, three factors drive design decisions: (i) strength of within-person and between-person confounding; (ii) whether circumstances exist that may predispose to misclassification of exposure or misclassification of the timing of the HOI; and (iii) whether the exposure of interest is predominantly transient or sustained. Additional design considerations include whether to focus on new users, the availability of appropriate active comparators, the presence of an exposure time trend, and the measure of association of interest. When the key assumptions of self-controlled designs are fulfilled (i.e., lack of within-person, time-varying confounding; abrupt HOI onset; and transient exposure), within-person comparisons are preferred because they inherently avoid confounding by fixed factors. The cohort approach generally is preferred in other situations and particularly when timing of exposure or outcome is uncertain because cohort approaches are less vulnerable to biases resulting from misclassification. Copyright © 2012 John Wiley & Sons, Ltd.


INTRODUCTION

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. SAFETY MONITORING DATA AS AN OPEN COHORT EVALUATION
  5. SAMPLING FROM THE OPEN COHORT
  6. FACTORS TO CONSIDER IN CHOOSING APPROPRIATE DESIGNS
  7. DESIGN RECOMMENDATIONS
  8. CONFLICT OF INTEREST
  9. ACKNOWLEDGEMENTS
  10. REFERENCES

Associations observed in data that stream through electronic healthcare databases can lead to erroneous conclusions: patients who initiate statins have more cardiovascular events than patients who do not; among those who experience heart attacks, rofecoxib may appear to prevent influenza; and highly active antiretroviral therapy may not appear to reduce mortality when looking within levels of on-treatment viral load. Of course, these associations do not necessarily reflect causal relations and, if misconstrued as such, can result in serious adverse public health consequences—exactly the opposite of what the Sentinel System, which will rely on these data, is intended to achieve.

The potential for false-positive and false-negative signals in active medical product monitoring is immense. The sheer size of electronic healthcare databases coupled with the large number of exposures and outcomes to be monitored, plus the multiplicity inherent in sequential monitoring over time, will result in many false positives because of chance alone.[1] Yet, the major limitation of observational data is that recorded treatments are not randomly assigned in real-world practice; thus, confounding can produce spurious signals. Indeed, confounding has been implicated in many incorrectly assumed associations in pharmacoepidemiology[2] and explains why those who initiate statins are more likely to have heart attacks than those who do not, although the drugs protect against them.[3] Importantly, confounding also can result in false-negative signals. In addition, other biases in observation, such as conditioning on or adjusting for (through stratification or inclusion in regression or propensity score models) factors affected by the exposure, can lead to false-positive and false-negative signals. Conditioning on a factor affected by both the exposure and outcome of interest (such as looking only at those with heart attacks when examining the relation between rofecoxib[6] and influenza[7]) can induce a statistical association between the exposure and outcome,[4, 5] even when they are marginally independent of each other. Stratifying on causal intermediates (i.e., factors on the causal pathway between exposure and the health outcome of interest [HOI]), such as measures of viral load during treatment for human immunodeficiency virus, can mute signals.[8-10]

Fortunately, the longitudinal electronic healthcare data that will be used for active medical product monitoring have been used for pharmacoepidemiology[11] for longer than the term “pharmacoepidemiology” has existed,[12] and strategies to monitor medical product safety can closely resemble ordinary pharmacoepidemiologic studies, particularly when focusing on pre-specified exposures and outcomes.[13] Over the last 25 years, pharmacoepidemiologists have developed, adapted, and tested many methods for ascertaining valid estimates of drug effects in these data. By employing appropriate design features, stakeholders can, in some situations, decrease bias in active medical product monitoring, which may reduce false-positive and false-negative signals because of confounding, selection, and information biases. When appropriate design and analytic methods can be used, observational evaluations can yield results consistent with randomized trials.[14-19]

A national system for active medical product monitoring requires implementing monitoring methods, including appropriate designs, in a transparent, intelligible, consensus-designed, and timely fashion. In this paper, we summarize the findings of the Mini-Sentinel Taxonomy Work Group,[20] which sought to (i) categorize potential medical product safety scenarios that could be monitored within Mini-Sentinel; and (ii) map these categories to appropriate design options for active safety monitoring using electronic healthcare data. The result is a structured decision table (available in the full report of the Taxonomy Work group[20]) that maps particular scenarios to preferred design options to facilitate timely, transparent decision making.

SAFETY MONITORING DATA AS AN OPEN COHORT EVALUATION

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. SAFETY MONITORING DATA AS AN OPEN COHORT EVALUATION
  5. SAMPLING FROM THE OPEN COHORT
  6. FACTORS TO CONSIDER IN CHOOSING APPROPRIATE DESIGNS
  7. DESIGN RECOMMENDATIONS
  8. CONFLICT OF INTEREST
  9. ACKNOWLEDGEMENTS
  10. REFERENCES

Longitudinal electronic healthcare databases reflect data that are captured prospectively from a dynamic population, and thus, active medical product monitoring activities conducted within these data are similar to prospective open cohort epidemiologic studies. However, in contrast to ordinary prospective cohort studies, in which data are specifically collected for research purposes, Sentinel stakeholders have limited or no influence on what data are collected and how they are captured. As a result, electronic healthcare databases often lack information on important factors, such as smoking status, body mass index, and over-the-counter drug use.[21] In addition, the data are collected primarily for administrative reasons or for reasons related to the provision of care, rather than for monitoring purposes. Nevertheless, the collected health utilization data serve as proxies for various exposures and outcomes that patients may incur. For example, prescription dispensing records are generally considered a valid proxy for actual drug exposure.[22] Indeed, electronic healthcare data are not subject to many of the limitations that plague prospective cohort studies with primary data collection, such as non-response bias and recall bias.[22]

Different monitoring design options are simply different approaches to sampling observation time from the underlying dynamic cohort. Monitoring designs can be thought of as sets of constraints applied to the total events and person-time experience in a dynamic cohort (i.e., a database or a network of databases) intended to yield the most valid comparisons for particular exposure and outcome pairs. For example, events and person-time could be restricted to the year following initiation of a particular drug and the year following initiation of a similar drug for similar patients with an estimate of association based on a comparison of rates of outcome between the two categories of person-time.[23] Alternatively, data could be constrained to only those covering a certain window of time preceding a specific HOI.[24] Restrictions on events and person-time can be implemented in many ways, such as through matching, patient-level inclusion and exclusion criteria, and specification of exposure-risk windows.

SAMPLING FROM THE OPEN COHORT

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. SAFETY MONITORING DATA AS AN OPEN COHORT EVALUATION
  5. SAMPLING FROM THE OPEN COHORT
  6. FACTORS TO CONSIDER IN CHOOSING APPROPRIATE DESIGNS
  7. DESIGN RECOMMENDATIONS
  8. CONFLICT OF INTEREST
  9. ACKNOWLEDGEMENTS
  10. REFERENCES

At the most basic level, two types of comparisons exist: between-person comparisons and within-person comparisons; that is, comparisons of person-time that arises from different people and comparisons of person-time that arises from the same person.

Between-person comparisons (cohort-type designs)

Classic epidemiologic designs employ between-person comparisons in which the person-time experience of one group (e.g., exposed patients) is compared with the person-time experience of another group (e.g., unexposed patients). These include cohort designs as well as case–control and case–cohort designs, both of which have been used in pharmacoepidemiology studies within electronic healthcare databases[25, 26] and yield consistent estimates but use efficient sampling techniques, rather than the full underlying cohort, to ascertain exposure and covariate distributions.[27-29]

Although efficient sampling strategies are useful for studies in which resources (e.g., blood samples) are limited and/or costly (e.g., genotyping), they offer no important practical advantages over full-cohort analyses for evaluations or studies within electronic healthcare data when focusing on pre-specified exposures and outcomes.[30] In fact, analyses nested within the full cohort evaluation or study that could have been performed, although consistent, may be less precise than approaches that use the full cohort.[28] Moreover, in a simulation study, McClure and colleagues found that for active surveillance of vaccines, a matched cohort design enabled the most rapid signal detection and produced the lowest false-positive rate and highest empirical power as compared with other approaches including the case–control design.[31] In most cases, and specifically when focusing on single pre-specified exposure and outcome pairs, the full cohort approach should be preferred to approaches that sample from the same cohort. Alternatively, when screening for possible risk factors of a specific HOI, a case–control approach may be useful because it can simultaneously examine many potential predictors. For example, such case–control surveillance may be useful for identifying previously unsuspected drug-induced birth defects.[32] However, establishing temporality with respect to possible medical product causes, causal intermediates between the medical product and the HOI, and potential confounders in the analysis can be problematic.[30]

The key assumption for valid between-person comparisons in active medical product monitoring is that between-person confounding can be adequately addressed. This entails decisions regarding use of truly unexposed comparison groups or active comparators. Below, we discuss how active comparators reduce between-person confounding and the corresponding assumptions required. Additional analytic methods, such as confounder summary scores,[33] instrumental variable approaches,[34] or methods to address time-varying confounding,[10] may be required to further mitigate between-person confounding.

Within-person comparisons (self-controlled designs)

In contrast to between-person designs, self-controlled designs use each person as his or her own control and compare person-time experience at different times, conditioned on the individual. Within-person approaches include the self-controlled case series (SCCS),[35] the case–crossover design,[36] the case–time–control design,[37] and sequence symmetry analysis.[38] Although the SCCS and case–crossover designs have different origins, they are analytically identical when focusing on specific exposure and outcome pairs.[39] In the SCCS and case–crossover design, for a given exposure-outcome pair, only individuals who experience the HOI and have variation in the exposure contribute to the analysis.[36, 40] Self-controlled designs are most valid when the exposure is transient, the onset of HOI is abrupt, and risk factors for the HOI are fixed within individuals over the (often short) observation window.[36, 41]

The major advantage of within-person comparisons is that confounding by fixed factors (e.g., genetic factors, family history), whether measured or unmeasured, are inherently controlled. Thus, only risk factors that vary within individuals over time need to be addressed. When time-varying confounding is unlikely to influence specific exposure-outcome relations, such as outcomes of childhood vaccinations where indications for the vaccination are exogenous, self-controlled evaluations can proceed expeditiously and with very little data aside from exposure and outcome information and their relative timing. However, self-controlled designs may be more sensitive to misclassifications of the exposure as compared with between-person comparisons.[40, 41]

FACTORS TO CONSIDER IN CHOOSING APPROPRIATE DESIGNS

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. SAFETY MONITORING DATA AS AN OPEN COHORT EVALUATION
  5. SAMPLING FROM THE OPEN COHORT
  6. FACTORS TO CONSIDER IN CHOOSING APPROPRIATE DESIGNS
  7. DESIGN RECOMMENDATIONS
  8. CONFLICT OF INTEREST
  9. ACKNOWLEDGEMENTS
  10. REFERENCES

Different characteristics of exposure-outcome pairs determine whether the assumptions of between-person or within-person comparisons are met for a given monitoring scenario. In some cases, the assumptions of both designs are fulfilled, whereas, in others, the assumptions of one may be more closely met. The main charge of the Mini-Sentinel Taxonomy Work Group was to identify and classify the types of exposure and outcome pairs that will likely be monitored by the Sentinel System and then to map these to appropriate design choices. We identified three distinct domains of monitoring scenarios that are likely to impact the appropriate design choice, which are exposure characteristics, outcome characteristics, and characteristics of the potential causal link between exposure and outcome. Below, we describe these characteristics, highlight several additional considerations, and then provide recommendations about design choice in light of these.

Exposure characteristics

Sentinel System monitoring activities may involve various types of regulated medical products with differing exposure patterns that may influence the choice of monitoring design. For example, use of some medical products is measured as fixed exposures at single points in time, such as vaccines and single-use devices (e.g., embolic protection device). Other products, such as intrauterine devices, also are recorded once but are left in the body to confer continuous exposure. Other medical products are used more than once but for short periods, such as on an intermittent, as needed, or episodic basis, including drugs such as antibiotics, triptans for migraine, and analgesics, and devices such as implanted cardioverters. Other exposures are intended for sustained continuous use, such as statins and other medications for chronic conditions, which will typically be identifiable in administrative claims data as repeated prescription refills from pharmacy dispensing records, although actual use may vary because of non-adherence.

The single most important exposure characteristic that drives the choice of design is persistence, that is, whether the exposure is transient or sustained. Self-controlled designs are not well suited for evaluating sustained exposures because the lack of variation in exposure status renders few patients eligible for analyses unless long case and control windows are used; however, long windows invite time-varying confounding. Moreover, when exposures are sustained, patients exposed during the control period but not the case period are those who discontinue the medication, resulting in a comparison of initiators to discontinuers—an unusual contrast. Many medical products have aspects of both transient and sustained exposures. For example, questions regarding the safety of an implanted device can pertain to the transient surgery and initial exposure to the device or to a sustained change in structure with ongoing presence of the implant. Drug initiation can be conceptualized as a transient event, during which the patient is adapting to the new substance, followed by sustained presence of the drug with repeated use.

For safety monitoring involving medical products that can be classified as either transient or sustained, the choice of which to use should be guided by the particular question. For example, in attempting to determine whether a newly marketed antibiotic causes anaphylaxis, the transient initiation of the drug is more relevant than sustained exposure to it. On the other hand, if the question regards cancer in relation to exposure to the antibiotic, initiation may be less relevant than sustained exposure. It is important to note that within electronic healthcare data, measurements of exposure persistence may not correspond with biological exposure persistence, which is the actual quantity of interest. For example, a vaccination may be recorded once and regarded as a transient exposure, whereas implantation of a device also may be recorded once but be regarded as a sustained exposure.

Outcome characteristics

Exposure characteristics alone are not sufficient for design recommendations and must be considered collectively with other characteristics. HOIs can be classified according to several characteristics that may influence the choice of monitoring design. HOIs may have an abrupt onset (e.g., stroke, acute myocardial infarction [AMI]) or they may be insidious in nature (e.g., diabetes, heart failure). The accurate ascertainment of the timing of HOI onset is critical to traditional epidemiologic designs based on incidence and risk. In cohort-type analyses, delayed identification of onset of an insidious HOI can result in misclassification of the HOI. In self-controlled analyses, incorrect identification of onset time can result in misattribution of exposures to case and control times. Because self-controlled designs are very sensitive to changes in risk with time, the deleterious effects of misclassification of outcome onset time may be more severe than in cohort-type approaches.[40, 41] Although non-differential misclassification of exposures and HOIs is generally considered conservative in epidemiologic research, because it typically biases estimates toward the null, such bias cannot be considered conservative in adverse event monitoring because it can result in false-negative signals, which can have adverse public health consequences.

Characteristics of the (potential) causal link between exposure and outcome

Some attributes relevant to design choice are characteristic of both the exposure and the HOI. For example, whether the interval over which the exposure confers risk (i.e., the exposure-effect period) is short or long and whether it is tied closely in time to initiation of exposure has implications for selecting the most appropriate contrast. In general, the later-appearing effects, such as cancer in relation to the use of immunomodulating agents, have a wider interval over which they may appear. For other pairs, such as anaphylaxis following a single dose of a medication, onset of the exposure-effect period is immediate, and its duration is relatively short.

The timing and duration of exposure-effect periods depend both on the nature (e.g., abrupt versus insidious) and timing (e.g., immediately after exposure or delayed) of HOI onset and on exposure use patterns (e.g., the medical product may confer risk only through continuous, or cumulative, exposure). Defining and using the optimal window representing the hypothesized effect period (i.e., exposure-risk window), with respect to both onset and duration, is critical to both between-person and within-person comparisons and can be the difference between true positive signals and false negatives that go undiscovered.

The probable nature of confounding in a particular monitoring scenario also is an important consideration for monitoring design choice. Confounding by indication can be strong or not strong depending on the way in which treatment decisions are made. Confounding by indication is likely to be strongest in situations in which the HOI (or correlates of the HOI) is anticipated and incorporated into the treatment decisions or in which risk factors for the HOI drive treatment decisions. Confounding, whether strong, can present on two axes—between-person confounding and within-person confounding. Within-person approaches are best suited for situations in which within-person confounding is absent, whereas between-person designs are best suited for scenarios in which independent comparison groups are exchangeable or can be made approximately so.[35, 40]

Additional considerations

New users

Patients who use medical products for a long time without experiencing adverse effects may differ in important way from patients who experience a HOI shortly after initiation of the product. Prevalent users may represent a population enriched with healthier patients or, at least, those less susceptible to the HOI. Focusing specifically on new users mitigates this prevalent-user bias, a form of depletion of susceptibles.[42] Moreover, many HOIs occur shortly after medical product initiation; events that cannot be observed when focusing only on prevalent users. Cohort-type analyses, particularly the full-cohort approach, can easily accommodate the new user design. Given their aptness for identifying triggers of HOIs,[36] self-controlled designs are particularly well suited for the identification of outcomes shortly after initiation of a medical product and the SCCS design inherently builds in aspects of the new user design. In addition, users with persistence of exposure over both the case and control windows do not contribute to the analysis of self-controlled designs.

Active comparators

Active comparators are a form of negative control[43] and can reduce bias because of confounding by indication and differences in health system use if two assumptions are met: (i) the reasons for treatment with the active comparator are similar to those for the monitoring product;[44]; and (ii) the active comparator does not have an effect on the HOI (otherwise, the question becomes one of comparative safety).[17] Thus, the appropriateness of an active comparator(s) depends on both the monitoring product of interest and on the HOI. A reasonable active comparator for a product when monitoring a given HOI may not be an appropriate comparator for the same product when monitoring a different HOI. In monitoring whether a newly marketed anti-diabetic drug causes pancreatitis, a reasonable active comparator might be another anti-diabetic drug that is not thought to be associated with HOI but is therapeutically interchangeable.[45] Not all monitoring scenarios may have reasonable active comparators on which to rely. For example, first-in-class products, including orphan drugs, may lack suitable controls. Active comparators can be used in both between-person and within-person comparisons.

Focusing on new users and employing active comparators can help establish clear temporality among pre-treatment variables that may confound the association between the monitoring product of interest and the HOI.[42, 46] As a general rule-of-thumb, conditioning on pre-treatment variables causes little mischief, as only two types of pre-treatment variables included in the conditioning set can adversely influence bias. For a detailed description of these, see the article by Rassen et al. in this supplement.[47] Conditioning on factors downstream from exposure should be avoided, lest the analysis be subject to collider stratification bias.[4, 5, 8, 9] Conditioning on a causal intermediate between the monitoring product and the HOI can lead to false negatives,[8, 9] as with the apocryphal conditional analysis of highly active antiretroviral therapy and mortality described in the introduction. Conditioning on an effect of both the exposure of interest and the HOI, such as when examining the association between rofecoxib and influenza among only those who experienced a heart attack, can lead to either false-positive or false-negative signals (in this case, a false-positive protective signal).[4, 5] Methods, such as marginal structural models, exist for appropriately handling effects of exposure that can confound future exposure and should be considered in active surveillance.[48, 49]

Exposure time trends

One of the main purposes of active medical product surveillance systems is to enable monitoring of products in their early marketing phase, a time when their uptake may increase rapidly. Such exposure time trends may bias self-controlled analyses that systematically order case and control windows temporally. For example, consider a case–crossover design that uses the 30 days preceding a HOI to define the case window and the 30 days preceding the case window to define the control window. In the absence of a true causal relation between the exposure and HOI, we would expect more exposures in the case window, as compared with the control window, if the incidence of exposure in the population under evaluation increases over time. This could result in a false-positive signal if the exposure time trend were not considered.[37] Suissa proposed the case–time–control design, which uses controls, much like in a case–control study, to estimate and offset the bias because of an exposure time trend in case–crossover studies.[37] Bi-directional self-controlled designs also can mitigate bias because of an exposure time trend.[50] Although case–control sampling from an underlying cohort can similarly be subject to bias because of an exposure time trend, this can be overcome by using risk set sampling where time is among the variables used to define the risk set. Full-cohort analyses are not subject to bias because of exposure time trends.

Measures of association

Although ratio measures are more familiar to pharmacoepidemiologists, absolute measures of association, such as the risk or rate difference, are the most relevant measures for informing public health decision making—the central purpose of the Sentinel System. These measures are readily available from full-cohort analyses. In case–control evaluations, absolute measures can be estimated if the sampling fraction is known, but this nonetheless requires enumerating the full cohort to determine.[30] Similarly, estimates of the baseline risk in the population under evaluation are required to estimate absolute effect measures in self-controlled analyses.

Interpretation

In some scenarios, it may be desirable to perform both within-person and between-person comparisons with the expectation that similar results would be reassuring. However, if the results of the two approaches do not concur, it does not necessarily mean that one approach gave the wrong answer, only that it gave an answer to a subtly different question. The between-person design answers the question “Why me?,” whereas the within-person design answers “Why now?”.[51] This difference arises from the fact that not all patients included in a between-person comparison contribute to the analysis of self-controlled designs. Constant users and nonusers drop out of the analysis, effectively leaving only patients who use the drug intermittently (i.e., those who experience the HOI shortly after initiation or those who use a product, stop, and then experience the HOI). Although self-controlled designs focus on the initiators and the discontinuers, new user cohort approaches typically focus on the initiators and the nonusers. The slightly different focus of the two approaches is not necessarily a limitation of either but merely a caveat in interpreting their results.

DESIGN RECOMMENDATIONS

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. SAFETY MONITORING DATA AS AN OPEN COHORT EVALUATION
  5. SAMPLING FROM THE OPEN COHORT
  6. FACTORS TO CONSIDER IN CHOOSING APPROPRIATE DESIGNS
  7. DESIGN RECOMMENDATIONS
  8. CONFLICT OF INTEREST
  9. ACKNOWLEDGEMENTS
  10. REFERENCES

Among all of the considerations for choosing a monitoring design, three key factors will drive design decisions: (i) strength of within-person and between-person confounding; (ii) circumstances that may predispose to misclassification of exposure or misclassification of the timing of the HOI, which leads to misclassification of exposure; and (iii) whether the exposure of interest is transient or sustained, where sustained exposures reduce short-term exposure variation. Table 1 summarizes the general design preferences based on these three considerations. This is a simplified version of the structured decision table available in the full Taxonomy Work Group report, which provides recommendations for all possible combinations of characteristics of exposure, HOI, and the exposure-HOI link, that may be used to inform decisions about the most appropriate design choice for active medical product safety monitoring.[20] The full table is intended to facilitate expeditious and transparent design selection for monitoring scenarios characterized by particular exposures, HOIs, and links between these. In Table 2, we present five hypothetical monitoring scenarios that we mapped to appropriate design choices based on consensus among the Work Group regarding each scenario's characteristics. We applied the characteristics indicated in Table 2 to the structured decision table to arrive at the design preference. Note that not all examples represent cases in which an association (causal or otherwise) is necessarily thought to exist. Below, we summarize the recommendations more generally. Although these recommendations cover a broad range of scenarios that may be of interest in active medical product monitoring systems, they do not cover special situations such as evaluation of fetal outcomes following in utero medical product exposure and identification of medical product interactions (e.g., drug–drug or drug–device interactions). Moreover, these recommendations pertain only to design considerations and do not cover subsequent analytic options.

Table 1. Preference for between-person versus within-person comparisons according to individual characteristics of a particular monitoring scenario
Characteristics of monitoring scenarioCohort approach preferredSelf-controlled approach preferred
  • indicates preferred approach.

  • ✔✔

    indicates that the approach is strongly preferred.

Exposure persistence:
TransientEither approach is suitable
Sustained 
Onset of exposure-effect period:
ImmediateEither approach is suitable
Delayed 
Duration of exposure-effect period:
ShortEither approach is suitable
Long 
Strength of between-person confounding:
NegligibleEither approach is suitable
Needs to be addressed ✔✔
Strength of within-person confounding:
NegligibleEither approach is suitable
Needs to be addressed✔✔ 
Health outcome of interest onset:
AbruptEither approach is suitable
Insidious 
Table 2. Examples of hypothetical monitoring scenarios mapped to design preference according to key characteristicsa
 Exposure persistenceOnset of exposure-effect periodDuration of exposure-effect periodStrength of within-person confoundingStrength of between-person confoundingbHOI onsetDesign preference
  • a

    Note that not all examples represent cases in which an association (causal or otherwise) is necessarily thought to exist.

  • b

    The values presented in this column are prior to selection of an active comparator. An appropriate comparator is one approach to reduce between-person confounding where it needs to be addressed.

  • c

    This could be classified as transient if the question or outcome of interest pertained to initiation of the product.

  • d

    If the outcome of interest were hypothesized to be related to some component of the vaccine that remained in the body for a long time, then the exposure could be classified as sustained.

  • e

    Some patients may be exposed repeatedly such that persistence could be regarded as sustained, making this a difficult decision; nevertheless, given the other characteristics of this particular scenario whether exposure persistence is classified as transient or sustained does not change the design choice recommendation.

  • f

    Certain questions could be formulated so as to pertain to the placement and initial exposure to the device, in which case persistence could be considered transient.

Sustained use of lisinopril and angioedemaSustainedcImmediateLongNegligibleNegligibleAbruptCohort preferred
Measles, mumps, rubella vaccination, and febrile seizuresTransientdImmediateShortNegligibleNeeds to be addressedAbruptSelf-controlled preferred but cohort to be considered
Rosuvastatin and rhabdomyolysisSustainedcImmediateLongNegligibleNegligibleAbruptCohort approach strongly preferred
Amphotericin B and acute liver failureTransienteImmediateLongNeeds to be addressedNeeds to be addressedAbruptCohort approach strongly preferred
Mechanical heart valve and thromboembolismSustainedfImmediateLongNeeds to be addressedNeeds to be addressedAbruptCohort approach strongly preferred

When the key assumptions of self-controlled designs are fulfilled (i.e., lack of within-person, time-varying confounding; abrupt onset of HOI; and transient exposure), this approach is preferred to cohort-based approaches because self-controlled designs inherently avoid confounding by fixed factors.

Assessing the extent to which timing issues (e.g., delayed onset, long duration of the exposure-effect period, and insidious nature of the HOI) may result in exposure misclassification (or misclassification of HOI timing) is important for design decisions because self-controlled designs are more susceptible to misclassification than are between-person comparisons.[36, 41] The cohort approach generally is preferred in situations in which issues that affect misclassification of exposure or timing of outcome onset is present, but this is considered secondary to issues of confounding.

Monitoring design selection also should consider situations that may reduce variation in exposure, namely, when the monitoring question pertains to a sustained exposure. This is important not only because lack of exposure variability can reduce the power of self-controlled designs but also because evaluating sustained exposures necessitates longer observation windows, which increases the likelihood that time-varying confounding can enter the analysis. Thus, cohort approaches are generally favored for sustained exposures.

In conclusion, active medical product monitoring will exploit data that arise from an underlying open cohort and are captured routinely and prospectively in electronic healthcare databases. Monitoring designs can be classified based on whether comparisons are made between or within patients. In deciding which approach is more appropriate for a given monitoring scenario, or when interpreting results within a given monitoring scenario, stakeholders must primarily consider the strength of within-person and between-person confounding, whether circumstances exist that may predispose to misclassification of exposure or misclassification of the timing of the HOI, which leads to misclassification of exposure, and whether the exposure of interest is transient or sustained.

CONFLICT OF INTEREST

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. SAFETY MONITORING DATA AS AN OPEN COHORT EVALUATION
  5. SAMPLING FROM THE OPEN COHORT
  6. FACTORS TO CONSIDER IN CHOOSING APPROPRIATE DESIGNS
  7. DESIGN RECOMMENDATIONS
  8. CONFLICT OF INTEREST
  9. ACKNOWLEDGEMENTS
  10. REFERENCES

Patrick B. Ryan is an employee of Johnson & Johnson Pharmaceutical Research and Development. Sebastian Schneeweiss has received investigator-initiated grants from Pfizer and Novartis. The other authors have no conflicts of interest to disclose.

KEY POINTS

  • Monitoring scenarios characterized by different types of exposures, outcomes, and links between these require different monitoring designs.
  • Key determinants of appropriate monitoring design include strength and nature of confounding, circumstances that predispose to misclassification, and whether exposure is predominantly transient or sustained.
  • Additional design considerations include whether to focus on new users, whether to use active comparators, and the measure of association of interest.
  • When key assumptions of self-controlled designs are fulfilled, within-person comparisons are preferred, otherwise cohort-type approaches are preferred.

ACKNOWLEDGEMENTS

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. SAFETY MONITORING DATA AS AN OPEN COHORT EVALUATION
  5. SAMPLING FROM THE OPEN COHORT
  6. FACTORS TO CONSIDER IN CHOOSING APPROPRIATE DESIGNS
  7. DESIGN RECOMMENDATIONS
  8. CONFLICT OF INTEREST
  9. ACKNOWLEDGEMENTS
  10. REFERENCES

Mini-Sentinel is funded by the Food and Drug Administration (FDA) through Department of Health and Human Services (HHS) Contract Number HHSF223200910006I. Many Mini-Sentinel collaborators, OMOP staff, and FDA staff contributed to this work through the Taxonomy Work Group. Specifically, the authors would like to thank Jeffery Brown, Melissa Butler, Andrea Cook, Besty Chrischilles, Stephen Crystal, Gregory Daniel, Robert Davis, Stephen Evans, Robert Gibbons, Tarek Hammad, Abraham Hartzema, Eric Johnson, Martin Kulldorff, David Madigan, David Magid, Richard Platt, Bruce Psaty, Judy Racoosin, Wayne Ray, Mark Regine, Marsha Reichman, Thomas Scarnecchia, Joseph Selby, Judy Staffa, and Alexander Walker.

REFERENCES

  1. Top of page
  2. ABSTRACT
  3. INTRODUCTION
  4. SAFETY MONITORING DATA AS AN OPEN COHORT EVALUATION
  5. SAMPLING FROM THE OPEN COHORT
  6. FACTORS TO CONSIDER IN CHOOSING APPROPRIATE DESIGNS
  7. DESIGN RECOMMENDATIONS
  8. CONFLICT OF INTEREST
  9. ACKNOWLEDGEMENTS
  10. REFERENCES