External Validity in Distributed Data Networks

,

Results: Each network approaches external validity differently Sentinel answers regulatory questions in the general US population using data from commercial health plans and Medicare fee-for-service beneficiaries and considers external validity when exploring outliers or performing subgroup analyses to examine potential heterogeneity of treatment effects.CNODES focuses on a Canadian target population but includes UK and US data and thus has to make decisions about which partners can be included in each analysis.PCORnet supports a wider array of studies including randomized trials and often assesses whether a given study will be representative of the wider US population.
Conclusions: There is no one-size-fits-all approach to external validity within distributed networks.With these networks and comparisons between their findings becoming a key part of pharmacoepidemiology, there is a need to adapt tools for improving external validity to the distributed network setting.

Plain language summary:
Much work has discussed how studies that partner with multiple separate databases can address things that create bias for estimates from specific databases.Little attention has been paid, however, to how these studies

Introduction
3][4] As a result, data on population-level drug safety and effectiveness generally come from a smattering of singledatabase studies with limited precision and in-sample diversity applying differing analytic approaches and statistical analyses. 5 healthcare data were digitized and information technology advanced, an alternative approach was proposed: analyses using distributed data.In 2008, the FDA launched the Sentinel Initiative to explore a system where database custodians, called "partners," maintained ownership of their data as separate "nodes" of the network but transformed it into a common data model to be analyzed in a consistent way. 6A similar effort started in Canada with the Canadian Network for Observational Drug Effect Studies (CNODES), 7,8 formally funded in 2011, and the Patient-Centered Outcomes Research Institute (PCORI) began to design its own distributed network of partner organizations, PCORnet (the National Patient Centered Clinical Research Network), in 2013. 9All three networks focus on generating one "network-wide" effect estimate in some fashion from the node-level data.Other distributed networks include the Data Analysis and Real World Interrogation Network (DARWIN-EU) project in Europe; 10 a network that leverages the infrastructure built by the Observational Health Data Sciences and Informatics (OHDSI) community; 11 the Asian pharmacoepidemiology network (AsPEN); 12 the Vaccine Safety Datalink (VSD); 13 and a distributed network created for the purposes of pregnancy research titled ConcePTION. 14ch has already been written about the steps these and other networks take to reduce confounding and information bias  in analyses within the individual nodes; after all, internal validity within nodes is necessary to generate unbiased estimates in nonexperimental research. 19 Concpts related to external validity -such as effect measure modification, target populations, generalizability, and transportability -have received comparably less attention in methodologic work on distributed data.Here, we describe the unique roles external validity and related concepts play in analyses of distributed data networks, especially those that seek to obtain a single "network-wide" effect estimate.We then provide an overview of the structure of Sentinel, CNODES, and PCORnet and describe how each network deals with these concepts.

Methods
Key terms and definitions.1][22][23] This has resulted in clearer definitions for "study population", "target population", "generalizability", and "transportability."The study population is the population supplying outcome, exposure, and covariate data for the analysis.The target population is a specific population in which researchers hope to estimate the effect of exposure on outcome, which may or may not be the study population.Generalizability is the extent to which a study population can be used to estimate a given effect in a specific target population it was sampled from (see Figure 1A ).Transportability is the extent to which a study population can be used to estimate a given effect in a specific target population it wasnot sampled from ( see Figure 1B ).This formalization has also resulted in the conceptualization of "target validity", defined as how well a given study estimates a treatment effect in a specific target population.Target validity takes into account internal validity, for the study population, and external validity, for the specific target population.This formalization has also helped clarify understanding of the connection between external validity and effect measure modifiers (i.e.variables that are associated with different treatment effect estimates on the scale of interest) 24 as well as ways to use analytic methods to "balance" effect measure modifiers between study and target populations. 25,26 tributed networks and external validity.The structure of distributed data networks (presented in Figure 2 ) makes these terms even more important.Unlike most epidemiologic research, there are multiple study populations -in fact, there are at least as many study populations as there are nodes in the distributed data network.Because of random error or differing degrees of confounding, effect estimates at the various nodes may not agree with one another.Of course, different distributions of effect measure modifiers at each node can also lead to differing effect estimates.Each node may represent a separate geographical region, a different insurance provider, or an entirely different type of data source (e.g.commercial claims vs electronic health records).It is entirely possible that an estimate in one node may not be externally valid, or may have poor target validity, for the population represented by another node.This is not a major concern if researchers report only node-level estimates and make few comparisons across nodes.The estimate of the treatment effect within each node that makes the fewest assumptions will generally be the estimate yielded by implementing the research study within that node.
If researchers want to combine results from the various nodes into one "network-wide" estimate and 95% confidence interval, which is usually the case, or assess whether different nodes exhibited different amounts of confounding, however, external validity becomes an important consideration when planning research.The target population for the combined network-level analysis affects the analytic strategies that can be used, both at individual sites and when combining results.Whether the findings of a given node can generalize to the whole network becomes an important consideration, as does the extent to which node-level estimates can be transported to one another or to external target populations.Finally, the network needs to consider whether their combined estimate has target validity for the population of interest.
Specific networks and external validity.We assessed how three major distributed data networks that specifically focus on creating network-wide estimates (Sentinel, CNODES, and PCORnet) address external validity in routine aggregate data projects.Researchers from each network provided background information on its purpose and general structure and answered questions related to external validity, target populations of interest, generalizability, transportability, and target validity.

Results
Describing the various networks.Table 1 summarizes key aspects of each network, including the number of individuals covered and the types of data partners included, the format of the data, the main users that can query the data partners, how analyses are generally conducted, and how the results of the analyses are compiled and combined.
Each network was designed to facilitate rigorous and reproducible public health research.Sentinel and CNODES were specifically created to help government agencies answer questions about drug safety and effectiveness, while PCORnet was created as a resource for a wider variety of stakeholders, including those interested in conducting pragmatic trials (i.e.randomized controlled trials including patients and conditions more in line with ordinary clinical practice than typical in randomized controlled trials). 27All three include a wide array of data partners providing claims and electronic health record data and each network has developed or implemented its own common data model (i.e., standardized way to store data to prepare for analysis). 28Sentinel has pursued the most standardized analytic approach, with almost every project using customizable and reusable SAS code created entirely within the operations center. 29The CNODES coordinating center prepares a high-level protocol and analysis plan implemented with code at each data partner for most projects, but also uses the Sentinel common data model.PCORnet allows even more flexibility with the potential for analyses to be performed centrally.Finally, the networks all allow for variation in the extent to which results are aggregated, with PCORnet once more allowing the most flexibility.
Sentinel Q&A: How does external validity come into play when planning and conducting projects in the database?A typical Sentinel analysis includes data from multiple large national and regional commercial health plans, a state Medicaid plan, and Medicare fee-for-service plan.Given the demographic and geographic diversity of the source data and the focus on internal validity, external validity is typically not an explicit consideration when planning or conducting Sentinel projects.That said, attention is paid to the relevance of the study population when finalizing the analysis plan and deciding on eligibility criteria (e.g.high risk individuals, true new users of the drug).These considerations also shape whether specific data partners are included in the analyses.Finally, if outlier results are observed from some data partners, the operations center may examine how the outlier populations differ from the other study populations.
What target population, if any, underlies most analyses?For the most part, the FDA is focused on studying treatment effects on individuals "treated" with drugs in the United States.The major gap with respect to this target is the lack of data on the uninsured and individuals with Medicaid, but there are ongoing efforts to fill this gap, such as the inclusion of more Medicaid data.
Are there ways to generalize the findings of the nodes to the network?Depending on the specific analysis, the operations center can always combine the outcome and treatment data across the sites to obtain a "network-wide" effect estimate.It is also possible to include a specific portion of the network in an analysis, e.g., running the analysis only in the Medicare fee-for-service data when examining the treatment effects in the elderly.
How easily can node-specific estimates be transported between nodes or to external populations?While it is theoretically possible to standardize node-specific estimates to one population (e.g., the entire US population), it has not yet been implemented within Sentinel.However, Sentinel routinely performs prespecified subgroup analyses to identify treatment effect heterogeneity.
Are choices ever made to maximize target validity, rather than internal validity or precision?Given that all the data partners are from the US, and the general focus is on general US population treated with the medications, requesters generally prioritize internal validity and precision..

CNODES Q&A:
How does external validity come into play when planning and conducting projects in the database?Just as with Sentinel, there is no explicit consideration of external validity during research.Still, the query refinement process helps establish whether limiting to specific groups of people (e.g.true new users of drugs, patients at high risk of the outcome) would reduce the relevance of the analysis, and whether some specific provinces and data partners like CPRD or MarketScan differ enough from the others in covariates and follow-up distribution to ultimately limit their use of the research question.External validity also comes up indirectly when combining the results, identifying outliers, and conducting meta-analyses.
What target populations, if any, underlies most analyses?Typically, the main target population of interest is the overall Canadian population.As a result of the fact that 97% of Canadian citizens reside in the provinces contributing to CNODES, the population represented in the Canadian portion of CNODES analyses and the Canadian target population are very similar; restrictions on drug coverage can change this for some analyses, however.
Are there ways to generalize the findings of the nodes to the network?The main way that findings are generalized from site-specific estimates to the broader network is typically by random-effects meta-analysis using inverse variance weighting. 30When using exclusively the Canadian portions of the network, this means that provinces with more events (and likely more individuals) tend to contribute more to the overall effect estimates.No attempt is made to generalize the results of each site, however, and effects are generally assumed to be constant across sites unless there is substantial heterogeneity.
How easily can node-specific estimates be transported between nodes or to external populations?Differences in demographics, the services and medications covered by each province, and the calendar time intervals each data source contributes can make it difficult to directly transport effect estimates between provinces.Because all these variables are measured, however, analytic methods like inverse odds weights or G-computation may be used to obtain more precise within-province estimates. 26Similar approaches could be used for researchers interested in using CNODES to estimate treatment effects in European or US populations, provided a target population was provided in the scientific and analytic protocols.
Are choices ever made to maximize target validity, rather than internal validity or precision?Internal validity is the core focus when preparing the scientific and analytic protocols.When choosing the provinces and data that will contribute to a given study, however, attention is paid to populations that are may differ greatly from the Canadian target population.
PCORnet Q&A: How does external validity come into play when planning and conducting projects in the database?PCORnet research teams take a variety of different approaches depending on the clinical question at hand.They often specifically discuss the degree to which in-network study populations are representative of the overall population of interest, which can vary substantially based on the goals of the research.This representativeness can include demographic factors, comorbidities, health care delivery-level factors, and broader societal factors.If new populations are being recruited into pragmatic trials or observational cohorts within PCORnet, the study coordinators may either target the overall PCORnet population or more specific recruiting targets (such as specific underserved populations).Finally, just like Sentinel and CNODES, differences between study sites and recruitment groups are frequently explored when different associations or effects are observed within nodes of the network.
What target populations, if any, underlie most analyses?Because of the broad mission of PCORnet, specific target populations vary.That said, most target populations are some subset of patients residing in the United States with access to healthcare.
Are there ways to generalize the findings of the nodes to the network?There are some existing ways to generalize site-specific estimates, like meta-analyzing the estimates after checking for heterogeneity.The focus is generally on estimating a network-wide treatment effect.How these methods are applied varies depending on the research team conducting the study and their importance for the clinical question at hand.
How easily can node-specific estimates be transported between nodes or to external populations?Currently, there are no out-of-the box solutions for transporting estimates in this fashion.That said, research groups can generate (if analyzing individual-level data) or request (if the PCORnet Coordinating Center is generating aggregate tables) cross-tabulation tables to stratify queries based on potential effect measure modifiers.
Are choices ever made to maximize target validity, rather than internal validity or precision?Target validity is frequently a major consideration when PCORnet's partner networks conduct large distributed pragmatic trials.The diversity of the data and the direct relationships with clinicians enable researchers to assess this target validity and perform in-depth evaluation of study results within participating sites.

Discussion
External validity, target populations, generalizability, transportability, and target validity were not explicit priorities when designing and constructing these distributed networks.As the networks have developed, however, differences in results between nodes of the distributed networks have forced them to confront external validity and related concepts, and each has chosen to adapt to this challenge in their own way.
Sentinel uses US-based data networks that cover a large portion of the population to respond to FDA queries about target populations treated with a given medication in the US.As a result, it has primarily interfaced with a lack of generalizability and transportability as explanations for differences in effect estimates between study sites.Its primary method for increasing representability has been adding new data (e.g., Medicaid claims).While CNODES is similarly designed to answer government queries, the fact that it targets a Canadian population (despite including non-Canadian data partners) has resulted in more consideration of how health systems can shape the generalizability and transportability of study results earlier during the study design process.PCORnet, in turn, was built for both the enrollment of pragmatic trials and observational research on cohorts in a wide array of disease states.PCORnet research questions and target populations vary more widely than with the other two targeted networks, making representativeness a major consideration for projects conducted within the network.PCORnet is also uniquely situated to explore external validity in these pragmatic trials, since confounding is very unlikely to be the origin of any differences in estimated treatment effects across nodes of the network.Of course, these are not the only noteworthy distributed networks.Exploring how AsPEN, ConcePTION, DARWIN-EU, and any other networks that may start performing research differ in their approaches to external validity can only further understanding of the utility of such analyses.Despite these differences, it is reassuring to see that all three networks consider external validity during the study design and enrollment and interpretation of the final results.Still, none of the three networks have established analytic tools for standardizing, generalizing, or transporting partner-or site-specific estimates to specific target populations (even if some ad hoc programming could technically be used within some of the systems).This is not surprising because these methods are still in the early stages of development and may not yet be sufficiently formalized to be trusted by regulatory bodies.Researchers need to adapt analytic methods for generalizing and transporting study results to these distributed settings and establish how they can improve the overall interpretability of findings.Such methods are also likely to play a key role in cross-network comparisons that contrast results from Sentinel, CNODES, PCORnet, or other distributed networks with one another.

Conclusion
External validity and related concepts present unique challenges to distributed data networks due to the diverse array of populations included in analyses.Fortunately, Sentinel, CNODES, and PCORnet recognize these challenges and have developed their own solutions and analytic strategies tailored to the needs and users of the network.Still, more research needs to be done on analytic methods for addressing external validity, target populations, generalizability, and transportability within the networks.

Figure 1 :
Figure 1: Graphical representation showing differences between generalizability (Panel 1A) and transportability (Panel 1B).The solid black study population is either contained within the target population (in generalizability) or drawn from a separate target population (in transportability).

Figure 2 :
Figure2: A graphical representation of distributed data networks.A central coordinating center sends analytic protocols or programs to workers at each of several databases; these protocols yield estimates or aggregated data, which is returned to the coordinating center to be consolidated and presented to those who requested the analysis.

Table 1 :
A summary of various aspects of the Sentinel, CNODES, and PCORnet distributed networks.Network Sentinel Primary goal Act as a resource for the US government and other interested parties to evalu Main users of the network The US Food and Drug Administration Total individuals covered and data partners More than 228 million individuals at 16 data partners including fee-for-service Data format Data is maintained in the Sentinel Common Data Model in SAS Analysis SAS code is written at the Sentinel Operations Center and then distributed a Consolidation and reporting Results are presented as network-wide estimates, with the option to present p