Considerations in characterizing real‐world data relevance and quality for regulatory purposes: A commentary

The 21st Century Cures Act of 2016 provided a framework to the US Food and Drug Administration (FDA) to rapidly move treatments to patients.1 The increased acceptability of real‐world data (RWD) sources allows for innovative ways to study products and has the potential to reduce trial costs. Published papers provide guidance regarding data quality issues, reproducibility, and validity assessment.2 Rapid evolvement of electronic health records (EHRs) encourages greater consideration of their use in research.1, 2, 3, 4, 5, 6 For years, the FDA has relied on epidemiological studies of postapproval product safety using RWD5, 6 (eg, administrative claims and EHR) and for device effectiveness studies4; however, regulatory use for evaluating drug effectiveness has been rare. As part of the Prescription Drug User Fee Act (PDUFA VI),3 use of RWD is being considered for potential contributions to evaluating effectiveness and safety of new indications for approved products and to satisfy postapproval study requirements. Recently, the Duke Margolis Center for Health Policy held workshops and issued two paper on this topic.5, 6 The first paper focused on defining RWD as data routinely collected pertinent to patient health status and/or delivery of care, and the use of RWD in regulatory and clinical contexts.5 
 
The second white paper from the October 1, 2018, workshop focused on data relevancy and quality, including cleaning, transforming, and linking RWD to characterize RWD sources as “fit for regulatory purpose.”6 These papers offer a practical “commonsense” high‐level view of primary data and methods considerations for RWD use from a regulatory perspective, facilitating discussion around regulatory uses of RWD within the research community and industry. However, salient points are missing from the papers and the RWD discussions among FDA, researchers, and industry. Here, we provide a commentary on the data considerations discussed in the white papers and highlight pertinent considerations with respect to RWD in the context of whether data are relevant, representative, and robust. 
 
1.1. Data relevance 
The recent white paper defines data relevance dimensions including representativeness of the population of interest, critical data field availability, accurate linking at the patient level with multiple data sources, and adequate sample size and follow‐up time to demonstrate expected treatment effects.6 Guidance from FDA on how to ensure RWD are fit for purpose and adequate to support regulatory decisions would be helpful on each dimension. 
 
Determining if RWD is fit for regulatory purpose is a “contextual exercise” where the specific research question, regulatory use, and data characteristics drive what meaningful conclusions can be drawn.6 Covariates may be critical for one research question but not another. Exposures and outcomes should be well defined when part of the research question but may not be critical for natural history studies. There is no “one‐size‐fits‐all” approach, and critical data components should be evaluated for each research question and regulatory use.7 A framework is needed to guide choice and evaluation of critical data elements for specific research questions for regulatory use. 
 
Representativeness of the population of interest is gauged in many ways. Recent FDA guidance on Patient Focused Drug Development suggests a statistical sampling approach be used to obtain patient experience data representative of the target population.8 However, most US real‐world databases use administrative claims or EHR for patients seeking medical attention. These RWD sources should be considered broadly representative of the population eligible for using most, if not all, new products and services. “Representativeness” should be assessed broadly in the context of likely product users with some diversity in geography, health status, and health care system as appropriate for the specific research question and regulatory context. While data linkage is likely to limit the eligible sample, it may be needed to increase the informative nature of RWD, especially with increasing evaluations to support precision medicine. 
 
Sample size should be derived based on anticipated treatment effects for studies of treatment effectiveness or safety, whether comparative or not, to ensure appropriate precision of estimates. For rare diseases, there should be flexibility given data sparseness worldwide, as indicated in the FDA guidance on rare disease.8 
 
Additional guidance would be useful regarding how “accurate linking” should be assessed since linking 100% of patients with administrative claims and EHR is impractical. Would FDA accept limited linked data if it was supplemental to cruder variables in the full dataset? Would a subset of 60% be adequate? In the context of probabilistic linkage, what level of certainty would constitute adequate linkage? Salience of linkable individuals to the specific research question should be considered in this determination and pre‐specified sensitivity analyses should help assess robustness of results and conclusions.9, 10


| INTRODUCTION
The 21st Century Cures Act of 2016 provided a framework to the US Food and Drug Administration (FDA) to rapidly move treatments to patients. 1 The increased acceptability of real-world data (RWD) sources allows for innovative ways to study products and has the potential to reduce trial costs. Published papers provide guidance regarding data quality issues, reproducibility, and validity assessment. 2 Rapid evolvement of electronic health records (EHRs) encourages greater consideration of their use in research. [1][2][3][4][5][6] For years, the FDA has relied on epidemiological studies of postapproval product safety using RWD 5,6 (eg, administrative claims and EHR) and for device effectiveness studies 4 ; however, regulatory use for evaluating drug effectiveness has been rare. As part of the Prescription Drug User Fee Act (PDUFA VI), 3 use of RWD is being considered for potential contributions to evaluating effectiveness and safety of new indications for approved products and to satisfy postapproval study requirements. Recently, the Duke Margolis Center for Health Policy held workshops and issued two paper on this topic. 5,6 The first paper focused on defining RWD as data routinely collected pertinent to patient health status and/or delivery of care, and the use of RWD in regulatory and clinical contexts. 5 The second white paper from the October 1, 2018, workshop focused on data relevancy and quality, including cleaning, transforming, and linking RWD to characterize RWD sources as "fit for regulatory purpose." 6 These papers offer a practical "commonsense" high-level view of primary data and methods considerations for RWD use from a regulatory perspective, facilitating discussion around regulatory uses of RWD within the research community and industry. However, salient points are missing from the papers and the RWD discussions among FDA, researchers, and industry. Here, we provide a commentary on the data considerations discussed in the white papers and highlight pertinent considerations with respect to RWD in the context of whether data are relevant, representative, and robust.

| Data relevance
The recent white paper defines data relevance dimensions including representativeness of the population of interest, critical data field availability, accurate linking at the patient level with multiple data sources, and adequate sample size and follow-up time to demonstrate expected treatment effects. 6 Guidance from FDA on how to ensure RWD are fit for purpose and adequate to support regulatory decisions would be helpful on each dimension.
Determining if RWD is fit for regulatory purpose is a "contextual exercise" where the specific research question, regulatory use, and data characteristics drive what meaningful conclusions can be drawn. 6 Covariates may be critical for one research question but not another.
Exposures and outcomes should be well defined when part of the research question but may not be critical for natural history studies.
There is no "one-size-fits-all" approach, and critical data components should be evaluated for each research question and regulatory use. 7 A framework is needed to guide choice and evaluation of critical data elements for specific research questions for regulatory use.

| Data quality
Data quality should be considered in terms of validity, conformance, plausibility, and consistency. 11 The acceptability of various degrees of accuracy and completeness depends on the specific research question and regulatory purpose. The white paper refers to data verification procedures, minimizing missing data, and consistency with source, often impractical given the anonymized nature of accessible data. RWD have proven valuable for specific purposes despite known limitations, when due attention is given to the adequacy of data elements, study design, and analysis. RWD used to support regulatory decisions must be of sufficient quality to ensure that it can be transformed to adequate and well-controlled real-world evidence.
Evaluations of data quality should be focused on fit-for-purpose design and methods, applying sensitivity analyses to support robustness and interpretation. 9,10 It is highly desirable to use a set of validated codes or algorithms (computable phenotypes) for critical fields, depending on study purpose. Decades of validation work in administrative claims have evaluated such algorithms relative to manual chart review. 12 Now that the chart and data for research may be the same (ie, EHR), we need to understand how and when such validation should be conducted. 12 Even if all available processes and SOPs for cleaning, transforming, and linkage are followed, overall data adequacy in the context of study and regulatory purpose should be assessed, preferably by a researcher experienced with RWD sources for regulatory decision making.
Missing data should be considered in the context of the impact on validity and generalizability of results. Whereas follow-up data can be critical for certain purposes such as use of RWD as a comparative arm or concurrent/historic control group, missingness may be less critical for other purposes (eg, missing health outcomes may be less likely to affect results of a product utilization study than an outcomes study).
That said, US RWD sources often are systematically missing followup data due to turnover in health insurance plans and the US health care system's transient nature. Thus, a key consideration for any realworld evidence research question is how much systematic loss of follow-up data or other missing data would influence study conclusions.

| Research framework
A fit-for-purpose framework starts with a well-defined research question and an assessment of relevance and quality of specific critical data elements within the RWD source (

KEY POINTS
• Guidance on use of RWD for regulatory purposes should outline circumstances and level of acceptable validation/reliability for outcome, population, and exposure definitions given the research question and intended regulatory use.
• Representativeness of a population of interest for RWD should be considered in the context of specific research questions for regulatory purposes.
• Producing interpretable results necessitates valid, relevant research definitions for disease, outcomes, and treatments.
• A framework for assessing the utility and usefulness of RWD with respect to a specific research question and the intended regulatory purpose is needed for use by agencies, companies, and researchers around the world.
A framework to assess usefulness of RWD in the context of specific research questions and intended regulatory purpose, along with published reporting guidelines, 9,14,15 could significantly help identify major components of well-designed studies in RWD to support specific product effectiveness and safety research questions for regulatory purposes.  What is the research question? In specifying the research question, include the relevant data elements 13 such as population, intervention and comparator (treatments), and outcome, as applicable (eg, to assess effects of intervention compared with comparator treatment on the incidence of outcome over 2 years (timing) in a population of patients with disease).  □ Absent specific feasibility numbers, the crude prevalence can be applied to the total person-lives in the database to crudely estimate sample size (without applying entry criteria).

Research data element definitions and validation
□ Essential data elements are coded consistently in the RWD health care system (codes capture the research data fields, eg, disease, outcome, treatment, critical covariates, if relevant, adequately). □ Systematic errors (eg, downcoding or upcoding) in the study population and essential data element definitions are identified and minimized and pre-specified sensitivity analyses can assess potential impact, if possible.
□ Definitions for essential data elements (eg, population and outcome) are unlikely to result from "screening" or "rule out" of a specific diagnosis in clinical practice.
□ Needed coding algorithms (eg, computable phenotypes) are available and validated for essential data elements. □ If additional validation is needed, given the research purpose and regulatory decision, then it can be performed within the data source.
□ Covariates or confounders are available that are critical to the research question. □ (If needed) variables that correlate highly with key missing confounders are available and can be used instead.
Missingness and completeness □ Consideration has been made regarding essential elements of the research question that may be systematically missing due to patients seeking care out of network or changes in insurance coverage and whether the outcome can be captured reliably over time within the RWD source. □ Level of systematic error will not substantially affect study interpretation.
□ Discrepancies between different sources of linked data (claims and EHR) for the data elements needed for specific research question will not affect interpretation of the study results.
□ In combining data from multiple health care systems, different coverage policies or benefit designs do not affect ability to address the research question.