The Clinical and Translational Science Award (CTSA) Workgroup for Comparative Effectiveness Research (CER) Education, Training, and Workforce Development identified a need to delineate the competencies that practitioners and users of CER for patient-centered outcomes research, should acquire. With input from CTSA representatives and collaborators, we began by describing the workforce. We recognize the workforce that conducts CER and the end users who use CER to improve the health of individuals and communities. We generated a preliminary set of competencies and solicited feedback from the CER representatives at each member site of the CTSA consortium. We distinguished applied competencies (i.e., skills needed by individuals who conduct CER) from foundational competencies that are needed by the entire CER workforce, including end users of CER. Key competency categories of relevance to both practitioners and users of CER were: (1) asking relevant research questions; (2) recognizing or designing ideal CER studies; (3) executing or using CER studies; (4) using appropriate statistical analyses for CER; and (5) communicating and disseminating CER study results to improve health. Although CER is particularly broad concept, we anticipate that these preliminary, relatively generic competencies will be used in tailoring curricula to individual learners from a variety of programmatic perspectives. Clin Trans Sci 2012; Volume 5: 476–479
This is an exciting time for researchers working to improve health care and health care delivery. Innovations in medical and surgical practice have led to a greater diversity in therapeutic approaches for many diseases. Although clinical trials are invaluable in assessing treatment efficacy, they often do not address important issues such as applicability, cost, or the ease of implementation for real-world health care practice. Until very recently, there has been limited evidence about the comparative effectiveness of medical interventions to help patients and health care practitioners make decisions.1–4 However, impelled by rising health care costs, the increasing number of diagnostic and therapeutic options available, evidence of variation in spending without benefit, and recent health care reform legislation, comparative effectiveness research (CER) for patient-centered outcomes research has been recognized as essential for sustainable health care delivery in the United States. In this paper, we adopt the definition of CER as described by the US Department of Health and Human Services Federal Coordinating Council at http://www.hhs.gov/recovery/programs/cer/execsummary.html: “Comparative effectiveness research is the conduct and synthesis of research comparing the benefits and harms of different interventions and strategies to prevent, diagnose, treat and monitor health conditions in “real world” settings. The purpose of this research is to improve health outcomes by developing and disseminating evidence based information to patients, clinicians, and other decision makers, responding to their expressed needs, about which interventions are most effective for which patients under specific circumstances.”5–11
A well-trained workforce is essential to identify pressing CER questions, design and conduct rigorous studies to answer these questions, and facilitate the implementation of research findings into real-world practices and policies. All of these activities are driven by the needs of patients, clinicians, researchers, as well as other stakeholders and decision makers who are vital partners in this research. To be successful, these vital partners must include representatives from the general public, health care entities, formulary committee members, payors, policy makers, industry, regulators, research reviewers and funders, and government. It is especially relevant to note the CER is a broad concept with the prospects of influencing prevention, diagnosis, treatments, and even the delivery of health care. Given the breadth of CER and its imminent importance to health care improvement and cost management, academic health programs need to consider the general competencies needed for those within and outside of the CER workforce.
Academic medical centers, especially institutions receiving support from the National Institutes of Health (NIH) Clinical and Translational Science Awards (CTSA), are well positioned to be leaders in training investigators, conducting CER, and disseminating results into real-world practices and policies. The CTSA Key Function Committee on CER was established to advance CER and facilitate communication and collaboration of activities across the CTSA consortium. This Committee established the CER Workgroup on Education, Training, and Workforce Development. The Workgroup members and collaborators identified the need to delineate the specific competencies that are fundamental to practitioners and users of CER. In this preliminary CER competencies paper, we incorporated conceptual themes from the 2010 CTSA CER White Papers on facilitating CER and CER workforce development.12,13 We describe our initial assessment of the workforce who will conduct, disseminate, and use the results of CER and the competencies that they will need to acquire to conduct CER. By developing these preliminary CER competencies, we aim to improve the health of individuals and communities through patient-centered outcomes research.
Ann Bonham of the American Association of Medical Colleges (AAMC) and colleagues wrote a commentary describing a new agenda for research that may emerge from the health care reform legislation.14 They observed that the legislation has led to an increased emphasis on implementation of the results of research. This science of implementation requires knowledge of human and organizational behaviors, skills in educating health professionals, knowledge of health care financing, skills of assessing patient and individual preferences, and familiarity with quality improvement methodologies. The authors encourage academic health centers to revisit their research missions and forge new partnerships “with [colleagues in the] economic, behavioral, anthropological, and managerial sciences, building not only on the skills of clinical epidemiologists and health economists but also those of decision scientists, health care educators, operations researchers, systems engineers, information technologists, behavioral psychologists, health services researchers, and other scholars.” They urge to strengthened ties with colleagues in pharmacy, nursing, and social work as well as administrative and community partners. Jonas and Crotty also observed that participants in CER come from diverse backgrounds including clinical medicine, pharmacology, epidemiology, biostatistics, public health, informatics, health policy, library science, health economics, and others. They suggested that it will be “challenging to provide the training necessary to standardize CER methods when those involved come from diverse fields using diverse terminology.”15
Given this view of an expanded workforce that includes individuals with diverse skills, we address the workforce needed for CER activities. Through a series of conference calls and email exchanges, we began by classifying groups of individuals as being within the CER workforce (i.e., practitioners directly involved in conducting CER), and as being outside of the workforce (i.e., end user stakeholders) along the pathway from knowledge discovery to improving health outcomes. These end user stakeholders may be patients and clinicians, as well as the general public, health care entities, formulary committee members, payors, policy makers, industry, regulators, research reviewers and funders, and government. Figure 1 depicts the workforce linearly with knowledge “moving” from left to right with information flows in bidirectionally. Ideally, the results of a given research study will, with input from end user stakeholders, inform the next CER projects that have the greatest potential for improving health.
As shown in Figure 1, individuals who are members of the CER workforce include: (1) Investigators with diverse backgrounds who design, support, and implement CER studies; (2) Technical staff (including project managers, research nurses, analysts, and research assistants) who execute CER studies; and (3) Knowledge translators who take the results of CER to the end user stakeholders. These knowledge translators may include the media, advocacy groups, patient educators, and knowledge translation researchers. CER investigators may also be included among the knowledge translators.
Individuals who are outside the CER workforce but who are on the translational pathway from discovery to health are: (1) Early-phase researchers who conduct research that is earlier on the translational pathway than CER; (2) Patients and clinicians who participate in studies that generate comparative effectiveness knowledge; and (3) other Stakeholders and decision makers who may reap the patient-centered outcomes research benefits from implementing the findings of CER. These other stakeholders may include the general public, health care entities, formulary committee members, payors, policy makers, industry, regulators, research reviewers and funders, and government.
We offer this framework to identify the competencies that may be needed by practitioners within the CER workforce and end users outside of the CER workforce.
Competencies are the knowledge and skills that individuals need to perform their work well. Competencies are ideally stated in behavioral terms and framed as something that is measurable.16
We reviewed the list of 10 broad competencies for clinical and translational research that were developed for the CTSA consortium. Using this list, we generated CER-specific competencies within each of these 10 categories. We generated 40 preliminary competencies that we shared with the CER representatives at each of the 55 CTSAs that existed at that time. The representatives were the individuals identified by the CTSA Principal Investigators as being most conversant in CER. We asked representatives to rank how essential (on a 5-point scale) each competency was to an investigator who conducts CER. We had 21 respondents; if fewer than 60% responded that the competency was essential or highly essential, it was eliminated from further consideration. As we were collecting these responses, we reviewed recent products from other organizations generating related competency criteria, including the International Society for Pharmacoepidemiology and the Pharmaceutical Manufacturers Association (PhRMA) Foundation.17 We also reviewed the competencies generated by the Key Function Committees (KFC) for Education and for Community Engagement in the CTSA consortium as they have complementary interests to the CER KFC. We employed the grades assigned by the CER representatives to prioritize the competencies that would be retained in the final set.
As we developed the competencies, we considered to whom they might apply and we defined two distinct competency levels: foundational and applied. In general, the competencies at a foundational level should be required of all investigators engaged in CER as well as all individuals who use the results of CER. The applied level competencies are required of the investigative team conducting CER. With the input of the CTSA CER representatives and collaborators through feedback at the NIH CTSA CER Stakeholder Forums in May and December 2010, CER competency preparation meetings, and from investigators in the Centers for Education and Research on Therapeutics (CERTs) program of the Agency for Healthcare Research and Quality, we trimmed the initial 40 competencies to 30 competencies (10 foundational and 20 applied) and categorized them in five domains paralleling the domains of clinical and translational research competencies as endorsed by the CTSA consortium. We also sought input from the Agency for Healthcare Research and Quality Evidence Based Practice researchers (Table 1).
Table 1. Foundational and applied competencies for comparative effectiveness research.
CER = Comparative effectiveness research.
Asks relevant research questions
•Formulates questions that will inform an end user stakeholder or decision maker about the relative effectiveness, safety, or costs of two or more interventions.
•Identifies the breadth of decision makers who would benefit from the results of CER and demonstrates ability to engage them in the development of relevant research questions.
•Seeks and identifies strong evidence for decision making based on evidence-based medicine and patient relevant outcomes research.
•Phrases questions about health care delivery as key questions for comparative effectiveness research.
•Phrases questions about prevention, screening, and diagnosis, in addition to treatment, as appropriate questions for CER.
Recognizes or designs ideal CER studies
•Describes the difference between efficacy and effectiveness studies.
•Identifies and designs studies using a noninferiority design.
•Identifies approaches to optimize the design and use of observational studies for CER.
•Describes key features of a systematic review of the literature.
•Explains the role of registries in CER and is able to design a registry.
•Identifies primary sources of high quality systematic reviews and meta-analyses.
•Explains key features of pragmatic trial design and is able to design a pragmatic trial
•Describes patient-relevant outcomes.
•Designs studies that evaluate test performance along the continuum from analytic validity to clinical validity to clinical utility with patient-relevant outcomes.
•Conducts a systematic review of the literature and/or meta-analysis with appropriate methodology.
•Design studies that account for underrepresentation of minority populations in existing data sources.
•Identifies and/or designs valid tools for measuring patient-reported outcomes.
Executes or uses CER studies
•Demonstrates appropriate human subjects protection including privacy protection in comparative effectiveness research.
•Demonstrates ability to write a research proposal incorporating the important elements of CER.
•Identifies key results and important limitations in a comparative effectiveness study.
•Demonstrates ability to assess patient relevant outcomes, using appropriate and culturally appropriate instruments or tools.
•Identifies sources of existing data for observational research including electronic health records and administrative claims.
•States the strengths and limitations of different data and the sources of bias and threats to internal validity in each.
Uses appropriate statistical analyses for CER
•Identifies the strengths and limitations of the different statistical methodologies used in CER.
•Applies methods to minimize threats to internal validity of studies such as in secondary data analyses.
•Applies appropriate methods for meta-analysis and/or meta-regression.
•Describes concept of heterogeneity of treatment effect (treatment response heterogeneity) and uses appropriate methods for exploration.
•Applies appropriate covariate balancing methods as needed in observational research.
Communicates and disseminates study results to allow for implementation of the results of research
•Communicates results of research to peer group.
•Prepares results in formats appropriate for the end user stakeholders who will make decisions with results.
•Identifies methods of dissemination of results that reach the end user stakeholders who will benefit from the results.
•Identifies remaining evidence needs and gaps and communicates them to appropriate decision makers.
We recognize the broad interests that different organizations have in curriculum development on the topic of CER. We do not presume that these are the definitive competencies, as different groups of trainees will likely have specialized needs that may not be captured in these competencies. We suggest, however, that this set may be a useful starting point for organizations that are developing competencies and the curricula to impart the necessary knowledge and skill set. Similarly, not every CER investigator will have applied expertise in all competency areas. For examples, investigators in systematic review should certainly have foundational expertise in issues of study design, but may not have applied expertise in the conduct of secondary data analyses.
These preliminary competencies are written at a fairly high level so that they would be flexible across settings from master’s level students in an introductory CER course to bench scientists advancing their translational skills. Meeting their diverse needs requires specialized curricula but the competency goals should be comparable.
Needed now is a process of vetting of these competencies by the organizations involved in training, which may well involve modifying those provided in Table 1. We hope that these preliminary competencies are considered as the CER workforce further develops and as CER is used to build evidence-based decision making more broadly toward improving patient-centered outcomes.
This project has been funded in whole or in part with Federal funds from the National Center for Research Resources and National Center for Advancing Translational Sciences (NCATS), National Institutes of Health (NIH), through the Clinical and Translational Science Awards Program (CTSA), part of the Roadmap Initiative, and Re-Engineering the Clinical Research Enterprise. The relevant contracts are UL1RR025747 (T.C., D.J., M.W.), UL1RR024153 (W.K.), UL1RR025761 (M.M), UL1RR025014 (P.M.), UL1RR025005 (J.S.), UL1RR029879 (G.S.), UL1 RR025752 (H.S.), UL1RR024131 (M.S.), and UL1RR025777 (K.S.). Other support comes from the Department of Veterans Affairs HSR&D Service Career Scientist Program [RCS 91–408] (M.W). The manuscript was approved by the CTSA Consortium Publications Committee.