• Open Access

Clinical and Translational Scientist Career Success: Metrics for Evaluation

Authors

  • Linda S. Lee Ph.D.,

    1. Department of Biostatistics & Bioinformatics, School of Medicine, Duke University, Durham, North Carolina, USA
    Search for more papers by this author
  • Susan N. Pusek M.S.,

    1. NC TraCS Institute, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA
    Search for more papers by this author
  • Wayne T. McCormack Ph.D.,

    1. Pathology, Immunology & Laboratory Medicine, College of Medicine, University of Florida, Gainesville, Florida, USA
    Search for more papers by this author
  • Deborah L. Helitzer Sc.D.,

    1. Department of Family and Community Medicine, University of New Mexico School of Medicine, Albuquerque, New Mexico, USA
    Search for more papers by this author
  • Camille A. Martina M.S., Ph.D.,

    1. Department of Community and Preventive Medicine, School of Medicine and Dentistry, University of Rochester, Rochester, New York, USA
    Search for more papers by this author
  • Ann M. Dozier Ph.D.,

    1. Department of Community and Preventive Medicine, School of Medicine and Dentistry, University of Rochester, Rochester, New York, USA
    Search for more papers by this author
  • Jasjit S. Ahluwalia M.D., M.P.H., M.S.,

    1. Department of Medicine and Center for Health Equity, Medical School, University of Minnesota, Minneapolis, Minnesota, USA
    Search for more papers by this author
  • Lisa S. Schwartz Ed.D.,

    1. Department of Clinical Research and Leadership, School of Medicine and Health Sciences, The George Washington University, Washington, District of Columbia, USA
    Search for more papers by this author
  • Linda M. McManus Ph.D.,

    1. Departments of Pathology and Periodontics, University of Texas Health Science Center at San Antonio, San Antonio, Texas, USA
    Search for more papers by this author
  • Brian D. Reynolds Ph.D.,

    1. Duke Translational Medicine Institute, School of Medicine, Duke University, Durham, North Carolina, USA
    Search for more papers by this author
  • Erin N. Haynes Dr.P.H., M.S.,

    1. Department of Environmental Health, Division of Epidemiology and Biostatistics, College of Medicine, University of Cincinnati, Cincinnati, Ohio, USA
    Search for more papers by this author
  • Doris M. Rubio Ph.D.

    1. Departments of Medicine, Biostatistics, Nursing, Institute for Clinical Research Education, School of Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania, USA.
    Search for more papers by this author

Linda S. Lee (linda.s.lee@duke.edu)

Abstract

Despite the increased emphasis on formal training in clinical and translational research and the growth in the number and scope of training programs over the past decade, the impact of training on research productivity and career success has yet to be fully evaluated at the institutional level. In this article, the Education Evaluation Working Group of the Clinical and Translational Science Award Consortium introduces selected metrics and methods associated with the assessment of key factors that affect research career success. The goals in providing this information are to encourage more consistent data collection across training sites, to foster more rigorous and systematic exploration of factors associated with career success, and to help address previously identified difficulties in program evaluation. Clin Trans Sci 2012; Volume 5: 400–407

Introduction

The provision of training and career development support in clinical and translational research is a core element of every Clinical and Translational Science Award (CTSA) program of the National Institutes of Health (NIH).1 The training programs developed by CTSA recipients are intended to equip the next generation of clinical and translational scientists with the tools not only to facilitate but also to accelerate the translation of basic science discoveries into clinical care and improvements in public health. Because a training component is required of all CTSA programs, a high priority is to evaluate the impact of these programs on the research productivity and success of trainees, both individually and collectively.

Through the CTSA Education Evaluation Working Group, a number of difficulties inherent in the evaluation of physician–scientist and graduate training programs have been identified. These include redundancies in data collection among different reporting systems locally and nationally, problems in identifying appropriate metrics for both process and outcome measures, and problems in dealing with the emerging gap between traditional measures of individual career success and productivity and newer measures that relate success and productivity to collaborative scientific endeavors. Over the past 18 months, this working group turned its attention to identifying and sharing methods and tools for assessing the impact of training on career success, using as a guiding framework the conceptual model proposed at the University of Pittsburgh’s Clinical and Translational Science Institute by Rubio and colleagues in their article entitled “A Comprehensive Career-Success Model for Physician-Scientists.”2

Here, we begin by presenting an overview of the conceptual model that framed our investigation of metrics and measures. We then describe selected tools and measures related to key components of the model, and we conclude with recommendations for further work in this area.

Overview of the Career Success Model

In light of the paucity of published information on factors related to the career success of clinician–scientists, the Research on Careers Group at the University of Pittsburgh embarked on an effort to evaluate the current literature and then create a comprehensive model of career success that includes predictive factors identified in pertinent literature in related fields, such as business and the social sciences. Through its work,2 this group identified two components of career success: extrinsic success (e.g., promotions, funded grants) and intrinsic success (e.g., career satisfaction). In addition, the group delineated two types of higher order contextual factors that affect career success. The first type is personal factors, which include demographic characteristics (e.g., gender, race, ethnicity, age), educational history (e.g., degrees, research experience), psychosocial factors (e.g., life events, family, dependent care, stress), and personality factors (e.g., motivation, passion, leadership). The second type is organizational factors, which include institutional resources (e.g., infrastructure, financial resources), training (e.g., didactics, research experience), conflicting demands (e.g., clinical and service responsibilities), and relational factors (e.g., mentoring, networking).

In the article describing the comprehensive model, Rubio et al.2 demonstrated how the model can be used to test different relationships, such as moderating and mediating factors. For example, dependent care can moderate the relationship between gender and promotion, whereas mentoring can mediate the relationship between dependent care and promotion.

A key attribute of the comprehensive model is that it allows each institution to modify it in ways that make it relevant to its own trainees and programs. The various CTSA programs have diversity in the type and intensity of training and in the level of trainees, who range from high school students to faculty. There is likely to be some overlap in the factors affecting the careers of CTSA-funded TL1 trainees (who are typically predoctoral students) and KL2 scholars (who are generally junior faculty members). However, the degree to which these factors vary or are affected by a specific training program is not known. In addition, there is diversity in the characteristics and priorities of the CTSA programs and their host institutions. Clearly, the environment in which a specific training program is based can influence the ultimate career success of its trainees. The model can be further refined to include such institutional factors as emphasis on multidisciplinary team science and utilization of CTSA resources.

Model Domains and Associated Metrics

Early in their discussions, members of the CTSA Education Evaluation Working Group acknowledged the tendency of individual programs, old and new, to “start from scratch,” work independently, and duplicate efforts when developing evaluation plans and metrics to meet grant or institutional requirements. By identifying domains and determinants of career success, the comprehensive model provides a starting framework to avoid potential missteps and redundancies. It fosters more efficient development of program evaluations that are grounded in the context of universally accepted metrics for career success while allowing for flexibility within the domains to incorporate metrics unique to specific programs. Brief descriptions of some metrics and measures associated with the domains and components of the model follow.

Domains of Career Success

Extrinsic career success factors

Extrinsic measures of career success are generally viewed as “objective” and “material” indicators, such as financial success, promotions, leadership positions, grant funding, and publications. Curriculum vitae (CVs) can be used to examine these factors, and bibliometric tools enable more detailed analyses of publication records. In addition, return-on-investment analysis (described later in this article) can help institutional leaders and sponsors of training programs with strategic planning for future training initiatives and measures to support the workforce.

Financial success

An individual’s interest in achieving financial success is one factor that may influence decisions made as his or her career progresses. An example of data that can be collected to reflect financial success is actual income. The measure of financial success is challenging because in academic medicine, as in other fields, salary is dependent on an individual’s discipline, institution, and geographic region. The inclusion of financial success in the model is relevant, however, because it may provide further data to address the perception that there are financial disincentives to the pursuit of careers in research. The extent to which individuals believe that financial success is possible in a particular career may influence their choice of discipline and training program and, ultimately, influence whether they are retained in the clinical and translational workforce.

Promotions

In academia, there are clearly defined and broadly accepted ranks through which individuals advance over their career. Participation in a research-training program may influence the promotion trajectory, either by increasing the likelihood of achieving certain milestones, such as grants or publications, or by increasing the speed of promotion. In addition, participation in a training program may increase the opportunities for collaboration with a research team. If an institution acknowledges contributions to team science in the promotion process, these opportunities may also influence an individual’s progress along the promotion pathway. An understanding of the institutional environment and perspective on promotion is essential in considering this factor as it relates to career success.

Leadership positions

One definition of success is whether a person achieves a position of leadership in a research team, program, or professional organization. From the perspective of advancing clinical and translational science, the extent to which trainees and scholars attain leadership positions is a reflection of their individual scientific accomplishments but may also be important from an institutional perspective, because the leaders may help determine policies and priorities for the next generation of investigators. Leadership skills may be inherent or may be developed through training. Although formal leadership training is often overlooked in research training programs, if it is present it should be included in the model.

Grants and publications

Two of the most commonly used metrics of career success are the number of grants and number of publications. Bibliometric analysis can provide additional data by illustrating the interdisciplinary nature and level of collaboration that may be evolving as individuals participate in training programs that emphasize translational science. In addition, many institutions are now using versions of faculty profiling systems not only to quantify grants received but also to collect data on faculty research networks. This information is often demonstrated by the number of outside contributors listed on grant proposals.

Intrinsic career success factors

Intrinsic career success, as noted in the comprehensive model, involves job, career, and life satisfaction. These factors interact in complex ways and are more difficult than extrinsic career success factors to define, quantify, and measure. Focus groups, interviews, and surveys are examples of approaches for obtaining information about intrinsic factors.

Life satisfaction

Life satisfaction is influenced by personal factors that include components outside the immediate work environment as well as financial factors. The Satisfaction With Life Scale (SWLS) developed by Diener et al.3 addresses the overall self-judgment about one’s life to measure the extent of life satisfaction. Although this 5-point Likert-type scale was developed using two different samples of undergraduate psychology students and one sample of elderly persons, its authors stress that the scale can be applied to other age groups.

Job satisfaction

Job satisfaction seems to derive from individual perceptions about the value of work being performed, the sense of accomplishment attained by performing the work, the quality of the relationships with close colleagues, and the perceived support provided by departmental and institutional leadership. When Mohr et al.4 conducted a study on the relationships between job characteristics and job satisfaction among physician–scientists in the Veterans Health Administration, they used a single item with a 5-point rating scale to assess overall job satisfaction by asking participants to compare their current level of job satisfaction with what they believed it should be in an ideal situation.

Another measure, the Job Satisfaction Survey (JSS),5 uses 36 items to measure nine related factors and to generate 10 different scores, one of which is for overall job satisfaction. The authors of this 1985 survey tested the measure’s psychometric properties and developed norms for diverse samples. Considering the age of the survey and today’s population of clinical and translational researchers, some items may need modification to produce stronger face validity, and some factors may not be applicable to this particular population.

Career satisfaction

Career satisfaction is closely related to life and job satisfaction. High levels of career satisfaction involve the ability to have input into the structure and functioning of the working environment. Career satisfaction extends to collegial relationships, trust in colleagues to do the right thing, the ability to tap colleagues as sources of expertise, and other aspects of interpersonal relationships in the workplace. Although much has been written about job satisfaction, there are few measures for career satisfaction. University of Pittsburgh evaluators created a single-item measure that asks trainees how satisfied they are with the direction of their careers (Doris M. Rubio, Ph.D., e-mail communication, October 26, 2011). This single item is then scored on a scale of 1 to 5, with 1 indicating not satisfied and 5 indicating very satisfied.

Determinants of Career Success

Personal factors

Among the factors that can affect an individual’s career success are demographics, educational background, psychosocial factors, and even personality. Demographic data, for example, show differences in the number of men and women in leadership positions. If someone experiences certain life events or has significant family stress, these factors can negatively affect career success, at least in the short term. Some personal factors are easier to measure than others. Further, the ability of training program directors to intervene with regard to these factors may be limited. Here, we present some suggestions for personal factors that are challenging to measure.

Leadership

Investigators have defined numerous competency sets for leadership, but most are in the context of business rather than scientific research. Recently, however, leadership competencies have been defined for public health professionals, nurses, and medical professionals.6–8 Regardless of the context, the measures indicate that similar attributes and competence domains are essential for successful leadership. These include demonstrating integrity, encouraging constructive dialogue, creating a shared vision, building partnerships, sharing leadership, empowering people, thinking globally, appreciating diversity, developing technological savvy, ensuring satisfaction of customers (which in the academic or research environment could consist of students, staff, faculty, or research participants), maintaining a competitive advantage (ensuring research success), achieving personal mastery, anticipating opportunities, and leading change. Within each of these broad domains, there are four or five specific competencies addressed by items in most of the instruments reviewed. One example is the Global Leader of the Future Inventory9 developed by Dr. Marshall Goldsmith, a 5-point Likert scale designed to be completed by an employer and colleagues that can be used to assess change over time.

The Turning Point National Program Office at the University of Washington School of Public Health and Community Medicine has identified six practices associated with effective collaborative leadership, and the related self-assessments can be used in evaluating leadership capacity or as educational tools.10

Motivation

Motivation to work affects career success. Whereas highly motivated faculty are likely to be resilient to negative feedback, grant rejections, or other challenges encountered during the research process, less motivated faculty may opt to terminate their research career.

The Work Preference Inventory11 is a 30-item measure that assesses two factors: intrinsic motivation and extrinsic motivation. These factors can be split into four secondary factors: the enjoyment, challenge, outward, and compensation scales. The inventory has been well tested with college students and working adults and has strong psychometric properties.

Passion and interest

Related to motivation are passion and interest in research. Many successful scientists believe that passion is a necessary factor for achievement. For example, dedication to research requires long hours, often to the detriment of other interests or responsibilities. The Grit scale12 is a 4-item measure that assesses consistency of interest and perseverance of effort.

Self-efficacy

Mullikin et al.13 developed a measure to assess research self-efficacy for clinical researchers. This 92-item scale measures an individual’s self-efficacy or level of confidence in conducting 12 steps in research, from conceptualizing a study through presenting results from a study. Additional work is currently being done to shorten the scale while maintaining its psychometric properties.

Professionalism

Much has been written about professionalism for medical students and residents, but research on how to measure professionalism among clinical and translational scientists is limited. Researchers at Penn State University developed and validated a survey instrument that measures attitudes toward professionalism. The 20-item scale, called the Penn State College of Medicine Professionalism Questionnaire,14 assesses seven factors: accountability, enrichment, equity, honor and integrity, altruism, duty, and respect. Although it may not be necessary to measure all of these factors for clinical and translational scientists, understanding the degree of professionalism and its impact on career success in research is certainly critical in the context of establishing and maintaining collaborations and successfully executing team-oriented research.

Burnout

In a highly demanding field such as academic medicine, burnout is an important issue to study. The challenges of developing a research career add to factors that may contribute to burnout among developing investigators and even predoctoral trainees. Rohland et al.15 developed a single-item measure of burnout and validated its results with the Maslach Burnout Inventory. This single-item measure was shown to perform well for physicians and, by extension, is expected to perform well for clinical researchers.

Organizational factors

Career trajectories can be influenced by organizational factors, such as extent of institutional resources, and by relational factors, such as mentoring, training, and conflicting demands at work. These factors are relevant for predoctoral and postdoctoral participants in training programs. Although program leaders may not be able to affect some of the personal factors that serve as barriers to success, they can often intervene to prevent organizational factors from interfering with success.

Financial resources and infrastructure

An individual’s ability to be successful can be affected by his or her institution’s financial resources. For example, it can be affected dramatically by whether faculty receive seed money for research as part of their hiring package, by whether departmental and institutional resources are focused on promoting translational research, and by whether an institute has funding through sources such as a CTSA. The resources and infrastructure required to be successful likely differ widely, depending on an individual’s training level and research approach. For instance, pilot program funding could positively impact the pace of an individual trainee’s research project, but if institutional priorities do not allow for people at entry training levels to compete for pilot funding, this may be perceived to negatively affect not only the individual’s research project but also his or her perception of the extent of institutional support. Moreover, clinicians have support needs that differ from those of clinician–scientists, basic scientists, and researchers engaged in clinical trials, and understanding the differences is essential in determining the extent to which the resources impact their career success.

The availability of and accessibility to research infrastructure and resources that facilitate translational research should be considered in evaluating the success of developing researchers. Including measures of infrastructure and level of access in the model for success allows program leaders to collect data that might be useful when thinking about the impact of large infrastructure programs, such as the CTSAs, on the career progression of trainees and scholars.

Mentoring

Although mentoring has been identified as critical for career development and furthering an institution’s research mission, faculty often do not receive the training and support necessary to be effective mentors.16 Described below is a selection of instruments that have been or are currently being developed to assess faculty mentoring relationships and the effectiveness of mentor training in the academic medicine setting.

The Mentorship Effectiveness Scale (MES), created by Berk et al.,17 is a 6-point Likert scale with 12 items that focus on the behavioral characteristics of mentors and that are intended to be completed by mentees. An accompanying instrument, the Mentorship Profile Questionnaire (MPQ), includes a qualitative section that explores the nature of the mentor–mentee relationship (e.g., communication frequency and methods, mentor’s perceived strengths and weaknesses) and a section to specify outcomes of the mentor–mentee relationship (e.g., publications, promotion). An advantage of these tools is that they are not intended to evaluate a particular mentoring development program. Instead, they assess the general characteristics and quality of the mentor–mentee relationship and can be tailored to an institution’s own program. A limitation noted by Berk et al. is that the tools are psychometric in nature, making precise determinations of their validity and reliability impossible.17

The Mentor Role Instrument, created by Ragins and McFarlin,18 examines the roles that mentors play, such as coach, protector, friend, parent, and role model. The measure has 33 items and has strong psychometric properties when used by clinical and translational scientists.19

The University of Wisconsin–Madison Institute for Clinical and Translational Research created the Mentoring Competency Assessment to measure the effectiveness of a specific mentor training program that was part of a national multisite study (Stephanie House, e-mail communication, February 3, 2012). The assessment instrument includes a list of 26 items that relate to mentoring skills and were rated on a 7-point scale by mentors and mentees before and after they participated in the training program. The validity and reliability of the instrument are currently being examined. However, its developers anticipate that the training program curriculum and associated instrument will be publicly available in the fall of 2012.

In a comprehensive literature review, Meagher et al.20 summarized the evaluation measures and methods that have been designed to assess the mentor–mentee relationship, mentor skills, and mentee outcomes. They also recommended the development of new evaluation tools to assess mentors prospectively, both objectively and subjectively. In a related article, Taylor et al.21 presented a six-part evaluation model that assesses multiple domains of the mentor–mentee relationship, including mentee training and empowerment, mentor training and peer learning, scholar advocacy, alignment of expectations, mentor self-reflection, and mentee evaluation of the mentor. In addition to providing a list of characteristics and potential questions that could be included in an instrument to address five mentoring domains, they also discussed potential barriers to evaluating the mentor–mentee relationship.

In an article describing a conceptual framework for mentorship at the institutional level, Keyser et al.22 outlined key domains necessary for effective research mentorship, including mentor selection criteria and incentives, facilitating the mentor–mentee relationship, strengthening the mentee’s capability to conduct responsible research, and professional development of both the mentor and mentee. They also provided a self-assessment tool to be used at the institutional or departmental level to conduct both formative and summative evaluations of the policies and infrastructure of mentorship programs.

Collaboration and team science

Much attention has been devoted to encouraging biomedical scientists to work in teams, both within and across disciplines. Given the emphasis of the CTSA program on consortium-based and collaborative initiatives, it is important to clarify attitudes toward collaboration and its perceived impact on career success in scholars and trainees participating in CTSA-based training programs.

The Cross-Disciplinary Collaborative Activities Scale23 is an instrument that has six items measuring the extent to which a respondent participates in cross-disciplinary collaborations. The scale was tested in an initial sample of 56 investigators participating in a National Cancer Institute funded study, the Transdisciplinary Research on Energetics and Cancer Initiative. The scale demonstrated an internal reliability (alpha score) of 0.81.

A second tool that could be used to complement quantitative data on the number and type of collaborative activities is the Research Collaboration Scale used to evaluate collaboration within the Transdisciplinary Tobacco Use Research Center Program.24 This is a 23-item tool that uses a 5-point Likert scale to measure respondents’ satisfaction with collaboration, perception of the impact of collaboration, and trust and respect for members of the research team. The scale was validated in a sample of 202 respondents, approximately half of whom were investigators and the remainder of whom were students, staff, and others in the research center. The three factor structures were validated with Cronbach alpha scores of 0.91 (satisfaction), 0.87 (impact), and 0.75 (trust and respect).

Used together, these tools may be valuable in comparing the extent of collaborative activities and the attitudes toward collaboration before and after participation in a specific training program.

Conflicting demands

It is not uncommon for clinical and translational scientists to have multiple responsibilities that are juggled on a daily basis. In a highly demanding field such as academic medicine, it is important to understand what these responsibilities are and the extent to which they ultimately affect career success. For example, noting what percentage of effort is dedicated to research versus other demands on time is an efficient way of exploring what types of demands influence the progress of a scholar.

Multifactor Measures Under Development

Investigators on national committees and at individual institutions are developing additional measures and tools to gather data on factors associated with researchers’ training and success. Each of these tools is being developed to meet the specific needs of a targeted population of investigators or trainees.

Experience Survey of the Association for Clinical Research Training

The Evaluation Committee of the Association for Clinical Research Training designed and is piloting a survey that explores facets of the training experience that may be associated with the pursuit of a translational, patient-oriented or population-based “thesis type” study by second-year research trainees in a master’s degree program (Ellie Schoenbaum, M.D., e-mail communication, May 17, 2011). Areas of inquiry include the trainees’ medical specialty, academic rank, protected time, mentoring support, quality of life, resource utilization, and research support. The final Web-based survey instrument, intended to be administered shortly before the end of training, will be hosted in RedCap at Albert Einstein College of Medicine for use by interested investigators and evaluators. When the study is completed, deidentified pilot data will be available to other program leaders and investigators who are interested in collecting and analyzing preliminary data for specific studies related to research training.

Research Support Survey of the Duke Translational Medicine Institute

The Duke Translational Medicine Institute (DTMI) developed the Research Support Survey to collect opinions of active academic biomedical investigators about their work environments and institutional support for research activities (Brian Reynolds, Ph.D., e-mail communication, January 5, 2012). Two sections of the survey inquire about factors considered to be influenced by institutional administrators. A professional alignment section explores the degree to which work experiences align with individual needs, interests, and motivation. Other sections assess perceptions regarding collegial relationships, departmental leadership, mentoring, research orientation, and barriers to recruitment, retention, and collaboration. In addition, the survey collects demographic information (age, gender, academic status, length of service, general responsibilities, funding status, and ethnicity) and provides for open-ended responses to relevant information not addressed directly elsewhere in the survey. Results of the DTMI’s institution-wide deployment of this survey in 2010 are under review, with publication anticipated within a few months. The survey instrument will be publicly available at that time.

Graduate Tracking Survey System

The Rockefeller University CTSA used open-source technologies to develop the Graduate Tracking Survey System.25 Designed specifically to collect data regarding the progress of individuals who have completed the CTSA training program, this Web-based system deploys two surveys: an initial survey, which gathers baseline data and is distributed at or soon after graduation, and an annual survey, which updates information every year thereafter. To reduce the burden on respondents, the survey is prepopulated with key data taken from public Web sites and related to publications and grants. The system is applicable to a variety of translational scientist training programs, and options are available for institutions to personalize the surveys to meet institutional needs.

Methods to Measure Career Success

Approaches that can be used to measure career success include citation analysis, return-on-investment analysis, social network analysis (SNA), and CV analysis.

Citation Analysis

Bibliometric tools can be used to facilitate evaluation of publication records as an indicator of career success. For example, Hack et al.26 identified 737 leaders, innovators, pioneers, and role models in Canadian academic nursing programs from 33 institutions. They then used the search engine Scopus to retrieve each individual’s number of published articles, number of first-authored published articles, and Scopus h-index score, as well as the number of citations to these articles that appeared to date in the literature. With this method, they were able to identify the top 20 nurses in Canada on the basis of the number of citations that all of their published articles and their first-authored articles received during their career.

Return-on-Investment Analysis

For purposes of evaluation, return on investment (ROI) is generally referred to as the net gain from an investment and is typically calculated as: ROI = (RevenueCost)/Cost.

ROI can be calculated to answer different questions. For example, to determine the ROI of CTSA education funding in terms of extramural funding received, evaluators start by tracking individual KL2 recipients by program type and determining who applied for and who received funding. The evaluators then determine the amount of education funding, the duration of the appointment period (entry and end dates), and the total amount of direct cost extramural funding that each individual received since entry into the KL2 program. In addition to determining the aggregate dollar amount, evaluators can determine the ROI of the program or the individual by comparing the total educational funding expended with the amount of extramural funding (direct costs only) received (i.e., extramural funding received minus educational funding expended, divided by educational funding). Alternatively, they can determine the per dollar invested ROI by dividing the extramural funding by the educational funding. The above methodologies can be applied to determine the impact of pilot funding or to quantify all funding and services received by an individual (e.g., pilot funding, consultation services) to calculate an aggregate amount of CTSA investment they have received.

To better understand the results of ROI analysis, program evaluators should also track other indicators of productivity of individual trainees over time. These data can be abstracted from CVs obtained annually and may include current appointment, degrees received, presentations, publications, funding proposals, and receipt of awards. Along with demographic data, these data should be entered into a database, then supplemented and verified with searches of PubMed, NIH Reporter, and internal grants management databases. To ensure a robust dataset, the same data should be collected from a comparison group consisting of other individuals who are early in their careers, such as those who applied for but did not receive KL2 funding or those who matriculated into degree programs but did not receive similar funding. The aggregated data can be used to compare the productivity of individual scholars or cohorts of scholars over time.

To ensure validity and to measure long-term outcomes, evaluators should use electronic databases to obtain and track individual data. Although this process is by nature labor intensive, it may be the only reliable means to collect data once an individual has left the program or institution. Tracking external funding for predoctoral or dual degree (e.g., M.D./Ph.D.) trainees may be challenging because the research careers of these trainees are delayed to allow for completion of degree and residency requirements. Thus, in the short term, it may be more feasible to calculate the ROI for the KL2 trainees than for predoctoral and M.D./Ph.D. trainees. Unfortunately, some individuals from all programs may be lost to follow-up over time.

Social Network Analysis

The origins of social network analysis (SNA) lie in the study of whether and how an individual’s actions are influenced by his or her place within a network.27 Originally applied mainly to social sciences and the explanation of scientific questions, SNA is now beginning to be used to measure the activities of researchers and the changing interdisciplinary landscape of biomedical research. SNA generates a description of the ways in which and the extent to which individuals, institutions, or other entities are connected. This information, in combination with other data such as academic productivity, may further elucidate factors contributing to career success. Examples of how SNA has been applied in clinical and translational research include describing the collaboration patterns of researchers over time, the structure of interdisciplinary research teams, and the degree to which programs that aim to foster interdisciplinary research accomplish this goal.27

Curriculum Vitae Analysis

The curriculum vitae (CV) is a universally available record of professional accomplishment and is gaining increased attention in research evaluation.28 Evaluators analyze CVs to study the career paths of scientists and engineers, researcher mobility, and faculty productivity and collaboration.29–32 CV analysis yields rich data not otherwise obtainable; however, evaluators should be mindful of the labor-intensive nature of this method.

Discussion

This overview of metrics and evaluation tools associated with factors comprising the comprehensive model for career success2 is intended to aid individual programs and institutions in assessing factors associated with the success of their research trainees and to inform the evaluation of the training programs themselves. The overview provides a clear conceptual and practical foundation for further identification and testing of measures and methods related to the many factors that affect the career success of clinical and translational scientists. Leaders and administrators of education and training programs often make assumptions about the degree to which individual factors impact career success. They commonly assume, for example, that mentoring is critical to successful research careers. Yet empirical work examining the extent to which aspects of mentoring influence career success is limited. Measuring this factor and testing its relationship to specific elements of career success can inform programs about the specific factors associated with mentoring that should be prioritized and incorporated into the education and training programs.

The model for career success can also be used to identify barriers to success. There is increasing evidence, for example, for one phenomenon that many investigators have long believed to be true: that individuals from racial and ethnic minority groups have less successful academic careers. A commonly used indicator, promotion and tenure, has consistently shown that ethnic minorities—specifically, African Americans and Latinos—are considerably underrepresented in the tenured and full professor categories of academia.33–36 Recent findings published in Science by Ginther et al.37 indicate that there are large racial disparities in the success rate for securing an NIH R01 award, which is the mark of an independent scientist and is often a necessary criterion for promotion and tenure. There likely are multifactorial reasons, some extrinsic and some intrinsic, for these disparities. Use of the career success model and its associated measures can be helpful in further exploring these factors and allowing training programs and their host institutions to identify and address specific barriers.

Application of the model and employment of selected measures can facilitate improved interpretation and understanding of program evaluations, particularly as they relate to different types of trainees. The clinical and translational research workforce consists of investigators from various disciplines in the basic and clinical sciences. These investigators can be traced through three emphases within the scientific pipeline: Ph.D. programs, dual-degree programs, and clinical degree programs (Figure 1).

Figure 1.

Pipeline for junior clinical and translational scientists. Clinical and Translational Science Awards (CTSAs) support Ph.D. training (TL1 programs) and junior faculty development (KL2 programs), highlighted in this diagram.

Despite the fact that a substantial proportion of Ph.D. faculty members in medical schools have primary appointments in clinical departments and are involved in clinical and translational research, Ph.D. scientists have limited opportunities for clinically relevant training during their graduate and postdoctoral years. In contrast, clinician–scientists receive extended education and training opportunities to acquire clinical knowledge, skills, and proficiency. Their predoctoral education may provide some research opportunities, but they subsequently engage in a prolonged course of specialization in residency or fellowship training to achieve certification and licensure as health care providers. Although they may have research opportunities during their advanced clinical training, their research experiences are often time limited and may not provide adequate preparation for success as an academic investigator. The challenges faced by individuals who pursue dual-degree programs are similar to those of the clinician-scientists and may be exacerbated by large periods of time when clinical training requirements prevent them from pursuing research.

Individuals in the three parts of the training pipeline generally share certain aspects of training, such as core courses, mentoring, and team science approaches, and have similar needs related to career satisfaction and success. Yet research-training programs vary in the extent of interaction with the different trainee types, and factors associated with success vary with the context, timeframe, and demands of particular trainees and situations. Understanding the similarities and differences in the training models may contribute to the interpretation of evaluation results.

Our goals in providing information about the model and measures for evaluation are to encourage more consistent data collection across training sites, to foster more rigorous and systematic exploration of factors associated with career success, and to help address previously identified difficulties in program evaluation. It is our hope that future work will focus on testing and adapting these measures to address specific questions and generate results that are generalizable and will help inform future training programs.

Acknowledgments

This project has been funded in whole or in part with Federal funds from the National Center for Research Resources and National Center for Advancing Translational Sciences (NCATS), National Institutes of Health (NIH), through the Clinical and Translational Science Awards Program (CTSA, part of the Roadmap Initiative, Re-Engineering the Clinical Research Enterprise. The manuscript was approved by the CTSA Consortium Publications Committee. The NCRR/NIH CTSA funding was awarded to Duke University School of Medicine (UL1RR024128), George Washington University/Children’s National (UL1RR031988/UL1TR000075), University of Cincinnati College of Medicine (UL1RR026314), University of Florida College of Medicine (UL1RR029890), University of Minnesota Medical School (UL1RR033183), University of New Mexico School of Medicine (UL1RR031977), University of North Carolina at Chapel Hill (UL1RR025747), University of Pittsburgh (UL1RR024153), University of Rochester (UL1RR024160), and University of Texas Health Science Center at San Antonio (UL1RR025767). The contents of this article are solely the responsibility of the authors and do not necessarily represent the official views of the NCRR, NCATS, or the NIH.

We thank all of the members of the Education/Career Development and Evaluation Key Function Committees of the CTSA Consortium and especially the members of the Education Evaluation Working Group who contributed to this work. We are particularly grateful to Carol Merchant, M.D., of the NCRR, who initiated this working group and provided continued support throughout the process.

Ancillary