• Open Access

Evaluating Research Mentors Working in the Area of Clinical Translational Science: A Review of the Literature

Authors


Emma Meagher (emma@exchange.upenn.edu)

Abstract 

The goal of this paper is to review the evaluation of mentors with a focus on training new investigators in clinical translational science. These scholars include physicians and Ph.D. scientists who are generally assistant professors in clinical departments. This white paper is one of a series of articles focused on the programmatic elements of effective mentoring practices and the “current state of the art.” Evaluating mentor performance and providing formative feedback can lead to stronger mentoring and ultimately lead to increased success of new clinical and translational investigators. While there is general agreement that mentor evaluation can be helpful, the process is difficult. Trainees are reluctant to share negative experiences and to rate their mentors. Mentors are not sure they want to be evaluated. Program leaders are not sure how to effectively use the information. This white paper provides mentees, mentors, and program leaders with new perspectives on mentor evaluation and ideas for future research. Clin Trans Sci 2011; Volume 4: 353–358

Introduction

Effective mentoring is a critical determinant of success in the world of academic medicine generally and clinical translational science specifically. A recent survey of 46 universities participating in the Clinical Translational Science Award (CTSA) reported the importance of the various elements of effective mentoring practices for K-funded scholars including mentor evaluation.1 The study interviewed leaders of the CTSA KL2 programs and found that assessing the effectiveness of research mentors is an area of increasing interest. And, while some of these institutions reported a process to evaluate mentors, there was limited consensus on which instrument to use, who should complete the questionnaire, how to protect a K scholar if the evaluation was negative, and how to use the information.

Statement of the Problem

There is an extensive literature base that presents individual personal characteristics and attributes that contribute to effective mentoring.2 In addition, there is no shortage of opinion as to what are the key elements that comprise a positive mentoring experience.3 However, there is no national-level consensus on specific criteria or competencies that could be used to assess effective versus ineffective individual level mentoring experiences. In addition, as mentoring training programs are evolving, a remaining challenge is how to accurately evaluate the effectiveness of such programs. The development of validated measures to assess the effectiveness of a mentoring relationship or a mentoring training program would, in the opinion of the CTSA mentor working group, have a positive impact on individual trainees, mentors, program directors, department chairs, and institutions who are trying to implement mentoring training programs and mentoring policies.

This white paper provides an overview of evaluation measures and methods reported in the literature. The focus of this review is on measures that assess the mentor-mentee relationship, mentor skills. and mentee outcomes. For the purposes of this article, we employ the Healy definition of a mentor-mentee relationship as: “A dynamic reciprocal relationship in a work environment between an advanced career incumbent (mentor) and a beginner (protégé) aimed at promoting the development of both.”4

Methods

In an attempt to grasp the range of approaches employed to evaluate mentoring, we reviewed 90 published peer-reviewed papers that had mentor evaluation in the title or abstract. Our review focused on evaluation methods and instruments. The initial list of 90 papers were read to determine if the reports discussed the evaluation of research mentors working in clinical translational science, contained specific measures used to assess the effectiveness of mentors, provided qualitative and/or quantitative data, and the psychometric properties of the measures. The authors selected 10 articles for this white paper that included some or all of the above information. There were no articles presenting the psychometric properties of the measures nor were there any studies designed to use individual-level data to improve mentor performance and change mentoring policies at a given institution. Nearly all studies were cross-sectional mailed surveys. None of the studies reported individual-level, longitudinal mentor assessments.

Review

Evaluating the effectiveness of any mentoring initiative is a daunting task. How does one measure what makes a relationship effective? Given the plethora of elements, both subjective and objective, that make up such a relationship adaptation of quantitative criteria alone fail in their ability to accurately assess the outcome of the interaction. Similarly, adoption of purely qualitative data would fall short of meeting the goal of effective evaluation.

A summary of the common variables discussed in the 10 papers selected for this review are presented in Table 1. The measures generally include questions that assess the mentor relationship, professional development, characteristics of good mentors and mentee outcomes. Quantitative measures of outcomes for individual mentees included accepted components of academic productivity—published papers, secured grant support, and secured faculty appointments. Faculty retention was a quantitative measure of the collective impact of mentoring. Qualitative data were derived from surveys and questionnaires that used either Likert scales or open-ended questions and focus groups. Little, if any, qualitative measures of mentor outcomes were discussed.

Table 1.  Common variables included in mentor evaluation measures reviewed.
Demographics of mentor/ menteeAssessing the mentoring relationshipProfessional developmentCharacteristics of a “good mentor”Mentee outcomes of mentoring relationship
AgeContact frequencyMentor facilitates opportunitiesCares about mentee as a “person”Research activity
GenderMode of communicationCritiques workTreated as a colleagueGrants
RaceLength of relationshipServes as resourceProvides supportPublications
DepartmentAccessibilityMakes connectionsActive listening skillsPresentations
Faculty rankMentee satisfactionProvides guidance and support Academic appointments
    Promotion
    National recognition

Table 2 summarizes the 10 articles selected for this review. The table includes information on primary group of interest, the outcome of interest, the instrument, design, and primary findings of each paper.

Table 2.  A detailed summary of the 10 mentor studies reviewed in this paper.
Author/ yearPrimary group of interestOutcome of interestInstrumentDesignResults
Levinson et al. (1991)Full-time women, 50 years and younger, in departments of medicine in the US (n = 862)Availability and experience with role models and mentorsMailed survey including demographics, questions about childbearing and child rearing, attitudes toward personal and professional issues, questions about mentors and role models (closed and open questions)Surveyed names taken from AAMC faculty roster (Sept 1987) 83% response rateWomen with mentors report more publications and more time allocated to research
     Barriers were also defined as difficulty finding a mentor, lack of senior women to be mentors, mentor only providing professional (not personal) guidance
Steiner et al. (2004)Graduated fellows from 25 National Research Service Award (NRSA) primary care research programs (n = 215)Association between mentorship and subsequent research productivity and career developmentMailed survey including demographics, training, current position, publications, research projects, and mentoring (yes/no and open ended)Survey to all NRSA post-docs who received funding between 1988 and 1997 (65% response rate)Those with influential mentors spent more time conducting research, published more papers, were more likely to be a PI on a grant and provide mentorship to others
     Influential and sustained mentorship enhances career development in research
Palepu et al. (1998)US Medical School Junior Faculty—stratified national sample (n = 3,013)Mentee perception of professional support, research, teaching, time allocation, career satisfactionSurvey including five domains (measured on 5–6-point Likert scale)—career satisfaction, work environment, research preparation, career sponsorship, psychosocial supportMailed survey 60% response rateThose with mentors rated their research preparation and research skills higher
     No significant difference between number of publications between those with/without mentors
     Those with mentors had significantly higher career satisfaction
Rammanan et al. (2002)Full-time house officers, instructors, and assistant professors at Harvard Medical School and affiliated independent hospitals (n = 8,397)Determine prevalence of mentoring, specific factors associated with having a mentor, and satisfaction with those being mentoredMailed survey including five domains of characteristics of mentoring relationships were used—personal communication, professional development, skill development, academic guidance, and research (5-point Likert scale)Participant names generated from internal admin databaseMentoring programs may be more likely to be successful if they have an emphasis on providing advice and building the mentor-mentee relationship
     Keeping in touch regarding progress and not abusing power were significantly associated with satisfaction with mentoring
     Lower academic rank and those planning an academic career were more likely to have a current mentor
Leppert & Artal (2002)Obstetrician-gynecologists who received research fellowship awards from professional associations (n = 107)Underlying factors leading to successful career pathsMailed survey including demographics, current professional position, current/past research time, research funding, current/past mentoring (research and clinical), publications, professional membership, factors that enhanced/ hindered career (face value, forced-choice, yes/no, Likert scale, open ended)Obstetrician-gynecologists who received research fellowships between 1971 and 1999 (62% response rate)40% (n = 24) respondents said that the most career-enhancing factor was mentoring
     Lack of mentoring was identified as a barrier to current participation in research
Berk et al. (2005) developed by Hopkins School of NursingNursing school trainees and mentorsMentor activities and skillsTwo instruments developed: mentorship profile questionnaire and mentorship effectiveness scale (evaluates 12 behavioral characteristics of the mentor)Paper addressed how tool was designed, data were not collectedPaper reviewed the challenges of variability and unique aspects of each mentee-mentor relationship
Illes et al. (2000) Department of Radiology, StanfordJunior faculty who completed a faculty mentoring program (n = 23)Evaluation of the effectiveness of mentoring program that intended to promote research & training.(10-point Likert scale)Evaluation of the program by all participants 83% response rateMentors and mentees rated research and academic advancement areas of greatest importance in mentoring meetings
Coleman et al. (2005) American College of Obstetricians and GynecologistsOB Residents (n = 4,721)Provide an in-depth look at mentoring attitudes and practices, with a focus on the role of race and genderSurvey included demographics, mentor characteristics, attitudes, and opinions of mentoring (4–5-point Likert scale)Cross-sectional national surveyHighest expectations of mentor: career guidance, academic guidance, personal advice
     Ethnic minorities are more likely than Caucasian residents to have a mentor
Wasserstein et al. (2007)Faculty at the Univ. of Penn. (n = 1,432)Investigate different aspects of mentoring in relation to faculty rank, track and genderSurvey included presence and structure of mentoring, types of mentoring received, satisfaction of mentoringCross-sectional mail survey rate 73% responseSatisfaction with mentoring correlated with number of types of mentoring received, job satisfaction, meeting frequency, and less expectation to leave university (assistant and associate rank)
     Few differences between men and women having a mentor, types of mentoring received, satisfaction with mentoring
Pfund et al. (2008)UW Madison Faculty and PhD graduate studentsInvestigate the effectiveness of an eight-session mentor training programSurvey focused on changes in mentor skills pre-/posttrainingPre-/postdesign before and immediately after eight sessionsSignificant changes in mentor communication skills and mentee skills rating of their mentors

The majority of these studies evaluated level of satisfaction and general opinions on mentoring during a single time period, most often in the last year. For example, Levinson et al. used the 1987 Association of American Medical Colleges (AAMC) faculty roster to survey all full-time women faculty, 50 years old and younger, in departments of medicine to examine experiences (as opposed to effectiveness) with role models and mentors.5 The large survey (110 items) included demographics, questions about childbearing and child rearing, attitudes toward both personal and professional issues, and questions about mentors and/or role models. Specific questions about mentoring included: presence of a mentor, sex of mentor, and open-ended description of their personal experiences with mentors. Quantitative outcome measures were not a focus of attention. Although the paper was generally informative, there was no deliberate attempt to align respondent’s experience with outcome.

In contrast, Steiner et al. surveyed fellows who graduated from 25 National Research Service Award primary care research programs to evaluate the association between mentorship and both subsequent research productivity and career development.6 Components of the survey included demographics, prior training, current position, publications, research projects, and mentoring. Specific questions about mentorship included: presence of a mentor during training, total number of mentors, time spent with mentor, status of relationship with mentor, current mentorship provided to others, and an open-ended description of what makes an influential mentor.

A more structured survey approach was employed by Palepu et al. who used a stratified random sample to survey 3,013 full-time faculty at 24 randomly selected US medical schools to assess the prevalence, quality, variation by gender or race, and relationship between mentoring and the mentees perceptions of professional support (institutional), research, teaching, development of clinical skills, allocation of time to professional activities, and career satisfaction.7 The survey included items broken into five scales: career satisfaction, work environment, research preparation, mentoring: career sponsorship (frequency of critique, promote participation, advise about promotion), and mentoring: psychosocial support. All responses were rated on a five- or six-point Likert scale.

Development of instruments to evaluate experiences has been a significant undertaking for many groups. For example, Rammanan et al. surveyed full-time house officers, fellows, instructors, and assistant professors at Harvard Medical School.8 The survey was compiled and validated jointly by the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine with the intent being to determine prevalence of mentoring, specific factors associated with having a mentor, and satisfaction with those being mentored. Five domains of characteristics of mentoring relationships were used: personal communication, professional development, skill development, academic guidance, and research.

In addition, Leppert and Artal utilized a measure, containing similar domains, to survey obstetrician-gynecologists who received a research fellowship award from a professional association between the years 1971 and 1999.9 The survey intended to look at underlying factors leading to successful career paths. Items included questions on current professional position, current/past research time, research funding, current/past mentoring, publications, professional memberships, factors that enhanced/hindered their career along with demographic information. Multiple question formats were used including face value, forced-choice, yes/no, Likert scale, and open-ended. Survey was designed with extensive consultation with professional education experts, was pretested, piloted, and endorsed by the American College of Obstetricians and Gynecologists.

Collectively, the results of these survey efforts are consistent. Papers uniformly attest to the positive impact of mentoring on a mentee’s perception of skills acquired, and general attitude. While these papers are informative, they represent but a “cross-sectional peek” at the question at hand. As a consensus of opinion regarding the importance of mentoring is close to being formed, academic institutions are recognizing the value and importance of providing mentoring and are implementing training programs.10 Thus, we sought out manuscripts that selectively reported on the specified impact of mentoring and mentoring training initiatives on selected mentoring outcomes through pre- and posttesting.

For example, in 2005, Berk et al. reported on the work of an ad hoc Faculty Mentoring Committee at Johns Hopkins University School of Nursing which was charged, among other things, to evaluate the effectiveness of the mentoring relationship.11 The group developed The Mentorship Profile Questionnaire (MPQ) and The Mentorship Effectives Scale (MES). The MPQ was used to illicit quantitative outcome measures of the mentoring relationship such as mentee publications, presentations, grant funding, engagement in research or clinical care, and academic promotion. The MES evaluates 12 behavioral characteristics of a mentor by using a well-defined six-point Likert scale. The 12 items were written to meet established scale-item criteria and reviewed for psychometric form and content-related validity. The authors intended the MES to be a comprehensive standardized tool to rate the mentorship experience as well as the effectiveness of the mentor. A similar initiative was undertaken by A National Center of Leadership in Academic Medicine.4 This group developed, implemented, and evaluated a 7-month structured mentoring program for junior faculty. The program was directed toward providing junior faculty with the knowledge, skills, attitude, and resources necessary for a successful career in academic medicine.

A unique feature of the evaluation of the program was an incorporation of an evaluation of the financial return on the investment in the program over a 10-year time span. Specifically, the program assessed four primary outcomes (three quantitative and one qualitative); retention within that academic institution, retention in academia in general, cost of the program as compared to the cost of recruitment of new faculty, and assessment of confidence in skills needed to succeed in academic medicine.12 A later quantitative survival analysis of this initiative was performed to compare retention of junior faculty who did and did not participate in the program over an 18-year time interval.13

There is, of course, the appeal of developing a program tailored to the needs of a specific discipline or department. In this regard, Illes and colleagues developed, implemented, and evaluated a Faculty Development Program in the Department of Radiology at Stanford University.14 Evaluation of the program was based on qualitative feedback: rating of overall satisfaction and the relative importance of five professional areas: academic progress, research, clinical, teaching, and administration. Responses were rated on a 10-point Likert scale and compared at two time intervals. The appeal of a department-specific initiative is clearly driven by the congruence of career objectives within a single department.

Another approach sought to examine the perceptions of physicians in training based on racial and gender differences between 1999 and 2004. Coleman and colleagues conducted a national cross-sectional survey of 4,721 obstetrics and gynecology residents to assess on the effectiveness of resident mentor relationships.15 The study found increased satisfaction with their mentors with ethnic minorities more like to report positive mentoring experiences. The University of Pennsylvania conducted a survey of 1,432 faculty members to assess mentoring practices and its relationship to career satisfaction.16 Having a mentor was strongly correlated with career satisfaction among the 1,046 respondents.

An even wider-reaching and sophisticated approach to mentoring undertaken by Pfund et al. incorporated a “train the mentor” component to a mentoring training program.17 An eight-session mentoring seminar series (“Entering Mentoring”) was undertaken to improve mentoring skills and strengthen the research experience of undergraduate students by training graduate and postdoctoral students on mentoring. Evaluation of this training program suggests positive changes in the mentors. This training model was used as the basis for a 16 site intervention study designed to test the effectiveness of the mentor training program. The trial is in progress.18

Discussion

Review of the current body of literature in this field has been illuminating. There have been ongoing efforts to evaluate mentoring for more than two decades. Extraordinary heterogeneity exists amongst the papers reviewed and the methods used therein. This paper does not do justice to the volume of excellent data that exists in the public domain. With very few exceptions, efforts to mentor have yielded impressive and impactful results. The evaluation methods employed are built upon the known facts that the mentoring experience is difficult to define and even more difficult to accurately measure. Measurement relies upon the appreciation that there are two facets to the mentor-mentee relationship. One, the quantitative aspect, is relatively easy to measure.

The other, the qualitative component, is more challenging. It relies in part on perception. It is limited in most examples by the use of scales such as the Likert scale. The latter is effective at capturing and interpreting the extremes of response range but is much less defined for measures close to the median. As such, the responses yielded close to the mean are prone to interpretation bias that renders them less reliable and thus less valuable. The majority of the papers analyzed lamented on the inadequacy of current mentoring practices. For those that reviewed the mentoring practices that occurred at their respective institutions comment on the inadequacy of evaluation methods used to measure the outcome of the mentoring efforts.

All papers comment on the inferred importance of the mentoring function. In general, there was a significant and maybe inappropriate degree of focus on the frequency of mentor-mentee “encounters.” Many articles commented on what they believed were important attributes of the mentor—such as accessibility, experience/seniority, etc. However, despite the limitations of the body of published work, one can draw from the obvious strengths of each and propose a model for evaluation.

Proposed Model for Evaluation

We propose developing a mentoring evaluation model that would allow measurement of outcome as a function of multiple factors that we consider are key determinants of success. We believe this model could be adapted to meet the needs of various constituencies, to include predoctoral, postdoctoral, fellowship, and faculty-level trainees. Determinants of success would be categorized as individual and environmental.

Explicitly the individual determinants of success would include: (1) demographic factors (age, gender, race/ethnicity, socioeconomic status); (2) education (level, quality, degrees, prior research experience); (3) personality traits (motivation, autonomy, creativity, leadership); and (4) personal circumstance (family life, children, financial pressures).

Environmental determinants of success would include: (1) institutional resources (research training programs, mentor training programs, formal mentoring program, financial support for mentors, organizational support and resources for research activity); (2) institutional attitudes (protected time for research, conflicting service demands); and (3) institutional policies (academic promotion, mentoring policies).

Measurement of outcome would be two dimensional—objective and subjective. Individual objective measures of success would include promotion, publications, grants submitted and awarded, academic awards and recognition, leadership and financial success relative to national norms. Collective quantitative/ objective measures would include retention of faculty and financial return on investment.

Subjective measures of success would include measures of personal satisfaction with (current job, rate of progress, career prospects/trajectory, work-life balance and overall quality of life). Additionally, the subjective measures of success should be correlated with the subjective qualitative assessment of the mentoring relationship. Ideally, we would propose that evaluation be commenced prior to the initiation of the mentoring relationship and/or training program. It should be a continuous process that is repeated at regular intervals throughout the lifecycle of the trainee/faculty member.

The optimum interval between evaluation periods is not known but could be modeled upon the interim analysis performed within each institution as part of the academic evaluation and promotion process. While several evaluation tools exist, no single survey comprehensively captures all elements described above. Therefore, there exists a need to extrapolate from the existing surveys and questionnaires and build upon these to develop a free-standing, adaptable, and customizable tool that can be applied across institutions.

Conclusions

There is a pressing need for new evaluation measures that can be used to assess individual mentors on an ongoing basis. We need to go beyond cross-sectional studies and one-time evaluations. The measurement process needs to be established to ensure that mentees are comfortable providing the information, mentors can receive specific feedback, and program directors are able to modify their programs based on the evaluation data. Like in so many areas of medicine, government, and industry, we need to apply continuous, strategic assessment, and feedback to the field of research mentoring in order to improve the training of new investigators.

Conflict of Interest

The authors have no conflict of interest.

Ancillary