SEARCH

SEARCH BY CITATION

Keywords:

  • psychometric validation;
  • mentorship;
  • professional development of physician-scientists

Abstract

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References

The Ragins and McFarlin Mentor Role Instrument (RMMRI) was originally developed to measure perceptions of mentoring relationships in research and development organizations. The current study was designed to evaluate the RMMRI's reliability and validity when the instrument was administered to clinical and translational science trainees at an academic medical center. The 33-item RMMRI was administered prospectively to a cohort of 141 trainees at the University of Pittsburgh in 2007–2008. Likert-scale items focused on perceptions of five mentoring roles in the career dimension (sponsor, coach, protector, challenger, and promoter) and six mentoring roles in the psychosocial dimension (friend, social associate, parent, role model, counselor, and acceptor). Outcome items included overall perceptions of mentoring satisfaction and effectiveness. Of 141 trainees, 53% were male, 66% were white, 22% were Asian, and 59% were medical doctors. Mean age was 32 years. Analyses showed strong within-factor inter-item correlations (Pearson Coefficients of 0.57–0.93); strong internal consistency (Cronbach alphas of 0.82–0.97); confirmatory factorial validity, as demonstrated by confirmatory factor analysis of the two mentoring dimensions, 11 mentoring roles, and 33 RMMRI items; and concurrent validity, as demonstrated by strong correlations (Pearson Coefficients of 0.56–0.71) between mentoring dimensions, satisfaction, and effectiveness. This article concludes that the RMMRI shows reliability and validity in capturing the multidimensional nature of mentoring when administered to clinical and translational science trainees in the academic setting. Clin Trans Sci 2010; Volume 3: 104–108


Introduction

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References

In academic medicine, mentors can play a critical role in guiding, nurturing, teaching, and supporting their mentees and in fostering the types of skills that mentees need for a successful career.1–4 In the rapidly emerging field of clinical and translational science, mentorship is particularly important for helping trainees develop, integrate, and apply skill sets related to basic science research and to clinical research. Because of the strong impact that mentoring can have on career productivity, career satisfaction, career mobility, promotion, and personal growth,1,3,5,6 it is crucial to ensure that the mentorship process is monitored and evaluated, that high-quality measures of the mentoring relationship are used in the evaluation process, and that these measures are able to identify the factors that are most predictive of professional success.

Although there is agreement in the literature about the importance of mentoring,7–9 key challenges in the assessment of mentoring have been noted. These challenges include the following: difficulties in obtaining a sufficient sample size in which to test hypotheses and validate measures; a paucity of psychometrically sound measurement tools that can be applied in and transferred across different fields, both within and outside the academic setting; the inability of measures to adequately capture the multidimensional nature of the mentoring construct; and the problem of response bias (e.g., the halo effect, in which the favorable perception of one characteristic is generalized to other characteristics).7–9

Several investigators have used Kram’s10 conceptual theory of mentoring roles to characterize mentoring relationships in terms of career development and psychosocial development.7,9 Career development involves components that promote professional advancement, such as networking and coaching. In contrast, psychosocial development involves interpersonal components of mentoring, such as friendship and acceptance. Based on Kram's theory, Ragins and McFarlin (1990) developed a mentor role instrument that assesses 11 roles and evaluates perceptions of mentoring relationships.7 They initially used their instrument to examine perceptions of mentoring in research and development organizations. Unlike other mentoring measurement tools, the Ragins and McFarlin Mentor Role Instrument (RMMRI) has undergone evaluations to establish its psychometric properties in several other settings, including private institutions.7 However, it has not been evaluated in the field of clinical and translational science.

Our prospective cohort study was designed to assess the RMMRI's psychometric properties in a sample of clinical and translational science trainees and, in particular, to examine the RMMRI's reliability, factorial validity, and concurrent validity.

Methods

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References

Study participants

The study sample consisted of trainees who were enrolled in 2007 in one of several degree or career development programs offered by the Institute for Clinical Research Education and sponsored by the Clinical and Translational Science Institute (CTSI) at the University of Pittsburgh. The goal of the CTSI is to develop clinical researchers with the skills needed to apply basic science discoveries to patient care and community health. The trainees in CTSI programs include predoctoral students, medical students, residents, fellows, and junior faculty members, each of whom conducts research in a laboratory or a program under the mentorship of one or more accomplished investigators.

Study procedures and study instrument

For our study, we recruited participants at the beginning of their training and asked them to provide informed consent. During their orientation session and annually thereafter, we asked them to complete a questionnaire about their sociodemographic characteristics, their frequency and type of contact with mentors, their perceptions of the mentoring relationship, their satisfaction with the relationship, and their overall perception of the relationship's effectiveness. For the study, we used data from participants with complete information from 2007 to 2008.

Sociodemographic questionnaire items included age, gender, race/ethnicity, educational background, and type of training program. The frequency of different types of mentoring contacts was measured on a 5-point Likert scale, with scores ranging from 1 (never) to 5 (more than weekly). The types of contacts were weighted to reflect the level of intensity, with e-mail messages weighted as 0.5, telephone calls as 1.0, impromptu meetings as 1.0, group meetings with primary mentors as 3.0, and group meetings with primary and secondary mentors as 3.0.

For perceptions of the mentoring relationship, the RMMRI was used in its entirety, without changes to the structure of its 33 items. 7 Of the 33 items, 15 items measured perceptions of five career roles (sponsor, coach, protector, challenger, and promoter) and 18 items measured perceptions of six psychosocial roles (friend, social associate, parent, role model, counselor, and acceptor). Each of the 33 items was assessed on a 7-point Likert scale, with scores ranging from 1 (strongly disagree) to 7 (strongly agree).

For the general rating of the mentoring relationship, trainees were asked two items, each rated on the same 7-point Likert scale. One of the items was designed to capture overall satisfaction with the relationship (“My mentor is someone I am satisfied with”). The other item was designed to capture the overall effectiveness of the relationship (“My mentor has been effective in his or her role”).

Statistical analyses

Descriptive statistics were used to characterize the baseline characteristics of participants. To assess the reliability of the RMMRI, we used internal consistency techniques. Initially, we used Pearson correlation coefficient to examine inter-item correlations. The distribution of inter-item correlations, the range, and the average inter-item correlation was used to measure the internal consistency reliability.

Next, we used Cronbach's alpha to examine the internal consistency of items purportedly belonging to each dimension (the career dimension and the psychosocial dimension) and also to examine the homogeneity of items concerning each of the five career roles and each of the six psychosocial roles. Although consistency is important, determining the appropriate level of homogeneity remains a complex issue.11 On the one hand, a high internal consistency is reflective of a unidimensional scale and can lead to redundancy of items in defining the factors within a measure.11 On the other hand, a scale that is more narrowly focused may be ideal if the goal is to have a construct that is specific rather than diverse in scope.11 In any case, since evaluating the psychometric properties of an instrument involves considerations of item cohesiveness, we tested for degree of consistency.

Third, to assess the confirmatory factorial validity of the study instrument, we used structural equation modeling and tested the higher order factor model proposed by Ragins and McFarlin.7 Career role is measured by five factors and each factor is measured by three items. Psychosocial role is measured by six factors and each factor is measured by three items.

Fourth, to assess the concurrent validity of the instrument, composite scores were created for each of the two higher order factors (Career Role and Psychosocial Role) by averaging the items that loaded onto the factors. We used Pearson correlation coefficients to calculate the degree to which the composite score for each dimension correlated with the following three outcomes: the overall mentoring satisfaction score (“My mentor is someone I am satisfied with”), the overall mentoring effectiveness score (“My mentor has been effective in his/her role”), and the composite frequency and type of mentoring score. This process was designed to determine whether perceptions of the mentoring relationship were related to the degree of satisfaction and effectiveness that mentees experienced with their mentors.

In higher order factor analysis, we used Mplus version 5.21.12 For all other analyses, we used SAS sof ware version 9.213 and R.14 We considered a p value of <0.05 to be significant.

Results

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References

Sample

Of the 188 individuals who enrolled in 2007–2008 in CTSI training programs, 141 (75%) had complete data and were included in the study.

Table 1 shows the characteristics of the sample. The average age of respondents was 32 years, with a standard deviation of 6.9 years. Of the 141 respondents, 74 (53%) were male and 76 (66%) were white, although a relatively large proportion (22%) were Asian. Eighty-one (59%) were medical doctors with a small percentage (12%) only having a B.S. or B.A.

Table 1.  Demographic characteristics of study participants.
CharacteristicsMean (SD) or n (%)
  1. *Subjects can belong to more than one program simultaneously.

Age (N= 135)32.1 (6.9)
Male (N= 140)74 (53%)
Race (N= 115)
 White76 (66%)
 Asian25 (22%)
 African American7 (6%)
 Hispanic6 (5%)
 Other1 (1%)
Degree Type (N= 137)
 M.D.81 (59%)
 Ph.D.15 (11%)
 B.S./B.A.17 (12%)
 D.M.D.7 (5%)
 Other degree15 (11%)
 No degree2 (1%)
Program (N= 205)*
 Certificate in Clinical Research41 (20%)
 Master's in Clinical Research37 (18%)
 Short-Term Access to Research Training Program (START UP)28 (14%)
 Clinical Research Scholars Program (CRSP)23 (11%)
 Clinical Scientist Training Program for Medical Students (CSTPM)22 (11%)
 Predoctoral Fellowships in Clinical and Translational Research (Predoc)20 (10%)
 Clinical Scientist Training Program for Residents (CSTPR)18 (9%)
 Other16 (8%)

Reliability

The Pearson correlations of items within factors ranged from 0.57 to 0.93. Seventy-five percent of the correlations were 0.69 or higher, indicating strong inter-item reliability.

The Cronbach alpha value was 0.95 for the career dimension and 0.93 for the psychosocial dimension, indicating strong internal consistency within these dimensions (Table 2). For career roles, the alpha values ranged from 0.87 (for the protector role) to 0.97 (for the challenger role), indicating item homogeneity. For the psychosocial roles, the alpha values ranged from 0.82 (for the counselor role) to 0.94 (for the friend and acceptor roles), again indicating item homogeneity.

Table 2.  Cronbach alpha values for internal consistency in the Ragins and McFarlin mentor role instrument.
Dimension, role, and itemCronbach alpha
Career dimension0.95
 Sponsor0.91
  Q1: Helps me attain desirable positions
  Q2: Uses influence in the organization for my benefi t
  Q3: Uses influence to support my advancement
 Coach0.89
  Q4: Suggests strategies for achieving career aspirations
  Q5: Gives me advice on how to attain recognition
  Q6: Helps me learn about other parts of the organization
 Protector0.87
  Q7: Runs interference
  Q8: Shields me from damaging contact with important people
  Q9: Protects me from those who are out to get me
 Challenger0.97
  Q10: Provides me with challenging assignments
  Q11: Assigns me tasks that push me
  Q12: Gives me tasks that require me to learn new skills
 Promoter0.95
  Q13: Helps me be more visible
  Q14: Creates opportunities for me to impress important people
  Q15: Brings my accomplishments to the attention of important people
Psychosocial dimension0.93
 Friend0.94
  Q16: Is someone I can confide in
  Q17: Provides support and encouragement
  Q18: Is someone I can trust
 Social associate0.87
  Q19: Has one-on-one social interactions with me outside work
  Q20: Socializes one-on-one with me outside work
  Q21: Gets together with me informally after work
 Parent0.89
  Q22: Reminds me of my parents
  Q23: Is like a father or mother to me
  Q24: Treats me like a son or daughter
 Role model0.88
  Q25: Serves as a role model
  Q26: Represents who I want to be
  Q27: Is someone I identify with
 Counselor0.82
  Q28: Guides my personal development
  Q29: Serves as a sounding board
  Q30: Guides my professional development
 Acceptor0.94
  Q31: Accepts me as a competent professional
  Q32: Thinks highly of me
  Q33: Sees me as being competent

Factorial validity

The higher order factor analysis model obtained a good fit to the data, with a chi square of 743.96, a df of 483, a p of < 0.0001, a root mean square error of approximation (RMSEA) of 0.06, a Tucker-Lewis index (TLI) of 0.93, and a comparative fit index (CFI) of 0.94.

As indicated by the t values shown in Figure 1, all of the parameter estimates were significant. The career dimension and psychosocial dimension were highly correlated (t= 4.9). The parameter estimates from dimensions to roles had t values ranging from 5.4 to 14.4, and the parameter estimates from roles to questionnaire items had t values ranging from 10.6 to 32.2.

image

Figure 1. Higher order factor analysis. Higher order factor analysis of the Ragins and McFarlin Mentor Role Instrument. The model obtained a good fit to the data, with a chi square of 764.4, a df of 483, a p of < 0.001, a root mean square error of approximation (RMSEA) of 0.06, and a Tucker-Lewis index (TLI) and a comparative fit index (CFI) both of 0.94. Numbers indicate t values for parameter estimates from dimension to dimension (t= 4.8) or from dimensions to roles (e.g., t= 7.9 from career dimension to coach role). Under each dimension, the first role item is delineated as the reference variable, so no estimates are shown for the first role. Parameter estimates from roles to questionnaire items (Q1, Q2, Q3, etc.) had t values ranging from 10.6 to 32.2. Questionnaire items are defined in Table 2.

Download figure to PowerPoint

The R2 values for the model were generally strong. Of the 11 roles, eight had R2 values that were 0.60 or higher. The three weakest roles were the social associate, parent, and challenger roles, with R2 values of 0.18, 0.30, and 0.41, respectively. Of the 33 items, all but one had R2 values above 0.57, indicating that at least 57% of the variance in the model was explained by the item. The weakest item, which had an R2 value of 0.44, was an item concerning whether the mentor guides the mentee's personal development.

Concurrent validity

As shown in Table 3, there were strong correlations between the following: career dimension and mentoring satisfaction (0.67); psychosocial dimension and mentoring satisfaction (0.71); career dimension and mentoring effectiveness (0.63); and psychosocial factor and mentoring effectiveness (0.56). These results, based on the Pearson correlation coeficient, indicate strong concurrent validity.

Table 3.  Pearson correlation Coefficients for concurrent and predictive validity in the Ragins and McFarlin mentor role instrument.
Mentoring outcomesCareer dimensionPsychosocial dimension
Coefficientp valueCoefficientp value
  1. *The types of mentoring contacts were weighted to reflect the level of intensity, with e-mail messages weighted as 0.5, telephone calls as 1.0, impromptu meetings as 1.0, group meetings with primary mentors as 3.0, and group meetings with primary and secondary mentors as 3.0.

Satisfaction (my mentor is someone I am satisfied with)0.67<0.0010.71<0.001
Effectiveness (my mentor has been effective in his or her role)0.63<0.0010.56<0.001
Composite score for type and frequency of mentoring*0.30<0.0010.190.03

The correlations for the relationship between the dimensions and the composite score for type and frequency of mentoring were significant, but not as strongly related as the other variables. The correlations were only 0.30 for the career dimension and 0.19 for the psychosocial dimension.

Discussion

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References

When we administered the RMMRI to 141 mentees involved in clinical and translational science programs, we found that this instrument showed good reliability and validity. Specifically, the instrument demonstrated the following characteristics: (1) strong inter-item correlations, as tested by the Pearson method; (2) strong internal consistency, as tested by the Cronbach alpha method; (3) factorial validity, as demonstrated by confirmatory factor analysis of mentoring dimensions, mentoring roles, and RMMRI items; and (4) concurrent validity, as tested by Pearson method used to correlate mentoring perceptions with several mentoring outcomes (satisfaction, effectiveness, and type and frequency of mentoring).

In our factor analysis, we were not surprised to find that the career dimension and psychosocial dimension were strongly correlated. We believe that the overlap in career and psychosocial roles fulfilled by mentors reflects the fact that the psychosocial aspects of the mentoring relationship create the type of supportive and nurturing environment that facilitates career development. Within the measurement model, we found that the weakest measure was an item concerning whether the mentor guides personal development. One explanation for this weak result may be that personal development, while linked to professional outcomes, is a construct fueled by other factors that are more reflective of the individual's personality than of the mentoring relationship. Our factor analysis results indicate that the model fit is good but could be improved, especially with respect to the social associate, parent, and challenger roles. However, because our work was strictly confirmatory, we did not test alternative models.

In our last set of analyses, we found that the career and psychosocial dimensions of mentoring each correlated well with overall mentoring satisfaction and effectiveness scores, indicating that the RMMRI has good predictive validity for these outcomes. However, the correlation between the dimensions and the composite score of frequency and type of mentoring activity was not particularly strong, perhaps indicating that good mentoring is more dependent on the quality of interactions than on the number and category of interactions.

In general, our findings corroborate the results of Ragins and McFarlin7 and provide additional evidence of the reliability and validity of the RMMRI. Moreover, our results in assessing the mentoring dynamic found within the physician-scientist context suggest that the RMMRI has the “transferability” characteristics needed to apply it across disciplines.

Nevertheless, several important study limitations deserve mention. First, our sample size was small and drawn from only one institution, and this could limit the generalizability of the results. In addition, minor heterogeneity with respect to participants’ educational background was noted, however, given the small sample size subgroup analysis was not warranted. Second, we performed our analyses only on mentee perceptions. Analyzing the responses and perspectives of mentors would add another dimension of richness to our analyses and interpretations. T ird, we have not yet explored the relationship between sociodemographic characteristics and mentoring perceptions. Fourth, the RMMRI does not explicitly include items addressing research mentoring or clinical role modeling, however, the instrument does contain items that reflect important and noncontext specific aspects of the mentee–mentor relationship. However, we view our current study of the psychometric properties of the RMMRI as just the first step in exploring the evaluation of mentoring. Once we have established the core elements of mentoring that we wish to evaluate, we plan to develop a comprehensive model to evaluate mentoring.

Conclusions

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References

In academic medicine and specifically in clinical and translational science, mentoring relationships are central to providing an environment that supports the career advancement of young researchers. Clinical and translational science emphasizes skills in conducting research and in moving research innovations from the bench to the bedside with the goal of improving the health of individuals and communities. Although many disciplines, including business and law, have been investigating the effects of mentoring for years, the assessment of mentoring in clinical and translational science is a relatively new activity. Given the unique environmental demands, competitiveness of the field, and the rapid advancement of the discipline, there is reason to believe that the needs of clinical and translational science researchers may be distinct from those in other areas. Our study of the psychometric properties of the RMMRI indicates that this instrument shows reliability and validity in capturing the multidimensional nature of mentoring when administered to clinical and translational science trainees in the academic setting. Future efforts should consider longitudinal analyses of the quality of the mentee– mentor relationship and the success of the mentee, analyses of subgroups of participants with dif ering educational backgrounds within clinical and translational science, and testing the stability of the instrument over time.

Acknowledgments

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References

This publication was made possible by Grant Number UL1 RR024153 from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH) and NIH Roadmap for Medical Research. The publication was also made possible by Grant Numbers 5 KL2 RR024154 and 5 TL1 RR024155 under the University of Pittsburgh Clinical and Translational Science Institute, funded by the NIH. Its contents are solely the responsibility of the authors and do not necessarily represent the official view of the NCRR or NIH.

References

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References