SEARCH

SEARCH BY CITATION

Keywords:

  • faculty development;
  • teaching skills;
  • learner centeredness;
  • feedback

Abstract

  1. Top of page
  2. Abstract
  3. METHODS
  4. RESULTS
  5. DISCUSSION
  6. Acknowledgments
  7. References

Background: The long-term impact of longitudinal faculty development programs (FDPs) is not well understood.

Objective: To follow up past participants in the Johns Hopkins Faculty Development Program in Teaching Skills and members of a comparison group in an effort to describe the long-term impact of the program.

Design and Participants: In July 2002, we surveyed all 242 participants in the program from 1987 through 2000, and 121 members of a comparison group selected by participants as they entered the program from 1988 through 1995.

Measurements: Professional characteristics, scholarly activity, teaching activity, teaching proficiency, and teaching behaviors.

Results: Two hundred participants (83%) and 99 nonparticipants (82%) responded. When participants and nonparticipants from 1988 to 1995 were compared, participants were more likely to have taught medical students and house officers in the last year (both P<.05). Participants rated their proficiency for giving feedback more highly (P<.05). Participants scored higher than nonparticipants for 14 out of 15 behaviors related to being learner centered, building a supportive learning environment, giving and receiving feedback, and being effective leaders, half of which were statistically significant (P<.05). When remote and recent participants from 1987 through 2000 were compared with each other, few differences were found.

Conclusions: Participation in the longitudinal FDP was associated with continued teaching activities, desirable teaching behaviors, and higher self-assessments related to giving feedback and learner centeredness. Institutions should consider supporting faculty wishing to participate in FDPs in teaching skills.

In this country, millions of dollars1 and thousands of hours of faculty time continue to be invested each year on faculty development programs (FDPs). Many of these educational programs are designed to enhance the teaching skills of clinicians, with an emphasis on community physicians and junior faculty members.2 Program types are varied and include brief workshops, teaching evaluations with consultation, and longitudinal courses, sometimes called part-time “fellowships.”3 Despite the significant national investment in faculty development, only a limited number of teaching hospitals offer FDPs, and most are limited in length or only offered sporadically.4

Longitudinal programs have the benefit of allowing time for reflection, practice, feedback, and self-directed learning methods that may impart a more lasting impact than do the more succinct courses.5 Several evaluations of longitudinal programs have occurred immediately or soon after the conclusion of the intervention.6–10 Some studies have examined the impact of these programs on the skills and behaviors of participants at a later date,11–17 but they have usually reported about a limited numbers of participants,12–14,17 and only one manuscript described long-term outcomes in a case–control manner.11

The Johns Hopkins Faculty Development Program in Teaching Skills (JHFDP/TS) is a 9-month, 1 half-day per week course that has been offered annually since 1987. The participants have included faculty and fellows associated with Johns Hopkins Medical Institutions and other institutions in the Baltimore–Washington region. Nearly 300 learners have participated in the program since the JHFDP/TS began in 1987, including general internists, family physicians, pediatricians, psychiatrists, and behavioral scientists. We conducted a study to better understand the long-term impact of the JHFDP/TS on its participants' professional lives and teaching roles, skills, and behaviors.

METHODS

  1. Top of page
  2. Abstract
  3. METHODS
  4. RESULTS
  5. DISCUSSION
  6. Acknowledgments
  7. References

Program Description

The goals of the JHFDP/TS are for participants to experience and to gain expertise in concepts believed to be critical to educating medical learners, such as learner centeredness, self-directed learning, and creating a supportive learning environment. Participants meet in groups of 5 to 8 participants with 1 to 2 facilitators between early September and late May to work on modules that vary in length from 1 to 6 weeks. Module topics include giving and eliciting feedback, precepting (one-on-one teaching), time management, giving lectures and presentations, and small-group leadership skills. A wide range of teaching methods are used but most commonly include large-group presentations, small-group discussions, and skills practice. Some time is spent each week on “personal awareness,” which involves the sharing of events, feelings, and meaning related to participants' personal and professional lives. The program is described in detail elsewhere.6

Study Population

We surveyed the 242 faculty and fellows who took part in the JHFDP/TS from 1987 through 2000 and 121 members of a nonparticipant comparison group selected by participants from 1988 through 1995 as they entered the program. Participants and nonparticipants were similarly dispersed when this follow-up survey was conducted, in more than 30 states, the District of Columbia, and Canada.

Participants from 1988 to 1995 were instructed as they entered the program to select a nonparticipant for inclusion in the comparison group who was similar to him or herself in terms of professional status, percent of time spent teaching, faculty appointment level, age, and gender. The nonparticipants could not be enrolled in or employed by any Johns Hopkins Medical Institution at the time of their selection, in order to avoid any effect from being exposed to program faculty or participants, and to minimize the likelihood that they would themselves become participants at a later date.

Despite efforts to make the two groups similar at baseline, when a preprogram evaluation was carried out, more nonparticipants had assistant or associate professor faculty appointments or prior training in teaching skills, and nonparticipants rated themselves significantly higher than participants for all teaching skills assessed, including giving feedback, precepting, lecturing, working in groups, and small-group teaching. When an immediate postprogram evaluation was carried out, participants rated their ability to give feedback significantly higher than nonparticipants, and they rated similarly their abilities to precept, work in groups, and teach in a small-group setting.6 Identification of nonparticipants for the comparison group was stopped after 8 years, when analysis showed that baseline and postprogram comparisons between recent participants and nonparticipants were mimicking comparisons between earlier groups.6

Four nonparticipants did not complete a baseline or postprogram questionnaire and were excluded from this follow-up study. At the time of this study, 1 participant and 2 nonparticipants were deceased. There were 3 nonparticipants who later became participants; they are included in the dataset as participants only.

Survey Development

The survey collected basic demographic information and details about respondents' training in teaching skills (other than that provided by the JHFDP/TS), professional and teaching roles and settings, time spent on various professional activities, and scholarly accomplishments. Respondents were asked to report their proficiency with various teaching skills and the frequency with which they demonstrate certain behaviors taught by the JHFDP/TS. Responses were in the form of short answers, yes-or-no, 4- and 5-point Likert-scale items, and percentage of effort. Pilot testing was performed to assess the clarity and relevance of each question. The study was approved by the Johns Hopkins Institutional Review Board.

Data Collection

Surveys were mailed in July 2002. Subjects were advised in a cover letter that participation was optional and that consent was implied by completing and returning the survey. Reminder postcards and repeat mailings of the questionnaires were sent to nonresponders in order to encourage full participation. Those who completed the survey were mailed a copy of the 6th edition of Principles of Ambulatory Medicine.18

Statistical Analysis

Responders and nonresponders were compared using χ2 tests for categorical variables and t tests for continuous variables. Responses from participants were compared with those from nonparticipants using χ2 tests and t tests as well. Only participants who entered the program from 1988 through 1995 were included in the comparison with nonparticipants. Responses to Likert-scale questions about teaching proficiency and behaviors were collapsed into two categories to facilitate reporting.

To determine whether time as participation had an impact on results, we stratified the entire group of participants into 3 groups by year of participation (1987–1991, 1992–1996, 1997–2000) and compared them with each other for all variables using χ2 tests.

RESULTS

  1. Top of page
  2. Abstract
  3. METHODS
  4. RESULTS
  5. DISCUSSION
  6. Acknowledgments
  7. References

Comparison of Responders and Nonresponders

Surveys were received from 200 past participants (83% response rate) and from 99 members of the comparison group (82% response rate). There was no difference between responders and nonresponders with respect to gender or year of participation, and only a small difference in age (41.6 vs 42.3 years, P=.0001).

Comparisons of Past Participants and Nonparticipants

Demographic Data and Professional Lives. Demographics and descriptors of the professional lives and scholarly activities of the responding participants and nonparticipants are shown in Table 1. Nonparticipants were more likely than participants to report that they had ever received “a moderate amount” or “a lot” of training (other than the JHFDP/TS) in precepting, giving feedback, giving lectures or presentations, and leading small groups (all P<.05). More participants reported having written down or reviewed a list of their values/priorities (57.4% vs 43.9%, P=.05) and their professional/work goals (70.4% vs 48.0%, P=.001) in the last year than had nonparticipants.

Table 1. Characteristics of Past Participants and NonParticipants
 Participants*NonparticipantsP Value
n=108n=99
  • *

    Only includes participants from years when nonparticipants were selected, 1988 to 1995.

  • Totals do not add to 100% owing to incomplete item responses.

Demographics
 Male (%)61.159.6.82
 Age, mean y (SD)42.442.9.38
 Non-Hispanic white (%)84.585.7.80
Professional characteristics
 MD (%)93.592.9.87
 MPH (%)29.613.1.004
 General Internal Medicine/Geriatrics (%)90.566.7<.001
 Works in a hospital or academic medical center (%)51.443.9.08
 Spent >60 h per week working in past year (%)30.640.2.15
 Currently has a medical school faculty appointment (%)76.965.7.07
 Faculty rank Associate or Full Professor28.946.2.03
 Has ever held an educational leadership role (%)39.344.4.45
 Percent of total work hours in the past year (%):
  Clinical46.246.5.69
  Teaching12.38.4.08
  Educational program development4.12.2.29
  Research (%)11.55.4.02
  Noneducational administration7.010.5.05
Scholarly activity (%)
 Ever authored a scholarly publication78.567.7.08
 Ever presented at a national or regional meeting80.666.3.02
 Funding during the past 2 y for research34.326.0.20

Teaching Responsibilities. There was no significant difference between the number of participants and nonparticipants who reported having taught or mentored any type of learner during the past year (88% vs 79%, P=.07); however, past participants were more likely to have taught medical students (75.5% vs 57.3%, P=.006) and house officers (77.4% vs 59.4%, P=.006) during the prior year. There was no significant difference between the number of participants and nonparticipants who reported teaching in their own practice (68.9 vs 59.4, P=.16) or in a dedicated housestaff clinic (38.7 vs 28.1, P=.11) in the past year. More participants reported having taught medical learners in a variety of other academic settings (inpatient wards, noon conferences, morning report) than the nonparticipants in the past year (P<.05 for all). Although a larger number of participants than nonparticipants reported enjoying mentoring “a lot” (59.1% vs 40.8%, P=.02), there were no differences between participants and nonparticipants in their reported enjoyment of precepting, giving lectures and presentations, or leading small groups (P>.05 for all). More past participants than nonparticipants reported having ever won a national teaching or education-related award (14.0% vs 4.0%, P=.01).

Teaching Proficiencies. Past participants and nonparticipants rated their overall teaching skills similarly (Table 2). Past participants rated themselves higher for all specific teaching skills assessed, and this difference was significant for giving feedback (Table 2).

Table 2. Percent of Participants and NonParticipants Who Rated Their Proficiency in the Following Areas as “Very Good” or “Excellent”*
SkillsParticipants (%)Nonparticipants (%)P Value
n=108n=99
  • *

    Respondents rated themselves on a 5-point scale (1=Poor, 2=Fair, 3=Good, 4=Very Good, 5=Excellent).

  • Only includes participants from years when nonparticipants were selected, 1988 to 1995.

Overall teaching skills89.790.5.85
Giving feedback86.075.0.04
Eliciting feedback68.255.2.06
Precepting97.293.8.24
Giving lectures and presentations87.782.1.26
Working in groups94.488.4.13
Leading small groups84.180.2.47

Teaching Behaviors. Participants reported themselves as being more learner centered, engaging more often in behaviors conducive to building a supportive learning environment, providing better feedback, and leading effectively more often than did the nonparticipants, and half of these differences were statistically significant (Table 3).

Table 3. Percent of Subjects Whose Behavior is “Frequently” or “Always” Described by the Statement*
 Participants (%)Nonparticipants (%)P Value
n=95n=78
  • *

    Respondents rated themselves on a 5-point scale, where 0=Never, 1=Rarely, 2=Sometimes, 3=Frequently, 4=Always.

  • Only respondents who had taught or mentored learners in the past year were included, and only includes participants from years when nonparticipants were selected, 1988 to 1995.

Learner centeredness
 I ask learners what they would like to get out of our interactions72.950.7.003
 In precepting encounters, I assess and focus on the learner's needs rather than my own agenda75.057.8.02
 I help learners identify resources that are available to meet their learning needs81.166.7.03
Supportive learning environment
 I spend time building supportive relationships with my learners87.469.3.004
 I try to detect the emotional response of my learners to various situations, and discuss this if appropriate70.260.0.16
 I express concern or support for my learners when they are struggling83.976.0.20
 I let learners know my limitations as a teacher53.241.3.13
 I let learners know how different situations make me feel51.146.0.51
Giving and receiving feedback
 I start feedback sessions by asking learners to assess their performance66.339.1<.001
 In feedback sessions I focus on specific areas needing improvement, rather than making generalizations75.867.1.22
 After receiving feedback from a learner, I reflect on whether or not that feedback is valid75.874.7.87
Effective leadership
 When leading small groups, I help to ensure that respect is conveyed to all participants98.087.5.005
 I make an effort to draw in those who don't participate much when I lead small groups73.364.8.21
 In working with others, I pay attention to others' needs in addition to my own92.683.5.04
 When I notice a problem that needs to be addressed in my workplace, I take time to analyze it67.679.6.05

Comparison of Remote and Recent Participants, 1987 to 2000

Remote participants were older and more were male than recent participants (both P<.05). In addition, remote participants reported spending more time on noneducational administration in the past year, were less likely to have taught medical students or to have taught in a dedicated housestaff clinic in the past year, and were less likely than recent participants to report that they let learners know their limitations as a teacher (P<.05 for all). There were no other significant differences between recent and remote participants.

DISCUSSION

  1. Top of page
  2. Abstract
  3. METHODS
  4. RESULTS
  5. DISCUSSION
  6. Acknowledgments
  7. References

Seven to 15 years elapsed between the time of JHFDP/TS participation and this study and yet we found several important differences between participants' and nonparticipants' self-assessed teaching skills and behaviors that might be attributed to a lasting influence from the program. The sustained impact is further highlighted by the few differences found when recent and remote participants were compared.

While similar numbers of participants and nonparticipants reported that they had taught or mentored any type of learner in the year prior to the study, participants were more likely to be teaching medical students and house officers in academic settings than nonparticipants. Additionally, more past participants have won national teaching or education-related awards. Although it is not an explicit objective of the JHFDP/TS for its participants to pursue academic careers, the program may have imparted participants with a greater interest in teaching or greater motivation to teach effectively.

This study is unique when compared with most prior long-term follow-up studies of FDPs, which have primarily focused on objective markers of academic success such as presentations, publications, and leadership positions.11–15,17 Instead, we evaluated participants' perceptions of their teaching. Educators value the ability to provide feedback and be learner centered.19,20 We are aware of several studies evaluating the feedback skills of participants in faculty training programs21–31 but only one that explicitly comments on the enhanced learner centeredness of its participants.16 Like others, we evaluated feedback skills, but we additionally assessed learner centeredness. Our past participants rated their abilities to show respect and support for learners, to be learner centered, and to give feedback significantly higher than nonparticipants, suggesting that the emphasis and methods of our FDP were effective and have had a lasting impact.

Several limitations of this study should be considered. First, we used a nonrandomized control group, and our self-selected participants may have had a greater interest in teaching. However, nonparticipants rated themselves significantly higher at baseline than participants for all teaching skills assessed,6 and have themselves pursued significant amounts of training in teaching skills. Second, the generalizability of the study's findings may be limited because it focuses on a single FDP; however, most past participants are currently in settings remote from Johns Hopkins University, which suggests that the JHFDP/TS has instilled learning with broad applicability. Finally, this study was dependent on self-assessment, and participants may have been “primed” by virtue of their participation to rate themselves more highly than nonparticipants. This effect is somewhat minimized by the amount of time elapsed since program participation. Very few evaluations of FDPs have been based on objective evidence, such as actual or videotaped observations of participants' teaching encounters,26,30,32,33 and it would not be feasible to conduct such a study with a population as large as ours. Studies based on learner or peer evaluations of FDP participants are also unusual.8,27,29,30,34 Self-assessments by participants in FDPs have been shown to correlate positively with trainee ratings,27 and clinician teachers may even underestimate their abilities when compared with learners' assessments.35

Our study represents one of the largest and longest follow-up studies to date of participants in a FDP in teaching skills. The length of the program, its emphasis on self-directed learning, and its use of reflection and experiential learning methods likely contributed to its lasting effect on participants' perceptions of their abilities to give feedback, support learners, and be learner centered. The apparent sustained impact of the JHFDP/TS on its participants, when compared with a group of nonparticipants, suggests that this longitudinal model for faculty development is effective. These findings support continued investment in FDPs to improve the skills of the medical educators who will be teaching the physicians of tomorrow.

Acknowledgments

  1. Top of page
  2. Abstract
  3. METHODS
  4. RESULTS
  5. DISCUSSION
  6. Acknowledgments
  7. References

We are grateful to Mrs. Darilyn Rohlfing for her administrative support of the project. This work was presented at the Annual Meeting of the Society of General Internal Medicine on May 14, 2004. Dr. Wright is an Arnold P. Gold Foundation Associate Professor of Medicine. The JHFDP/TS is supported by the Health Resources and Services Administration, Grant #5 D55HP00049-05-00.

References

  1. Top of page
  2. Abstract
  3. METHODS
  4. RESULTS
  5. DISCUSSION
  6. Acknowledgments
  7. References
  • 1
    Diamond RD. Creating Opportunities in Medical Education and Training: Health Resources and Services Administration Bureau of Health Professions for the General Internal Medicine Faculty Development Project; 2000.
  • 2
    Skeff KM, Stratos GA, Mygdal WK, et al. Clinical teaching improvement: past and future for faculty development. Fam Med. 1997;29: 2527.
  • 3
    Wilkerson L, Irby DM. Strategies for improving teaching practices: a comprehensive approach to faculty development. Acad Med. 1998;73: 38796.
  • 4
    Clark JM, Houston TK, Kolodner K, Branch WT, Levine RB, Kern DE. Teaching the teachers: National survey of faculty development in departments of medicine of U.S. teaching hospitals. J Gen Intern Med. 2004;19: 20514.
  • 5
    Skeff KM, Stratos GA, Mygdal W, et al. Faculty development. A resource for clinical teachers. J Gen Intern Med. 1997;12 (suppl 2): S5663.
  • 6
    Cole KA, Barker LR, Kolodner K, Williamson PR, Wright SM, Kern DE. Faculty development in teaching skills: an intensive longitudinal model. Acad Med 79: 46980.
  • 7
    Elliot DL, Skeff KM, Stratos GA. How do you get to the improvement of teaching? A longitudinal faculty development program for medical educators. Teach Learn Med. 1999;11: 527.
  • 8
    Teherani A, Hitchcock MA, Nyquist JG. Longitudinal outcomes of an executive-model program for faculty development. Acad Med. 2001;76: S6870.
  • 9
    Gruppen LD, Frohna AZ, Anderson RM, Lowe KD. Faculty development for educational leadership and scholarship. Acad Med. 2003;78: 13741.
  • 10
    Pololi LH, Knight SM, Dennis K, Frankel RM. Helping medical school faculty realize their dreams: an innovative, collaborative mentoring program. Acad Med. 2002;77: 37784.
  • 11
    McGaghie WC, Bogdewic S, Reid A, Arndt JE, Stritter FT, Frey JJ. Outcomes of a faculty development fellowship in family medicine. Fam Med. 1990;22: 196200.
  • 12
    Marks MB. Academic careers in medical education: perceptions of the effects of a faculty development program. Acad Med. 1999;74: S724.
  • 13
    Morzinski JA, Simpson DE. Outcomes of a comprehensive faculty development program for local, full-time, faculty. Fam Med. 2003;35: 4349.
  • 14
    Rust G, Taylor V, Morrow R, Everett J. The Morehouse Faculty Development Program: methods and 3-year outcomes. Fam Med. 1998;30: 1627.
  • 15
    Anderson WA, Stritter FT, Mygdal WK, Arndt JE, Reid A. Outcomes of three part-time faculty development fellowship programs. Fam Med. 1997;29: 2048.
  • 16
    Armstrong EG, Doyle J, Bennett NL. Transformative professional development of physicians as educators: assessment of a model. Acad Med. 2003;78: 7028.
  • 17
    Steinert Y, Nasmith L, McLeod PJ, Conochie L. A teaching scholars program to develop leaders in medical education. Acad Med. 2003;78: 1429.
  • 18
    BarkerLR, BurtonJR, ZievePD, eds. Principles of Ambulatory Medicine. 6th edn. Philadelphia, PA: Lippincott Williams & Wilkins; 2003.
  • 19
    Ludmerer KM. Learner-centered medical education. N Engl J Med. 2004;351: 11634.
  • 20
    Branch WT, Paranjape A. Feedback and reflection: teaching methods for clinical settings. Acad Med. 2002;77: 11858.
  • 21
    Salerno SM, Jackson JL, O'Malley PG. Interactive faculty development seminars improve the quality of written feedback in ambulatory teaching. J Gen Intern Med. 2003;18: 8314.
  • 22
    Salerno SM, O'Malley PG, Pangaro LN, Wheeler GA, Moores LK, Jackson JL. Faculty development seminars based on the one-minute preceptor improve feedback in the ambulatory setting. J Gen Intern Med. 2002;17: 77987.
  • 23
    Stone S, Mazor K, Devaney-O-Neil S, et al. Development and implementation of an objective structured teaching exercise (OSTE) to evaluate improvement in feedback skills following a faculty development workshop. Teach Learn Med. 2003;15: 713.
  • 24
    Skeff KM, Stratos GA, Bergen MR. Evaluation of a medical-faculty development program—a comparison of traditional pre/post and retrospective pre/post self-assessment ratings. Eval Health Profess. 1992;15: 35066.
  • 25
    Quirk ME, DeWitt T, Lasser D, Huppert M, Hunniwell E. Evaluation of primary care futures: a faculty development program for community health center preceptors. Acad Med. 1998;73: 7057.
  • 26
    Sheets KJ, Henry RC. Evaluation of a faculty development program for family physicians. Med Teach. 1988;10: 7583.
  • 27
    Hewson MG, Copeland HL, Fishleder AJ. What's the use of faculty development? Program evaluation using retrospective self-assessments and independent performance ratings. Teach Learn Med. 2001;13: 15360.
  • 28
    Skeff KM, Campbell M, Stratos G, Jones HW III, Cooke M. Assessment by attending physicians of a seminar method to improve clinical teaching. J Med Educ. 1984;59: 94450.
  • 29
    Skeff KM, Stratos GA, Berman J, Bergen MR. Improving clinical teaching. Evaluation of a national dissemination program. Arch Intern Med. 1992;152: 115661.
  • 30
    Skeff KM, Stratos G, Campbell M, Cooke M, Jones HW III. Evaluation of the seminar method to improve clinical teaching. J Gen Intern Med. 1986;1: 31522.
  • 31
    Hewson M. A theory-based faculty development program for clinician-educators. Acad Med. 2000;75: 498501.
  • 32
    Nasmith L, Steinert Y, Saroyan A, Daigle N, Franco E. Assessing the impact of a faculty development workshop: a methodological study. Teach Learn Med. 1997;9: 20914.
  • 33
    Skeff KM. Evaluation of a method for improving the teaching performance of attending physicians. Am J Med. 1983;75: 46570.
  • 34
    Nathan RG, Smith MF. Students' evaluations of faculty members' teaching before and after a teacher-training workshop. Acad Med. 1992;67: 1345.
  • 35
    Windish DM, Knight AM, Wright SM. Clinician-teachers' self-assessments versus learners' perceptions. J Gen Intern Med. 2004;19: 5547.