Long-Term Follow-Up of a Longitudinal Faculty Development Program in Teaching Skills
Address correspondence and requests for reprints to Dr. Knight, CIMS Division, ASC Bldg., Rm. 262, Johns Hopkins Bayview Medical Center, 4940 Eastern Avenue, Baltimore, MD 21224 (e-mail: email@example.com).
Background: The long-term impact of longitudinal faculty development programs (FDPs) is not well understood.
Objective: To follow up past participants in the Johns Hopkins Faculty Development Program in Teaching Skills and members of a comparison group in an effort to describe the long-term impact of the program.
Design and Participants: In July 2002, we surveyed all 242 participants in the program from 1987 through 2000, and 121 members of a comparison group selected by participants as they entered the program from 1988 through 1995.
Measurements: Professional characteristics, scholarly activity, teaching activity, teaching proficiency, and teaching behaviors.
Results: Two hundred participants (83%) and 99 nonparticipants (82%) responded. When participants and nonparticipants from 1988 to 1995 were compared, participants were more likely to have taught medical students and house officers in the last year (both P<.05). Participants rated their proficiency for giving feedback more highly (P<.05). Participants scored higher than nonparticipants for 14 out of 15 behaviors related to being learner centered, building a supportive learning environment, giving and receiving feedback, and being effective leaders, half of which were statistically significant (P<.05). When remote and recent participants from 1987 through 2000 were compared with each other, few differences were found.
Conclusions: Participation in the longitudinal FDP was associated with continued teaching activities, desirable teaching behaviors, and higher self-assessments related to giving feedback and learner centeredness. Institutions should consider supporting faculty wishing to participate in FDPs in teaching skills.
In this country, millions of dollars1 and thousands of hours of faculty time continue to be invested each year on faculty development programs (FDPs). Many of these educational programs are designed to enhance the teaching skills of clinicians, with an emphasis on community physicians and junior faculty members.2 Program types are varied and include brief workshops, teaching evaluations with consultation, and longitudinal courses, sometimes called part-time “fellowships.”3 Despite the significant national investment in faculty development, only a limited number of teaching hospitals offer FDPs, and most are limited in length or only offered sporadically.4
Longitudinal programs have the benefit of allowing time for reflection, practice, feedback, and self-directed learning methods that may impart a more lasting impact than do the more succinct courses.5 Several evaluations of longitudinal programs have occurred immediately or soon after the conclusion of the intervention.6–10 Some studies have examined the impact of these programs on the skills and behaviors of participants at a later date,11–17 but they have usually reported about a limited numbers of participants,12–14,17 and only one manuscript described long-term outcomes in a case–control manner.11
The Johns Hopkins Faculty Development Program in Teaching Skills (JHFDP/TS) is a 9-month, 1 half-day per week course that has been offered annually since 1987. The participants have included faculty and fellows associated with Johns Hopkins Medical Institutions and other institutions in the Baltimore–Washington region. Nearly 300 learners have participated in the program since the JHFDP/TS began in 1987, including general internists, family physicians, pediatricians, psychiatrists, and behavioral scientists. We conducted a study to better understand the long-term impact of the JHFDP/TS on its participants' professional lives and teaching roles, skills, and behaviors.
The goals of the JHFDP/TS are for participants to experience and to gain expertise in concepts believed to be critical to educating medical learners, such as learner centeredness, self-directed learning, and creating a supportive learning environment. Participants meet in groups of 5 to 8 participants with 1 to 2 facilitators between early September and late May to work on modules that vary in length from 1 to 6 weeks. Module topics include giving and eliciting feedback, precepting (one-on-one teaching), time management, giving lectures and presentations, and small-group leadership skills. A wide range of teaching methods are used but most commonly include large-group presentations, small-group discussions, and skills practice. Some time is spent each week on “personal awareness,” which involves the sharing of events, feelings, and meaning related to participants' personal and professional lives. The program is described in detail elsewhere.6
We surveyed the 242 faculty and fellows who took part in the JHFDP/TS from 1987 through 2000 and 121 members of a nonparticipant comparison group selected by participants from 1988 through 1995 as they entered the program. Participants and nonparticipants were similarly dispersed when this follow-up survey was conducted, in more than 30 states, the District of Columbia, and Canada.
Participants from 1988 to 1995 were instructed as they entered the program to select a nonparticipant for inclusion in the comparison group who was similar to him or herself in terms of professional status, percent of time spent teaching, faculty appointment level, age, and gender. The nonparticipants could not be enrolled in or employed by any Johns Hopkins Medical Institution at the time of their selection, in order to avoid any effect from being exposed to program faculty or participants, and to minimize the likelihood that they would themselves become participants at a later date.
Despite efforts to make the two groups similar at baseline, when a preprogram evaluation was carried out, more nonparticipants had assistant or associate professor faculty appointments or prior training in teaching skills, and nonparticipants rated themselves significantly higher than participants for all teaching skills assessed, including giving feedback, precepting, lecturing, working in groups, and small-group teaching. When an immediate postprogram evaluation was carried out, participants rated their ability to give feedback significantly higher than nonparticipants, and they rated similarly their abilities to precept, work in groups, and teach in a small-group setting.6 Identification of nonparticipants for the comparison group was stopped after 8 years, when analysis showed that baseline and postprogram comparisons between recent participants and nonparticipants were mimicking comparisons between earlier groups.6
Four nonparticipants did not complete a baseline or postprogram questionnaire and were excluded from this follow-up study. At the time of this study, 1 participant and 2 nonparticipants were deceased. There were 3 nonparticipants who later became participants; they are included in the dataset as participants only.
The survey collected basic demographic information and details about respondents' training in teaching skills (other than that provided by the JHFDP/TS), professional and teaching roles and settings, time spent on various professional activities, and scholarly accomplishments. Respondents were asked to report their proficiency with various teaching skills and the frequency with which they demonstrate certain behaviors taught by the JHFDP/TS. Responses were in the form of short answers, yes-or-no, 4- and 5-point Likert-scale items, and percentage of effort. Pilot testing was performed to assess the clarity and relevance of each question. The study was approved by the Johns Hopkins Institutional Review Board.
Surveys were mailed in July 2002. Subjects were advised in a cover letter that participation was optional and that consent was implied by completing and returning the survey. Reminder postcards and repeat mailings of the questionnaires were sent to nonresponders in order to encourage full participation. Those who completed the survey were mailed a copy of the 6th edition of Principles of Ambulatory Medicine.18
Responders and nonresponders were compared using χ2 tests for categorical variables and t tests for continuous variables. Responses from participants were compared with those from nonparticipants using χ2 tests and t tests as well. Only participants who entered the program from 1988 through 1995 were included in the comparison with nonparticipants. Responses to Likert-scale questions about teaching proficiency and behaviors were collapsed into two categories to facilitate reporting.
To determine whether time as participation had an impact on results, we stratified the entire group of participants into 3 groups by year of participation (1987–1991, 1992–1996, 1997–2000) and compared them with each other for all variables using χ2 tests.
Comparison of Responders and Nonresponders
Surveys were received from 200 past participants (83% response rate) and from 99 members of the comparison group (82% response rate). There was no difference between responders and nonresponders with respect to gender or year of participation, and only a small difference in age (41.6 vs 42.3 years, P=.0001).
Comparisons of Past Participants and Nonparticipants
Demographic Data and Professional Lives. Demographics and descriptors of the professional lives and scholarly activities of the responding participants and nonparticipants are shown in Table 1. Nonparticipants were more likely than participants to report that they had ever received “a moderate amount” or “a lot” of training (other than the JHFDP/TS) in precepting, giving feedback, giving lectures or presentations, and leading small groups (all P<.05). More participants reported having written down or reviewed a list of their values/priorities (57.4% vs 43.9%, P=.05) and their professional/work goals (70.4% vs 48.0%, P=.001) in the last year than had nonparticipants.
Table 1. Characteristics of Past Participants and NonParticipants
| Male (%)||61.1||59.6||.82|
| Age, mean y (SD)||42.4||42.9||.38|
| Non-Hispanic white (%)||84.5||85.7||.80|
| MD (%)||93.5||92.9||.87|
| MPH (%)||29.6||13.1||.004|
| General Internal Medicine/Geriatrics (%)||90.5||66.7||<.001|
| Works in a hospital or academic medical center (%)||51.4||43.9||.08|
| Spent >60 h per week working in past year (%)||30.6||40.2||.15|
| Currently has a medical school faculty appointment (%)||76.9||65.7||.07|
| Faculty rank Associate or Full Professor||28.9||46.2||.03|
| Has ever held an educational leadership role (%)||39.3||44.4||.45|
| Percent of total work hours in the past year (%):†|
| Educational program development||4.1||2.2||.29|
| Research (%)||11.5||5.4||.02|
| Noneducational administration||7.0||10.5||.05|
|Scholarly activity (%)|
| Ever authored a scholarly publication||78.5||67.7||.08|
| Ever presented at a national or regional meeting||80.6||66.3||.02|
| Funding during the past 2 y for research||34.3||26.0||.20|
Teaching Responsibilities. There was no significant difference between the number of participants and nonparticipants who reported having taught or mentored any type of learner during the past year (88% vs 79%, P=.07); however, past participants were more likely to have taught medical students (75.5% vs 57.3%, P=.006) and house officers (77.4% vs 59.4%, P=.006) during the prior year. There was no significant difference between the number of participants and nonparticipants who reported teaching in their own practice (68.9 vs 59.4, P=.16) or in a dedicated housestaff clinic (38.7 vs 28.1, P=.11) in the past year. More participants reported having taught medical learners in a variety of other academic settings (inpatient wards, noon conferences, morning report) than the nonparticipants in the past year (P<.05 for all). Although a larger number of participants than nonparticipants reported enjoying mentoring “a lot” (59.1% vs 40.8%, P=.02), there were no differences between participants and nonparticipants in their reported enjoyment of precepting, giving lectures and presentations, or leading small groups (P>.05 for all). More past participants than nonparticipants reported having ever won a national teaching or education-related award (14.0% vs 4.0%, P=.01).
Teaching Proficiencies. Past participants and nonparticipants rated their overall teaching skills similarly (Table 2). Past participants rated themselves higher for all specific teaching skills assessed, and this difference was significant for giving feedback (Table 2).
Table 2. Percent of Participants and NonParticipants Who Rated Their Proficiency in the Following Areas as “Very Good” or “Excellent”*
|Overall teaching skills||89.7||90.5||.85|
|Giving lectures and presentations||87.7||82.1||.26|
|Working in groups||94.4||88.4||.13|
|Leading small groups||84.1||80.2||.47|
Teaching Behaviors. Participants reported themselves as being more learner centered, engaging more often in behaviors conducive to building a supportive learning environment, providing better feedback, and leading effectively more often than did the nonparticipants, and half of these differences were statistically significant (Table 3).
Table 3. Percent of Subjects Whose Behavior is “Frequently” or “Always” Described by the Statement*
| I ask learners what they would like to get out of our interactions||72.9||50.7||.003|
| In precepting encounters, I assess and focus on the learner's needs rather than my own agenda||75.0||57.8||.02|
| I help learners identify resources that are available to meet their learning needs||81.1||66.7||.03|
|Supportive learning environment|
| I spend time building supportive relationships with my learners||87.4||69.3||.004|
| I try to detect the emotional response of my learners to various situations, and discuss this if appropriate||70.2||60.0||.16|
| I express concern or support for my learners when they are struggling||83.9||76.0||.20|
| I let learners know my limitations as a teacher||53.2||41.3||.13|
| I let learners know how different situations make me feel||51.1||46.0||.51|
|Giving and receiving feedback|
| I start feedback sessions by asking learners to assess their performance||66.3||39.1||<.001|
| In feedback sessions I focus on specific areas needing improvement, rather than making generalizations||75.8||67.1||.22|
| After receiving feedback from a learner, I reflect on whether or not that feedback is valid||75.8||74.7||.87|
| When leading small groups, I help to ensure that respect is conveyed to all participants||98.0||87.5||.005|
| I make an effort to draw in those who don't participate much when I lead small groups||73.3||64.8||.21|
| In working with others, I pay attention to others' needs in addition to my own||92.6||83.5||.04|
| When I notice a problem that needs to be addressed in my workplace, I take time to analyze it||67.6||79.6||.05|
Comparison of Remote and Recent Participants, 1987 to 2000
Remote participants were older and more were male than recent participants (both P<.05). In addition, remote participants reported spending more time on noneducational administration in the past year, were less likely to have taught medical students or to have taught in a dedicated housestaff clinic in the past year, and were less likely than recent participants to report that they let learners know their limitations as a teacher (P<.05 for all). There were no other significant differences between recent and remote participants.
Seven to 15 years elapsed between the time of JHFDP/TS participation and this study and yet we found several important differences between participants' and nonparticipants' self-assessed teaching skills and behaviors that might be attributed to a lasting influence from the program. The sustained impact is further highlighted by the few differences found when recent and remote participants were compared.
While similar numbers of participants and nonparticipants reported that they had taught or mentored any type of learner in the year prior to the study, participants were more likely to be teaching medical students and house officers in academic settings than nonparticipants. Additionally, more past participants have won national teaching or education-related awards. Although it is not an explicit objective of the JHFDP/TS for its participants to pursue academic careers, the program may have imparted participants with a greater interest in teaching or greater motivation to teach effectively.
This study is unique when compared with most prior long-term follow-up studies of FDPs, which have primarily focused on objective markers of academic success such as presentations, publications, and leadership positions.11–15,17 Instead, we evaluated participants' perceptions of their teaching. Educators value the ability to provide feedback and be learner centered.19,20 We are aware of several studies evaluating the feedback skills of participants in faculty training programs21–31 but only one that explicitly comments on the enhanced learner centeredness of its participants.16 Like others, we evaluated feedback skills, but we additionally assessed learner centeredness. Our past participants rated their abilities to show respect and support for learners, to be learner centered, and to give feedback significantly higher than nonparticipants, suggesting that the emphasis and methods of our FDP were effective and have had a lasting impact.
Several limitations of this study should be considered. First, we used a nonrandomized control group, and our self-selected participants may have had a greater interest in teaching. However, nonparticipants rated themselves significantly higher at baseline than participants for all teaching skills assessed,6 and have themselves pursued significant amounts of training in teaching skills. Second, the generalizability of the study's findings may be limited because it focuses on a single FDP; however, most past participants are currently in settings remote from Johns Hopkins University, which suggests that the JHFDP/TS has instilled learning with broad applicability. Finally, this study was dependent on self-assessment, and participants may have been “primed” by virtue of their participation to rate themselves more highly than nonparticipants. This effect is somewhat minimized by the amount of time elapsed since program participation. Very few evaluations of FDPs have been based on objective evidence, such as actual or videotaped observations of participants' teaching encounters,26,30,32,33 and it would not be feasible to conduct such a study with a population as large as ours. Studies based on learner or peer evaluations of FDP participants are also unusual.8,27,29,30,34 Self-assessments by participants in FDPs have been shown to correlate positively with trainee ratings,27 and clinician teachers may even underestimate their abilities when compared with learners' assessments.35
Our study represents one of the largest and longest follow-up studies to date of participants in a FDP in teaching skills. The length of the program, its emphasis on self-directed learning, and its use of reflection and experiential learning methods likely contributed to its lasting effect on participants' perceptions of their abilities to give feedback, support learners, and be learner centered. The apparent sustained impact of the JHFDP/TS on its participants, when compared with a group of nonparticipants, suggests that this longitudinal model for faculty development is effective. These findings support continued investment in FDPs to improve the skills of the medical educators who will be teaching the physicians of tomorrow.
We are grateful to Mrs. Darilyn Rohlfing for her administrative support of the project. This work was presented at the Annual Meeting of the Society of General Internal Medicine on May 14, 2004. Dr. Wright is an Arnold P. Gold Foundation Associate Professor of Medicine. The JHFDP/TS is supported by the Health Resources and Services Administration, Grant #5 D55HP00049-05-00.