SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. The Shift Toward Competency-Based Medical Education
  4. Exploring the Potential Contribution of the Social and Behavioral Sciences
  5. Support for Scholarship in Medical Education
  6. Trends in Simulation
  7. Conclusions
  8. References

This article describes opportunities for scholarship in medical education, based on a brief overview of recent changes in medical education. The implications arising from these changes are discussed, with recommendations for focus, and suggestions and examples for making progress in this field. The author discusses 1) the historical context of the current shift toward competency-based medical education, 2) the potential contribution of social and behavioral sciences to medical education scholarship, 3) methods and approaches for supporting scholarship in medical education, and very briefly 4) trends in simulation. The author concludes with a call for quality in medical education scholarship and argues that the most promising and fruitful area of medical education scholarship for the future lies in the field of assessment of individual competence.

In developing a broad overview of research in medical education, it is helpful to draw from not only the usual peer-reviewed literature consisting of primary research, but also nontraditional sources, such as conference discussions, proposed books and chapters, and successful, long-running courses and curricula, as well as those that are being developed to address an identified need in the field. Typical of these nontraditional sources are a course at the University of Ottawa called the “Healthcare Education Scholars Program” (based on a course the author directed earlier at the University of Michigan)[1] and a new book series entitled Advances in Medical Education. In developing both the course and the book series, the goal was similar: to introduce clinician educators to the “possibilities” of medical education. Specifically, the goal was to encourage people to learn from history and extrapolate beyond the current state of the field, to imagine where the field of medical education can go. We can benefit immensely if we consider both the historical context of academic medicine and the potential for contribution from the social sciences. At the same time, it is critical to integrate theory and practice. We need to make the scholarship of medical education relevant to the program director or clinician educator involved in the everyday teaching and administration of curricula. This latter point underscores a final concern to be addressed in this article, and that is to develop mechanisms and cultural contexts to support educational scholarship in the academic health center. This final task is no small feat, given that the traditional emphasis for academic productivity does not typically reward innovation in teaching, learner assessment, and curriculum development.[2] This will continue to be an uphill battle, but one worth fighting for the sake of future generations of health care practitioners.

Out of the planning and consultation with colleagues for the book series and the course came a blueprint for future directions for research in medical education, as attempts were made to summarize the current state of the field and decide how exactly to dispense this broad and complex overview. At last count, this list had evolved[3] to contain more than 60 topics (see Table 1).

Table 1. Future Directions for Scholarship in Medical Education
  • 1.
    Contextual issues
    • The history of medical education
    • Philosophical foundations in medicine and medical education
    • The influence of medical practice on medical education
      • Medical administration and its impact on medical education
      • The intersection of medical education and health services research
      • The underused potential of morbidity and mortality rounds on medical education
    • Academic “silos” in health care centers and their impact on medical education
    • The influence of social sciences and substantive academic disciplines on medical education
    • Sociocultural issues in clinical teaching
      • Psychosocial aspects of medicine and the doctor–patient relationship
    • The effect of technological developments on medical education
      • Preparing the life-long learner
    • Medical sociology—bioethics and medical education
      • Teaching and assessment of humanism and professionalism
  • 2.
    Cognition and educational theory
    • Principles of adult learning
    • The nature and nurture of medical expertise
    • Transfer of learning
    • Clinical reasoning and medical decision-making
    • Motivation and deliberate learning
    • Individual learning styles
    • Self-assessment and confidence
  • 3.
    Assessment of individuals
    • Selection of medical students and residents
      • Noncognitive criteria
      • Use of simulation scenarios
    • Assumptions about individual competency assessment
    • Workplace-based assessment
    • Simulation-based high-stakes assessment
    • Assessment of noncognitive skills
      • Assessment of teamwork and communication skills
    • Assessment of teacher effectiveness
  • 4.
    Teaching and learning
    • Enhancing effective teaching in a clinical setting
    • Communication skills
      • Breaking bad news
      • Establishing a therapeutic relationship
    • Designing a curriculum for continuity of care training
    • Community-oriented medical education
    • Residents as teachers
    • The hidden curriculum
    • Simulation in medical education
      • Effectiveness of simulators as training platforms
      • Use of simulation for remediation
      • Trainee buy-in, motivation
      • Cost-effectiveness
      • Integration of simulation technology with simulated patients
      • Where to situate simulators within curricula

In this article, the following major points will be addressed, which will provide the reader with some context as well as direction for moving the field forward: 1) competency-based medical education, 2) potential contribution of the social and behavioral sciences, and 3) support for scholarship in medical education.

The Shift Toward Competency-Based Medical Education

  1. Top of page
  2. Abstract
  3. The Shift Toward Competency-Based Medical Education
  4. Exploring the Potential Contribution of the Social and Behavioral Sciences
  5. Support for Scholarship in Medical Education
  6. Trends in Simulation
  7. Conclusions
  8. References

In 1910, Abraham Flexner revolutionized medical education with his report “Medical Education in the United States and Canada: A Report to the Carnegie Foundation for the Advancement of Teaching.”[4] In this report, he recommended the alignment of medical schools with universities (a shift that is currently happening in China) and the development of a new curriculum that emphasized preclinical science followed by clinical training.

Today, there is once again a revolution under way in medical education. This time, it was not commissioned by the American Medical Association, but emerged from changes in the practice and the business of medicine. Ludmerer,[5] in his seminal work “Time to Heal,” outlines the impact that managed care had on the way we train health care professionals. In particular, he noted that the era of the apprenticeship model is fading into the past because of the lack of time necessary to support such a model. With an emphasis on patient throughput and efficiency of hospital-based care, there is less opportunity for clinician educators to spend time guiding trainees through the process of becoming professional caregivers. These forces, along with the movement toward duty hour restrictions, error reduction, and patient safety, as well as increased individual accountability, have created a new landscape for medical education that involves the assessment of individual competence according to a standardized expectation of roles or competencies,[6, 7] with more focus on the learner.

This new focus on the assessment of individual competence is changing not only the landscape for our learners, but also for the clinician educators and administrators who run our medical schools and residency programs.[8] Indeed, the discussion of competence seems to appear in every forum on medical education,[9-12] to the point where Lingard[13] has commented that the word competence is used as a “god term or rhetorical trump card, regularly played as the last word in debates about how health professions should function.”

One of the by-products of this shift toward competency-based medical education is the development of expectations and standards for the maintenance of competence. Whatever means clinicians had used in the past to stay up to date with knowledge and skills, this will change dramatically in the next generation.[14] Individual practitioners will be regularly called on to demonstrate proficiency in all aspects of their clinical practice, according to the competencies or expected roles derived by organizations such as ACGME or frameworks such as CanMEDs.

The Emerging Primacy of Valid and Defensible Learner Assessment

The major implication of this shift toward competency-based medical education, and the major feature of this revolution in medical education, is the emphasis on individual assessment of learners. It is perhaps not an exaggeration to predict that an entire new industry of psychometric assessment is opening up to support this new emphasis. Given the imperatives of duty hour restrictions[15] and patient safety,[16] it is essential that we not only ensure effective training of our learners, but also employ valid and reliable assessment instruments to verify that students and practitioners can provide appropriate and compassionate care. Certainly, as the field moves toward a competency-based system, valid methods of assessment will be required to ensure both achievement and maintenance of these competencies.[14, 17]

With this new emphasis on learner assessment, there will be an increased need for faculty to be trained in many facets of education, including high-stakes summative assessment, psychometrics, simulation, methods for deliberate practice, and formative assessment in a patient safe environment.[18]

Aside from knowledge-based tests for certification and licensure, the breadth of valid and reliable assessment instruments available in medical education to date is relatively modest. Examples of success do exist in some areas,[19] and a review is beyond the scope of this article. For the assessment of decision-making and technical skills, instruments with consistent evidence for validity and reliability include the OSCE (Objective Structured Clinical Examination),[20] OSATS (Objective Structured Assessment of Technical Skills),[21-23] Mini-CEX,[24] and some simulation-based scenarios,[25] among others. In surgery, for example, since 1994, Reznick and colleagues[26-28] have developed and provided repeated validity evidence for the OSATS, typically using checklists and expert global ratings to examine surgical trainees on operations involving bench-top models and cadavers. The available battery of reliable and valid approaches to technical skills assessment has since been expanded to include final product ratings,[29] off-line videotape assessment,[30, 31] hand motion analysis,[32-34] and clinically relevant measures of performance.[35]

Pass rating and completion time also exhibit validity because these show changes in performance after educational intervention similar to those obtained with the expert ratings. The pass rating typically asks, “Would you allow this candidate to carry out this procedure on your next patient?” which confers some sense of clinical relevance. Completion time is also considered an important clinical variable in the operating room, but its relevance has been challenged recently on the grounds that it may fail to distinguish between efficiency and lack of attention to detail.[36] Operative training simulators have incorporated metrics specific to each trainer, and there are currently several studies under way examining whether these metrics of performance meet the test for reliability and validity and transfer to the clinical setting.[37-41]

In sum, validation evidence has accrued for several dimensions of competence and may provide a blueprint going forward for a structured approach to the development of valid and reliable assessment tools. Included here is a checklist to help researchers through the process of generating an assessment instrument which yields reliable and valid data (see Table 2).

Table 2. Seven-step Checklist for Developing a Good Assessment Instrument
Note:
  1. We can never achieve perfect validity, so consider this as an ongoing process whereby you are constantly checking performance statistics for reliability and validity.

1. Determine the purpose of your assessment
A. Formative, summative (standard setting/criteria) research
B. Knowledge, skills, attitudes (e.g., performance, teamwork, anxiety)
2. Content validity—identify main construct of interest and stakeholders
3. Review with content experts—focus group
A. Representative sample: different institutions and disciplines
B. Thematic saturation, address political issues
C. Set preliminary standards—what does perfect/borderline performance look like?
4. Item writing/development (based on related existing tests?)
5. If necessary, train the raters (and assess inter-rater reliability)
6. Pilot test the instrument (representative sample) for validity
A. Feasibility check—length, clarity, cost
B. If necessary, go back to Step 4 (modify items and pilot test again)
7. Implement modified test—measure reliability, validity based on larger sample
A. Assess construct validity

Exploring the Potential Contribution of the Social and Behavioral Sciences

  1. Top of page
  2. Abstract
  3. The Shift Toward Competency-Based Medical Education
  4. Exploring the Potential Contribution of the Social and Behavioral Sciences
  5. Support for Scholarship in Medical Education
  6. Trends in Simulation
  7. Conclusions
  8. References

One of the areas that has been largely neglected in medical education is the adoption of theories and methods from the social and behavioral sciences. This is possibly an artifact of the focused training that clinicians receive during and prior to medical school, with an emphasis on the traditional life sciences such as biology and physiology. However, social and behavioral sciences have much to offer the field. Recently, there have been examples of significant gains in our understanding of how clinicians think and behave during the practice of medicine,[42] with implications for training. For example, Moulton and colleagues[43] adopted methods and approaches from cognitive and social psychology to explore why surgeons sometimes continue to proceed with difficult cases in the operating room despite the presence of known warning signs to slow down and exercise more deliberate care. The model, widely known in the field of cognitive psychology, suggests that we all have limited attentional capacity in short-term working memory, such that we can become overloaded when presented with too much information. As experts, we learn to organize and manage complex “chunks” of information efficiently such that our attentional capacity is not overtaxed during quite sophisticated procedures. However, during cases in which something goes unexpectedly wrong, the expert's attentional capacity can suddenly become taxed to the limit, thus rendering the clinician susceptible to otherwise harmless distractions. The implication for training and safe patient care is that clinicians should recognize these situations using metacognitive strategies[44] and “slow down” to eliminate unnecessary distractions during these moments. In addition to the contribution of cognitive psychology during these situations, there are also social psychological forces at play in the clinical setting that may serve to inhibit “slowing down,” e.g., when there is time pressure to finish a case, or pressure to appear “in control” in front of one's peers. This is but one example in a burgeoning field exploring alternative ways of viewing both the practice of clinical medicine and medical education.

Support for Scholarship in Medical Education

  1. Top of page
  2. Abstract
  3. The Shift Toward Competency-Based Medical Education
  4. Exploring the Potential Contribution of the Social and Behavioral Sciences
  5. Support for Scholarship in Medical Education
  6. Trends in Simulation
  7. Conclusions
  8. References

Traditionally, scholarship has been defined narrowly in terms of publication and grant capture, while teaching has been accorded much less weight. In particular, proficiency in clinical practice is often regarded as sufficient for effective teaching. More recently, academic scholarship has been reconceptualized as consisting of four components: discovery, integration, application, and teaching.[45] The scholarship of discovery is the discovery and creation of new knowledge, traditionally associated with research. The scholarship of integration involves making connections of knowledge across and between disciplines, placing knowledge in a larger and richer context. The scholarship of application is the use of knowledge to solve problems and answer questions, as in knowledge translation. The scholarship of teaching involves not only communication but also the transformation of knowledge, according to Boyer.[45] By identifying teaching as a scholarly activity, Boyer stressed that teaching should be valued as a specific skill, important for the academic mission, and he stressed equal weighting with that of the scholarship of discovery. Scholarly teaching goes beyond excellent teaching by critically assessing the relevant educational literature to choose the most appropriate educational intervention, applying the intervention, observing and analyzing the outcomes, obtaining peer review, and then using the results to improve or modify teaching.[46] For it to be categorized as the scholarship of teaching, three additional criteria must be satisfied: it must be in a tangible form that is publicly disseminated, it must be open to review and critique, and it must allow for others to use it or build on it to advance the field.[47, 48]

How Does One Move From Scholarly Teaching to Educational Scholarship?

The topic of quality of research in medical education has been covered by others,[49] so will not be repeated here. Suffice it to say that dissemination through publication may not always be the most appropriate method of knowledge translation for this discipline. One idea for changing this culture, so that innovators are rewarded, is to make use of other metrics such as impact statements or other types of data based on qualitative research. One useful framework for assessing the impact of educational interventions is a modification of the Kirkpatrick framework, which was originally developed for use in human resources in business[50, 51] (see Table 3).

Table 3. Modified Hierarchical Version of the Kirkpatrick Framework for Assessing Impact of Educational Intervention
Note:
  1. This modified framework extends both the original work of Kirkpatrick[50] and the hierarchy developed by the BEME group[51] and is presented here for the first time.

Level 1—Reaction
Level 1a—Participation
Level 1b—Satisfaction
Level 2—Learning
Level 2a—Change in attitudes/confidence
Level 2b—Change in knowledge/skills
Level 3—Transfer to the workplace
Level 3a—Change in clinical practice
Level 3b—Successful application of learned knowledge/skills
Level 4—Results
Level 4a—Improvements in patient satisfaction
Level 4b—Improvements in patient outcome

Scholars in medical education should not be satisfied with measuring the impact of their educational intervention at Level 1 (i.e., reaction) and instead should strive to measure a change in knowledge or skill (Level 2) or beyond. When used in combination with standard educational principles and practices, the modified Kirkpatrick hierarchy can be a powerful approach for enhancing the accountability of educational scholarship and evaluating the impact of educational innovations. This modified hierarchy seems to have gained significant traction in medical education.[52] The advantage of the hierarchy is that it encourages scholars to consider raising the threshold for the type of evidence they accumulate in support of their innovations.

How Do We Support Career Development?

Although still quite rare in North American medical schools, medical education research units have developed to support innovation and scholarship. These units are typically populated with full- or part-time PhD educators and a core group of part-time clinician educators with a commitment to developing scholarship in medical education. The larger and more established of these units provide some type of support for the clinician educators at those institutions, for example, education rounds, grant programs, and support for fellowships and graduate degrees in education.[53]

Four broad strategies for promoting scholarship in medical education have recently been proposed by Naik et al.[3] These are: 1) promote excellent teaching and scholarly teaching, 2) promote broad recognition that teaching is a valued and considered scholarly activity, 3) promote a strong institutional culture for the support of educational scholarship, and 4) foster educational scholarship in residents and other trainees.

In addition, while it seems reasonable to support the attainment of advanced degrees in education to respond to the recent changes in medical education mentioned, sometimes a master's degree or PhD in education does not help the institution cope with local responses to these changes. Instead, Goldszmidt et al.[54] have convincingly argued for a local supplementary structure to support those clinician educators who are considering advanced training in medical education.[54] In this way, we can make the scholarship of medical education relevant to the program director or clinician educator involved in the everyday teaching and administration of curricula.

Trends in Simulation

  1. Top of page
  2. Abstract
  3. The Shift Toward Competency-Based Medical Education
  4. Exploring the Potential Contribution of the Social and Behavioral Sciences
  5. Support for Scholarship in Medical Education
  6. Trends in Simulation
  7. Conclusions
  8. References

Although this topic was addressed in this keynote presentation, it has recently been summarized elsewhere, and the reader is referred to that work.[55, 56] An emerging opinion from this work is that simulation should be seen as just another educational tool, not “separate” from other educational activities. Taken in this light, the major questions in this field are similar to those in other fields of education: 1) is simulation effective, 2) what type of simulation is effective for a given clinical skill, 3) how can we make it more effective, 4) is simulation cost-effective, and 5) how can we make it more cost-effective? A final question could be added, based on the primacy of individual learner assessment that is the focus of this article: is simulation useful for assessment of skills? These questions will shape the future uptake and support for simulation as a worthwhile medical education activity as questions start to be raised about the cost of the technology involved in this enterprise.[57, 58]

Conclusions

  1. Top of page
  2. Abstract
  3. The Shift Toward Competency-Based Medical Education
  4. Exploring the Potential Contribution of the Social and Behavioral Sciences
  5. Support for Scholarship in Medical Education
  6. Trends in Simulation
  7. Conclusions
  8. References

It can be argued that the most promising and fruitful area of medical education scholarship for the future lies in the field of assessment of individual competence. This is due to the revolution we are currently experiencing in the field toward competency-based education and the drive toward a regulated system of maintenance of competence.

While simulation can certainly assist in providing a highly structured framework for both delivering curricula and providing for standardized assessment, it is in the end, simply another educational tool and should be regarded as such. This means that we should continue to develop an evidence-based rationale for its use, much like any other educational tool, and develop curricula around the same basic principles for any effective educational intervention. These principles include developing a proper needs assessment, learning objectives, and lesson plans, as well as a plan for evaluating the effectiveness of the intervention.

As researchers in medical education, we should continue to look outside our field for methods, ideas, and theoretical structure to guide us. Many advances that have been made in the social and behavioral sciences could be brought to bear directly on the practice of medical education by virtue of the fact that ours is a field influenced heavily by issues relevant to sociology, social psychology, and cognitive science.

Finally, to continue to grow, we will need a continued emphasis on quality scholarship in our field, and efforts to advocate for the importance of educational scholarship in academic medicine should continue to be made, both at the local and at the national levels. The AEM consensus conference in Chicago on May 9, 2012, was a step in the right direction.

I thank the organizers of the AEM consensus conference on medical education research, Drs. Lalena Yarris, Joe LaMantia, and Nicole Deiorio for their guidance in preparing my comments for this keynote address. I thank the audience for asking critical questions and informed discussion both following the presentation and later during informal conversations.

References

  1. Top of page
  2. Abstract
  3. The Shift Toward Competency-Based Medical Education
  4. Exploring the Potential Contribution of the Social and Behavioral Sciences
  5. Support for Scholarship in Medical Education
  6. Trends in Simulation
  7. Conclusions
  8. References