At the 2012 Academic Emergency Medicine consensus conference, “Education Research in Emergency Medicine: Opportunities, Challenges, and Strategies for Success,” a breakout session convened to discuss postgraduate fellowship training in emergency medicine (EM), which would focus on education research. Graduates will form a growing cadre of education scholars who conduct and publish quality education research. This proceedings article reports the consensus findings of a breakout session subgroup whose goal was to construct a needs assessment for the proposed 2-year education scholarship fellowship. The authors describe, based on expert and participant consensus, a framework for a large-scale, mixed-methods needs assessment for such a fellowship.
The quality of medical education research has been criticized for its inexact methods and standards of reporting. Challenges have been posed to the medical education community to perform more outcomes-based research, which demonstrate measurable outcomes with validity evidence and go beyond descriptive studies of individual learner satisfaction.[1-6] In emergency medicine (EM), despite the growth of departmental and national faculty development programs,[7-15] most faculty develop education research skills either on their own or through an unstructured apprenticeship approach with experienced medical education researchers.
The 2012 Academic Emergency Medicine consensus conference “Education Research in Emergency Medicine: Opportunities, Challenges, and Strategies for Success” aimed to promote and further the field of EM education scholarship. One specific objective of the conference was to develop guidelines for a rigorous, postgraduate “education scholarship fellowship.” The goal is that graduates of these programs will form a growing cadre of education scholars who meet high standards in the conduct of education scholarship in EM. Having a critical mass of education scholars is a key to success for highly productive medical education research groups, as measured by publications and funding.[16, 17] A comprehensive literature review revealed no publications describing postgraduate EM education fellowships. Based on Web-based search strategies, a few EM education fellowships were identified. A review of their public descriptions indicated that they are nonuniform in training duration, curricular content, and education research emphasis. All, however, require a significant investment by the trainee, educators, mentors, statisticians, and the department. Because an education scholarship fellowship will also likely be resource-intensive, careful consideration is warranted before implementing such training programs. For the consensus conference, we thus concluded that a comprehensive needs assessment should be conducted to explore the advantages, disadvantages, and feasibility of developing a 2-year EM fellowship in education scholarship that would focus heavily on education research.
Overview and Methods of a Needs Assessment
A needs assessment is a systematic process to identify gaps between current and desired performance to make informed decisions. Common to all needs assessment approaches is eliciting the perspectives and preferences of key stakeholders. We broadly define stakeholders as those groups or individuals that are “affected by organizational performance.” Before recommending the implementation of an adult educational activity or a revision of existing curricula, a formal needs assessment that includes key stakeholder input should be conducted. Such judicious planning can help determine feasibility, tailor curricular design, and optimize resource utilization. Furthermore, the needs assessment may also identify benchmark measures for program effectiveness, which will be useful for tracking future effect, as well as deficiencies that may not be remedied with the proposed training solution. Finally, gaining the support of the stakeholders prior to implementation enhances the likelihood of their endorsement of the program.
The first step of a needs analysis is to determine what specific needs are to be investigated. These can take the form of why, what, who, how, and when. For example, the overarching goal of the consensus conference was to improve the quality of medical education research led by EM faculty. In this context, the needs analysis can answer a variety of questions, which might include the following:
- Why conduct fellowship training in medical education research?
- What skills do graduating fellows need to be successful in educational scholarship?
- Who are best to train education fellows—MD or PhD educators?
- How do we improve the education scholarship of medical educators in EM?
- When is the optimal time to develop education research skills—residency, fellowship, or during faculty career?
It is critical to clearly identify the specific areas of need to be addressed. In this needs assessment we will address two key questions: 1) why have a medical education scholarship fellowship and 2) what competencies are necessary for a graduating fellow to be successful and academically productive?
To answer these questions, learning gaps should be identified. This process assesses the current state of the training program and learning needs and then identifies the ideal state, thus describing the gap. There are five types of learning needs: normative, prescribed, perceived, expressed, and comparative. Normative needs are discrepancies between standards established by experts and a group's current performance or skills. Prescribed needs are training discrepancies defined by educators or programmatic leaders that require an educational intervention. Perceived needs are the needs that learners articulate. Expressed needs are the learners' needs expressed through their actions. Comparative needs are identified by comparing discrepancies between two similar groups, rather than between a single group and the ideal standard.[22, 23] A gap analysis examines discrepancies between these learning needs and the ideal state.
Once the learning needs are identified, a data collection strategy must be determined. There is a variety of qualitative and quantitative needs assessment tools that exist to evaluate learning needs (Table 1). The choice of instruments will depend on the targeted learning needs, stakeholder accessibility, and implementation feasibility. Surveys provide both quantitative and qualitative data through scaled responses and free-text comments, respectively. Focus groups and interviews provide qualitative data; as such, they are helpful for eliciting information representing the meaning and values participants ascribe to their experiences, complementing predefined constructs used in qualitative approaches. Environmental scans use existing data sets to answer targeted questions.[22, 24]
Table 1. Needs Assessment Tools Useful in Analyzing Training Programs[19, 21, 23]
|Survey||Quantitative(scaled survey),Qualitative(comment servey)||Surveys or polls conducted on paper or electronically with a variety of question formats|| |
- Can sample large groups in a short time
- Data easily summarized
- Opportunity for response without fear of embarrassment
- Time-intensive to develop an effective survey
- May not effectively explore the root cause of problems or achieve potential solutions
| || || |
|Interview||Qualitative||A conversation with individuals to gain in-depth insight of their perspectives|| |
- Exploration of unique qualitative information (e.g., knowledge, skills, attitudes) from an individual's perspective
- Spontaneous feedback
- Time-consuming for interviewer and interviewee
- Difficult to analyze and quantify data
- Requires skilled interviewer
|Focus group||Qualitative||A group discussion typically focusing on a particular problem, goal, or task|| |
- Real-time interaction between different perspectives
- Focus on consensus-building
- Time-consuming for facilitator and participants
- Difficult to analyze and quantify data
- Requires skilled facilitator
|Environmental scan||Qualitative, Quantitative||Assessment of already existing data, either internal or external to the institution (e.g., performance data, literature review)|| |
- Inexpensive because already existing data sets
- Often automatically updated data sets
- Does not require contact with target audience
- Externally verifiable data
- Data sources may provide too broad an answer and not exactly answer targeted question
The objective of this article is to propose and describe, based on expert and participant consensus, a framework for a large-scale, mixed-methods needs assessment for a postgraduate education scholarship fellowship. We focus on whether a formal 2-year, education scholarship fellowship should be developed, codified, and supported in EM. To meet standards for scholarship, a needs assessment will aim to: 1) clearly state and justify the research question, 2) represent the target populations, 3) have outcome measures with evidence supporting reasonable claims of validity, and 4) highlight a clear, concise take-home message.
Based on information about existing EM education fellowships, three members of the breakout group (WC, ML, LY) developed priorities for discussion at the consensus conference breakout session, addressing education researcher training at the postgraduate fellowship level. The session started with a 15-minute introduction on the purpose, concept, and instruments available to conduct a needs assessment by an external medical education expert. Subsequently, the breakout participants were divided into smaller focus groups for 45-minute facilitated discussions about building a needs assessment framework for an education scholarship fellowship program. The sessions were audiorecorded, and dedicated notetakers transcribed comments. The objectives were to determine the key stakeholder roles, their potential responses, and the optimal process to solicit their opinions.
There were 23 participants who attended the consensus conference breakout session. The participant pool included expert EM faculty with an academic focus in medical education, medical education researchers, educational leaders, current and past education fellowship participants, and two external expert consultants in the field of medical education. A framework was constructed to inform the design of the formal needs assessment and was based on the opinions that emerged during the consensus-building focus group sessions.
We identified seven primary stakeholders, which included department chairs, education leaders, faculty members with an interest in education research, medical education fellowship directors, graduates of education fellowship programs, current education fellows, and directors of faculty development programs in education scholarship. Because a common pitfall in conducting needs assessments is overreliance on a single assessment approach or a limited population sample, we constructed a large-scale, mixed-methods needs assessment of education scholarship fellowship training in EM by targeting the stakeholders through different assessment strategies (Table 2).[19, 22]
Table 2. Training Needs Assessment Overview for the Seven Key Stakeholders in the Development of an EM Education Scholarship Fellowship
|Department chairs||Survey (scaled and open-ended responses)||X||X|
|Education leaders||Survey (scaled and open-ended responses)||X||X|
|Faculty interested in education research||Survey (scaled and open-ended responses)||X||X|
|Directors, EM education fellowship||Structured interview with SWOT analysis|| ||X|
|Graduates, EM education fellowship||Semistructured interview|| ||X|
|Environmental scan||X|| |
|Current fellows, EM education fellowship||Semistructured interview|| ||X|
|Directors, faculty development programs in education scholarship||Semistructured interview|| ||X|
STAKEHOLDER: Department Chairs in EM
Department chairs play an integral role in shaping the future of postgraduate fellowships by hiring graduates and supporting the development of new fellowships. Their hiring practices indirectly drive how academically minded residents, who wish to become faculty, plan their training. It is unclear what criteria chairs use to hire new faculty and how they value education research and scholarship in their departments. This information will be gathered and assessed using an online survey tool through the Association of Academic Chairs of Emergency Medicine listserv. The chairs will be queried about the following issues:
- The current and desired level of scholarly performance of their faculty members in the various academic series, especially within the realm of research (basic science, clinical, education, epidemiology, translational, and health policy).
- Why the department does or does not currently have an education fellowship.
- Their perceived value of a 2-year education scholarship fellowship or an advanced education degree.
- The desired competencies for entering faculty within the various tracks, especially those who focus on education.
The answers to these questions will be critical in determining the feasibility, interest, and value of such a fellowship from the perspective of those who hire faculty. It is possible that some chairs have not considered an education scholarship fellowship and that our query may lead to future implementation in their departments and hiring of fellowship graduates.
STAKEHOLDER: Education leaders
There are a variety of leadership roles in the academic educational arena at the national, academic health center, medical school, and department levels targeting undergraduate, graduate, and continuing medical education learners. Such roles include deans, vice-chairs of education, residency directors, and clerkship directors. These leaders continually shape the field of education research in EM through publications, policy-making, and mentorship of the new generation of education scholars. This consensus conference, which had 175 attendees, represents a groundswell of support among education leaders in trying to improve the quality of education research in EM.
The consensus conference attendees were queried as a large group at the conclusion of the day using audience response system technology. They indicated that the development of fellowship programs that focuses on education scholarship is a worthwhile pursuit. To delve more deeply into their attitudes and opinions, we will use an online survey tool. We plan to query the group on their opinions about the current and desired level of performance of EM faculty in education research, the preparedness of new education scholars to conduct high-quality education scholarship, the possible metrics for success of a 2-year education scholarship fellowship, and whether these proposed metrics could be better achieved through a different mechanism. Ultimately, these education leaders will provide a broad perspective in clarifying whether a fellowship meets the specialty's call for improved education research.
STAKEHOLDER: Faculty With Interest in Education Research
The consensus breakout session identified faculty members with interest in education research as key stakeholders for the needs assessment. Specifically, we plan to survey participants of the Medical Education Research Certificate (MERC) faculty development program hosted by the Council of EM Residency Directors (CORD). This program was originally created by the Association of American Medical Colleges' section on Research in Medical Education to enhance faculty skill development in education research.[26, 27] Participants will be surveyed about whether, in retrospect, they would have pursued a postgraduate education scholarship fellowship, if it were available, or an advanced degree program. Early participants have already identified major obstacles in conducting and publishing education research, which include insufficient training in education research, inadequate protected time from other departmental responsibilities, inadequate funding, lack of mentorship, insufficient networking or a need for great numbers to complete a project, and lack of journal support for education research. Because an education scholarship fellowship will provide a rigorous training curriculum, protected time from other duties, and mentorship in addition to departmental support, we hypothesize that several of the MERC participants would have been interested in such a fellowship.
STAKEHOLDER: Directors of Existing EM Fellowships In Medical Education
Currently several education fellowships exist in EM, which vary in focus, curriculum, and even number of years of training. Some focus primarily on teaching skills, while others specialize in developing education scholars with expertise in research methods. Some also offer or require completion of an advanced postgraduate degree. Because these fellowship directors are familiar with the intricacies of finances, operations, and implementation for a training program in education, they will be able to provide a unique insight into the feasibility, challenges, and opportunities of implementing a formal education scholarship fellowship.
Fellowship directors will be contacted individually for semistructured phone interviews. Although interviews are resource- and time-intensive to conduct, they provide an opportunity for in-depth insight and allow for clarification of individual perspectives. Objective data will first be collected about their fellowships (years in training, funding sources, support services such as statisticians, and access to simulation or other educational venues), key allies (e.g., medical school or hospital leadership), curricular focus and content, and training requirements. Fellowship directors then will be asked about the current and desired level of performance of their fellowship graduates in teaching, learning theories, and education research. Additionally, fellowship directors may have unique ideas that challenge, modify, or fall outside the interview framework that may further inform our process. An open-ended design allows for this expanded data gathering potential.
While focusing on whether a 2-year education scholarship fellowship should be developed, codified, and supported in EM, the needs assessment for the fellowship directors will be framed within a strengths, weaknesses, opportunities, threats (SWOT) analysis approach. The SWOT analysis was originally developed to help implement strategic change in businesses, but has been widely adapted for programmatic evaluation and quality improvement projects. For the fellowship directors, the strengths and weaknesses as well as opportunities and threats for a potentially new education scholarship fellowship will be discussed. Given that a fellowship experience somewhat mirrors an advanced education degree program, a specific discussion item will be to compare the fellowship experience to the attainment of an advanced degree outside of the formal fellowship structure for graduating residents interested in pursuing faculty positions with an academic niche in education scholarship.[30-32] This SWOT analysis allows the fellowship directors to define their challenges and provide an agenda to the education community on what an ideal fellowship structure would entail.
From an operations perspective, the fellowship directors whose programs focus primarily on teaching will also be asked about the feasibility, interest, and value of converting their current fellowships into or creating separate 2-year education scholarship fellowships. We hypothesize that some will continue to favor the retention of a specialized track to develop high-quality educators who do not engage in research to serve as expert educators, while others will welcome the opportunity to offer researcher training.
STAKEHOLDER: Graduates of EM Education Fellowships
Fellowship graduates are in a unique position to comment on their fellowship experience, including both the perceived benefits of spending additional time in focused, specialized training and any potential drawbacks, including opportunity costs and clinical productivity. We are especially interested in gaining their perspectives on the utility of their training curricula as it relates to their current positions and on areas that could be improved to help their transition to full-time academic faculty positions.
Graduates from education fellowships in EM, including those whose fellowships focused primarily on teaching methods, will be assessed through semistructured phone interviews. They will be asked about their reasons for becoming education fellows and whether the fellowships met their expectations, especially with regard to their training in education research. A key question will address why they enrolled in fellowships rather than accepting faculty positions immediately following residency training. Retrospectively, they will be asked to rank and comment on the following training models that would have been acceptable to them, if available: 1) a 1-year education fellowship, which focuses heavily on teaching principles and adult learning theories; 2) a 2-year education scholarship fellowship, which focuses heavily on education research in addition to teaching principles and adult learning theories; 3) a 2-year education scholarship with unfunded enrollment in a graduate degree program in education; and 4) a 2-year education scholarship with funded enrollment in a graduate degree program in education.
Additionally, a quantitative environmental scan will be conducted through a literature search. Using PubMed, ERIC, PscyhInfo, and MedEdPortal, the number of education-related publications per graduate per year postfellowship will be determined. This objective, a baseline productivity metric, will determine the current state of fellowships and could be tracked longitudinally should EM departments build education scholarship fellowships. We hypothesize that there is a need for increased peer-reviewed publications for graduating fellows and education faculty in general and that such a fellowship could not only lead to increased number of publications, but also could affect the quality.
STAKEHOLDER: Current Fellows in EM Education Fellowships
For current fellows, semistructured interviews will be conducted to explore their motivations and goals for becoming education fellows. We will focus on their decision-making strategies by asking how they learned of the availability of education fellowships and what advice they received from the faculty at their residency training institutions about career planning. They will also be asked about whether 2-year education scholarship fellowships would align with their learning priorities and training goals. Similar to the fellowship graduates, they will be surveyed regarding which of the four training models would be acceptable to them, if available.
STAKEHOLDER: Directors of Longitudinal Faculty Development Programs in Medical Education Scholarship Outside of EM
Several academic medical centers have longitudinal faculty development programs, which provide faculty with resources and skills to become scholarly educators. Because these mirror education-based fellowships except that they target faculty rather than postgraduate trainees, best practices from these programs may help inform the needs analysis despite not being specifically targeted to EM. Based on a national survey, most faculty development programs primarily focus on teaching skills. Scholarly dissemination and education research methods were also reported as primary foci in 58 and 47% of the programs surveyed, respectively. To elicit lessons learned from these programs, we will conduct semistructured interviews of several directors of these programs, which primarily focus on scholarly dissemination and education research. These interviews will focus on metrics of success, including the use of national databases that can provide a context or benchmark for interpreting, for example, scholarly productivity.
It is possible that we have omitted some key questions from our needs assessment and that the addition of more viewpoints or other stakeholders could have yielded more complete results. We focused our needs assessment strategy on key stakeholders and will use the gathered information to develop an initial framework. If necessary, additional information sources will be collected to augment the proposed needs analysis.
Breakout group discussions have led to consensus opinion that we must develop a large-scale, mixed-methods training needs assessment focusing on education scholarship fellowships in EM. This framework will help determine the value, feasibility, and structure of the fellowship. Implementation will require the alignment of stakeholder needs with the normative needs expressed by the greater medical education community for higher level education research that can lead to improved learning, and subsequently, patient care outcomes.
The authors acknowledge the contributions of the participants of the consensus building session on May 9, 2012 (in alphabetical order): Samuel Clarke, Wendy Coates, Leigh Evans, Sharon Griswold, Todd Guth, Stanley Hamstra, Phyllis Hendry, Jaime Jordan, Umbar Khan, Michelle Lin, Katie Lind, Daniel Martin, Katie Nacca, Steven Rougas, Dan Rusyniak, Sally Santen, Nancy Searle, Jonathan Sherbino, Mike Smith, John Vozenilek, Elizabeth Walters, Julie Welch, and Lalena Yarris.