Three Steps in Planning a Scholarly Project
Beckman and Cook have presented a three-step approach to developing scholarly projects in medical education:
1. Define the Study Question. The most important part of any study is the research question (or study goal or hypothesis). Even the most rigorous study will fail to have an effect on the field if it does not answer a question that is both important (i.e., the answer would influence practice or future research) and novel (i.e., the answer is unknown).
The best way to establish a question's importance is to employ a strong conceptual framework, which is a theory, approach, or model for how things work. This allows educators at future times and in other settings to adopt and build on a study's results. Conceptual frameworks also assist educators in defining and selecting the study variables and in predicting and interpreting the results. A question's importance also derives from its timeliness or effect on current practice, but in the absence of a conceptual framework such answers will usually have limited influence over time. Cook and others[14, 15] have argued that the most important questions—those with the greatest potential for long-term effect—are those that clarify how things work, for whom, and in what circumstances.
The best way to establish a question's novelty is to rigorously review the literature. Even if no studies are found for the question being considered, the researcher should diligently seek studies that address similar questions or that may offer suggestions for key elements of the framework, training intervention, or assessment. Such relevant work may often be found in other fields such as surgery, nursing, or non–health professions literature. The literature review should culminate in a focused problem statement—a clear summary of what remains unknown. The research question (or objective) then follows naturally from the problem statement: “The evidence thus far shows ____. What remains unknown is _____. Thus, we sought to _____.”
Educators can identify important questions in their daily teaching activities each time they ask, “I wonder if we should do it [this way] or [that way]?” If a literature search for evidence and theory-based principles does not answer the question, this may be the start of a worthwhile project.
2. Identify Study Designs and Methods. Three guiding principles for research study design are:
First, identify the general class of study design most relevant to the research question. This is essential, because the standards for excellence and potential pitfalls vary depending on the design. Studies of educational interventions (e.g., a new course or training approach) will require an experimental study design, studies that evaluate educational assessments (e.g., a new testing approach or tool) will invoke a validity study design, studies seeking to understand current attitudes or learning needs may employ a cross-sectional (e.g., survey) design, and studies attempting to summarize published literature will use a systematic or nonsystematic literature review.[16-20]
Second, realize that all studies have flaws. However, some flaws matter more than others. In addressing study flaws it is helpful to focus on reproducibility and validity threats, i.e., threats to the validity of study interpretations. Ask, “If another researcher at another institution were to conduct a study using different methods but addressing the same question, would he or she be likely to arrive at the same answer?” Answering this question will focus primarily on threats to study validity—issues such as preexisting differences among participants, selection bias, instrumentation problems, and implementation bias. The most important threats, and the specific ways in which they can manifest, vary for different research designs.
Third, be realistic. Carefully consider factors such as the number of potential participants, the logistics of implementing an intervention, and the feasibility of assessing outcomes. The standards of Glassick et al. for scholarship are useful in judging the rigor of your scholarly project, however small. Estimate the needed sample size using a credible approximation of the anticipated effect. It is also important that they ensure that institutional review board approval is obtained, as learners are a vulnerable study population.
3. Select Outcomes. Nearly all research studies require some type of outcome. To avoid prematurely settling on an inferior outcome measure, we suggest first identifying the broad class of outcomes (e.g. “knowledge” or “skill” [in a controlled setting] or “behaviors” [at the bedside]), then considering different measures available for a given outcome, and then selecting a specific instrument. Ten tips for success in education research have been published previously and are summarized in Table 1.
Table 1. Tips for Getting Started in Education Research
|Get some training||Training can occur locally (local workshops), at national meetings (in particular at education-oriented conferences), and through formal degrees (e.g., masters-level training).|
|Find a mentor||The importance of a mentor cannot be overemphasized. If no single person at your institution possesses the needed skills or sufficient time, consider working with multiple mentors or looking outside your institution. |
|Ask important questions||See main text.|
|Start small and grow||Powerhouse investigators did not start off that way; they started with small projects and grew into their current position. A poster at a national meeting may be the first step in a series of progressively insightful studies.|
|Aim high||It is okay to start small, but don't settle for mediocrity. Whatever you do, do it well. The standards of Glassick et al. for scholarship are useful in judging the rigor of your scholarly project, however small.|
|There's no such thing as a perfect study||If you wait for the perfect study (or perfect opportunity), you'll be waiting a very long time. If the question is important and novel, any answer (even if incomplete) will be better than none.|
|Where will you find the time?||Be realistic in budgeting time, money, and other resources. |
|Remember ethical issues||Learners are a vulnerable population. Many education journals now require institutional review board approval for all studies involving human subjects.|
|Network||Build relationships locally and nationally through involvement in committees and professional organizations. Approach new contacts with a specific purpose (e.g., a question).|
|This is hard work||Education research is fun and rewarding, but it is not easy.|