Changes in anatomy instruction and USMLE performance: Empirical evidence on the absence of a relationship

Abstract

Anatomy instruction has evolved over the past two decades as many medical schools have undergone various types of curricular reform. To provide empirical evidence about whether or not curricular changes impact the acquisition and retention of anatomy knowledge, this study investigated the effect of variation in gross anatomy course hours, curricular approach (stand-alone versus integrated), and laboratory experience (dissection versus dissection and prosection) on USMLE Steps 1 and 2 Clinical Knowledge (CK) scores. Gross anatomy course directors at 54 United States schools provided information about their gross anatomy courses via an online survey (response rate of 42%). Survey responses were matched with USMLE scores for 6,411 examinees entering LCME-accredited schools in 2007 and taking Step 1 for the first time in 2009. Regression analyses were conducted to examine relationships between gross anatomy instructional characteristics and USMLE performance. Step 1 total scores, Step 1 gross anatomy sub-scores, and Step 2 CK scores were unrelated to instructional hours, controlling for MCAT scores. Examinees from schools with integrated curricula scored slightly lower on Steps 1 and 2 CK than those from stand-alone courses (effect sizes of 2.1 and 1.9 on score scales with SDs of 22 and 20, respectively). Examinees with dissection and prosection experience performed slightly better on Step 2 CK than examinees in courses with dissection only laboratories (effect size of 1.2). Results suggest variation in course hours is unrelated to performance on Steps 1 and 2 CK. Although differences were observed in relation to curricular approach and laboratory experience, effect sizes were small. Anat Sci Educ 6: 3–10. © 2012 American Association of Anatomists.

INTRODUCTION

Training in the anatomical sciences provides a foundation upon which medical students' subsequent acquisition of clinical knowledge and skills can be built (Sugand et al.,2010). An understanding of anatomy also helps to ensure safe and effective patient care (Older,2004; Ahmed et al.,2010, 2011). Therefore, appropriate and effective instruction in anatomy during medical school is important and can lead to fewer medical errors and increased patient safety and satisfaction (Ellis,2002; Waterston and Stewart,2005; Bagley et al.,2011)

Anatomy instruction has evolved over the past two decades as many medical schools have undergone various types of curricular reform (Drake et al.,2009; Sugand et al.,2010). Such changes have included a decrease in the number of course hours devoted to anatomy (Drake at al.,2009; Craig et al.,2010), a shortening of the time available for cadaver dissection (McLachlan,2004), and a shift toward integrated curricula in which anatomy is presented with other disciplines in an organ systems approach (Jacobson et al.,2009; Klement at al.,2011; Johnson et al.,2012). These curricular changes have raised questions about their potentially negative impact on medical students' knowledge of anatomy. In fact, a few studies have indicated that graduating medical students, residents and newly qualified doctors (Monkhouse,1992; Prince et al.,2005; Ahmed et al.,2010, 2011) feel poorly prepared for everyday practice because of their perceived lack of anatomical knowledge (Gupta et al.,2008; Bagley et al.,2011; Hall et al.,2013). Additionally, insufficient anatomical knowledge may negatively influence medical graduates in career choice and professional advancement (Smith and Mathias, 2011; Hall et al.,2013).

Although a definite effect of gross anatomy instructional hours on medical students' anatomical knowledge has not been demonstrated, several studies suggest a relationship. For example, studies have shown a positive relationship between course hours and test performance, where students who received more hours of anatomy instruction outperformed students who received fewer hours of anatomy instruction (Verhoeven et al.,2002; Bergman et al.,2008). Additionally, when the level of anatomical knowledge was compared among students from different medical schools, students with longer anatomy training and those with multiple exposures to anatomy had better retention of anatomical knowledge (Blunt and Blizard,1975; Prince et al.,2003; Bergman et al.,2008). Yet, few studies have explored the impact of course hours on anatomy knowledge at the national level.

With a decline in total course hours, time spent dissecting cadavers in the gross anatomy laboratory also has decreased. To correct this situation, medical schools increased the development of anatomical resources through technological advances and innovations in multimedia (Trelease,2008; Alexander et al.,2009; McNulty et al.,2009; Johnson et al.,2012; Mayfield et al.,2013). As a result, many medical students currently receive anatomy instruction supplemented by prosected or plastinated specimens (Yeager,1996; Cornwall,2011), plastic and clay models (Oh et al.,2009), clinical images, that is, MRI, CT, and ultrasound (Gregory et al.,2009; Lufler et al.,2010; Brown et al.,2012;), and/or computer-based multimedia training (Reeves et al.,2004; McNulty et al.,2009). Proponents of dissection have argued that dissection aids students in understanding three-dimensional structures (Lachman and Pawlina,2006; Collins,2008), develops students' professionalism, teamwork skills, and manual dexterity (Pawlina and Lachman,2004; McLachlan and Patten,2006; Pawlina et al.,2006), and increases students' overall anatomical knowledge (Biasutto et al.,2006; Winkelmann,2007; Sugand et al.,2010). Despite these claims, some have argued that instruction via dissection may be unnecessary in undergraduate medical education and that it may be best suited for postgraduate training (Wong and Stewart,2004; Collins2008).

Finally, many medical schools have implemented integrated curricula where basic science is learned in the context of its clinical application (Louw et al.,2009; Klement et al.2011). One study compared anatomy outcomes in a traditional discipline-based course and an integrated system-based model where teaching of basic medical sciences was concurrent with clinical training (McKeown et al.,2003). Students taught in the traditional discipline-based curriculum scored better than students in the system-based curriculum, especially in the area of surface anatomy (McKeown et al.,2003). Another study demonstrated an increase in anatomy examination scores at a single institution that moved to a self-directed learning approach (Findlater et al., 2012). There are also studies that revealed no difference in anatomy test scores between students receiving traditional and integrated problem-based learning curricula (Prince et al.,2000; Custers and Cate,2002; Bergman, et al.,2008). Thus, there are no clear conclusions regarding the effect of integrated curricula on the acquisition and retention of anatomical knowledge (Bergman, et al.,2011).

With respect to knowledge retention, there are limited data that show a positive relationship between long-term retention in distributed versus mass learning in medical education (Kerfoot et al.,2007; Raman et al.,2010). Mass learning in short intensive didactic blocks is associated with progressively increasing cognitive fatigue (Sweller,1998) that leads to increasing cognitive load that can reduce performance on knowledge transfer tasks (Persson et al.,2007; Raman et al.,2010). Furthermore, other studies suggest that retention of the basic science information acquired in the pre-clinical years of the traditional curriculum is lost during final clinical years (Kennedy et al.,1981; Norman,2000; Ling et al.,2008; Custers and Ten Cate,2011; Lazarus et al.,2012).

Given the lack of empirical evidence related to the effect of changes in anatomy instruction on student acquisition and retention of anatomy knowledge, particularly at the national level, it seemed worthwhile to examine the impact of current curricular approaches on scores on national assessments of basic and clinical science knowledge.

The United States Medical Licensing Examination^® (USMLE^®) is a three-step examination used in granting initial licensure to practice medicine in the United States. The first examination in the sequence, Step 1, assesses an understanding and application of basic science concepts, while the second examination in the sequence, Step 2 Clinical Knowledge (CK), assesses the application of medical knowledge, skills, and understanding of clinical science essential for the provision of patient care under supervision. Recent changes in the Step 1 examination emphasize the use of patient-based test items that require examinees to apply their basic science knowledge to particular case scenarios (Swanson et al.,1991; Ling et al.,2008).

To understand the relationships between anatomy instruction and USMLE performance, this study investigated the effect of variation in course hours, curriculum type, and laboratory experience on Steps 1 and 2 CK scores. The purpose of this study was to provide empirical evidence about whether or not curricular changes related to anatomy instruction impact the acquisition and retention of anatomy knowledge.

METHODS

In 2009, an online survey was administered to gross anatomy course/program directors to obtain descriptive information about gross anatomy instruction at their medical schools. Several different survey administration approaches were used. Information about and a link to participate in the survey were posted to the American Association of Anatomists (AAA) and the American Association of Clinical Anatomists (AACA) electronic mailing lists. In addition, members of the Association of Anatomy, Cell Biology, and Neurobiology Chairpersons (AACBNC) and anatomy course/program directors identified by the AAA were invited individually to participate in the survey. Out of 130 allopathic and 25 osteopathic medical schools, 65 gross anatomy course/program directors responded to the survey, resulting in an overall survey response rate of 42%. See Drake et al. (2009) for a summary of survey design and results.

Survey content included questions related to gross anatomy instructional characteristics such as total course hours (excluding examination hours), curricular approach, and laboratory experience. With respect to curricular approach, respondents were asked to indicate if their gross anatomy course was best described as a stand-alone course or part of an integrated curriculum. In terms of laboratory experience, respondents were asked to indicate if their laboratories consisted of dissection only, dissection and prosection, or prosection only. Only two course/program directors indicated that their course included prosection only, so these two schools were dropped from analysis.

Survey responses from 54 Liaison Committee on Medical Education (LCME)-accredited medical schools were matched with USMLE scores for 6,411 examinees entering medical school in 2007 and taking Step 1 for the first time in 2009. The USMLE scores included Step 1 total test scores, Step 1 gross anatomy sub-scores, and Step 2 CK total test scores. At the time the data were matched, 629 of the 6,411 examinees did not have Step 2 CK scores (roughly 10%).

With respect to USMLE scores, this study was considered exempt from Institutional Review Board inspection because data were collected as part of routine operational activities, all records were de-identified, and examinees who requested that their data be withheld from research (less than 1% of examinees) were excluded.

Ordinary least squares (OLS) regression analyses were conducted to examine the relationships between gross anatomy instructional characteristics and USMLE performance. Three separate OLS regression models were estimated with three different dependent variables: Step 1 total test scores, Step 1 gross anatomy sub-scores, and Step 2 CK total test scores. In all three models, the independent variables included Medical College Admission Test^® (MCAT^®) total scores, total gross anatomy course hours, a dichotomous variable representing a laboratory experience including dissection and prosection (versus dissection only), a dichotomous variable representing a gross anatomy course that is part of an integrated curriculum (versus a stand-alone course), and an interaction term between MCAT total scores and the dichotomous variable indicating an integrated course. MCAT total scores were used to control for examinees' knowledge and skills prior to entry into medical school and were calculated as the sum of three MCAT scores: Biological Sciences, Physical Sciences, and Verbal Reasoning. Scores on the MCAT Writing Sample were not included in the sum because preliminary analyses indicated that they were unrelated to USMLE performance.

The interaction term was included to examine if the relationship between anatomy instructional approach (stand-alone course versus integrated curriculum) and USMLE scores were consistent across MCAT score ranges. For example, it may be that students with higher/lower MCAT scores performed better/worse on Step 2 CK if gross anatomy was a stand-alone course. MCAT scores and the dichotomous variable representing curriculum type were grand-mean centered prior to creating the interaction term. The interaction term and the centered component terms then were included together in the models. This was done so that the constant terms were interpreted as the average Step scores at the average MCAT score and average percentage of schools with integrated curricula. The centering of the component terms prior to creating the interaction term and the inclusion of all three terms in the models also aids interpretation of the coefficients for the component terms. See Aiken and West (1991) for more detailed information on the use and interpretation of interaction terms in regression analysis.

To augment the primary analyses, a supplemental set of analyses was conducted using OLS regression. Like the primary analyses, three models were estimated with three different dependent variables: Step 1 total test scores, Step 1 gross anatomy sub-scores, and Step 2 CK total test scores. All of the independent variables were identical to those included in the primary analyses, except the predictor for total course hours was replaced by the three variables used in calculating total course hours: number of lecture, laboratory, and small group hours. These supplementary analyses were undertaken to examine if the use of total course hours in the primary analyses obscured significant relationships between USMLE and more specific indicators of time spent in gross anatomy instruction. For example, if a significant positive relationship exists between the number of lecture hours and Step 1 scores, and a significant negative relationship between the number of small group hours and Step 1 scores also exits, these two relationships could have cancelled each other out in the primary analyses where only the total number of course hours was examined. Thus, it is possible to have a non-significant finding for total course hours, but still see a significant effect for the more specific indicators of time spent in gross anatomy instruction.

Potential violations of the assumptions of OLS regression analysis were investigated within models. All of the Variance Inflation Factors were less than 2 and the bivariate correlation coefficients between the independent variables were all less than 0.4 indicating that multicollinearity is unlikely to be an issue. An examination of residuals suggested that the errors were approximately normally distributed and the Cook's D influence statistics were all significantly below 1 demonstrating a desired lack of influential outlier observations. Thus, the basic assumptions for OLS regression analysis appear to be met. Statistical analysis was performed using SPSS (PASW) Statistics, version 18.0 (SPSS Inc., Chicago, IL).

RESULTS

As discussed in greater detail below, results indicated that the gross anatomy instructional characteristics examined were essentially unrelated to Steps 1 and 2 CK scores. No practically meaningful relationships were found, suggesting that performance on Steps 1 and 2 CK is unrelated to variation in gross anatomy course hours, curriculum type and laboratory experience.

Table 1 presents summary statistics for the MCAT and USMLE scores used in the analyses. As shown, examinees had a mean Step 1 score of 224.6 with a standard deviation of 21.7, a mean Step 1 gross anatomy sub-score of 220.7 with a standard deviation of 22.1, and a mean Step 2 CK score of 234.8 with a standard deviation of 20.1. The mean MCAT score was 29.6 with a standard deviation of 4.8. These summary statistics were quite similar to analogous summary statistics for the total group of examinees who entered LCME-accredited medical schools in 2007 and took Step 1 for the first time in 2009. The study group performed better on Step 1 than the total group of examinees from LCME-accredited medical schools who took Step 1 for the first time in 2009 regardless of when they entered medical school or what school they attended, with the total group obtaining a mean Step 1 score of 223.5 with a standard deviation of 23.1 and a mean Step 1 gross anatomy sub-score of 219.8 with a standard deviation of 23.3. This was an expected result because restricting the study sample to reflect the typical time period between entry into medical school and taking Step 1 excluded students that ran into academic problems during the first 2 years of medical school, delaying their first Step 1 attempt. MCAT and Step 2 CK scores for the study and total groups were similar.

Table 1. Summary Statistics for Examinee-level Score Variables

Table 2 shows summary statistics for school-level gross anatomy instruction variables. On average, gross anatomy courses included 151.8 total course hours, excluding examination hours. Approximately 40% of schools included both dissection and prosection as part of their laboratory experience and approximately 30% of schools indicated that their gross anatomy course was part of an integrated curriculum.

Table 2. Summary Statistics for School-level Gross Anatomy Instruction Variables

Figures 1, 2, 3 illustrate observed relationships between MCAT scores and USMLE performance by curriculum type with the examinee as the unit of analysis. More specifically, Figure 1 plots the relationship between MCAT scores and Step 1 total test scores by curriculum type; Figure 2 plots the relationship between MCAT scores and Step 1 gross anatomy sub-scores by curriculum type; and Figure 3 plots the relationship between MCAT scores and Step 2 CK total test scores by curriculum type. For all three USMLE scores, it appears that examinees from schools with stand-alone gross anatomy courses slightly outperform examinees from schools with integrated curricula (differences in intercepts), controlling for differences in MCAT scores. Furthermore, a unit increase in MCAT scores appears to be associated with a greater gain in USMLE scores for examinees from schools with integrated curricula (difference in slopes). However, these differences in intercepts and slopes are quite small and may be practically meaningless. The results of the regression analyses help to understand these relationships more fully.

Figure 1. Observed relationship between MCAT score and USMLE Step 1 total test score by curriculum type.

Figure 2. Observed relationship between MCAT score and USMLE Step 1 gross anatomy score by curriculum type.

Figure 3. Observed relationship between MCAT score and USMLE Step 2 clinical knowledge total test score by curriculum type.

Table 3 provides the results of the primary regression analyses predicting Step 1 total test scores, Step 1 gross anatomy sub-scores, and Step 2 CK total test scores. For Step 1 total test scores, the independent variables explained 18% of the variance. This value was lower for Step 1 gross anatomy sub-scores (8%), in part reflecting the lower reliability of this sub-score. The independent variables explained 12% of the variance in Step 2 CK scores.

Table 3. Regression Estimates of Step 1 Total Test Scores, Step 1 Gross Anatomy Sub-scores, and Step 2 Clinical Knowledge (CK) Total Test Scores

As reported previously, there was a statistically significant positive relationship between MCAT scores and USMLE performance, with increases in MCAT scores resulting in increases in USMLE scores (Basco et al.,2002; Donnon et al.,2007; Gohara et al.,2011). For example, a one-point increase in MCAT score is associated with an expected increase of 1.88 on Step 1 total test score. Controlling for MCAT scores, total course hours were unrelated to USMLE performance. In other words, the number of hours in gross anatomy instruction that an examinee received had no effect on how well they performed on Step 1 or Step 2 CK. Also, examinees from medical schools with laboratory experiences including dissection and prosection fared no differently on Step 1 than examinees from schools with dissection only laboratories. However, examinees that were exposed to both dissection and prosection outperformed examinees that experienced dissection only on Step 2 CK by 1.21 score points. Although this effect is statistically significant, it is unlikely to be practically important since it is only 1/16 of a Step 2 CK score standard deviation.

In terms of curriculum type, results indicated that there are statistically significant negative relationships between (1) integrated course and Step 1 total test scores and (2) integrated course and Step 2 CK total test scores. On average, examinees from schools with integrated gross anatomy instruction perform approximately 2 points lower on Step 1 and Step 2 CK when compared with examinees from schools with stand-alone gross anatomy courses. In both cases, the effect is about 1/10 of a USMLE score standard deviation. Thus, while these effects are statistically significant, they are small and may lack practical importance. The interaction term between MCAT score and integrated course was not significantly related to USMLE performance, demonstrating that the effect of anatomy instructional approach on USMLE scores is consistent across the MCAT score range.

Table 4 provides the results of the supplementary OLS regression analyses. Results revealed the same general pattern of findings in terms of the significance, magnitude, and direction of the coefficients as in the primary analyses. Lecture hours and laboratory hours were unrelated to USMLE performance. Small group hours were unrelated to Step 1 performance; however small group hours were statistically significantly and negatively related to Step 2 CK performance, with a 0.042 decrease in Step 2 CK total test scores expected for every 1 hour increase in small group hours. Thus, an increase of 10 small group hours would have an expected effect of less than one point on the Step 2 CK score scale.

Table 4. Regression Estimates of Step 1 Total Test Scores, Step 1 Gross Anatomy Sub-scores, and Step 2 Clinical Knowledge (CK) Total Test Scores (Supplementary Analyses)

DISCUSSION

This study provides empirical evidence related to the effect of changes in anatomy instruction on student acquisition and retention of anatomy knowledge. The primary finding is that the gross anatomy instructional characteristics examined were essentially unrelated to performance on Step 1 and Step 2 CK. In other words, how well examinees perform on Step 1 and Step 2 CK is not impacted by variations in course hours, curricular approach or laboratory experience. This is important, given that these areas of anatomy instruction have undergone changes in recent years, and theoretical debates about the impact of these changes on student learning continue. No empirical research at the national level testing the relationships between approaches to anatomy instruction and subsequent anatomy knowledge existed prior to this study.

The observed lack of association between the instructional approach used for anatomy and USMLE performance should not be interpreted to mean that it does not matter how anatomy is taught. For example, more hours of anatomy instruction may not impact performance on Steps 1 or 2 CK, but longer exposure to anatomy instruction may better prepare students for clinical practice. Similarly, dissection only laboratories may not yield higher Step 1 scores, but they may lessen medical errors and increase patient satisfaction in everyday practice. In addition, use of small-group sessions may demonstrate the clinical relevance of anatomy to students, increasing student interest in and satisfaction with basic science instruction.

This study has a number of limitations. First, while the study sample was fairly large, it only included students from 54 medical schools who entered medical school in 2007 and took Step 1 for the first time in 2009. Effect sizes for missing schools and for other cohorts may vary from those observed in the study. Second, the study did not investigate school-to-school variation in the nature of the relationships among gross anatomy instructional approaches, MCAT scores, and USMLE performance. Differences in medical schools' selectivity, educational environments, and curricular approaches at a more granular level may affect the nature and the strength of the relationships between gross anatomy instructional characteristics and USMLE performance. Additional multilevel analyses exploring the impact of this variation are planned. Third, the knowledge, skills, and attitudes learned in gross anatomy instruction are not necessarily well measured by USMLE and the relationship between gross anatomy instructional characteristics and USMLE performance may be masked (and attenuated) because of the importance of licensing examination scores from students' perspectives, due to their use in making intramural promotion/graduation decisions and screening applicants for resident training: students may make up for perceived shortcomings in anatomy instruction by studying. Fourth, neither the current study nor previous studies, which indicate that students feel their training in anatomy may be insufficient (AMA,2006; Mitchell and Batty,2009), demonstrate a direct impact on clinical performance or patient safety. Finally, it should be mentioned that there may also be a single significant equalizing factor that has nothing to do with curricular course hours, curricular approach or laboratory experience that has not been factored into this report. Most medical schools in the United States provide a study break of 4 to 6 weeks at the end of the second year for their students to prepare for the Step 1 Examination. It is unclear what type of general equalizing effect this has on USMLE scores. While this study provides preliminary information regarding the effect of alternate approaches to anatomy instruction on acquisition of anatomy knowledge, more research is needed to fully explore these relationships.

NOTES ON CONTRIBUTORS

MONICA M. CUDDY, M.A., is a measurement scientist at the National Board of Medical Examiners (NBME) in Philadelphia, Pennsylvania. Her research interest is in using multilevel modeling techniques to explore school- and student-level effects for important outcomes within medical education.

DAVID B. SWANSON, Ph.D., is the Vice President for Program Development and Special Projects in the Assessment Programs unit at the National Board of Medical Examiners (NBME) in Philadelphia, Pennsylvania. His research interests are in assessment of medical decision making with multiple-choice tests clinical simulations; assessment of clinical skills with standardized patients; patterns of performance on admissions, licensure and specialty certification examinations; issues in computer-based testing. In 2011, he was awarded the Richard Farrow Gold Medal by the Association for the Study of Medical Education in the United Kingdom for outstanding contributions to medical education.

RICHARD L. DRAKE, Ph.D., is a professor in the Department of Surgery and Director of Anatomy at the Cleveland Clinic Lerner College of Medicine of Case Western Reserve University, Cleveland, Ohio. He teaches gross anatomy, embryology, and neuroanatomy to first-year and second-year medical students.

WOJCIECH PAWLINA, M.D., is a professor of anatomy and medical education and chair of the Department of Anatomy at Mayo Medical School, College of Medicine, Mayo Clinic, Rochester, Minnesota. He serves as Assistant Dean for Curriculum Development and Innovation and teaches gross anatomy and histology to first-year medical students, residents, and fellows.