SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References

Medical Education 2011: 45: 818–826

Context  The Association of American Medical Colleges’ Institute for Improving Medical Education’s report entitled ‘Effective Use of Educational Technology’ called on researchers to study the effectiveness of multimedia design principles. These principles were empirically shown to result in superior learning when used with college students in laboratory studies, but have not been studied with undergraduate medical students as participants.

Methods  A pre-test/post-test control group design was used, in which the traditional-learning group received a lecture on shock using traditionally designed slides and the modified-design group received the same lecture using slides modified in accord with Mayer’s principles of multimedia design. Participants included Year 3 medical students at a private, midwestern medical school progressing through their surgery clerkship during the academic year 2009–2010. The medical school divides students into four groups; each group attends the surgery clerkship during one of the four quarters of the academic year. Students in the second and third quarters served as the modified-design group (n = 91) and students in the fourth-quarter clerkship served as the traditional-design group (n = 39).

Results  Both student cohorts had similar levels of pre-lecture knowledge. Both groups showed significant improvements in retention (p < 0.0001), transfer (p < 0.05) and total scores (p < 0.0001) between the pre- and post-tests. Repeated-measures anova analysis showed statistically significant greater improvements in retention (F = 10.2, p = 0.0016) and total scores (F = 7.13, p = 0.0081) for those students instructed using principles of multimedia design compared with those instructed using the traditional design.

Conclusions  Multimedia design principles are easy to implement and result in improved short-term retention among medical students, but empirical research is still needed to determine how these principles affect transfer of learning. Further research on applying the principles of multimedia design to medical education is needed to verify the impact it has on the long-term learning of medical students, as well as its impact on other forms of multimedia instructional programmes used in the education of medical students.


Introduction

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References

In 2007, the Association of American Medical Colleges’ Institute for Improving Medical Education (AAMC-IME) issued a report entitled ‘Effective Use of Educational Technology in Medical Education’,1 which highlighted the importance of multimedia learning and suggested that medical educators should utilise multimedia learning principles when designing instructional presentations for medical students. A critical point emphasised by the IME’s report was that faculty development programmes should encourage faculty members to understand how people learn best from words and pictures, based on the theoretical underpinnings of multimedia learning theory, and, accordingly, to understand how to design effective multimedia instructional messages.

Multimedia learning refers to learning from words and pictures. Words can be seen or heard because they can be communicated in print or through spoken narration, respectively. Pictures can be presented in static forms (illustrations, photographs, diagrams, charts and maps) or in dynamic forms (animation and video). PowerPoint™ (ppt™) software is one of the multimedia tools most commonly used by medical faculty members when lecturing medical students, residents and peers. Through its default function, ppt™ structures content into a single-word or short-phrase headline followed by a bulleted list of relevant factual points that can be enhanced with charts, pictures, figures or animations. ppt™ is a popular multimedia resource because it is readily available, convenient to use, can reduce complicated messages to simple bullet-point items and can function as a script to keep the topic and presenter organised.2 Although ppt™ presentations can be configured in any number of ways, the vast majority of faculty staff consistently rely on the bullet-point format, propagating a state of boredom and fatigue among learners, who also complain of information overload and confusion, a phenomenon described as ‘death by PowerPoint’.3,4 In fact, some researchers have shown that learner outcomes improve when ppt™ presentations are not used in the default mode and certain design rules are closely observed.5 These suggested design rules include: changing the ppt™ default headline from a short-phrase headline to a short sentence that states the main assertion of the slide; replacing the bulleted list with visual representations of the evidence; reducing the number of words on a slide, and, finally, ending presentations with conclusion slides. Alley and his team described significant improvements in examination scores of science and engineering college students who received lectures using this new approach to ppt™, which they termed ‘assertion-evidence structure design’.5 This finding suggests that multimedia design changes can lead to improvements in learning by students in actual classrooms.

Most clinical faculty staff involved in medical education lack formal or structured training in the science of teaching prior to commencing their duties as educators. Given this lack of structured training of faculty staff, it is more likely than not that their well-intentioned efforts may not be as effective as they might be in the education of their own students.6 Successful faculty development programmes should provide educators with insight into the cognitive processes involved in how humans learn and understanding of how to design instructional materials that enhance the learning process for students.6,7 The emphasis on understanding the cognitive process of multimedia learning, rather than simply following certain design rules, is key. This process is especially relevant in medical education as a significant proportion of medical learning occurs through a variety of multimedia formats, such as didactic lectures, small-group sessions and web-based modules, among other applications.

The principles for designing effective multimedia presentations are based on cognitive theory of multimedia learning.8 Cognitive theory of multimedia learning and cognitive load theory, from which the former is derived, provide a well-researched account of how people learn from words and pictures.9 These theories suggest there are two distinct channels in the human information processing system, of which one processes information presented in a visual or pictorial format and the other processes information presented in an auditory or verbal format. This is also known as the dual-channel theory of multimedia learning.10,11 Each channel has predetermined limited capacity to process incoming information.12,13 The cognitive process of learning progresses through distinctive pathways of the human memory system, as noted in the diagrammatic representation in Fig. 1. Sensory memory is exposed to an unlimited amount of incoming information presented as verbal and pictorial stimuli, but only a limited number of the incoming stimuli can be processed through either channel at any given time. The selected information progresses through the system to reach the working memory area in the nervous system. There, the information is organised into distinct cognitive representations. This phase is a rate-limiting step in the system and consumes a significant amount of time, until an appropriate cognitive representation emerges that faithfully represents elements of the selected verbal and pictorial stimuli. Then, the newly constructed cognitive representations can be integrated with relevant prior knowledge activated from long-term memory, and ultimately stored in the long-term memory of learners.14,15

image

Figure 1.  A diagrammatic representation of dual-channel theory for multimedia learning, adopted with permission from the American Psychological Association21

Download figure to PowerPoint

How to design multimedia presentations based on the principles of cognitive theory of multimedia learning has been extensively studied and publicised by Mayer.16,17 These principles are displayed in Table 1. To guide the process of learning, Mayer emphasised that educators need to design the instructional message in a manner that will facilitate students’ cognitive learning processes. As such, educators should make the objectives for the educational activity clear to students ahead of time and students should be encouraged to familiarise themselves with the topic of interest prior to attending the lecture. External distracters, both verbal and pictorial, should be eliminated when designing the slides. Educators should match the information load they intend to present to the allotted time for the educational activity. That information load should not exceed the learner’s cognitive capacity to form appropriate mental frames of the topic at hand. Educators should take the initiative to establish what students already know about the topic from other disciplines and use that information to build on and expand this knowledge. This should facilitate the integration of the newly presented material with the previously stored information. The positive effects of the design changes proposed earlier by Alley and his team5 can be easily explained by their adherence to the cognitive processes involved when students are learning.

Table 1.   Mayer’s principles for designing effective instructional multimedia materials
  1. Adapted from Mayer22

Eliminate external distracters
 Coherence principleExclude extraneous words, pictures and sounds
 Signalling principleHighlight essential material
 Redundancy principleDo not add on-screen text to narrated animation
 Spatial contiguityPlace printed words next to corresponding graphs
 Temporal contiguityPlace corresponding narration and animation at the same time
Encourage learners to establish ‘mental frames’ for the material
 Segmenting principlePresent animation in learner-paced segments
 Modality principlePresent words as narration instead of printed text
 Pre-training principlePrepare/read ahead of time
Facilitate integration of new material with prior established knowledge
 Multimedia principlePresent words and pictures rather than words alone
 Personalisation principleEmploy conversational style instead of formal dry style to present words

Educators and education research experts have suggested that applying multimedia learning principles when designing educational materials for students facilitates deep learning, also known as meaningful learning. 17 Learning can be measured using retention tests and transfer tests. Retention tests measure how much of the presented material is remembered, whereas transfer tests measure how well the learner can apply what was learned to new situations.

Most of the published studies that assessed the effects of these multimedia design principles involved college-level students as participants, used very short lessons (i.e. < 3 minutes) and were conducted in laboratories under very controlled conditions.17 The effects of multimedia design principles have not been studied with medical students in a real classroom environment in which lectures last for 45–60 minutes; therefore, it is unknown whether the findings are generalisable to medical education. We believe that providing evidence that those findings are generalisable to medical education would encourage more clinical faculty staff to utilise multimedia principles to improve their educational programmes. Ultimately, we believe that incorporating the principles of multimedia design into faculty development programmes will decrease faculty members’ dependence on bulleted slide design and create, on their part, a need to know more about the cognitive processes of how students learn and the best practices by which to improve the learning experiences of students.

The purpose of this study was to test the applicability of the proposed multimedia learning principles to presentations given to Year 3 medical students. More specifically, this study was designed to address the following research question: does multimedia instruction that applies the principles of multimedia learning result in increased retention and transfer by medical students? We hypothesised that ppt™ lecture slides that were designed based on multimedia principles would result in an increase in short-term retention and transfer of learning in medical students compared with instruction delivered using the traditional design.

Methods

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References

Participants and materials

Participants included three groups of Year 3 medical school students at a private, midwestern medical school progressing through their surgery clerkship during the academic year 2009–2010. The medical school divides students into four groups; each group attends the surgery clerkship during one of the four quarters of the academic year. Each week the surgery clerks attend a series of core curriculum lectures given by surgical faculty members. The required weekly lecture series served as the venue for this research. For the purposes of this study, the lecture that addressed shock was selected because it is the clinical authors’ area of expertise. This lecture is given once each quarter. Students in the second- (Q2) and third-quarter (Q3) clerkships served as the modified-design group (n = 91) and students in the fourth-quarter (Q4) clerkship served as the traditional-design group (n = 39). All students were in good academic standing. To establish that both groups were similar academically, we compared the mean scores achieved by the two cohorts of students on the National Board of Medical Examiners (NBME) Surgery Subject Examination, better known as the ‘Surgery Shelf Exam’; we also compared the mean Surgery Clerkship Composite scores achieved by the two groups. The latter tool was developed and validated at our programme and was shown to have positive correlations with other clerkship test scores and with scores on the US Medical Licensing Examination Step 2, Clinical Knowledge.18 This score is computed from three weighted variables, consisting of: written assessments that include the locally developed mid-term examination and the NBME Shelf Test; clinical and skills performance assessments including an objective standardised clinical examination (OSCE) and global performance scores given by faculty members, residents and nurses, and, lastly, the professional behaviour and accountability score. It provides an accurate global assessment of each student.

Procedure and design

Two versions of the same lecture containing identical factual information were delivered. The traditional lecture used ppt™ slides that had been developed previously and used for the lecture on shock given during the previous academic year (2008–2009). The clinical author (NI) and the senior education specialist (DD) redesigned the traditional lecture using principles of multimedia learning as described by Mayer.16 Examples of the modified slides are shown in Fig. 2.

image

Figure 2.  Examples of slide modifications using Mayer’s design principles

Download figure to PowerPoint

Both lecture designs were given by the same faculty member (NI) to control for lecturer style differences. The factual content of the two lectures was identical, although the design of the slides differed. To help control for lecturer bias, lectures were observed by an education specialist (SS), who is also the associate clerkship director and holds a Masters degree in health professions education. She verified that all clerkship objectives for the designated topic were met, and that the lecturer used a consistent and comparable presentation delivery style, regardless of the multimedia design format. She also verified that the presentations took the same length of time and that voiced invitations by the instructor were given consistently every 10–15 minutes at each session to encourage questions. The slides in both sessions served as a ‘storyboard’ to keep the lectures consistent in their sequence of content. We also tracked the global evaluations, perceptions and comments given by the medical students to the presenter after each lecture.

Institutional review board approval and informed consents were obtained from all participating students in the academic year 2009–2010. A pre-test/post-test control group research procedure was used. The clerkship learning objectives relevant to the topic of shock were used as a blueprint to develop lecture content and test items for the traditional and modified groups. The lecture objectives were: definition of shock; mechanisms of cellular dysfunction in shock; aetiology and classification of shock, and principles of diagnosis and management of shock. The lecture was 50 minutes long and the pre- and post-tests lasted 10 minutes each. The pre-test was administered at the start of the lecture and the post-test was administered 1 hour after the completion of the lecture.

The pre- and post-test questions were designed using the question format published in Mayer’s studies on the effects of multimedia learning.17 A total of 10 open-ended questions were developed. The open-ended question format was chosen because it does not cue learners, as do multiple-choice questions. One point, half a point or no point was given for each of the questions when scores were tallied. Five questions were written to test retention of information and five to test transfer of knowledge. Retention questions asked learners to define some phenomenon presented in the lecture (e.g. ‘Define shock’ or ‘Write the oxygen delivery equation and highlight the relevant clinical variables’). Transfer of learning questions asked learners to write possible solutions to new patient problems relevant to shock. A clinical vignette format was used. Participants were asked to identify the disease process of a patient in shock based upon the clinical data provided. Participants were then asked a series of questions regarding the management of the patient through a changing clinical scenario. The pre- and post-test questions were identical. Content validity of test items was established through revision and review by two board-certified trauma and critical care surgeons (NI and MSh), both of whom were active in undergraduate medical education, were familiar with the lecture topic and had lectured on it for several years and to different levels of learners, including medical students, residents, fellows and peers. These same two faculty members agreed on the scoring key for each question. The test questions were piloted for clarity with students in the first-quarter (Q1) clerkship (n = 50). Pilot-test findings guided the refinement of the words used when administering the examination to the students. Data from the pilot group were excluded from the analysis. Each of the two surgeons independently and blindly scored the tests. Inter-rater reliability of scoring was calculated as the percentage of agreement between the two raters.

Statistical analysis

Retention (0–5 possible points), transfer (0–5 possible points) and total scores (0–10 possible points) were summed for each student’s pre-test and post-test. Test scores for each group were summarised using descriptive statistics including means and standard deviations (SDs). Comparison of scores was accomplished using paired t-test analysis and repeated-measures analysis of variance (anova) statistics. Scores were considered significantly different at an alpha of ≤ 0.05. All statistical analyses were performed using sas Version 9.2 (SAS, Inc., Cary, NC, USA).

Results

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References

A comparison of the two student cohorts showed no differences in the mean scores achieved on either the NBME Surgery Subject Examination (traditional-design group: mean = 76.8, SD = 7.5; modified-design group: mean = 75.5, SD = 7.0; = 0.82, p = 0.4146) or the Surgery Clerkship Composite (traditional-design group: mean = 99.5, SD = 10.7; modified-design group: mean = 99.0, SD = 12.3; = 0.82, p = 0.8576). This suggests that students in both groups had comparable levels of knowledge and were similar academically. The inter-rater reliability between the two surgeons who scored the examinations showed 93% agreement in Q2, 96% agreement in Q3 and 94% agreement in Q4. The few test-scoring discrepancies were resolved through discussion between the two scorers. The educator who observed each lecture reported no differences in presenter style between traditional or modified lectures. Medical student evaluations, perceptions and comments about the lecturer and his style were consistent through the quartiles; global scores given to the lecturer ranged from 4.71 to 4.97 out of 5.

Table 2 summarises the scores achieved by the traditional-design and modified-design groups on the pre-test and post-test. Students in the traditional-design and modified-design groups had similar pre-test scores (p > 0.05), indicating similar topic-specific pre-test levels of knowledge between the cohorts. Both groups showed significant improvements in retention (p < 0.0001), transfer (p < 0.05) and total scores (p < 0.0001) between pre- and post-test scores. A repeated-measures anova analysis showed statistically significant interactions between group (traditional versus modified) and time (pre-test versus post-test) for retention scores (F = 10.2, p = 0.0016) and total scores (F = 7.13, p = 0.0081), but not for transfer scores (F = 1.18, p = 0.2780), as noted in Table 3. These results show that pre-test–post-test improvements in retention and total scores were greater for those students instructed using the modified lecture design than for those instructed using the traditional lecture design. Differences between the groups in pre-test–post-test improvements on transfer did not reach statistical significance.

Table 2.   Pre-test and post-test scores achieved using a traditional slide design compared with modified slides designed using multimedia principles*
 Traditional designModified design
Pre-test (n = 39)Post-test (n = 39)Paired t-testPre-test (n = 91)Post-test (n = 91)Paired t-test
ScoreMeanSDMeanSDtp-valueMeanSDMeanSDtp-value
  1. * No significant differences in pre-test scores were found between the two groups

  2. SD = standard deviation

Retention2.260.613.330.649.0< 0.00012.050.913.870.7014.7< 0.0001
Transfer2.320.882.910.992.80.00742.31.093.201.066.7< 0.0001
Total4.581.256.241.177.1< 0.00014.31.666.991.2812.4< 0.0001
Table 3.   Repeated-measures anova analysis showing the effect of modified lecture design on the retention, transfer and total scores of medical students
SourceRetention scoreTransfer scoreTotal score
MSd.f.Fp-valueMSd.f.Fp-valueMSd.f.Fp-value
  1. MS = mean square; F = ratio; d.f. = degrees of freedom

  2. Bold value indicates organize data, highlight deferences.

Group (traditional versus modified)0.818.10.00220.911.440.15043.017.750.0064
Time (pre-test versus post-test)107.71185.4< 0.000130.01280.0124257.31129.9< 0.0001
Group × time5.9110.20.00161.311.180.278014.117.130.0081
Error0.6254  1.1254  2.0254  

Discussion

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References

There is a well recognised paucity of medical education research on the effects of utilising cognitive load theory and multimedia design principles on the learning of medical professionals.6,19,20 This study builds on the work that cognitive psychologists such as Mayer,16 Paivio,10 Sweller et al.9 and others11,12,14,20 have already carried out in the science of instructional design. Didactic lectures using ppt™ slides are one form of multimedia instruction and they remain a mainstream instructional strategy in medical education. They require preparation time on the part of faculty staff, who tend to use multimedia slides that are largely dependent on text and bullet points. The current study shows that departing from the default, bullet-point template for ppt™ software and applying multimedia principles when designing slides for medical students results in greater improvement in student knowledge retention compared with traditional slide design. These results are generally consistent with those obtained by Mayer16,17 with undergraduate students as participants, and show that multimedia design principles can be successfully applied in medical education. However, the results of this study did not show that multimedia design principles improved students’ ability to apply the newly acquired knowledge to new clinical problems compared with students instructed using the traditional design.

Several differences are noted between our study design and the design used in Mayer’s experiments that may explain the differences in outcomes. In Mayer’s design, separate experiments using very short lessons (i.e. < 3 minutes each) were used to study the effects of individual multimedia principles on retention and transfer.16,17 The present study employed a combination of multimedia principles during a 50-minute lecture; as such, our slides were modified to adhere to multiple principles rather than only one. Another difference that may have hampered our results from mimicking those of Mayer was that Mayer’s experiments were conducted in laboratory-like conditions and were attained by students for the purposes of the study.16,17 Our study was carried out during an actual core curriculum session with uncontrolled seating and student distractions. In addition, in Mayer’s design, students were given the post-tests immediately after the designate lecture had concluded and there was no pre-test.16,17 In our study, post-tests were administered an hour after the designated lecture; in the interim period, our students sat through a non-related lecture that may have caused interference or overload.

Conclusions

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References

Faculty development curricula should introduce multimedia learning principles and cognitive load theory to faculty staff in order to heighten their awareness of these principles and discourage the overuse of the default ppt™ bullet-point template.

Our study shows that modifying lecture slides based on multimedia principles is more likely to result in improved short-term retention among medical students, but empirical evidence is still needed to determine how these principles affect transfer of learning. Outcome measures in future studies should also evaluate the effects of applying multimedia learning principles on long-term learning in medical students and other medical professionals.

Contributors:  NI contributed to the conception of the study and its hypothesis, the study design, wrote the test questions and delivered the lectures, and contributed to data acquisition and interpretation, and the drafting of the article. MSc contributed to the conception of the study and its hypothesis, the study design, data acquisition and interpretation, and the drafting of the article. SS monitored the lectures and contributed to the collection, acquisition and interpretation of data, and the drafting of the article. MSh contributed to the design of the study, wrote the test questions, scored the tests and contributed to data acquisition and interpretation, and the drafting of the article. EW contributed to the study design, conducted the power analysis and contributed to data analysis and interpretation, and the drafting and critical revision of the article. REM contributed to data analysis and interpretation, and the critical revision of the article. DAD contributed to the conception of the study and its hypothesis, the study design, the analysis and interpretation of data, and the critical revision of the article. All authors approved the final manuscript for publication.

Acknowledgments

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References

Acknowledgements:  the authors thank Julie Randolph and Joy Serletic-Freeman for their great help in preparing the manuscript.

Funding:  none.

Conflicts of interest:  none.

Ethical approval:  this study was approved by the Institutional Review Board of Northwestern University Feinberg School of Medicine (STU00016328).

References

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References