Using eye movements to model the sequence of text–picture processing for multimedia comprehension
Abstract
This study used eye movement modeling examples (EMME) to support students' integrative processing of verbal and graphical information during the reading of an illustrated text. EMME consists of a replay of eye movements of a model superimposed onto the materials that are processed for accomplishing the task. Specifically, the study investigated the effects of modeling the temporal sequence of text and picture processing as shown in various replays of a model's gazes. Eighty‐four 7th graders were randomly assigned to one of the four experimental conditions: text‐first processing sequence (text‐first EMME), picture‐first processing sequence (picture‐first EMME), picture‐last processing sequence (picture‐last EMME) and no‐EMME (control). Online and offline measures were used.
Eye movement indices indicate that only readers in the picture‐first EMME condition spent significantly longer processing the picture and showed stronger integrative processing of verbal and graphical information than students in the no‐EMME condition. Moreover, readers in all EMME conditions outperformed those in the control condition for recall. However, for learning and transfer, only readers in the picture‐first EMME condition were significantly superior to readers of the control condition. Furthermore, both the frequency and duration of integrative processing of verbal and graphical information mediated the effect of condition on learning outcomes.
Lay Description
What is already known about this topic?
- Successful learning from illustrated text requires the integration of verbal and graphical information.
- Students usually do not pay much attention to pictures.
- Eye‐tracking technology is a research tool to track cognitive processing while reading and observing.
What this paper adds?
- Students' visual behaviour during reading an illustrated text can be modeled using eye movement modeling examples (EMME).
- Students were modeled varying the sequence of text–picture processing: text‐first EMME, picture‐first EMME and picture‐last EMME.
- Students in the picture‐first EMME condition showed stronger integrative processing than no‐EMME students.
- Students in the picture‐first EMME condition obtained greater learning and transfer than no‐EMME students.
Implications for practice and/or policy
- Once the video of a model's eye movements is available, it can be used easily in the classroom without an eye tracker.
- A short and simple intervention of a perceptual nature has potential to support multimedia processing and learning.
- A video showing a model's gaze replay that connects words and graphical elements is effective for recall of text content, a necessary step towards deep comprehension.
- Integrative processing that establishes correspondences between textual and pictorial information is crucial to learn from illustrated texts.
Number of times cited: 2
- Marie-Christin Krebs, Anne Schüler and Katharina Scheiter, Just follow my eyes: The influence of model-observer similarity on Eye Movement Modeling Examples, Learning and Instruction, 10.1016/j.learninstruc.2018.10.005, (2018).
- Katharina Scheiter, Carina Schubert and Anne Schüler, Self-regulated learning from illustrated text: Eye movement modelling to support use and regulation of cognitive processes during learning from multimedia, British Journal of Educational Psychology, (2017).
