Effects of a modelling example for teaching information problem solving skills
Abstract
Although students often appear to be skilled in retrieving and making use of information from the internet, research shows that their information problem solving skills are overestimated. They show deficiencies in many of the necessary skills, such as generation of search terms, selection of sources, and critical processing of information. It is therefore necessary to design and develop effective instruction to foster information problem solving skills. Research shows that learning from examples can be an effective approach for teaching complex cognitive skills in ill‐structured domains, such as writing or communicating. To explore whether this also holds for information problem solving, this study investigates the effects of presenting a modelling example in an online information problem solving training. Results of two experiments show that viewing a modelling example, presented as a screencast of an expert thinking out loud and interspersed with cognitive prompts, leads to a higher posttest performance than performing a practice task. The effect persisted on a delayed posttest 1 week later. The results imply that information problem solving instruction in an online setting can benefit from employing video‐based modelling examples.
Lay Description
What is currently known?
- Example‐based learning is effective for initial skill acquisition.
- Information problem solving (IPS) is a complex cognitive skill often underdeveloped.
- No research was found investigating effects of example‐based learning on IPS.
What this paper adds?
- Two experiments investigate effects of a video‐based modelling example on IPS performance.
- The modelling example was compared with a practice task in a short online training.
- Results show a modelling example leads to higher IPS performance than the practice task.
What are implications for practitioners?
- Findings corroborate the effectiveness of example‐based learning on an ill‐structured domain, specifically IPS.
- Teachers and researchers should consider using modelling examples in IPS instruction.
1 INTRODUCTION
1.1 Information problem solving
Information problem solving (IPS) is a skill often required from students in today's educational programs, as it is common for teachers to provide assignments requiring students to search for information on the internet. These assignments can be characterized as information problems: problems that require more information to solve than is currently available to the learner. They pose an information gap, because students must first search for the missing information and then process it in order to solve problem. Teachers might assume that searching and processing information automatically leads to learning, but such information problems are often ill‐defined and present unknown or unclear task demands, goals, or solution paths. Although it is tempting to regard students as “digital natives” and expect that they automatically acquired skills to solve such problems, research shows that most students' IPS skills are underdeveloped. Students struggle to systematically search for information, evaluate it critically, and produce an adequate solution for an information problem (Frerejean, van Strien, Kirschner, & Brand‐Gruwel, 2016; Walraven, Brand‐Gruwel, & Boshuizen, 2008, 2009).
An effective approach to solving an information problem can be summarized in five steps (e.g., the IPS‐I model; Brand‐Gruwel, Wopereis, & Vermetten, 2005; Brand‐Gruwel, Wopereis, & Walraven, 2009). First, learners build a problem representation by reviewing the task demands, activating prior knowledge on the topic, and identifying which information is needed. They form an idea of the extent and structure of the domain, and formulate a question. Research shows that this step is often neglected entirely or performed only partially (Brand‐Gruwel et al., 2005). In the second step, learners determine a search strategy and start searching for information sources. In the case of an online search, they generate search terms, execute the search query in a search engine, and evaluate the search engine results page. Here, learners use ineffective strategies (Hölscher & Strube, 2000; Van Deursen & van Diepen, 2013), have problems generating relevant search terms, and formulate unproductive queries (Zhou, 2013). The third step is often executed in parallel with the second step and involves the evaluation of information and sources. In this step, learners determine whether a source is relevant, recent, and credible. This kind of critical scrutiny is essential to avoid irrelevant and unreliable sources, yet it is often lacking (Fogg et al., 2003; Gerjets, Kammerer, & Werner, 2011; Keil & Kominsky, 2013). The sources that make it through the selection process are processed in the fourth step. In this step, learners are often seen making annotations, highlights, or summaries as they critically study the contents to find similarities and differences between the sources. Research shows novices spend less time on processing the source contents than experts (Brand‐Gruwel et al., 2005). In the final step, learners create a product such as an essay, presentation, or poster that integrates information from the sources in order to solve the information problem and answer the question. During these steps, it is important that learners regulate their process by monitoring their progress, gauging the needed information, and steering the process if necessary. Again, research shows novices monitor and steer their process less often than experts and pay little attention to task time constraints (Brand‐Gruwel et al., 2005; Zhou, 2013).
The many problems that researchers and educators have discovered indicate that there is a need for formal instruction on IPS in schools in order to foster and improve students' IPS skills. However, as Badke (2010) illustrates, information literacy and IPS instruction is often lacking or not implemented effectively in educational programs, for a variety of reasons. Teachers may lack the necessary digital and IPS skills themselves and cannot teach them to their students, or hold a misplaced belief that such skills do not need to be trained because they develop naturally (Kirschner & van Merriënboer, 2013). And teachers who are equipped with the skills and willing to teach them may be unaware of how to provide effective instruction and integrate it in their lessons. Reports investigating the Dutch educational context underline that there is little structural attention for the integration of digital skills, and little is known about effective implementation in practice (Platform Onderwijs2032, 2016; Thijs, Fisser, & van der Hoeven, 2014). From these findings, it becomes clear that there is a need for empirically tested instructional interventions and best practices to guide teachers and instructional designers. Fortunately, research on IPS instruction is now growing.
As a relevant example, a study by Frerejean et al. (2016) presented students with a standalone online training session, comprising an instructional video, a modelling example, and four learning tasks presenting an information problem. The study investigated how learners could be supported while working on whole tasks and which type of task support was most effective to teach IPS skills. It compared four conditions, each with a different training design. In the emphasis manipulation condition, instructional emphasis was placed on only one aspect in each learning task. For example, the design included a series of prompts for teaching source evaluation in one task, search skills in the next task, and problem definition skills in another task. A second condition used the completion strategy, an approach presenting a series of tasks with a fully worked‐out first task, and a decreasing number of worked‐out steps in each subsequent task until the final task contained no worked‐out steps. A third condition presented a combination of these two approaches and a fourth condition served as a control condition applying no task support. Although the significant increase in performance after the training indicated the training was effective, the gains in scores were similar in all conditions and no differential effects of task support methods were found. The authors suggested that the modelling example that was presented in all conditions could be partly responsible for the learning effect. This suggests that providing demonstrations of effective IPS by experts can be an effective instructional method for teaching novices how to approach and solve information problems. Example‐based learning thereby presents itself as an interesting direction for future research in the domain of IPS.
1.2 Example‐based learning
Example‐based learning finds support in disciplines such as Bandura's (1977) social learning theory and cognitive load theory. From the perspective of Bandura's social learning theory, skills learning takes place by observing others perform the skill. Observational learning can be realized by presenting learners with modelling examples, typically showing a model performing the skill while thinking out loud and, in contrast to traditional paper‐based worked examples, providing important insight into the thought‐processes and decision‐making processes that otherwise remain covert. In the social learning account of observational learning, Bandura (1977) posits four processes that govern learning: attentional, retention, reproduction, and motivational. In order for modelling examples to be effective, a learner's attention should be focused on the essential features of the model, the actions should be stored in memory so they are retained and not forgotten, there should be opportunity for reproduction to practice the skills, and the learner should be motivated to display the correct behaviour. Similar processes are identified in the cognitive load perspective on example‐based learning.
In cognitive load theory, the worked example effect states that learning from fully and/or partially worked examples (i.e., problems presented with a full or partial solution) leads to more effective and/or efficient learning than conventional problem solving, as novice learners often lack the specific knowledge and problem solving strategies necessary to solve problems without support (Sweller, 2006). Consequently, they mostly fall back to naïve strategies such as trial‐and‐error or means‐ends analysis, which place a high demand on working memory and leave few mental resources to devote to learning (Sweller, 1988). Short on working memory capacity, novice learners focus primarily on irrelevant problem features and build a superficial representation of the problem. Experts, on the other hand, identify structural problem features, such as relevant domain principles, to create a more elaborate problem representation (Chi, Feltovich, & Glaser, 1981; Sarsfield, 2014).
A provided worked example traditionally contains an initial problem state, a goal state, and a written account of the solution steps leading to a solution, such as a step‐by‐step description to solve a mathematics problem. Providing a worked example that shows the solution steps toward the goal state relieves learners of the search for a solution path and reduces the burden on working memory (Renkl, Hilbert, & Schworm, 2009). It provides an example of the correct procedure to solve a problem, which frees up cognitive resources to use activities that are germane to the construction of knowledge schema and solution procedures. Van Gog, Paas, and van Merriënboer (2004) argue that examples can be improved by providing not just a step‐by‐step process, but also process‐oriented information. Elaborating on the rationale behind the problem solving process—the “how” and “why”—can enhance the transfer of these skills to other problem contexts (Van Gog, Paas, & van Merriënboer, 2008). In reviews on the effectiveness of example‐based learning, Van Gog and Rummel (2010) and Renkl (2014) give an overview of the parallels between the social learning and the cognitive load accounts. Providing an exhaustive discussion on these accounts is outside the scope of this article, and therefore this section will focus on some of the factors affecting the effectiveness of examples that are relevant for the presented study.
First, effectiveness of example‐based learning depends on the degree to which the information in the example is processed. Learning from a model is improved when learners actively process the example by elaborating on the presented information and evaluating the process (Braaksma, Rijlaarsdam, van den Bergh, & van Hout‐Wolters, 2006; Braaksma, van den Bergh, Rijlaarsdam, & Couzijn, 2001). Without performing these essential activities, learners might observe without trying to understand. Their attention might be diverted away from the relevant information or focused on less important elements in the example, with the risk of a decreased learning effect (Renkl, 1999; Stark, Mandl, Gruber, & Renkl, 2002). This can be overcome by directing learners' attention to important elements and to ensure active processing of the modelling example, for example, with self‐explanation prompts (Renkl, 2002; Renkl & Atkinson, 2002). Such prompts are considered an integral part of example‐based learning from a cognitive load perspective (see Renkl et al., 2009), and are widely considered as an effective learning mechanism (Chi, Bassok, & Lewis, 1989; Chi, De Leeuw, Chiu, & Lavancher, 1994).
As answering such prompts requires that learners pay attention to the example and attempt to follow the solution procedure (Aleven & Koedinger, 2002), it reduces the possibility that they passively watch the example without cognitive investment (Atkinson, Derry, Renkl, & Wortham, 2000; Renkl, 1997). Other types of prompts can have additional benefits, such as metacognitive prompting to stimulate metacognitive thinking (Stadtler & Bromme, 2008), or comparison prompts, asking the learner to compare and contrast their own approach to an expert's systematic approach. The latter is particularly beneficial if the learner starts out with intuitive strategies that are less effective, as such prompts can stimulate learners to think critically about the problem domain, the problem structure, and the demonstrated approach to problem solving (Van Merriënboer & Kirschner, 2013).
Second, example‐based learning is effective if retention is ensured and the learner is able to remember and apply the observed skills in situations where the model is no longer present (Bandura, 1971). Enactment, or practice, is necessary for strengthening and automating the required skills without the presence of the model. Worked examples and modelling examples contain a high degree of guidance and support, which is beneficial for novice learners who lack domain knowledge and solution strategies. However, when progressing through the learning phase, schemas become increasingly more elaborate and more strategies are formed to cope with varying problem situations (Atkinson, Renkl, & Merrill, 2003). At some point, examples will offer little new information and much redundant information. When this occurs, learning from examples can lose its benefit over solving practice problems and induce an expertise reversal effect (Kalyuga, Ayres, Chandler, & Sweller, 2003; Kalyuga, Rikers, & Paas, 2012) where providing too much support to advanced learners can be detrimental to learning. From this, it follows that examples should precede a period of practice, where learners get a chance to apply the observed knowledge and skills. This improves retention and avoids diminishing learning effects caused by the expertise reversal effect.
These findings dictate that examples are most effective for novice learners and when presented with incorporated methods to stimulate active processing, such as prompting. In addition, learners should be able to practice the observed skills after watching the examples in order to promote retention.
1.3 Examples in IPS instruction
Much of the research on example‐based learning has taken place in structured, well‐defined domains such as mathematics and physics where there are often fixed procedures for solving a problem, but some research exists on the effects of example‐based learning in ill‐defined domains. These problems cannot be solved by following a strict procedure with discrete solution steps; instead, learners will have to reason through the problem and make the right decisions relying on heuristics, strategies, and an evaluation of the currently available information. In these cases, a worked example showing only a step‐by‐step solution procedure is not sufficient, because learners need more information about how decisions are made and which knowledge is used to make these decisions. For ill‐defined problems, modelling examples containing process information are preferred over traditional written worked examples to demonstrate how the problem solver reasons through the solution steps (Van Gog & Rummel, 2010). Seeing the solution steps being performed and hearing the reasoning behind them can help learners improve or create knowledge schemas and solution procedures.
Research has shown that modelling examples can be effectively employed to teach complex skills in unstructured domains. For example, novice psychotherapists improved their communication skills the most when watching a video of an experienced psychotherapist, when compared with watching a video of an expert lecture, novice model, or their performance during an interview (Baum & Gray, 1992). An experiment on problem solving showed that students' problem solving strategies improved the most when watching a teacher thinking aloud in a problem‐based learning setting when compared with a teacher giving direct instruction or giving no advice (Pedersen & Lui, 2003). And research on creativity in art showed student designers delivered more creative work after watching videos of peers thinking aloud during a design task than after receiving direct instruction on strategies (Groenendijk, Janssen, Rijlaarsdam, & van den Bergh, 2013). Although further research shows example‐based learning is beneficial for the acquisition of complex skills, such as academic writing (Braaksma et al., 2001; Braaksma, Rijlaarsdam, van den Bergh, & van Hout‐Wolters, 2004) and problem solving (Van Gog et al., 2004), no research was found investigating modelling examples in IPS instruction. From these findings, it can be expected that example‐based learning in the form of modelling examples is also effective for teaching the complex skill of IPS.
In the context of IPS instruction, a modelling example might consist of an expert solving a problem while explaining the reasoning about each step and skill in the process. For example, the expert explains why a certain strategy is chosen, how search terms are generated, and how the different results and sources are evaluated. A recorded screencast can show the screen and activities (i.e., clicking) of the expert while concurrently playing the expert's narration. Online learning environments provide an easy opportunity for embedding modelling examples in the form of video demonstrations. However, instructional designers should be aware that multimedia materials can easily create unwanted cognitive load, which carries the danger of impairing the learning process (Paas, Tuovinen, Tabbers, & van Gerven, 2003). It is therefore wise to follow the principles derived from research on multimedia learning to reduce hindering load on working memory and increase activities that lead to learning (Mayer, 2014).
2 THIS STUDY
This experiment is a follow‐up to the research by Frerejean et al. (2016) that investigated the effect of task support on acquiring IPS skills in a short online intervention. It follows up on the suggestion that modelling examples were responsible for the learning effect found in the former study, and attempts to answer the question “What are the effects of providing a modelling example on the acquisition of IPS skills in a short online training?” Two experiments test the hypothesis that students receiving a modelling example display higher performance on an IPS test than students receiving no example, but engage in a practice task. To investigate whether and how viewing a modelling example also puts strain on working memory and affects the learning process, subjective mental effort ratings are collected during the learning phase.
3 EXPERIMENT 1
3.1 Method
3.1.1 Participants
A total of 39 first‐year university students participated in the individual, computer‐based online training session at a Belgian university (27 female, 12 male). All students had the Belgian nationality, and their age varied between 16 and 38 years old (Mage = 19.67, SD = 3.47). In the modelling example condition, 15 students were female, 5 were male, and the age varied between 17 and 38 years (Mage = 20.15, SD = 4.25). In the practice task condition, 12 students were female, 7 were male, and the age varied between 16 and 22 (Mage = 19.16, SD = 1.8).
3.1.2 Design
The experiment was a pretest‐posttest design with two conditions. All students received a 2‐hr online training in IPS, consisting of an instruction video, a modelling example for one half of the students or a practice task for the other half, and four learning tasks. Students' skill level was measured before and after the training.
3.2 Materials
3.2.1 Online training
The training was presented in an online learning environment and consisted of three elements. First, a 14‐min instructional video introduced a systematic approach to solving information problems, based on the IPS‐I model (Brand‐Gruwel et al., 2009): define the problem, search for information, select information, process information, and present the solution. The video was presented to provide students in a short amount of time the necessary domain knowledge and problem‐solving approach to complete the upcoming learning tasks. Then, either the modelling example or the practice task was presented, further explained below. Finally, the students received four learning tasks, each consisting of a problem description and a textbox to enter an answer. Students had to search the web for information to reach a solution. The learning tasks contained no further support or guidance. The presented problem descriptions handled disputed socioscientific topics: the effect of stretching before sports, the dangers of electromagnetic radiation from cellphones, the consequences of violence in videogames, and the influence of using media devices on sleep quality.
3.2.2 Modelling example
The modelling example was presented as a 10‐min screencast in which a fictitious expert demonstrated how to solve an information problem about the effect of GPS navigation systems on traffic safety. The model was a 23‐year old Dutch female speaking in a standard‐accented voice. Although earlier research suggests a speaker/gender effect, stating that learning outcomes from video modelling examples are higher when narration is presented by a female speaker rather than a male speaker (e.g., Linek, Gerjets, & Scheiter, 2010), more recent research finds gender has no beneficial effects on learning outcomes, though it may influence affective aspects of learning (Hoogerheide, Loyens, & van Gog, 2015). The model in this example was not visible to the viewer, but narrated the actions on‐screen by thinking aloud and explaining her reasoning behind each decision. The modelling example was split into four short fragments and interspersed with prompts. Before viewing each fragment, students first activated their prior knowledge by answering the prompt “Where will you focus your attention while executing the next step?” This prompt served as a method to activate the relevant principles and strategies pertaining to that step before the student watches the model.
The first fragment showed the expert reasoning about the problem description and generating a brief and clear problem statement. The fragment ended with a prompt that included the questions “What do you think of the actions of the expert?” and “How does this differ from your current approach?” These questions were intended to stimulate comparisons of solution procedures between the student and the expert and an active processing of the example. Students entered the answers to these questions in a textbox before clicking through to the next fragment.
The second fragment demonstrated how the expert chose search terms and entered them into the Google™ search engine. The subsequent results page was analysed by thinking aloud while showing relevant on‐screen elements with the cursor. The fragment ended with the questions “What do you think of the actions of the expert?,” “Would you have chosen the same keywords?,” and “Do you agree with the evaluation of the search results?”
The third fragment started with a short reflection by the expert on her reasons for selecting a particular website. These additional comments served as a feedback component, so students could compare their answers to the expert's reasoning. The fragment continued with a demonstration on how to quickly scan and evaluate a source. The information in the source was deemed relevant and reliable and subsequently added to the bookmarks. The expert noted that the information was a bit outdated, so she returned to the search results to find a more recent source. After evaluating and saving a second source, the expert made some changes to the keywords and evaluated two additional pages. The fragment concluded with the following prompts: “What do you think of the actions of the expert?,” “Would you have done the same?,” and “What would you do differently?”
The final fragment showed the expert's formulated answer to the problem. Students were advised to pause the video to read the answer in their own pace. Afterwards, they were prompted with the questions “What do you think of the expert's answer?” and “Would you have given a similar answer?”
The screencast was a complete yet condensed application of the five steps of the systematic approach introduced in the instruction video. The design of the video followed several instructional design principles: Schema construction was promoted by activating the learner's knowledge prior to each fragment, and active processing was promoted by adding prompts after viewing each fragment. In addition, students were stimulated to compare the expert approach with their own. At the beginning of the third fragment, the modelling example included some general reflection remarks that serve as feedback on the students' answers to the prompts. In addition, care was taken to design the modelling example by following principles for effective multimedia instruction. Appendix A gives an overview of these principles and how they were applied to the screencast. After the example, students received four learning tasks to practice the demonstrated approach. These aspects (prompting, segmenting, etc.) are considered integral parts of a well‐designed modelling example and are therefore implemented and analysed as one intervention.
3.2.3 Practice task
The practice task contained the same problem description as the worked‐out example and a textbox to enter an answer. Students were asked to spend approximately 15 min searching the web for information before formulating a short answer; comparable with the amount of time it took the other students to process the modelling example. After the practice task, students received the same four learning tasks for more practice.
3.2.4 Preliminary questionnaire
To collect demographic data, including age, nationality, and prior education, a short questionnaire was administered before the pretest. The questionnaire also included items about the amount and pattern of internet and computer usage. Students were asked to indicate their perceived level of competence in solving information problems on a scale of 1 to 10.
3.2.5 Measurement of IPS skill
IPS skill was measured using a situational judgment instrument developed by Frerejean et al. (2016). The online measure consisted of seven fabricated situations that occur during IPS and asked students to describe how they would act in the presented situation. Table 1 provides a schematic overview of the seven questions in these skill tests. To ensure content and face validity, the items correspond to the skills and subskills in the IPS model by Brand‐Gruwel et al. (2009). For example, to measure the skill “selecting information,” a fabricated search engine results page was presented and students were asked to select three results and give reasons for their selection. The answers were scored blindly using the scoring rubric in Appendix B. Students could obtain a subscore for each of the four skills: defining the problem, searching information, selecting information, and processing information. The skill presenting information was not included in the tests for two reasons. First, presenting can be done in countless ways and concerns a multifaceted skill that is difficult to measure in a short timeframe. Second, the training presented little instruction on presenting information, so little improvement is expected. The four subscores were then averaged to obtain the total test score and expressed in a percentage for ease of interpretation. The items on the posttest were identical to those on the pretest, but on a different problem domain. In the pretest, gender‐specific education was used as a problem domain, whereas the malleability of intelligence was used in the posttest. A second rater rescored 20 randomly selected cases to allow interrater reliability analysis. The two‐way, mixed, absolute, single‐measure intraclass correlation of .878 indicated a high interrater agreement and therefore a reliable measurement.
| Item | Skill | Subskill | Given | Question |
|---|---|---|---|---|
| 1 | Defining the problem | Problem orientation | A problem description | How would you start this task? What is your first step and why? |
| 2 | Defining the problem | Formulating a problem statement | A problem description | Which problem statements would you formulate? Why do you choose these? |
| 3 | Searching information | Generating search terms | A problem description | Which search query would you type into Google? Formulate two alternative search queries. |
| 4 | Selecting information | Evaluating search results | A fabricated search engine results page | Which three websites would you select? Why did you select these websites? |
| 5 | Processing information | Scanning a source | A screenshot of a text‐rich website, zoomed so the text is unreadable | What do you do when you visit a text‐rich website and want to find out if it contains relevant information? How do you proceed? |
| 6 | Processing information | Evaluating information | A short text fragment containing an argument given by an expert | Which criteria do you use to determine whether information is useful for your task? What are your conditions for use? |
| 7 | Processing information | Dealing with conflicting information | Two short, contradicting arguments | How do you deal with contradicting information? How does this affect your solution? Explain. |
3.2.6 Mental effort
Solving an information problem is a complex task imposing a high cognitive demand, especially when the required skills are insufficiently developed. To investigate whether viewing a modelling example alters the experienced cognitive demand during practice, mental effort was measured four times during the training phase. At the end of each learning task, students answered the item How much effort did it take to perform this task? on a 9‐point scale (Paas, 1992).
3.3 Data analysis
An analysis of covariance was conducted on the posttest scores with modelling example (yes vs. no) as a between subjects factor and the pretest score as a covariate. A repeated measures analysis of variance was conducted on the mental effort ratings, with learning task as a within subjects variable and modelling example (yes vs. no) as a between subjects variable.
3.4 Procedure
Students participated in the experiment as a practical assignment in their curriculum. The 2‐hr session took place in the university computer room where students received log‐in credentials to access the online experimental environment. They received instructions to work individually through the tasks and to spend approximately 15 min on each learning task, which is a realistic time limit for finding information online (Lazonder, Biemans, & Wopereis, 2000). They were further informed that their screen content could differ from that of the other students and then presented with the preliminary questionnaire. After filling out the questionnaire, students were automatically randomly assigned to one of the two conditions. They then received the pretest and the instructional video. Half of the students received the modelling example and half received the practice task. Afterwards, students could practice the skills in four learning tasks, followed by the posttest. After the posttest, students signed a form to obtain course credit and were subsequently dismissed. A debriefing followed 8 weeks later.
3.5 Results
3.5.1 Preliminary analysis
Analysis of the demographic data revealed no significant differences on any of the variables measured, such as age and computer use. Students reported they use the internet for 2.72 hr per day (SD = 1.32) and estimated their own IPS ability with a 6.28 out of 10 (range = 4–8). On the posttest, nine scores (<3%) that make up each of the four subscores were missing, probably due to students accidentally skipping questions or some incidental technical issues. In these cases, the scores were replaced by the corresponding pretest score. Subscores and total posttest scores were then calculated as described. There were no other missing values.
3.5.2 IPS skill tests
Both conditions obtained higher scores on the posttest than on the pretest, indicating that learning took place. As shown in Table 2, students receiving a modelling example scored higher on the posttest than those receiving a practice task. The difference between the conditions was statistically significant when controlling for pretest scores: F(1, 39) = 5.64, p = .023, η2partial = .135.
| Condition | Pretest (SD) | Posttest (SD) |
|---|---|---|
| Modelling example | 38.44 (10.97) | 55.94 (10.03) |
| Practice task | 39.47 (11.23) | 49.34 (6.88) |
3.5.3 Mental effort
The ratings on the 9‐point mental effort scale collected after each of the four learning tasks are displayed in Figure 1. The repeated measures analysis shows that reported mental effort changed significantly over time: F(3, 37) = 3.01, p = .033, but with a small effect size: η2partial = .079. Subsequent contrast analysis indicates only a significant decline from learning task 3 to 4: F(1, 37) = 6.68, p = .014, η2partial = .160. There was no difference between the conditions: F(1, 39) = .22, p = .645.
3.6 Conclusion
These findings provide support for the hypothesis that students receiving a modelling example achieve higher learning outcomes than students receiving a practice task. Although both groups show improved scores on the posttest, the group receiving a modelling example increased nearly twice as much as the practice task group. Although this provides an answer to the research question, we sought to replicate and further investigate the learning effect. First, it would be interesting to investigate transfer over time. A delayed posttest would reveal if modelling examples have potential for robust learning. Furthermore, a larger sample size would increase confidence in the findings. For these reasons, a replication experiment was conducted in a slightly larger student sample and with an added delayed posttest.
4 EXPERIMENT 2
The design and materials were identical to those in the first experiment, but included a delayed posttest to measure the delayed learning effect. Additionally, the pretest was administered at home in the week before the training session, reducing possible priming effects on the learning phase.
4.1 Method
4.1.1 Participants
A total of 60 first‐year Psychology students from a Dutch university participated in the replication (41 female, 19 male). Two students had the Belgian nationality, and 24 had the German nationality. The remainder was Dutch. The age ranged from 18 to 32 (Mage = 20.63, SD = 2.14). The modelling example condition contained 21 female students and 8 male students with an age range of 18 through 24 years (Mage = 20.43, SD = 1.61), of which 1 was Belgian, 13 were German, and 17 were Dutch. The practice task condition consisted of 20 female students and 11 male students with an age between 18 and 32 years (Mage = 20.83, SD = 2.57), of which 1 was Belgian, 11 were German, and 17 were Dutch. Participation was voluntary, but strongly stimulated by granting research participation credit and informing students that the content of the training was relevant for the current topic in their curriculum (problem solving). Students could choose one of eight different timeslots. Furthermore, students were informed that an online pretest and delayed posttest had to be filled out in their own time. A debriefing followed in a lecture 2 weeks after the delayed posttest.
4.2 Materials
4.2.1 Measurement of IPS skill
A delayed posttest was added after the posttest. It was identical to the existing pretest and posttest, but handled the topic of health benefits of drinking red wine. The pretest and posttest were the same as in Section 3.
4.3 Data analysis
The pretest, posttest, and delayed posttest were scored as in Section 3. An analysis of covariance was conducted on the posttest scores with modelling example (yes vs. no) as a between‐subjects factor and the pretest score as a covariate. This analysis was repeated on the delayed posttest scores. A repeated measures analysis of variance was conducted on the mental effort ratings, with learning task as a within‐subjects variable and modelling example (yes vs. no) as a between subjects variable.
4.4 Procedure
The procedure and design were largely identical to the procedure of the first experiment, with the exception that the pretest was filled out at home in the week before the training and the delayed posttest was filled out at home, 1 week after the training. Because it was known that a large proportion of students were German, the online environment was programmed to divide students in conditions on a random basis, yet to stratify for nationality. This was done as a precaution in case the German students' performance suffered because the materials were all in Dutch. This resulted in conditions containing approximately the same proportion of Dutch and German speaking students. Before starting the training session, the experimenter stimulated students to spend approximately 20 min on each learning task. After finishing the final evaluation, students signed an informed consent form and obtained research participation credit. They were reminded to fill out the delayed posttest 1 week later and were then dismissed. The same conditions were used as in the first experiment.
4.5 Results
4.5.1 Preliminary analysis
No differences arose on any of the variables in the demographic questionnaire. Students reported they use the internet for 4.40 hr per day (SD = 1.95) and estimate their IPS ability with a 6.32 out of 10 (range = 3–8). For missing data, substitution of a missing posttest value for its corresponding pretest value occurred three times (<1%). For one student, one subscore was classified as missing and the posttest score was calculated as the average of the remaining three subscores. One student had several missing values, so the posttest score was classified as missing. Four students did not show and dropped out. On the delayed posttest, two additional students dropped out. For one student, the delayed posttest score was calculated as the average of three subscores due to one missing subscore.
4.5.2 IPS skill tests
On the pretest, students obtained an average score of 35.02% (SD = 11.45), which increased to 57.90% (SD = 10.04) on the posttest. Table 3 shows an overview of scores per condition. The analysis revealed a significantly higher posttest score in the modelling example group when controlling for pretest scores: F(1, 55) = 4.46, p = .040, η2partial = .079. Running the same analysis on the delayed posttest scores indicated that the effect of modelling example remains significant: F(1, 54) = 5.51, p = .023, η2partial = .097.
| Condition | Pretest (SD) | Posttest (SD) | Delayed posttest (SD) |
|---|---|---|---|
| Modelling example | 34.15 (11.35) | 60.71 (9.45) | 58.93 (15.72) |
| Practice task | 36.00 (11.73) | 54.75 (9.93) | 52.42 (16.59) |
4.5.3 Mental effort
Reported mental effort ratings are displayed in Figure 2. The repeated measures analysis showed that reported mental effort changes significantly over time: F(3, 49) = 2.76, p = .045, but with a small effect size: η2partial = .055. As in Section 3, subsequent contrast analysis indicated a significant decline from learning task 3 to 4: F(1, 49) = 6.58, p = .014, η2partial = .123. Although mean mental effort scores were higher on each learning task in the modelling example condition, there was no significant difference between the conditions: F(1, 49) = 2.79, p = .102.
4.6 Conclusion
The findings in this replication study resemble those of the first experiment. Students receiving a modelling example achieve higher learning outcomes (i.e., scores on the skill tests) than students receiving a practice task. This supports the hypothesis that modelling examples, in which students actively process an example of problem solving, are more effective for teaching IPS skills than practice tasks in which students practice the newly acquired knowledge by themselves. The analysis of delayed posttest scores reveals that the learning effect of the modelling example persists at least 1 week after the training.
Students in the first experiment reported mental effort scores around or slightly below the midpoint on the 9‐point scale. In the second experiment, these scores are lower and are scattered around the 4‐point mark, yet they follow the same pattern as in the first experiment declining significantly in the final learning task. The lack of a difference between both conditions indicated that receiving a modelling example does not alter the amount of reported mental effort during the learning phase.
5 GENERAL DISCUSSION
These experiments were designed to investigate the effect of learning from a modelling example in which learners see the application of a solution procedure accompanied by additional procedural information (how) and application of domain‐specific knowledge (why), on the acquisition of IPS skills. Section 3 showed that students who receive a modelling example significantly outperform students who receive a practice task. Section 4 showed similar findings in a larger yet comparable sample, and revealed that this effect persisted after 1 week. Compared with a practice task, a single modelling example was found more effective for the formation of cognitive models and strategies needed for IPS performance. The results in this study also illustrate the low level of performance of first‐year university students on IPS tasks. Untrained, they obtain average scores of under 40% on the skill tests. This observation directly opposes claims that students are digitally native and naturally develop the necessary skills to deal with information technologies (Prensky, 2001; Tapscott, 1999). The ease with which they seemingly manage to retrieve information online seems to mislead those who present these claims. These results once again underline the necessity for formal training in the area of IPS (Bennett, Maton, & Kervin, 2008; Kirschner & van Merriënboer, 2013; Smith, 2012).
For teachers and researchers in the domain of information literacy or IPS, this study shows that example‐based learning is an effective approach for training students to solve information problems. This finding adds a data point to the body of research showing that modelling examples are effective for teaching complex skills in ill‐structured domains (e.g., Braaksma et al., 2004; Van Gog et al., 2004). Although beneficial effects of modelling examples were found in other domains, we found no research confirming the same for the domain of IPS. This research fills that gap, and shows that an instructional design containing an instruction video, a single modelling example followed by a period of practice leads to higher skill acquisition than an instruction video followed by mere problem solving. More specifically, a 10‐min video of a modelling example, segmented and interspersed with cognitive prompts, taking multimedia principles into account and followed by four learning tasks, leads to a higher learning effect than instruction followed by mere practice, is effective for the development of an important 21st century skill such as IPS. Teachers, instructional designers, or researchers interested in developing effective IPS instruction are therefore advised to consider including well‐designed modelling examples in IPS instruction (Hilbert, Renkl, Schworm, Kessler, & Reiss, 2008).
In addition, the results show that viewing a modelling example did not affect reported mental effort during the practice phase. Solving information problems effectively and efficiently requires the integration of knowledge, skills, and attitudes and the coordination of several constituent skills. Because these experiments focused on the effects of the modelling example, the learning tasks were intentionally stripped of all support and guidance—such as worked‐out steps or prompts—to avoid confounding effects. For novices, solving information problems without receiving any form of built‐in task support should be cognitively demanding. Yet, average experienced mental effort ratings were scattered around the 4‐point mark on the scale, which corresponds to rather low mental effort. Students in the second experiment had approximately 5 min longer to complete each learning task and reported less mental effort than students in the first experiment. This is likely causal: more time means less time pressure, which means lower cognitive demand Frerejean et al. (2016). Not much research exists on experienced mental effort during search tasks, making it difficult to compare these ratings, but they seem to be slightly lower than in other studies (Kim & Rieh, 2005; Rieh, Kim, & Markey, 2012).
These low mental effort ratings might indicate low investment. It may be the case that students regarded the tasks as simple teacher‐imposed obligations with little relevance, which lowered their motivation and lead them to invest little energy in performing the tasks (De Vries, van der Meij, & Lazonder, 2008; Russell & Grimes, 2007). Although effort was made to create learning tasks on socioscientific topics with relevance to the study domain of the students, they were not topics that were integrated in the curriculum outside of the presented IPS training session. Although some students informally expressed they perceived the training as somewhat boring and long, motivation and perceived relevance were not measured in the study, making it difficult to draw any solid conclusions from these statements.
The pattern of reported mental effort was similar in both experiments: it remained stable over the first three tasks in both conditions, then significantly dropped in the final learning task. Perhaps working on several conventional tasks in a row might have demotivated students, making them decide to rush through the final task to end the session. As motivation is one of the four governing processes as identified by Bandura (1977), one might expect that task content more aligned to the students' curriculum might increase motivation and thereby learning effects. However, when interpreting mental effort ratings, it is important to remember that without knowing whether the cognitive demand refers to load that leads to learning (i.e. germane load) or hinders learning (i.e. extraneous load), one cannot explain effects on learning outcomes (Van Merriënboer & Ayres, 2005). No data was collected on students' motivation, so these findings merely warrant a suspicion that students invested less energy in performing the task to their best abilities.
As an alternative explanation, the low mental effort ratings can be caused by overestimation. The ease with which students find information online might lead them to overestimate their ability to solve information problems in an effective and efficient way. A Dunning–Kruger effect can occur, where the unskilled learners are unable to assess their own level of competence and consider themselves more skilled than they are (Dunning, Johnson, Ehrlinger, & Kruger, 2003; Kruger & Dunning, 1999). Indeed, students' perceptions of their own competence were higher than their objective scores in the pretest: an estimation of 6.28 on a scale of 10 compared with a score of 38.94% in Section 3 and 6.32 on the same scale compared with 35.02% in Section 4. This contrast between skills perception and actual performance points in the direction of a Dunning–Kruger effect (Kruger & Dunning, 1999). After the training, approximately one‐third of the students informally stated they already knew much of what was taught in the training, showing that students might think of themselves as competent, while in reality their scores after the training are still below 60%. Students are apparently unable to correctly judge their IPS performance.
Several general limitations should be considered when interpreting these results. First, due to time constrains, the training session could only include one modelling example. Results from previous research suggest using multiple examples allow students to detect structural and surface features (Atkinson et al., 2000; Renkl, 2014). Multiple examples can improve the abstraction of knowledge schema because students have more opportunities for encoding information from examples and comparing their schema with the expert performance (Alfieri, Nokes‐Malach, & Schunn, 2013; Gerjets, Scheiter, & Schuh, 2008). Additionally, only one type of search task was included: an information collection task using a general search engine. This prevents any conclusions about transfer and generalizability to tasks with a different level of complexity (Becerril & Badia, 2015), such as tasks that require specific information (e.g., academic articles) or specific strategies (e.g., using an academic literature database). Researchers and instructional designers need to further investigate how employing sequences of examples can lead to transfer and contribute to teaching skills in a way that allows students to apply them in different contexts (Fyfe, McNeil, Son, & Goldstone, 2014; Johnson, Reisslein, & Reisslein, 2014).
Second, assessment of IPS skills was done using a short skills test using seven items presenting realistic situations and recording students' intended actions and reasons. The instrument has not undergone formal validation, though content validity and interrater reliability after scoring were deemed sufficient. For a skill as extensive and complex as IPS, the current instrument only gives a global and superficial view of students' performance level. To achieve a fine‐grained view of a students' aptitude, the assessment methods should not only focus on overt information such as the quality of search terms or selected sources, but also on latent information such as decision‐making strategies, reasoning, or discarded sources. A combination of concurrent or retrospective thinking aloud while logging all visible actions would help uncover the necessary information to make detailed assessment possible.
Third, although this study investigated effects of an integral modelling example, variations in the design of that example can impact those effects. Instructional designers can make a myriad of design choices concerning length, visual design, application of multimedia principles, method of presentation, and information provided in each example (Hoogerheide, Loyens, & van Gog, 2014; Van Gog, Verveer, & Verveer, 2014). The investigated modelling example was optimized to achieve maximum effect in the ecologically valid setting of this study, based on known best practices in instructional design. For that reason, it is not possible to attribute the learning effect to one of the design choices or the application of a specific principle (i.e., segmenting and prompting). Similarly, the degree to which the modelling example had a direct effect on learning or an indirect effect through an improved practice phase cannot be deduced from the current experiment. Further research is necessary to disentangle and isolate these effects to detect which design choices are most effective. To achieve this, researchers could measure process variables such as mental effort and attention continually during the processing of the example, in addition to learning (Spanjers, Wouters, van Gog, & van Merriënboer, 2011). Such methodology is already employed, for example, in research by Kammerer, Bråten, Gerjets, and Strømsø (2012), which combines eye tracking methodology, process logging, and verbal protocols. With information on learning processes that occur during example processing, and by comparing different designs, conclusions can be drawn about the effects of individual instructional principles and design choices.
To conclude, the intervention in this study is a short, one‐shot, standalone training and yields only small effect sizes, yet it shows a promising result: modelling examples are effective tools for fostering IPS skills. Based on the findings, it can be predicted that these skills can be developed with a well‐designed training program including modelling examples and providing sufficient time for practice. A longitudinal approach, where IPS instruction is embedded in a curriculum and combined with domain‐specific instruction, might be a fruitful design to achieve this challenging goal (Argelagós & Pifarré, 2012; Rosman, Mayer, & Krampen, 2016; Wopereis, Brand‐Gruwel, & Vermetten, 2008).
APPENDIX A
Overview of multimedia principles and how they were applied in the modelling example
| Multimedia principle | Description | Application |
|---|---|---|
| Split‐attention principle | Avoid formats where learners have to split their attention between multiple sources | All information was contained in the video screen and zoomed on relevant elements where possible |
| Modality principle | Learners learn better from audio narration than on‐screen text | The screencast contains a voice‐over narration and no on‐screen instructions |
| Redundancy principle | Avoid simultaneous presentation of verbal and visual text | On‐screen text and verbal narration do not overlap |
| Segmenting principle | Media should be segmented or allow learner to process in own pace | The example was split into fragments and could be paused and replayed |
| Pre‐training principle | Learners should be familiar with domain‐specific key concepts | An instruction video prior to the modelling example explained all concepts |
| Coherence principle | Avoid extraneous, nonrelevant material | Not possible to remove these elements from a realistic screencast, but zooming was used to focus on relevant information |
| Signalling principle | Focus the leaner's attention to essential material | The mouse cursor was accentuated and was often used to “point” at on‐screen elements the expert was talking about. Zooming was used when possible to move distractions (such as advertisements) off‐screen |
| Contiguity principles | Visually and temporally align words and graphics | In addition to signalling methods, relevant information was always on‐screen when it occurred in the narration |
| Personalization principle | Deliver instruction in a conversational tone | Although scripted, the narration resembled an expert who thinks out loud during the search |
| Voice principle | Learners learn better from narration in a standard‐accented human voice | The speaker was a standard‐accented Dutch woman |
| Image principle | Adding the speaker's image on screen does not necessarily lead to better learning | The image of the speaker was not included |
APPENDIX B
Scoring rubric for information problem solving assessment
|
Question 1: What is your first step and why? Maximum points: 2 |
|
|---|---|
| 0 points | For statements that reflect that the student starts searching right away |
| Add 1 point | For statements reflecting orientation activities: activating prior knowledge, planning, thinking, etc. |
| Add 1 point | For statements concerning task demands: determining information needs, types of sources, formulating a question, etc. |
|
Question 2: Which problem statements would you formulate? Maximum points: 2 |
|
| 0 points | For statements that are irrelevant for the task |
| 1 point | For statements that are relevant, but incomplete or formulated vaguely |
| 2 points | For statements that contain all three relevant concepts (comparable with “What is the influence of X on Y?”) |
|
Question 3: Which search query would you type into Google? Maximum points: 4 |
|
| Award a point for each relevant search term or synonym thereof. If the student shows a systematic search pattern, award an additional point. | |
| Pretest | Gender‐specific education, influence, school performance |
| Posttest | Intelligence, change, age |
| Delayed posttest | Red wine, health, influence |
|
Question 4: Which three websites would you select? Why? Maximum points: 4 |
|
| Pretest | Sources #3, #4, and #7 yield 2 points, sources #6 and #8 yield 1 point |
| Posttest | Sources #4, #5, and #6 yield 2 points, sources #3 and #8 yield 1 point |
| Delayed posttest | Sources #3, #6, and #8 yield 2 points, sources #4 and #5 yield 1 point |
|
If the sum of these points is 5 or 6, award 2 points for this question. If the sum of these points is 2, 3, or 4, award 1 point for this question. If the sum of these points is lower than 2, award no points for this question. |
|
| Award an additional point, but no more than 2 points, for all selection criteria that are mentioned in the comment that do not refer to “relevance.” For example: reliability, author, publication date, reputation, etc. | |
|
Question 5: What do you do when you visit a text‐rich website and want to find out if it contains relevant information? Maximum points: 1 |
|
| 1 point | For mentioning a scanning strategy, such as reading headlines only or using the search function (Ctrl + F) |
|
Question 6: Which criteria do you use to determine whether information is useful for your task? Maximum points: 2 |
|
| 1 point for each of the following criteria | Goal of the text, reliability, author reputation, publication date, language/style, compares with other sources |
|
Question 7: How do you deal with contradicting information? Maximum points: 1 |
|
| 1 point | For statements that reflect critical scrutiny, for example, searching for more information or investigating reliability, or if the answer reflects that both sides of the story are incorporated in the solution |
| Calculating the score | |
| Subscore for Step 1: Define the problem | The sum of scores for Questions 1 and 2 |
| Subscore for Step 2: Search information | The score for Question 3 |
| Subscore for Step 3: Select information | The score for Question 4 |
| Subscore for Step 4: Process information | The sum of scores for Questions 5, 6, and 7 |
| Total score | The average of these four subscores forms the final score for the test and is expressed as a percentage of the maximum score (4 points) |
