Individual differences in the preference for worked examples: Lessons from an application of dispositional learning analytics

Summary Worked-examples have been established as an effective instructional format in problem-solving practices. However, less is known about variations in the use of worked examples across individuals at different stages in their learning process in student-centred learning contexts. This study investigates different profiles of students' learning behaviours based on clustering learning dispositions, prior knowledge, and the choice of feedback strategies in a naturalistic setting. The study was conducted on 1,072 students over an 8-week long introductory mathematics course in a blended instructional format. While practising exercises in a digital learning environment, students can opt for tutored problem solving, untutored problem solving, or call worked examples. The results indicated six distinct profiles of learners regarding their feedback preferences in different learning phases. Finally, we investigated ante-cedents and consequences of these profiles and investigated the adequacy of used feedback strategies concerning ‘ help-abuse ’ . This research indicates that the use of instructional scaffolds as worked-examples or hints and the efficiency of that use differs from student to student, making the attempt to find patterns at an overall level a hazardous endeavour.


| INTRODUCTION
'One important direction for future research on example-based learning is to start addressing the effects over time, in real classroom contexts,' because currently '[m]ost studies on the effectiveness of example-based learning have been of the highly controlled single session variety in a lab or school.' (van Gog & Rummel, 2018, p. 206).This summarising recommendation in a recent review study of examplebased learning is unmatched in terms of outlining our research rationale.Empirical research studying worked examples typically follows the golden rule of pre-test and post-test experimental design principles during a relatively short learning episode in their learning context (van Gog & Rummel, 2010, 2018).This type of research often assigns students to specific instructional modes, representing different treatments of the experimental design, which may limit its external validity in real classrooms.
The rise of multi-modal research combining self-report measures with trace data generated in digital learning platforms has created new venues to explore worked examples in addition to the experimental design (Aleven, McLaren, Roll, & Koedinger, 2004;Noroozi et al., 2019).By capturing temporal changes in learning behaviour over a full course period, multi-modal data enable researchers to ask new questions that were not feasible in an experimental setup, for example, the antecedents and consequences of individual differences in preferences for instructional format and the existence of profiles representing characteristic patterns in the use of instructional scaffolding.
Studies into the merits of example-based education, and in the specific worked examples case, set-up in line with experimental design principles (see e.g., Lusk & Atkinson, 2007;Pachman, Sweller, & Kalyuga, 2014;Spanjers, van Gog, & Merriënboer, 2012) generally conclude that example-based education is an efficient and effective instructional format for novice learners (Baars, van Gog, de Bruin, & Paas, 2014;Renkl, 2014;van Gog & Rummel, 2010, 2018;van Gog, Rummel, & Renkl, 2019).In many studies of this type, untutored problem solving acts as the control condition: feedback provided to the learner is restricted to the evaluation of provided answers at the end of the problem-solving steps (McLaren, van Gog, Ganoe, Karabinos, & Yaron, 2016;McLaren, van Gog, Ganoe, Yaron, & Karabinos, 2014).A potentially stronger choice for control is the instructional format of tutored problem solving, where students receive feedback in the form of hints and evaluation of provided answers, both during and at the end of the problem-solving steps (Koedinger & Aleven, 2007;McLaren et al., 2014McLaren et al., , 2016;;Winne, 2017).With this new controls introduced, differences in effectiveness and efficiency between the instructional formats diminish, although efficiency benefits of the worked example instruction type remains (Salden, Aleven, Renkl, & Schwonke, 2009;Salden, Koedinger, Renkl, Aleven, & McLaren, 2010;Schwonke et al., 2009).A subsequent logic step in this development is to investigate efficiency and effectiveness of instructional formats combining tutored problem solving with the use of worked examples (McLaren, Lim, & Koedinger, 2008;Schwonke et al., 2009).
The type of research we propose is closely linked with the development sketched above: investigating the use of worked examples, tutored, and untutored problem solving in an authentic learning context.However, our research was conducted in an observational setting instead of following the traditional experimental design.As a result, we sacrificed the ability to make causal claims about the effectiveness and efficiency of different instructional formats in problemsolving practices.Instead, a longitudinal and observational design over a sustained period of time (in our case 8 weeks) allows new questions to be asked, such as: which students and in what contexts opt for learning by worked examples, tutored or untutored problem solving?
What are the antecedents of these choices, in terms of prior knowledge, prior schooling, and learning dispositions?Moreover, what are the consequences of these choices, in terms of cognitive and noncognitive learning outcomes?Moreover, inspired by a learning analytics based context where the answers to these questions are formulated in terms of learning feedback to groups of students demonstrating similar learning behaviours: are we able to distil characteristic patterns of revealed preferences for instructional scaffolding?
When students have access to different feedback formats, another question arising from an observational design is how students self-regulate their use of worked-examples (Noroozi et al., 2019;Rienties, Tempelaar, Nguyen, & Littlejohn, 2019) and whether the help-seeking mechanisms the students apply are the most efficient and effective options available in obtaining their learning goals.

| THEORETICAL BACKGROUND: SELF-REGULATED LEARNING AND ITS SCAFFOLDS OF EXAMPLES AND HINTS
A wide body of self-regulated learning (SRL) literature has looked at how learners make decisions in how and when to learn (Winne, 2017).
A critical review of six prominent SRL models by Panadero (2017) showed that learners iteratively go through three main phases: the preparatory phase (i.e., planning and goal-setting), the performance phase (i.e., performing the task and monitoring and controlling their own cognition), and the appraisal phase (i.e., reflecting and adapting on their SRL process, as part of self-reflection, by peers, by a computer, or via a teacher).Numerous empirical studies have shown the benefits of SRL on academic performance in both online and blended learning environments, in which learners have more autonomy over their own learning process (Broadbent & Poon, 2015;Fincham, Gaševi c, Jovanovi c, & Pardo, 2019;Kizilcec, Pérez-Sanagustín, & Maldonado, 2017;Littlejohn, Hood, Milligan, & Mustain, 2016;Rienties et al., 2019).In the context of example-based learning, previous studies proposed that novice learners would be more likely to benefit from using worked examples prior to problem solving than vice versa, or only using problem solving (Leppink, Paas, van Gog, van der Vleuten, & van Merriënboer, 2014;van Gog, Kester, & Paas, 2011;van Merrienboer & Sweller, 2010).The theoretical underpinning behind this is that worked examples are more beneficial to novice learners at the stage of schema acquisition because learners can focus their limited cognition on understanding the principle or concept.However, when learners are given autonomy over their choice of help-seeking, they do not always choose the most optimal learning strategies as proposed in the literature.
For example, through a series of three experiments, Foster, Rawson, and Dunlosky (2018)  Having access to multiple instructional scaffolds gives way to another phenomenon: that of opting for non-optimal forms of scaffolding, also coined as 'help abuse' (Aleven et al., 2004;Price, Zhi, & Barnes, 2017;Shih, Koedinger, & Scheines, 2010).The most common, or at least most frequently investigated, form of help abuse is bypassing hints that are more abstract and going straightforwardly to concrete solutions (e.g., Aleven et al., 2004;Aleven, McLaren, & Koedinger, 2006;Aleven, Roll, McLaren, & Koedinger, 2016) Rienties et al., 2019;Tempelaar, Rienties, & Giesbers, 2015;Tempelaar, Rienties, Mittelmeier, & Nguyen, 2018;Tempelaar, Rienties, & Nguyen, 2017).Learning dispositions represent individual difference characteristics that affect all learning processes and include affective, behavioural, and cognitive facets (Rienties, Cross, & Zdrahal, 2017).Although the merits of including individual difference characteristics are recognised ('As yet research has not systematically addressed the influence of other individual differences than prior knowledge on the effectiveness of example-based learning', van Gog & Rummel, 2018, p. 203), not much empirical research into example-based education has followed that route.One of the exceptions is a recent study by van Harsel, Hoogerheide, Verkoeijen, and van Gog (2019), in which individual difference variables such as motivation, interest, and self-efficacy act as consequences of alternative instructional formats, differing in the order examples and problems are offered to students.In our study, the role of the individual difference characteristics is much broader: some act as consequences of student learning behaviours, some act as antecedents of the learning behaviours, and a third group is taken as defining the student profiles, together with learning behaviour indicators.In all of these cases, we aim to investigate what the relationship is between the use of worked examples and hints and the measured facets of individual differences.
In line with research practices in the LA field (Fincham et al., 2019;Rienties, Toetenel, & Bryan, 2015), we will focus our research questions on the level of groups of students demonstrating similar learning behaviours: the profiles, what brings us to personoriented modelling approaches.The rationale for doing so is twofold.
First: the ultimate aim of the research is to support students in shaping their learning process, and do so at a group or profile level, rather than individual or generic level.Second: trace data of learning behaviours tend to give rise to heterogeneous data sets.That heterogeneity is at odds with the application of variables oriented methods that require homogeneity.Decomposing the full data set into more homogeneous clusters is a further aim of this research.The novelty of this study is the combination of learning dispositions and other individual learner characteristics as instruments in the cluster analysis to detect student profiles.For example, while Fincham et al. (2019) were able to successfully identify different self-regulation patterns of engagement using cluster analyses based upon trace behaviour data, whether the learning dispositions of these students actually matched the cluster analysis results was not explored.These factors help explain the composition of the profiles as well as the differences in the use of feedback at different stages throughout a course.

| Context of the empirical study
This study took place in a large-scale introductory course in mathematics and statistics for first-year business and economics students at a public university in the Netherlands.This course followed a blended learning format for over 8 weeks.In a typical week, students attended a 2-hr lecture that introduced the key concepts in that week.After that, students were encouraged to engage in self-study activities, such as reading textbooks and practising solving exercises using the two etutorial platforms SOWISO (https://sowiso.nl/)and MyStatLab (MSL).This design is based on the philosophy of student-centred education, in which the responsibility for making educational choices lies primarily with the student.There were two 2-hr face-to-face tutorials each week based on the problem-based learning (PBL) approach in small groups (14 students), coached by expert tutors.Since most of the learning takes place outside the classroom during self-study through the e-tutorials or other learning materials, the class time is used to discuss how to solve advanced problems.The educational format, therefore, has most of the characteristics of the flipped-classroom design in common (Nguyen et al., 2016).
The use of the e-tutorials can be distinguished in three different phases.In Phase 1, students prepared for the next tutorial session.
Knowing that they would face the discussion of 'advanced' maths problems in that tutorial session, students were expected to prepare by selfstudy outside class, for example, by studying the literature together with some peers or practising in the e-tutorials.Phase 1 was not formally assessed, but instead allowed students to actively participate in the discussion of the problem tasks in the face-to-face tutorial session.
Phase 2 was the preparation of the quiz session, 1 or 2 weeks after the respective tutorial.The three quizzes were taken every 2 weeks in 'controlled' computer labs that consisted of test items from the same pool of questions in the practising mode.Although the assessment through quizzes was primarily for formative purposes, students could score a bonus point in each respective quiz, which was afterwards added to their written exam score.To incentivise the preparation of these quizzes through practising in the e-tutorials, and to diminish the tension of completing a quiz as a semi-high stake assessment, students could compensate part of 'lost bonus score' by achieving adequate mastery levels in the e-tutorials in Phases 1 or 2.
Phase 3 was the preparation of the final exam, at the end of the course.Phase 3 included formal, graded assessments.The written exam was a multiple-choice test of 20 questions on mathematics, as well as 20 questions on statistics.These questions could be practised using textbook materials and e-tutorial modes.The final exam was mostly summative of nature and had by far the largest weight in the course score.
Due to the compensation mechanisms, weights of the three performance categories could only be expressed as on average ex-post contributions to the course score: 86% for the final exam, 11% for the aggregated quiz scores, and 3% for the tool mastery level.Students' timing decisions, therefore, related to the amount of preparation in each of the three consecutive phases and is graphically summarised in Table 1.Cells below the diagonal are left empty but belong to Phase 1.However, in contrast to some other studies conducted in distance learning settings (Nguyen, Huptych, & Rienties, 2018), very few students in our context practised more than 1 week in advance.
The subject of this study is the full cohort of students 2018/2019 (1,072 students).The student population was diverse: only 21% of the student population was educated in the Dutch secondary school system, compared to 79% educated in foreign systems, with 50 nationalities.A large part of the students had a European nationality, with only 4.0% of the students from outside Europe.Secondary education systems in Europe differ widely, particularly in the fields of mathematics and statistics.Therefore, it is crucial that this introductory module is flexible and allows for individual learning paths.On average, students spent 27 hr connect time in SOWISO and 32 hr in MSL, which is 30-40% of the 80 hr available to learn both subjects.Although students worked in two e-tutorial platforms, this analysis will focus on student activity in one of them, SOWISO, because of the availability of fine-grained, time-stamped feedback data, missing for the other e-tutorial.
One component of the course assessment was an individual student project, in which students analyse a data set and report on their findings.That data set consisted of students' own learning disposition data, collected through the self-report surveys, explaining the full response of our survey data (students could opt-out and use alternative data, but no student made use of that option).Repeated students who failed the exam the previous year and redid the course are excluded from this study.

| Instruments and procedure
Both e-tutorial systems followed a test-driven learning and practice approach.Each step in the learning process was initiated by a problem and students were encouraged to (try to) solve each problem.If a student had not (fully) mastered a problem, he or she could ask for hints to solve the problem step by step or ask for a fully worked out example.Upon receipt of feedback, or upon starting a second attempt to solve the problem, a new version of the problem was loaded (parameter based, thus with new data) to enable the student to demonstrate his or her newly acquired mastery.
Figure 1 shows the implementation of the alternative feedback strategies that students can choose in a sample problem: • Check: the unstructured problem-solving approach, which only provides correctness feedback after solving a problem; • Hint: the tutored problem-solving approach, with feedback and tips to help the student with the different problem-solving steps; • Solution: the worked examples approach; • Theory: ask for a short explanation of the mathematical principle.

T A B L E 1
The three learning phases: preparing the tutorial session as Phase 1 (green), preparing the quiz session as Phase 2 (orange), and preparing the exam as Phase 3 (red) [Colour table can be viewed at wileyonlinelibrary.com]Our study combined trace data from the SOWISO e-tutorial with self-report measures of learning dispositions, and course performance data.Azevedo et al. (2013) distinguished between trace data of product type and process type, whereby click data are part of the process data category.In this study, we combined both types of process data, such as the clicks, to initiate the learning support mentioned above of Check, Hint, Solution, and Theory, as well as product data, such as the mastery in the tool, as discussed below.Trace data from SOWISO were processed as follows.Firstly, all dynamic trace data were assigned to the three consecutive learning phases, in line with the scheme depicted in Table 1.Afterwards, the data were aggregated over time, to arrive at static, full course period accounts of trace data.
Secondly, a selection was made from the wide range of trace variables by focusing on the process variables that were most closely related to the alternative learning strategies.A total of four trace variables were selected: • Mastery: the proportion of the exercises that have been successfully solved as a product indicator; • Attempts: total number of attempts at individual exercises; • Hints: the number of hints called; • Examples: the number of worked examples called.Averaging over all students and all problems implies that each problem is seen 2.7 times, 0.8 times as example, and 1.9 times in an attempt.• Surprise: neutral epistemic emotion, • Curiosity: positive, activating epistemic emotion, • Confusion: negative, deactivating epistemic emotion, • Anxiety: negative, activating epistemic emotion, • Frustration: negative, deactivating epistemic emotion, • Enjoyment: positive, activating epistemic emotion, • Boredom: negative, deactivating epistemic emotion.Frustration, .78 for Enjoyment, .and.86 for Boredom (Pekrun et al., 2017).

| Attitudes to Learning
The attitudes towards learning mathematics and statistics were assessed using the SATS instrument, based on the expectancy*value framework of student learning choices (Tempelaar, Gijselaers, Schim van der Loeff, & Nijhuis, 2007).The instrument contained six quantitative methods-related learning attitudes: • Affect: students' feelings about mathematics and statistics, • Cognitive competence: cognitive competence, or the students' self-perceptions of their intellectual knowledge and skills when applied to mathematics and statistics, • Value: the attitude of students towards the usefulness, relevance and value of mathematics and statistics in their personal and professional lives, • Difficulty: students' perception that mathematics and statistics as subjects are not difficult to learn, • Interest: the individual interest of students in learning mathematics and statistics, • Effort: the amount of work that students are willing to do to learn the subjects.(Tempelaar et al., 2007).

| Motivation and engagement wheel
The instrument Motivation and Engagement Wheel (Martin, 2007) breaks down learning cognitions and learning behaviours into four categories of adaptive versus maladaptive types and cognitive versus behavioural types.
• Self-belief, the value of school (ValueSchool), and Learning focus (LearnFocus) shape the adaptive, cognitive factors, or cognitive boosters.
The timing of a fourth instrument, learning activity emotions, was different.This survey was administered exactly halfway the course to give students sufficient exposure to the course itself.If it were monitored too close to the exam, there was a risk that students would mix up learning activity emotions with test emotions.Learning activity emotions are seen as the affective consequences of learning behaviours, the other consequences being of cognitive type: the course performance data.

| Learning activity emotions
The control-value theory of achievement emotions (CVTAE, Pekrun, 2000) postulates that emotions that arise in learning activities differ in valence, focus, and activation.Emotional valence can be positive (enjoyment) or negative (anxiety, hopelessness, boredom).CVTAE describes the emotions experienced about an achievement activity (e.g., boredom experienced while preparing homework) or outcome (e.g., anxiety towards performing at an exam).The activation component describes emotions as activating (i.e., anxiety leading to action) versus deactivating (i.e., hopelessness leading to disengagement).
Different from the other factors described above, learning activity emotions are not only a learning disposition but also an outcome of the learning process.Published reliability scores, Cronbach's alphas, are .85for Enjoyment, .86 for Anxiety, .93 for Boredom, and .90 for Hopelessness (Pekrun et al., 2011).
All self-report surveys applied the 1…7 Likert measurement scale.
Course performance data were based on the final written exam and

| Statistical analyses
For both practical and methodological arguments, we have opted for a where we sample data of students having different learning behaviours with the aim to generate different types of learning feedback, this condition is not often satisfied.In such a case, the application of variablesoriented models creates an 'average' learner that may not even exist in reality and generate feedback rules that are highly inadequate.
The aim of person-oriented modelling is splitting the heterogeneous sample into (more) homogeneous subsamples and investigates characteristic differences between these profiles.
Learning profiles were estimated with k-means cluster analysis.In Variables-oriented methods, like regression models or structural equation models, focus on the relationships between variables rather than forming profiles of students demonstrating similar learning behaviours.
The profiles being the main outcome of our study, lead us to apply cluster analysis rather than probabilistic approaches as latent class analysis.
The richness of the latent class approach, where all individual students are assigned probabilities of belonging to each of the several latent classes, would logically imply a feedback generation structure far too complex for the context of the learning analytics application.As a method for clustering, we opted for k-means cluster analysis or nonhierarchical cluster analysis, one of the most applied clustering tools in the LA field (Rienties et al., 2015).The number of clusters was chosen based on several practical arguments: to have maximum variability in profiles (based on the minimum distance between cluster centres for cluster solutions ranging from 2 to 12 clusters), not going into very small clusters and maintaining the interpretability of cluster solutions.
We opted for a six-cluster solution, as solutions with higher dimensions did not strongly change the characteristics of the clusters, but tended to split the smaller clusters into even smaller ones.As a next step in the analysis, differences between profiles were investigated with ANOVA.

| Cluster analysis
The interpretation of the final six-cluster solution, of which cluster centres are provided in Table 3, is primarily based on differences in overall activity in the e-tutorial.Six variables describe that overall activity: the number of Attempts in each of the three learning phases (preparing tutorial session, preparing quiz session, and preparing final exam) and the number of Examples called in these three learning phases.The first cluster in Table 3

| Profile differences in activity levels per learning phase
Further differences between the several profiles are found when we disaggregate overall activity levels into levels of the three consecutive   The temporal pattern of the call for Hints, see Figure 5, demonstrates that Phase 2 is the crucial one here.Except, once again, students of the profile of high prior knowledge: they use most of their Hints calls already in Phase 1. Students of fourth and fifth profiles spread the use of Hints over the first two learning phases.

| Learning dispositions as antecedents
Learning dispositions measured at the start of the course, as well as the two demographic data of gender and international status, provide partial explanations of the composition of the profiles, be it that the contribution to explained variation is modest, ranging between 2.0 and 4.3% for individual antecedents.International students and female students are overrepresented in the three high activity and extreme activity profiles, and underrepresented in the profiles labelled as inactive and low activity (ANOVA significance levels below .0005).
No effects of demographic antecedents are visible in the sixth profile of students with high prior knowledge.The effect of a selection of learning dispositions type of antecedents is made visible in Figure 6.
In Figure 6, we find three profiles standing out in different ways.
Students in the profile with lack of activity in the e-tutorial, besides the overrepresentation of male, local students, score above average on the maladaptive behaviours Disengagement and Self-sabotage, but not on the maladaptive epistemic emotions Frustrated and Anxious.It

| Learning activity emotions as consequences
The four learning activity emotions discussed here not only differ from the epistemic learning emotions in that they focus on specific learning tasks, rather than the learning of mathematical topics in general, but also in that they are measured halfway the course, and not at the start of the course.Where the timing of the measurements makes the epistemic emotions antecedents of learning behaviours, the activity emotions are best seen as consequences of learning behaviours (in the first half of the course).Although these are important differences, the measured scores of activity emotions demonstrate similarity to those of the epistemic emotions: see Figure 7.
Students of the profile of inactivity develop relative high levels of Boredom, low levels of Enjoyment.Students in the profile with extreme activity distinguish by high Anxiety and Hopelessness levels.
Students of the profile of high prior knowledge are at the opposite pole: low levels of all negative emotions Anxiety, Boredom, and Hopelessness, relatively high levels of Enjoyment.
The explanation of activity emotions by profile membership generates effect sizes ranging from 3.4% for Hopelessness, 3.7% for Anxiety, 3.9% for Enjoyment to 5.2% for Boredom.In comparing the distribution of Attempts, left panel Figure 2, with the distribution of ToolMastery, we find that students in the second, fourth, and last profiles are efficient learners: relative to the number of Attempts, these students achieve high mastery levels.Least efficient learners are the students in the fifth profile: their levels of mastery are high, beyond 90%, but students of four out of six profiles while using fewer attempts reach similar levels of mastery.

| Course performance as consequences
Differences in efficiency of learning are also visible from the other two panels of Figure 8, where scores in the exam and quizzes have been re-expressed as a proportion of the maximum score, to ease the comparison of the three performance categories.Given the levels of ToolMastery, students in profiles three and five do less well than students in the other profiles in providing evidence of that mastery in the quiz sessions and the final exam.Given that the passing benchmark in Dutch grading systems is typically 55%, differences in passing rates are larger than differences in performance levels, profile means of exam and quiz scores being not far from this 55% in most cases.In terms of final grading, the weights of the three performance categories are best expressed in terms of their contribution to the final math score.Given the large weight of the exam score, not only students of profiles two, four, and six, but also students of the first profile, that of the inactive students, perform relatively well.

| DISCUSSION
Being an 'ideal student' has several dimensions.From the perspective of the instructional method of PBL, the ideal student is the one who enters well prepared the tutorial session, ready to discuss and solve advanced problems.From this perspective, the students of the sixth profile make up the ideal students: it is the single cluster where most of the learning takes place in Phase 1, preparing the tutorial session (Tempelaar, Rienties, Mittelmeier, et al., 2018).Profiles four and five students also do quite well from this perspective, spreading learning over the first two phases, but the first three profiles of students, counting a majority of students, are far from ideal: most of their preparation is after the tutorial session has already taken place.
Research into students' use of learning scaffolds has a different tradition in defining the ideal student (McLaren et al., 2008(McLaren et al., , 2014(McLaren et al., , 2016)).
Choosing the most appropriate scaffold, that is, not abusing less appropriate forms of help, is the main characteristics of 'good learning'.The tells that students in all profiles become less anxious, implying that the profile of extremely active students still retains their top position in this respect.
In terms of 'good and bad' learning behaviours from the perspective of potential 'help abuse', carrying in mind that 'The ideal student behaves as follows: If, after spending some time thinking about a problem-solving step, a step does not look familiar, the student should ask the tutor for a hint.' (Aleven et al., 2004, p. 229), the outcomes of this study suggest that high levels of help abuse may be present.Examples relatively extensively, suggests that the penalty explanation is at best a very partial explanation.
Another explanation might relate to the quality of the hints, as suggested by Price et al. (2017).They find that the quality of the first few hints is positively associated with future hint use, and that initial hint quality is associated with help abuse.If that mechanism is at work in our context, it suggests that the hints might not be easily accessible by most students other than the best prepared.The phrasing of the hints might be too abstract or at a too high difficulty level for less well-prepared students.The fact that the students in the profile of extreme activity, combine highest levels of Attempts, highest levels of Examples with lowest levels of Hints, except for the profile of inactive students, strengthens the plausibility of this explanation: these students are least prepared of all.(see e.g., Kim & Hannafin, 2011).With a few exceptions that take for instance learner attributes into account (Kim & Wei, 2011), studies into the empirics of blended learning aim to establish generic patterns about the use and efficiency of different instructional scaffolds.The contribution of our study to this field of research is that in blended contexts where students self-regulate the use of scaffolds, not only students attributes but also temporal patterns play a crucial role in how learning behaviours get shape.

| CONCLUSION
demonstrated that self-regulated learners were more likely to attempt problem solving before seeking any help, and were more likely to seek partial examples than worked examples after an unsuccessful problem-solving attempt.Using cluster analyses of 1,138 students' engagement in an Engineering course, Fincham et al. (2019) found four clusters of students who differed in terms of their self-regulation strategies.Furthermore, when looking at how students self-regulated learning over a longer period of time, we found temporal variances in the use of worked examples over different phases of the study, which subsequently influenced academic performance (Rienties et al., 2019).In technology-enhanced learning environments, self-regulated learning is facilitated by the availability of instructional scaffolding.Worked-examples, the step-by-step demonstration of the solution to a problem, is only one of them.The facility to request for hints that provide concrete help in proceeding with a problem-solving step when students get stuck shapes another type of scaffold.Salden et al. (2010) define problem solving with a hint request facility as tutored problem solving, where untutored problem solving represents the situation without instructional scaffolds.In a comparison of tutored problem solving with and without the support of workedexamples, McLaren et al. (2016) conclude that the main difference is in the efficiency gain resulting from having access to workedexamples.
. Analysing log behaviours of students, distinguishing 'proper use and abuse' of worked examples, Shih et al. (2010) created profiles of adaptive and maladaptive learning behaviours.Such profiling based on differences in learning behaviours is also the aim of our current study and builds on previous research of the authors (Nguyen, Tempelaar, Rienties, & Giesbers, 2016).However, we do not seek to demonstrate the difference between proper use and abuse of worked examples, but rather to find different patterns in the use of worked examples and hints, and connect these patterns to antecedents and consequences.Another difference with Shih et al. (2010) is the dispositional learning analytics (DLA) dimension of our study.The DLA infrastructure, introduced by Buckingham Shum and Deakin Crick (2012), combines learning data (trace data generated in logs of learning activities through technology-enhanced systems) with learner data (e.g., student dispositions, values, and attitudes measured through self-report surveys;

F
I G U R E 1 Sample of Sowiso problem with feedback options Check, Theory, Solution, and Hint.The yellow part represents the Hint for this problem, the green part the full Solution [Colour figure can be viewed at wileyonlinelibrary.com] These numbers were the outcome of the combination of student learning preferences and the incentive structure embedded in the instructional design.Although the bonus that came with tool mastery was restricted in size, the majority of students strived to achieve high mastery scores.Solving a problem in the untutored problem-solving mode brought full mastery.Solving a problem in the tutored problemsolving mode brought only partial mastery: every hint that was called by a student came with a penalty in the mastery score and calling more than three hints would result in a zero score.Calling an example does not count as a problem-solving attempt and therefore does not come with any score.Striving for high mastery by many students will, therefore, trigger students who have called an example but did not start problem solving, who started problem solving but were not successful or who started problem solving but used more than three hints, these are all students having still a zero mastery-score, to revisit the problem.As it will trigger students with an only partial score, whogave the correct answer but only after calling up to three hints, to revisit the problem.In addition, it triggers students to use as few hints as possible in their final attempt.Our analysis is based on the counts of different learning activities derived from the time-stamped activity logs, where we use the time data to categorise activities in three learning phases.The time duration of the activity was excluded for two reasons: highly collinear with the count data, and subjective in terms of the judgement of idle time.In this study, we focussed on a selection of self-report surveys for measuring students' learning dispositions.As part of the dispositional learning project, more than a dozen instruments have been monitored, ranging from affective learning emotions to cognitive learning processing strategies.Three instruments, all known to be predictors of course performance from previous studies(Nguyen et al., 2016;Rienties et al., 2019;Tempelaar, Rienties, & Giesbers, 2015;Tempelaar, Rienties, & Nguyen, 2017;Tempelaar, Rienties, Mittelmeier, & Nguyen, 2018; Tempelaar, Rienties, &   Nguyen, 2018), were selected as potential antecedents of learning behaviours.All three surveys were administered in the first week of the course.Due to that timing, the responses to these instruments represented student experiences in learning mathematics and statistics before entering university:• Epistemic learning emotions• Subject-specific (mathematics and statistics) learning attitudes• Motivation and engagement constructs3.2.1 | Epistemic emotionsWhile achievement emotions arise from doing learning activities, like doing homework, epistemic emotions are related to cognitive aspects of the task itself.Prototypical epistemic emotions are curiosity and confusion.In this study, epistemic emotions were measured with the Epistemic Emotion Scales(Pekrun, Vogl, Muis, & Sinatra, 2017), which was distributed at the start of the course.That instrument included the scales: the three intermediate quizzes.The quiz scores were averaged, and for both the exam and the quiz score, we focussed on the topic score for mathematics, in line with the focus on the math e-tutorial SOWISO.That resulted in MathExam and MathQuiz as the relevant performance indicators.As explained above, a third performance indicator with very little weight in the final score was the ToolMastery.On the first day of the course, students wrote a diagnostic entry test, of which MathEntry indicated the score; together with the level of prior math education, MathMajor, this served the role of cognitive antecedents of learning behaviours.
previous research, focusing on differences in the temporal characteristics of feedback use(Rienties et al., 2019), we made use of trace-based process variables only.In this research, we opted for estimating profiles on a very broad range of educational measurements: trace variables as the number of Attempts, Solutions, and Hints to prepare the tutorial sessions, to prepare the quiz sessions, and to prepare the final exam (the three phases), next to dispositional variables and prior knowledge indicators.Eight dispositional variables were selected, four of adaptive type and four of maladaptive type, known to be predictive of academic success from previous research(Tempelaar, Rienties, & Giesbers, 2015;Tempelaar, Rienties, & Nguyen, 2017;Tempelaar, Rienties, Mittelmeier, et al., 2018; Tempelaar, Rienties, & Nguyen, 2018): motivation and engagement dispositions Persistence, StudyManagement, Disengagement, and Self-sabotage, epistemic emotions Anxious and Frustrated, and attitudinal dispositions Cognitive Competence and Interest.This person-oriented modelling approach allowed us to profile students based on the combined trace, disposition, and prior knowledge data.

4. 3 |
Profile differences in overall activity levelsThe comparison of overall levels (overall implying that counts referring to the three learning phases are summed) of Attempts, Examples, and Hints for the six profiles is best made with Figure2.Appreciating the large differences in total counts among Attempts, Examples, and Hints, visible from the vertical axes, the outstanding position of the Extreme activity profile is clear from the first two panels.However, not in the third panel: where Hints are concerned, the High prior knowledge profile takes the lead, and the profile of most active students is in one-but-last position.
Figure 4.In all profiles but the last one, the majority of Examples calls is positioned in Phase 3, the preparation of the exam.Only students from the last three profiles use substantial amounts of Examples to prepare their quizzes.The profile of students having high prior knowledge is again that of the ideal students: they make intensive use of examples in the preparation of the very first assessment they need to write, which frees them from further preparation for the final exam.

F
I G U R E 3 Profile differences in the number of Attempts during the first learning phase, preparing the tutorial group, during the second learning phase, preparing the quiz session, and during the third learning phase, preparing the final exam, with error bounds based on standard errors [Colour figure can be viewed at wileyonlinelibrary.com]F I G U R E 4 Profile differences in the number of Examples during the first learning phase, preparing the tutorial group, during the second learning phase, preparing the quiz session, and during the third learning phase, preparing the final exam, with error bounds based on standard errors [Colour figure can be viewed at wileyonlinelibrary.com]F I G U R E 5 Profile differences in the number of Hints during the first learning phase, preparing the tutorial group, during the second learning phase, preparing the quiz session, and during the third learning phase, preparing the final exam, with error bounds based on standard errors [Colour figure can be viewed at wileyonlinelibrary.com] is as if they are too disengaged to be anxious.They score low on adaptive disposition Interest, Persistence and Study-management, but not on self-perceived competence: Cognitive Competence.That makes these students hard to reach out for educational support: they feel disconnected, and do not see a good reason to become connected.Students from the profile with extreme activity levels provide the opposite pattern, in most respects.Already from Day 1 of the course, they feel highly Frustrated and Anxious about learning math and statistics, suggesting their prior schooling may have had an unfavourable impact on the epistemic learning emotions.They regard themselves as highly Persistent with adequate Study management skills, but their self-perceived competence is the lowest of all students.The most straightforward pattern of dispositions is visible in the profile of students with high levels of prior knowledge.They score positive on all adaptive dimensions, especially Cognitive competence, and score negatively on all maladaptive dimensions, especially Frustrated and Anxious.
Students' mastery in the e-tutorial is strongly related to Attempts as main activity indicator of learning intensity in the digital platform: the F I G U R E 6 Profile differences in learning dispositions measured at the start of the course, with error bounds based on standard errors [Colour figure can be viewed at wileyonlinelibrary.com] bivariate correlation of Attempts and ToolMastery, at the individual level, equals .78.The relationship at the profile level between the two variables is similarly strong: profile memberships explain 63.2% of the variation in ToolMastery.That relationship at profile level is depicted in the right panel of Figure 8. (ANOVA significance levels are below .0005.) most studied format of 'help abuse' is that of calling worked examples, where a concrete hint would have been sufficient to overcome the obstacles of solving a problem.Looking at the ratios of calls for Hints and calls for Examples, and looking at the temporal patterns of these calls, there is only one profile of students nominated for being 'ideal students'.Again, it is the same last profile of students gaining that nomination.It is remarkable that the same students best satisfy both of these two rather different conditions.Even more remarkable, it is the high prior knowledge profile of students who qualify for this 'good student behaviour'.Given the large heterogeneity of prior knowledge and F I G U R E 8 Profile differences in course performance variables, with all performance scores expressed as proportion of the maximum score, with error bounds based on standard errors [Colour figure can be viewed at wileyonlinelibrary.com]F I G U R E 7 Profile differences in activity emotions measured at halfway the course, with error bounds based on standard errors [Colour figure can be viewed at wileyonlinelibrary.com]prior schooling of students, these top 10% of candidates, of whom 55% had taken science preparing math classes in high school, were not supposed to be active learners in the e-tutorial, and if active, maybe just to check their proficiency levels shortly before quiz or exam.The scaffolds offered in the e-tutorials were primarily directed at the students in the other tail of the distribution of prior knowledge, to allow them to optimally prepare their tutorial sessions.The other somewhat unexpected observation is that in checking their proficiency levels, these highly prepared students are one of the least frequent users of the Examples option, but the most frequent users of the Hints option.There are some characteristic differences in levels of learning dispositions amongst the six profiles, be it that the amount of variation that learning dispositions contribute in explaining profile membership is limited.The two profiles that stand out most are the two profiles at the opposite poles of activity in the e-tutorial.Students at the inactive pole score relatively high on the two maladaptive engagement constructs: Disengagement and Self-sabotage.That suggests that students in this profile are more likely to drop out than students from other profiles, what turns to be true: out of 26 dropouts, 18 are from this profile, against eight from the other profiles together.However, the other students from the profile of inactive students score relatively well in the final exam, suggesting that these students studied outside the digital platform and/or profited from their prior knowledge (the cluster counts 40% of students having prior schooling at the advanced level).The other profile that stands out is the one characterised by very high activity levels.This profile counts only 9% of students educated at the advanced level, and far worse from prior knowledge level only, these students seemed to have built relative high levels of epistemic Anxiety and Frustration towards learning math during their prior schooling experiences.It is reassuring that anxiety levels dropped for about 10% during the course, on average from clearly above the neutral level of the scale (4.41 at the start of the course), to about the neutral level of the scale (3.96 halfway the course, measured as learning activity emotion).However, the confrontation of Figures6 and 7 Existing studies on example-based education point in the direction of worked-out examples being an efficient and effective instructional technology.These are generic conclusions that do not distinguish between types of academic tasks and types of students.The most important contribution of this research is the emphasis on individual student preferences: when taking the digital learning environment out of the lab, bringing it to an authentic context where students themselves decide what learning scaffolds to use over a long period of 8 weeks, we observe large differences in the intensity our 1,072 students use different instructional scaffolds: worked examples, hints as part of tutored problem solving, or untutored problem solving.These large differences, mostly in the timing of the use of instructional scaffolds, are associated with individual differences in prior knowledge and learning dispositions; therefore, it requires an observational type of study, rather than an experimental type of study, to discover these different learning profiles based on individual differences in knowledge and learning dispositions.Next, the disparity in learning patterns of students with different profiles suggests strong heterogeneity in the composition of this population of learners, endangering a traditional variables-oriented modelling approach.Modelling the way an 'average' student applies instructional scaffolds as worked-examples or hints might be a less meaningful endeavour because of the lack of homogeneity.In our observational study, we could identify six profiles of students learning mathematics and statistics, based on traces of learning behaviours, learning disposition variables, and prior knowledge and schooling.One of these cluster-based profiles is composed of students who demonstrate 'ideal learning behaviour' in many respects: they are ideal PBL students, optimally prepared for their tutorial sessions, and they are ideal users of feedback, using relatively large amounts of hints, relatively low amounts of examples, in their initial learning.When using examples, it is for the preparation of the assessments, rather than the learning.These ideal learners happen to be the best-prepared learners, with high levels of prior knowledge, certainly not novice learners.Our novice learners are concentrated in the cluster of extreme activity.In line with the evidence brought by empirical studies on example-based education, these novices are strongly oriented on the use of examples, as part of intensive use of the e-tutorial.As a side effect of the authentic nature of our study, our novice learners are also the learners bringing the highest levels of epistemic frustration and anxiety, making any investigation into effectivity or efficiency of learning to a very complex endeavour.Research into instructional scaffolding within the context of blended learning using technology-enhanced learning environments has investigated the role different types of scaffolds can play in problem solving, with an emphasis on alternative visualisation supports

Table 2
provides an impression of the size of these numbers.On average, students made 737 Attempts, called 322 Examples, and called 29 Hints (in 387 problems available for the practice mode).
Descriptive statistics of learning disposition variables Published reliability scores, Cronbach's alphas, are .84forSurprise, .88 for Curiosity, .78 for Confusion, .76 for Anxiety, .77forTA B L E 2 All analyses were done using IBM SPSS statistical package.Ethics approval for this study was achieved by the Ethical Review Committee Inner City faculties (ERCIC) of the Maastricht University, as file ERCIC_044_14_07_2017.All participants provided informed consent to use the anonymized student data in educational research.Descriptives of the survey-based scales are contained in Table2, together with a sample item for each of the scales.All variables of adaptive nature score above the neutral level of four: Persistence, Study management, Cognitive competence, Interest, and Learning enjoyment.Most variables of maladaptive nature score below the neutral level: Disengagement, Self-sabotage, Frustated, Learning anxiety, Boredom, and Hopelessness.The exception to this pattern is the maladaptive epistemic emotions Anxious that scores above the neutral anchor.Reliabilities range from satisfactory to good, with the exception of the Cronbach alpha value of Disengagement.Means, standard deviations, and correlations of all variables in the analysis, both for the full sample and for each of the six clusters, are contained in the statistical Appendix.
is labelled as the profile of 'Inactive' ing.Students in the 'High prior knowledge' profile score highest on the diagnostic entry test and are the single profile with a majority of students educated at the advanced level in high school.When comparing rows rather than columns inTable 3, we see that the counts of Attempts are much higher than those of Examples, and Examples are again dominating Hints in number.
profiles in this respect, with the most 'healthy' ratios of Hints to Attempts and Hints to Examples in the sixth profile of students with high levels of prior knowledge.The circumstance that these best-prepared students use Hints relatively intensively, and The incentives provided to students, directed towards demonstrating mastery, may partly explain the low number of Hints.If full mastery can only be acquired in the untutored problem-solving mode and the use of hints results in a penalty towards the mastery score, saving on hints is understandable.However, there are large differences between